Air combat command decision-making intelligent agent training platform and method
By adopting a generalized software and hardware tool environment and a multi-agent parallel training method, the problem of low training efficiency of air combat command and decision-making agents was solved, and efficient agent training and improved model generalization ability were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI INST OF ELECTROMECHANICAL ENG
- Filing Date
- 2023-02-22
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies for training air combat command and decision-making intelligent agents suffer from low training efficiency, poor generalization, and weak scalability. This is mainly due to a lack of real combat and experimental data, a single training mode, a long simulation system construction cycle, and a limited number of training samples.
A generalized software and hardware tool environment is adopted, including a general system adversarial simulation platform, a general reinforcement learning platform, and a general communication middleware, to quickly build simulation models and agent models, decompose actual command and decision-making problems, build a sub-problem agent training application system, conduct multi-agent parallel training, and improve the diversity of training samples by rapidly batch-producing training scenarios.
It improves the efficiency of training system construction, enhances the adaptability and generalization ability of the agent model, and improves training efficiency and agent adaptability.
Smart Images
Figure CN116070922B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of military intelligence, specifically to a training platform and method for intelligent agents for air combat command and decision-making. Background Technology
[0002] With the rapid development of artificial intelligence (AI) technology, countries around the world have successively carried out research and application of AI and related technologies in the military field in order to enhance combat capabilities in the future intelligent context.
[0003] Air combat command and decision-making is a typical dynamic and interactive combat activity. In the traditional model, commanders formulate combat plans and issue combat orders based on subjective considerations and historical experience, according to the battlefield situation. Due to the complexity and variability of the combat situation, commanders need to be able to quickly analyze the battlefield environment, respond to battlefield uncertainties, and accelerate decision-making. By introducing artificial intelligence and combat simulation technologies, and using experimental and simulation data to train intelligent agents with learning algorithms, the efficiency of command processes such as military situation processing and decision optimization will be improved.
[0004] Due to the lack of real combat and experimental data, the current main method for training command and decision-making agents is to use simulated data generated by combat simulation systems as sample data. However, due to the highly specialized design of simulation models and agent models, the long construction cycle of simulation and training systems, and the use of serial training of a single high-state / action-dimensional agent, command and decision-making agents face the problem of low training efficiency. Furthermore, due to the limited number of training scenario samples and the separation of training and application phases, the trained agents suffer from poor generalization and limited scalability.
[0005] Therefore, a new technical solution is needed to improve the above-mentioned technical problems. Summary of the Invention
[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a training platform and method for an intelligent agent for air combat command and decision-making.
[0007] A training platform for air combat command and decision-making intelligent agents, provided by the present invention, includes:
[0008] Support layer: Provides general software and hardware environment support for modeling, training, demonstration and verification of air combat command and decision-making intelligent agents, as well as system confrontation modeling and simulation experiments;
[0009] Resource Layer: To address different training needs, a general simulation platform is used to quickly build a combat simulation model library and a combat rule library, and integrate combat simulation application systems; a general reinforcement learning platform is used to quickly build an intelligent agent model library and an intelligent game algorithm library; providing a resource foundation for building a learning and training environment;
[0010] Application Layer: Based on the intelligent application requirements of air combat command and decision-making, the actual command and decision-making problems are broken down, and a training application system for sub-problem intelligent agents is integrated and constructed. A set of combat training scenarios is built, and parallel training of multiple command and decision-making sub-problem intelligent agents and demonstration and verification applications of intelligent air combat command and decision-making related technologies are carried out.
[0011] Preferably, the support layer includes general software support tools and general hardware support tools, wherein the general software support tools include a general system adversarial simulation platform, a general reinforcement learning platform, and a general communication middleware;
[0012] The general-purpose system confrontation simulation platform provides a design framework, model scheduling, scenario creation, situation display, simulation operation control, and internal communication interaction for simulation models and simulation systems; specifically, it includes a combat simulation engine, combat scenario editing tools, situation display tools, simulation model configuration tools, operation control tools, and simulation agent software; the simulation agent software can receive operation control commands sent by the reinforcement learning platform and control the simulation engine to start, stop, and advance;
[0013] The general reinforcement learning platform includes an agent model generation module, an agent training module, a training agent software module, and a training process demonstration module. The training agent software module can send simulation operation control commands to the simulation platform according to the training process and receive information on the transition to simulation operation status and results.
[0014] Preferably, the general-purpose hardware support tools include a training server, a data and model storage server, a simulation computer, a situation display and control system, an algorithm development and debugging display and control system, a training display and control system, an agent model design display and control system, an agent model development and debugging display and control system, a training control system, a scenario editing display and control system, a situation display and control system, a data acquisition display and control system, a simulation model design display and control system, a simulation model development and debugging display and control system, and a combat rule design display and control system.
[0015] The data and model storage server is used to store simulation data, combat simulation models, intelligent agent models, and training scenarios;
[0016] The training control display is used for training mode control and training business management. The training modes include single simulation node single agent training, single simulation node multi agent training, multi simulation node single agent training, and multi simulation node multi agent training.
[0017] The aforementioned operational rules design and control system is used to design and formulate military operational rules for air combat command and decision-making.
[0018] Preferably, the resource layer includes a combat simulation model library, a combat rule library, a training scenario set, an agent model library, and an intelligent game algorithm library;
[0019] The combat simulation model library stores and manages simulation models of combat platforms, command and control system equipment, early warning and detection equipment, fire interception equipment, electronic warfare equipment, and strike weapons and equipment, as well as combat mission simulation models; the strike weapon and equipment models are used to simulate the strike patterns and processes of incoming targets; and the combat mission simulation models are used to configure air combat missions.
[0020] The application layer integrates an intelligent agent training application system based on the training requirements for air command and decision-making, and carries out training and verification applications of the intelligent agent for command and decision-making.
[0021] Preferably, the simulation model configuration tool can provide a modeling framework and interactive interface template for simulation model construction, and the models are integrated through a unified framework and interaction mechanism;
[0022] The intelligent agent model generation module provides an intelligent agent model design framework and template, supports the design and construction of model state space, action space, value function and policy function, generates command and decision-making intelligent agent neural network model, and integrates it through a unified framework and interaction mechanism;
[0023] The intelligent agent training module provides general artificial intelligence reinforcement learning algorithms and algorithm design frameworks, supports the integration and construction of combat command and decision-making business training algorithms according to standard interfaces, and controls the training business process.
[0024] Preferably, the general communication middleware includes an interaction protocol formulation module and a communication transmission module; the interaction protocol formulation module customizes the interfaces between the simulation platform and the reinforcement learning platform, and between the simulation model and the intelligent agent, according to the training business logic of the command and decision-making intelligent agent.
[0025] Preferably, the interface between the simulation platform and the reinforcement learning platform includes simulation operation control commands and propulsion status information; the interface between the simulation model and the intelligent agent is the command and decision training business interaction content; and the communication transmission module provides multiple mainstream network communication protocols.
[0026] Preferably, the combat rule base is used to store and manage various air combat command and decision rules, including troop movement rules and equipment usage rules, and can query and recall historical rules; the training scenario set stores and manages various combat command and decision combat simulation scenarios for agent training; the intelligent game algorithm library is used to store and manage game training algorithms that have been released and applied; and the agent model library is used to store and manage agent models that have been released and applied.
[0027] This invention also provides a training method for an air combat command and decision-making intelligent agent, characterized in that the method applies the aforementioned air combat command and decision-making intelligent agent training platform, and the method includes the following steps:
[0028] Step S1: Prepare training resources for combat simulation models, combat rules, agent models, and game learning algorithms using tools such as general system simulation platforms, general reinforcement learning platforms, and general middleware;
[0029] Step S2: Based on the air command and decision-making link, the actual command and decision problem is broken down into multiple command and decision sub-problems. Training resources are retrieved for each command and decision sub-problem, and the intelligent agent training application system for the command and decision sub-problem is integrated.
[0030] Step S3: Based on the command and decision-making training requirements, use the scenario editing and management tools of the intelligent agent training application system to batch create training scenario sets;
[0031] Step S4: Conduct parallel training of multi-sub-problem intelligent agent models. In the sub-problem intelligent agent training application system, in addition to the decision-making instructions generated by the intelligent agent for the sub-problem, other decision-making links can be automatically decided by setting combat rules.
[0032] Step S5: Separate the intelligent agent models of each sub-problem from the reinforcement learning platform, while retaining the external business interfaces for modular encapsulation, and synthesize the encapsulated sub-problem intelligent agent models into the command and control system equipment model;
[0033] Step S6: Retrieve application scenarios from the scenario set, and conduct intelligent decision-making combat concept demonstration and verification, system confrontation simulation evaluation, and application of intelligent equipment combat based on the air combat command and decision-making intelligent agent training platform.
[0034] Preferably, step S4 includes the following steps:
[0035] Step S4.1: Set the training mode. Based on the actual training needs and hardware and software resources, select the single simulation node single agent training, single simulation node multi-agent training, multi-simulation node single agent training, or multi-simulation node multi-agent training mode; where multi-agent refers to multiple agents with different air combat command and decision-making sub-problems.
[0036] Step S4.2: Determine the training order. When training the same agent using multiple scenarios, retrieve the scenario file from the training scenario set and specify the sequence of scenarios for training and the number of training iterations for each scenario.
[0037] Step S4.3: Allocate hardware and software training resources. According to the training mode settings, allocate the hardware resources of the simulation node machines. According to the set training order, retrieve the target files in sequence and allocate them to each simulation node. Multiple simulation nodes can train and simulate the same / different intelligent agents in parallel.
[0038] Step S4.4: Training and parameter tuning. After allocating training resources, the training system trains the agent according to the training order. When multiple simulation nodes train the same agent using the same scenario sequence, during the training process, after multiple simulation nodes have completed a certain scenario training task, the platform schedules multiple simulation nodes to carry out the next scenario training. After the training reaches the predetermined stage, the training parameters are adjusted according to the training results to continue the training.
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] 1. This invention adopts a generalized software and hardware tool environment, including a generalized system adversarial simulation platform, a generalized reinforcement learning platform, and a generalized communication middleware, which can quickly build simulation models and intelligent agent models according to actual needs; it utilizes simulation model libraries and intelligent agent model libraries to realize the rapid integration of simulation systems and training systems, thereby improving the efficiency of training system construction; at the same time, the training system has strong scalability.
[0041] 2. Based on the air-to-air decision-making command chain, this invention decomposes the actual command and decision-making problem into command and decision-making sub-problems, constructs a sub-problem intelligent agent training application system, and combines "expert rule decision-making" and "intelligent decision-making" to carry out multi-agent parallel training, thereby improving the training efficiency of intelligent agents;
[0042] 3. The platform of this invention rapidly creates training scenarios in batches and uses the scenario set for parallel training to improve the diversity of training samples. By conducting self-game training of the agent model, it enhances the adaptability and generalization ability of the agent model. Attached Figure Description
[0043] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0044] Figure 1 This is a schematic diagram of the overall platform architecture of the present invention;
[0045] Figure 2 This is a flowchart of the training process of the present invention;
[0046] Figure 3 This is a diagram illustrating the internal interaction and progression of the present invention. Detailed Implementation
[0047] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0048] Example 1:
[0049] A training platform for air combat command and decision-making intelligent agents, provided by the present invention, includes:
[0050] Support layer: Provides general software and hardware environment support for modeling, training, demonstration and verification of air combat command and decision-making intelligent agents, as well as system confrontation modeling and simulation experiments;
[0051] Resource Layer: To address different training needs, a general simulation platform is used to quickly build a combat simulation model library and a combat rule library, and integrate combat simulation application systems; a general reinforcement learning platform is used to quickly build an intelligent agent model library and an intelligent game algorithm library; providing a resource foundation for building a learning and training environment;
[0052] Application Layer: Based on the intelligent application requirements of air combat command and decision-making, the actual command and decision-making problems are broken down, and a training application system for sub-problem intelligent agents is integrated and constructed. A set of combat training scenarios is built, and parallel training of multiple command and decision-making sub-problem intelligent agents and demonstration and verification applications of intelligent air combat command and decision-making related technologies are carried out.
[0053] The support layer includes general software support tools and general hardware support tools. The general software support tools include a general system adversarial simulation platform, a general reinforcement learning platform, and general communication middleware.
[0054] The general-purpose system confrontation simulation platform provides a design framework, model scheduling, scenario creation, situation display, simulation operation control, and internal communication interaction for simulation models and simulation systems; specifically, it includes a combat simulation engine, combat scenario editing tools, situation display tools, simulation model configuration tools, operation control tools, and simulation agent software; the simulation agent software can receive operation control commands sent by the reinforcement learning platform and control the start, stop, and advancement of the simulation engine;
[0055] The general reinforcement learning platform includes an agent model generation module, an agent training module, a training agent software module, and a training process demonstration module. The training agent software module can send simulation operation control commands to the simulation platform according to the training process and receive information on the transition to simulation operation status and results.
[0056] General hardware support tools include training servers, data and model storage servers, simulation computers, situation display and control systems, algorithm development and debugging display and control systems, training display and control systems, agent model design and control systems, agent model development and debugging display and control systems, training control and control systems; scenario editing and control systems, situation display and control systems, data acquisition and control systems, simulation model design and control systems, simulation model development and debugging display and control systems, and combat rule design and control systems.
[0057] Data and model storage servers are used to store simulation data, combat simulation models, intelligent agent models, and training scenarios;
[0058] The training control display is used for training mode control and training business management. The training modes include single simulation node single agent training, single simulation node multi-agent training, multi-simulation node single agent training, and multi-simulation node multi-agent training.
[0059] Operational rules design and control is used to design and formulate military operational rules for air combat command and decision-making.
[0060] The resource layer includes a combat simulation model library, a combat rule library, a training scenario set, an agent model library, and an intelligent game algorithm library;
[0061] The combat simulation model library stores and manages simulation models of combat platforms, command and control system equipment, early warning and detection equipment, fire interception equipment, electronic warfare equipment, and strike weapons and equipment, as well as combat mission simulation models; the strike weapon and equipment models are used to simulate the strike patterns and processes of incoming targets; and the combat mission simulation models are used to configure air combat missions.
[0062] The application layer integrates an intelligent agent training application system based on the training needs of air command and decision-making, and carries out training and verification applications of command and decision-making intelligent agents.
[0063] The simulation model configuration tool can provide a modeling framework and interactive interface templates for simulation model construction, and the models can be integrated through a unified framework and interaction mechanism;
[0064] The agent model generation module provides an agent model design framework and template, supports the design and construction of model state space, action space, value function and policy function, generates command and decision-making agent neural network model, and integrates it through a unified framework and interaction mechanism;
[0065] The intelligent agent training module provides general artificial intelligence reinforcement learning algorithms and algorithm design frameworks, supports the integration and construction of operational command and decision-making business training algorithms according to standard interfaces, and controls the training business process.
[0066] The general communication middleware includes an interaction protocol formulation module and a communication transmission module; the interaction protocol formulation module customizes the interfaces between the simulation platform and the reinforcement learning platform, and between the simulation model and the intelligent agent, based on the training business logic of the command and decision-making intelligent agent.
[0067] The interface between the simulation platform and the reinforcement learning platform includes simulation operation control commands and propulsion status information; the interface between the simulation model and the intelligent agent is the command, decision-making, training, and business interaction content; and the communication transmission module provides various mainstream network communication protocols.
[0068] The combat rule library stores and manages various air combat command and decision-making rules, including troop movement rules and equipment usage rules, and allows for historical rule queries and recalls; the training scenario set stores and manages various combat command and decision-making combat simulation scenarios for agent training; the intelligent game algorithm library stores and manages game training algorithms for published applications; and the agent model library stores and manages agent models for published applications.
[0069] The present invention also provides a training method for an air combat command and decision-making intelligent agent, characterized in that the method applies the air combat command and decision-making intelligent agent training platform described above, and the method includes the following steps:
[0070] Step S1: Prepare training resources for combat simulation models, combat rules, agent models, and game learning algorithms using tools such as general system simulation platforms, general reinforcement learning platforms, and general middleware;
[0071] Step S2: Based on the air command and decision-making link, the actual command and decision problem is broken down into multiple command and decision sub-problems. Training resources are retrieved for each command and decision sub-problem, and the intelligent agent training application system for the command and decision sub-problem is integrated.
[0072] Step S3: Based on the command and decision-making training requirements, use the scenario editing and management tools of the intelligent agent training application system to batch create training scenario sets;
[0073] Step S4: Conduct parallel training of multi-sub-problem intelligent agent models. In the sub-problem intelligent agent training application system, in addition to the decision-making instructions generated by the intelligent agent for the sub-problem, other decision-making links can be automatically decided by setting combat rules.
[0074] Step S4.1: Set the training mode. Based on the actual training needs and hardware and software resources, select the single simulation node single agent training, single simulation node multi-agent training, multi-simulation node single agent training, or multi-simulation node multi-agent training mode; where multi-agent refers to multiple agents with different air combat command and decision-making sub-problems.
[0075] Step S4.2: Determine the training order. When training the same agent using multiple scenarios, retrieve the scenario file from the training scenario set and specify the sequence of scenarios for training and the number of training iterations for each scenario.
[0076] Step S4.3: Allocate hardware and software training resources. According to the training mode settings, allocate the hardware resources of the simulation node machines. According to the set training order, retrieve the target files in sequence and allocate them to each simulation node. Multiple simulation nodes can train and simulate the same / different intelligent agents in parallel.
[0077] Step S4.4: Training and parameter tuning. After allocating training resources, the training system trains the agent according to the training order. When multiple simulation nodes train the same agent using the same scenario sequence, during the training process, after multiple simulation nodes have completed a certain scenario training task, the platform schedules multiple simulation nodes to carry out the next scenario training. After the training reaches the predetermined stage, the training parameters are adjusted according to the training results to continue the training.
[0078] Step S5: Separate the intelligent agent models of each sub-problem from the reinforcement learning platform, while retaining the external business interfaces for modular encapsulation, and synthesize the encapsulated sub-problem intelligent agent models into the command and control system equipment model;
[0079] Step S6: Retrieve application scenarios from the scenario set, and conduct intelligent decision-making combat concept demonstration and verification, system confrontation simulation evaluation, and application of intelligent equipment combat based on the air combat command and decision-making intelligent agent training platform.
[0080] Example 2:
[0081] like Figure 1 As shown, the present invention provides a training platform for an intelligent agent for air combat command and decision-making, comprising:
[0082] Support layer: Provides general software and hardware environment support for modeling, training, demonstration and verification of air combat command and decision-making intelligent agents, as well as system confrontation modeling and simulation experiments;
[0083] Resource Layer: To address different training needs, a general simulation platform is used to quickly build a combat simulation model library and a combat rule library, and integrate combat simulation application systems; a general reinforcement learning platform is used to quickly build an intelligent agent model library and an intelligent game algorithm library; providing a resource foundation for the rapid construction of a learning and training environment;
[0084] Application Layer: Based on the intelligent application requirements of air combat command and decision-making, decompose actual command and decision-making problems, integrate and build an intelligent agent training application system, construct a set of combat training scenarios, and carry out applications such as parallel training of intelligent agents for multiple command and decision-making sub-problems and demonstration and verification of intelligent air combat command and decision-making related technologies.
[0085] The support layer includes general software support tools and general hardware support tools. The general software support tools include a general system adversarial simulation platform, a general reinforcement learning platform, and a general communication middleware.
[0086] The general-purpose system-on-systems simulation platform provides a design framework, model scheduling, scenario creation, situation display, simulation operation management and control, and communication interaction for simulation models and systems. Specifically, it includes a combat simulation engine, combat scenario editing tools, situation display tools, simulation model configuration tools, operation management tools, and simulation agent software.
[0087] The operational scenario editing and management tool can create training scenarios in batches according to training needs, generate large samples of training scenarios, and can be used for the maintenance and management of training scenario sets.
[0088] The simulation agent software can receive operation control commands sent by the reinforcement learning system and control the start, stop, and advance of the simulation engine.
[0089] Simulation model configuration tools can provide model frameworks and interaction interface templates for simulation model construction, and models can be quickly integrated through a unified model framework and interaction mechanism.
[0090] The general reinforcement learning platform includes an agent model generation module, an agent training module, an agent software module, and a training process demonstration module.
[0091] The agent model generation module provides an agent model design framework and template, designs the model's state space, action space, value function, and policy function, and generates a command and decision-making agent neural network model.
[0092] The intelligent agent training module provides general artificial intelligence reinforcement learning algorithms and algorithm design frameworks, and supports the integration of combat command and decision-making business training algorithms according to standard interfaces.
[0093] The training agent software module can send simulation operation control commands to the simulation platform.
[0094] The general communication middleware includes an interaction protocol design module and a communication transmission module. The interaction protocol design module customizes the interfaces between the simulation platform and the reinforcement learning platform, and between the simulation model and the agent, based on the training business logic of the command and decision-making agent. Specifically, the interface between the simulation platform and the reinforcement learning platform contains simulation operation control commands, while the interface between the simulation model and the agent contains the command and decision-making training business interaction content, including status information and command and control commands.
[0095] The communication transmission module provides a variety of mainstream communication protocols.
[0096] The general hardware support tools include training servers, data and model storage servers, simulation computers, situation display and control, algorithm development and debugging display and control, training display and control, agent model design and control, agent model development and debugging display and control, training control and control; scenario editing and control, situation display and control, data acquisition and control, simulation model design and control, simulation model development and debugging display and control, and combat rule design and control.
[0097] Data and model storage servers are used to store simulation data, simulation models, agent models, training scenarios, etc.
[0098] The training control display is used for training mode control and training business management. The training modes include single simulation node single agent training, single simulation node multi-agent training, multi-simulation node single agent training, and multi-simulation node multi-agent training.
[0099] Operational rules design and control is used to formulate military operational rules for air combat command and decision-making.
[0100] The resource layer includes a combat simulation model library, a combat rule library, a training scenario set, an agent model library, and an intelligent game algorithm library.
[0101] The combat simulation model library stores and manages simulation models of combat platforms, command and control system equipment, early warning and detection equipment, fire interception equipment, electronic warfare equipment, strike weapons and equipment, and combat mission simulation models.
[0102] The command and control system equipment is used to complement the functions of the command and decision-making intelligent agent, and to simulate command and decision-making functions outside the intelligent agent based on combat rules; in addition to various air defense combat equipment models, the strike weapon equipment models are used to simulate the strike patterns and strike processes of incoming targets.
[0103] Combat mission simulation models are used to configure air combat missions.
[0104] The operational rules base is used to store and manage various operational command and decision-making rules, mainly including troop movement rules and equipment usage rules, and supports historical rule query and retrieval.
[0105] The training scenario set includes various combat command and decision-making simulation scenarios, which are used for intelligent agent training.
[0106] The agent model library is used to store and manage agent models for published applications; the intelligent game algorithm library is used to store and manage intelligent game algorithms for published applications.
[0107] The application layer, based on the air-to-air command and decision-making chain, breaks down the actual command and decision-making problem into multiple sub-problems, including force action command and decision-making, situation generation command and decision-making, and interception and engagement command and decision-making. For each sub-problem, an intelligent agent training application system is integrated, and multi-agent parallel training is conducted in conjunction with operational rules. In the sub-problem intelligent agent training application system, apart from the decision-making instructions generated by the intelligent agent for that sub-problem, other decision-making stages can be automatically decided by setting operational rules.
[0108] The sub-problem agents are integrated to form a combat command and decision-making agent, which is used for demonstration and verification of intelligent decision-making combat concept design, system confrontation simulation evaluation, and intelligent equipment combat application. During the demonstration and verification phase, the agent model is separated from the reinforcement learning platform and directly integrated into the simulation system to carry out demonstration and verification experiments.
[0109] This invention provides a method for training an intelligent agent for air combat command and decision-making, comprising:
[0110] The training of air combat command and decision-making agents can be broadly divided into four phases: training resource preparation, training system integration, agent training, and agent application. The workflow of the air combat command and decision-making agent training method and the relationships between each phase are as follows: Figure 2 As shown.
[0111] (1) Training resource preparation stage: Using the general simulation platform model configuration tool, carry out the design and development of combat simulation models based on the simulation model framework, and store the simulation models in the combat simulation model library;
[0112] The operational rules were designed and the model was developed using the general simulation platform model configuration tool, and the operational rules were stored in the operational rule library.
[0113] Using the model generation module of the general reinforcement learning platform, we design and develop intelligent agent models based on the intelligent agent model framework, and store the intelligent agent models in the intelligent agent model library.
[0114] Using the training module of a general reinforcement learning platform, we designed and developed game learning algorithms based on the algorithm framework, and stored the intelligent game algorithms in the game algorithm library.
[0115] (2) Training system integration stage: First, based on the air-to-air decision-making command link, the actual command decision problem is broken down into multiple command decision sub-problems, including force action command decision, situation generation command decision, interception and combat command decision, etc. For each sub-problem, an intelligent agent and training application system are built, and finally merged to obtain the air-to-air command decision intelligent agent.
[0116] Then, the construction process of the intelligent agent and training application system for the sub-problem is as follows:
[0117] Based on the needs of the training scenario, combat simulation models are retrieved from the combat simulation model library. According to the air combat process and combat command and decision-making process, business relationship interface protocols are formulated between the combat simulation models, including sensor target intelligence interface information, sensor status interface information, electronic countermeasures interface information, interceptor weapon status information interface, command and decision-making instruction interface information, and damage result interface information. Various combat simulation models are linked together through these business relationship interface protocols to integrate the air combat training simulation application system.
[0118] At the same time, based on the needs of the training scenario, relevant combat command and decision rules are retrieved from the combat rule base to complement the decision points of the sub-problem agent, including troop action rules, equipment application rules, etc., and configured in the combat command and control system simulation model; taking the interception and engagement command and decision intelligent agent training system as an example, the combat rules that need to be configured include troop action rules, sensor application rules, etc.
[0119] According to the needs of the training scenario, the combat command and decision-making intelligent agent model is retrieved from the intelligent agent model library, the initialization parameter configuration of the intelligent agent model is completed, and the integration with the training module is completed based on interfaces such as state input and reward output.
[0120] Based on the needs of the training scenario, a suitable intelligent reinforcement learning algorithm is retrieved from the game algorithm library, and the algorithm is integrated with the training module based on the algorithm input and output interface.
[0121] Then, based on the needs of the training scenario, using scenario editing and management tools, and based on the simulation model library and training simulation application system, combat training scenario files are created and stored in the training scenario set.
[0122] Then, using a general-purpose communication middleware, an interactive interface protocol is customized between the general-purpose simulation platform and the general-purpose reinforcement learning platform, and between the combat decision-making agent and the combat simulation model. Interfaces can be reloaded from historical interface protocols or new interactive interfaces can be created and distributed to the simulation agent software and learning agent software for communication. The interface protocol between the general-purpose simulation platform and the general-purpose reinforcement learning platform includes a simulation control command interface and a simulation status interface; the interactive interface between the combat decision-making agent and the simulation model includes simulation engagement status information and combat command decision command information; the integrated interactive process between the simulation platform and the general-purpose reinforcement learning platform, and between the combat decision-making agent and the combat simulation model, is as follows: Figure 3 As shown.
[0123] The simulation control command interface includes runtime command information such as simulation start, simulation pause, and simulation end. These are sent by the general reinforcement learning platform to the general simulation platform's simulation agent software through the learning agent software based on the training progress. The simulation agent software then controls the simulation engine to run according to the commands. The simulation status interface includes simulation progress status information, which is sent by the general simulation platform to the general reinforcement learning platform through the simulation agent software based on the progress of the simulation engine.
[0124] Simulated engagement status information includes sensor status, hard and soft weapon system status, and operational status information during the air combat simulation process. This includes target tracking information, radar target tracking resource status, interceptor weapon resource status, weapon interceptability information, and interception engagement results. This information is sent from the simulation model to the learning agent software via simulation agent software and then forwarded to the intelligent agent model. Operational command and decision-making instructions include various air combat decision-making instructions, such as target allocation, target interception allocation, target guidance allocation, and operational mode selection. These instructions are sent from the intelligent agent model to the general simulation agent software via the learning agent tool and then forwarded to the intelligent agent.
[0125] (3) Agent Training Phase: Using the agent training systems for each air combat command and decision-making sub-problem integrated in the training system integration phase, agent training is conducted for each sub-problem. The specific process is as follows:
[0126] Step 1: Set the training mode. Based on the timing of training requirements and the availability of hardware and software resources, select either single-simulation node single-agent training, single-simulation node multi-agent training, multi-simulation node single-agent training, or multi-simulation node multi-agent training mode; where multi-agent specifically refers to multiple agents for different air combat command and decision-making sub-problems.
[0127] Step 2: Determine the training sequence. When training the same agent using multiple scenarios, retrieve the scenario files from the training scenario set and determine the sequence of scenarios for training and the number of training iterations for each scenario;
[0128] Step 3: Allocate hardware and software training resources. Based on the training mode settings, allocate hardware resources to the simulation nodes; according to the set training order, sequentially retrieve the target files and distribute them to each simulation node. Multiple simulation nodes can perform parallel training simulations of the same or different intelligent agents.
[0129] Step 4: Training and Parameter Tuning. After allocating training resources, the training system trains the agent according to the training sequence. When multiple simulation nodes train the same agent using the same scenario sequence, during training, after multiple simulation nodes have completed a certain scenario training task, the platform schedules multiple simulation nodes to start the next scenario training. After the training reaches the predetermined stage, the training parameters can be adjusted according to the training results to continue training.
[0130] (4) Application stage of intelligent agents:
[0131] Step 1: Agent Encapsulation. The trained air combat command and decision-making agents for each sub-problem are encapsulated as modules, while maintaining their external input / output interfaces unchanged during the encapsulation process.
[0132] Step 2: Agent Merging. Based on the air combat command process logic and agent interaction relationships, the trained agent modules for each air combat command decision sub-problem are integrated into the command and control system equipment model;
[0133] Step 3: Application of intelligent agents;
[0134] Step 3.1: Create a demonstration and verification scenario and store it in the scenario set;
[0135] Step 3.2: Retrieve typical scenarios from the scenario set as application scenarios, and carry out applications such as intelligent decision-making combat concept design, system confrontation simulation evaluation, and intelligent equipment combat application based on the air combat command and decision-making intelligent agent training platform.
[0136] Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1.
[0137] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0138] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A training platform for an intelligent agent for air combat command and decision-making, characterized in that, include: Support layer: Provides general software and hardware environment support for modeling, training, demonstration and verification of air combat command and decision-making intelligent agents, as well as system confrontation modeling and simulation experiments; Resource Layer: To address different training needs, a general simulation platform is used to quickly build a combat simulation model library and a combat rule library, and integrate combat simulation application systems; a general reinforcement learning platform is used to quickly build an intelligent agent model library and an intelligent game algorithm library; providing a resource foundation for building a learning and training environment; Application Layer: Based on the intelligent application requirements of air combat command and decision-making, decompose actual command and decision-making problems, integrate and construct a training application system for sub-problem intelligent agents, construct a set of combat training scenarios, and carry out parallel training of multiple command and decision-making sub-problem intelligent agents and demonstration and verification applications of intelligent air combat command and decision-making related technologies. In the application layer, based on the air command and decision-making link, the actual command and decision problem is broken down into multiple command and decision sub-problems, including force action command and decision, situation generation command and decision, and interception and engagement command and decision; for each sub-problem, an intelligent agent training application system is integrated, and multi-agent parallel training is carried out in conjunction with combat rules; In the aforementioned sub-problem intelligent agent training application system, apart from the sub-problem where the intelligent agent generates decision instructions, other decision-making processes can be automated by setting operational rules.
2. The air combat command and decision-making intelligent agent training platform according to claim 1, characterized in that, The support layer includes general software support tools and general hardware support tools, wherein the general software support tools include a general system adversarial simulation platform, a general reinforcement learning platform, and a general communication middleware; The general-purpose system confrontation simulation platform provides a design framework, model scheduling, scenario creation, situation display, simulation operation control, and internal communication interaction for simulation models and simulation systems; specifically, it includes a combat simulation engine, combat scenario editing tools, situation display tools, simulation model configuration tools, operation control tools, and simulation agent software; the simulation agent software can receive operation control commands sent by the reinforcement learning platform and control the simulation engine to start, stop, and advance; The general reinforcement learning platform includes an agent model generation module, an agent training module, a training agent software module, and a training process demonstration module. The training agent software module can send simulation operation control commands to the simulation platform according to the training process, and receive information on the transition to simulation operation status and results.
3. The air combat command and decision-making intelligent agent training platform according to claim 2, characterized in that, The general hardware support tools include training servers, data and model storage servers, simulation computers, situation display and control systems, algorithm development and debugging display and control systems, training display and control systems, agent model design and control systems, agent model development and debugging display and control systems, and training control and control systems. Scenario editing and display control, situation display and control, data acquisition and display control, simulation model design and display control, simulation model development and debugging and display control, and combat rule design and display control; The data and model storage server is used to store simulation data, combat simulation models, intelligent agent models, and training scenarios; The training control display is used for training mode control and training business management. The training modes include single simulation node single agent training, single simulation node multi agent training, multi simulation node single agent training, and multi simulation node multi agent training. The aforementioned operational rules design and control system is used to design and formulate military operational rules for air combat command and decision-making.
4. The air combat command and decision-making intelligent agent training platform according to claim 1, characterized in that, The resource layer includes a combat simulation model library, a combat rule library, a training scenario set, an agent model library, and an intelligent game algorithm library. The combat simulation model library stores and manages simulation models of combat platforms, command and control system equipment, early warning and detection equipment, fire interception equipment, electronic warfare equipment, and strike weapons and equipment, as well as combat mission simulation models; the strike weapon and equipment models are used to simulate the strike patterns and processes of incoming targets; and the combat mission simulation models are used to configure air combat missions. The application layer integrates an intelligent agent training application system based on the training requirements for air command and decision-making, and carries out training and verification applications of the intelligent agent for command and decision-making.
5. The air combat command and decision-making intelligent agent training platform according to claim 2, characterized in that, The simulation model configuration tool can provide a modeling framework and interactive interface template for simulation model construction, and the models can be integrated through a unified framework and interaction mechanism. The intelligent agent model generation module provides an intelligent agent model design framework and template, supports the design and construction of model state space, action space, value function and policy function, generates command and decision-making intelligent agent neural network model, and integrates it through a unified framework and interaction mechanism; The intelligent agent training module provides a general artificial intelligence reinforcement learning algorithm and algorithm design framework, and supports the integration and construction of operational command and decision-making business training algorithms according to standard interfaces. And control the training process.
6. The air combat command and decision-making intelligent agent training platform according to claim 2, characterized in that, The general communication middleware includes an interaction protocol formulation module and a communication transmission module; the interaction protocol formulation module customizes the interfaces between the simulation platform and the reinforcement learning platform, and between the simulation model and the intelligent agent, based on the training business logic of the command and decision-making intelligent agent.
7. The air combat command and decision-making intelligent agent training platform according to claim 6, characterized in that, The interface between the simulation platform and the reinforcement learning platform includes simulation operation control commands and propulsion status information; the interface between the simulation model and the intelligent agent is the command decision training business interaction content; the communication transmission module provides a variety of mainstream network communication protocols.
8. The air combat command and decision-making intelligent agent training platform according to claim 1, characterized in that, The combat rule base is used to store and manage various air combat command and decision rules, including troop movement rules and equipment usage rules, and can query and recall historical rules; the training scenario set stores and manages various combat command and decision combat simulation scenarios for intelligent agent training. The intelligent game algorithm library is used to store and manage the game training algorithms for published applications; The agent model library is used to store and manage agent models for published applications.
9. A training method for an intelligent agent for air combat command and decision-making, characterized in that, The method applies the air combat command and decision-making intelligent agent training platform as described in any one of claims 1-8, and the method includes the following steps: Step S1: Prepare training resources for combat simulation models, combat rules, agent models, and game learning algorithms using tools such as general system simulation platforms, general reinforcement learning platforms, and general middleware; Step S2: Based on the air command and decision-making link, the actual command and decision problem is broken down into multiple command and decision sub-problems. Training resources are retrieved for each command and decision sub-problem, and the intelligent agent training application system for the command and decision sub-problem is integrated. Step S3: Based on the command and decision-making training requirements, use the scenario editing and management tools of the intelligent agent training application system to batch create training scenario sets; Step S4: Conduct parallel training of multi-sub-problem intelligent agent models. In the sub-problem intelligent agent training application system, in addition to the decision-making instructions generated by the intelligent agent for the sub-problem, other decision-making links can be automatically decided by setting combat rules. Step S5: Separate the intelligent agent models of each sub-problem from the reinforcement learning platform, while retaining the external business interfaces for modular encapsulation, and synthesize the encapsulated sub-problem intelligent agent models into the command and control system equipment model; Step S6: Retrieve application scenarios from the scenario set, and conduct intelligent decision-making combat concept demonstration and verification, system confrontation simulation evaluation, and application of intelligent equipment combat based on the air combat command and decision-making intelligent agent training platform.
10. The training method for an air combat command and decision-making intelligent agent according to claim 9, characterized in that, Step S4 includes the following steps: Step S4.1: Set the training mode. Based on the actual training needs and hardware and software resources, select the single simulation node single agent training, single simulation node multi-agent training, multi-simulation node single agent training, or multi-simulation node multi-agent training mode. Among them, multi-agent refers to multiple intelligent agents with different sub-problems of air combat command and decision-making; Step S4.2: Determine the training order. When training the same agent using multiple scenarios, retrieve the scenario file from the training scenario set and specify the sequence of scenarios for training and the number of training iterations for each scenario. Step S4.3: Allocate hardware and software training resources. According to the training mode settings, allocate the hardware resources of the simulation node machines. According to the set training order, retrieve the target files in sequence and allocate them to each simulation node. Multiple simulation nodes can train and simulate the same / different intelligent agents in parallel. Step S4.4: Training and parameter tuning. After allocating training resources, the training system trains the agent according to the training order. When multiple simulation nodes train the same agent using the same scenario sequence, during the training process, after multiple simulation nodes have completed a certain scenario training task, the platform schedules multiple simulation nodes to carry out the next scenario training. After the training reaches the predetermined stage, the training parameters are adjusted according to the training results to continue the training.