Layered command control method combining large model and reinforcement learning

Abstract

The invention provides a large model and reinforcement learning combined hierarchical command control method, which is based on agents of a hierarchical structure, and is characterized in that the first layer of agents are RAG-based large model intelligent formation command agents; the second-layer intelligent agent is composed of an air strike marshalling intelligent agent and a reconnaissance and strike mixed marshalling The second-layer intelligent agent judges which subtask is executed according to the instruction sent by the first-layer intelligent agent; comprising the following steps that S1, an RAG-based large-model intelligent formation command system comprises an offline knowledge preparation stage and an online decision execution stage; s2, setting a state space, an action space and a reward and punishment function; and S3, respectively training the air strike marshalling agent and the reconnaissance and strike mixed marshalling agent, and constructing an intelligent game model based on a multi-agent reinforcement learning algorithm. According to the invention, exclusive knowledge support is provided for the large model through the RAG technology, then the structured decision instruction is generated through the large model, training is carried out based on multi-agent reinforcement learning to obtain the agent of a specific task, and formation-level command control is realized.

Description

Technical Field

[0001] This invention relates to the field of unmanned swarm game adversarial technology, specifically to a hierarchical command and control method that combines a large model with reinforcement learning. Background Technology

[0002] As a core combat style in modern warfare, joint operations require precise coordination from top-level mission planning to lower-level tactical execution, based on the dynamic battlefield situation. This not only requires command decisions to be supported by knowledge to ensure rationality and explainability, but also requires the execution level to have dynamic coordination capabilities to cope with complex adversarial environments.

[0003] In the context of informationized warfare, formation command and control has shifted from experience-driven to a dual-driven approach of data and knowledge. On the one hand, the dynamics and uncertainties of the battlefield situation have increased significantly, making traditional command models relying on expert rules ill-suited to real-time changing adversarial scenarios, prone to rigid decision-making and delayed responses. On the other hand, large-scale formations involve multi-group and multi-task coordination, making it difficult for single intelligent technologies to simultaneously provide knowledge support for top-level decision-making and dynamically optimize lower-level execution. Therefore, combining the knowledge processing capabilities of large language models with the dynamic decision-making capabilities of reinforcement learning to construct a hierarchical and collaborative command and control system has become the core technological direction for solving these problems.

[0004] A hierarchical command and control method combining large-scale models and reinforcement learning is proposed. Existing technical solutions include: "A Modeling Method for Cooperative Autonomous Behavior Decision-Making of Multi-Squadrons in Air Force Based on QMIX" (authors: Li Ni; Wang Ze; Gong Guanghong; Dong Liwei); "A Multi-UAV Cooperative Air Combat Maneuver Decision-Making Method Based on Multi-Agent Reinforcement Learning" (authors: Yang Qiming; Zhang Jiandong; Shi Guoqing; Wu Yong; Zhu Yan; Zhang Yaozhong); and "A Multi-Entity Asynchronous Cooperative Decision-Making Method and Device for Wargaming Based on Reinforcement Learning" (authors: Zhang Yu; Jiang Chaoyuan; Luo Junren; Li Tingting; Liu Yun; Yang Jingzhao; Liu Guo; Li Xin; Liu Yifeng). Chen Jiaxing), "Human-Machine Strategy Generation Method" (Authors: Tian Jing; Liu Feiyang; Shi Tianjie; Feng Yuxin; Zhao Yixuan; Wen Minhua), "Multi-Aircraft Cooperative Air Combat Planning Method and System Based on Deep Reinforcement Learning" (Authors: Feng Yanghe; Cheng Guangquan; Shi Wei; Huang Kuihua; Huang Jincai; Liu Zhong), "Unmanned Swarm Intelligent Behavior Decision-Making Method and System Based on Large Model" (Xiong Gang; Zhao Yunlong; Zheng Haoyuan; Dong Xisong; Shen Zhen; Wang Weixing; He Chang; Lü Yisheng), "An Intelligent Generation Method for Unmanned Swarm Formation Control Algorithm Based on Large Model" (Li Xuelong; Zhang Tao; Yu Dengxiu; Wang Zhen; Gao Chao). Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a hierarchical command and control method that combines a large-scale model with reinforcement learning. This method outputs decision commands through a large-scale intelligent formation command system based on RAG (Reinforcement Aggregate Logic), and combines this with multi-agent reinforcement learning training technology to solve the problems mentioned in the background. This method provides dedicated knowledge support to the large-scale model using RAG technology, then generates structured decision commands through the large-scale model, and simultaneously trains agents for specific tasks using multi-agent reinforcement learning, ultimately achieving formation-level command and control.

[0006] The specific plan is as follows:

[0007] A hierarchical command and control method combining large-scale models and reinforcement learning is proposed, based on an agent with a hierarchical structure. The first layer agent is a large-scale intelligent formation command agent based on RAG (Reinforcement Algorithm), responsible for mission planning and formation configuration. The second layer agents consist of air strike formation and reconnaissance-strike hybrid formation agents, each executing tasks within its respective formation and cooperating to complete reconnaissance and strike missions. The second layer agents determine which sub-task to execute based on instructions sent by the first layer agents. The method includes the following steps:

[0008] S1. A large-scale intelligent formation command system based on RAG, including an offline knowledge preparation phase and an online decision-making and execution phase;

[0009] S2, define the state space, action space, and reward / punishment functions;

[0010] S3. Train the air strike formation and the reconnaissance-strike mixed formation agents separately, and build an intelligent game model based on the multi-agent reinforcement learning algorithm.

[0011] Furthermore, in the offline knowledge preparation stage of step S1, in order to provide the intelligent formation command system with complete and readily available decision-making basis, it is necessary to first complete the localization construction of the military knowledge base; the specific implementation steps are as follows:

[0012] Step 1: Local Document Collection

[0013] Professional knowledge files, including historical battle examples, tactical regulations, sensor feature descriptions, and real-time intelligence reports, covering multiple formats such as PDF, Word, and Txt, are uniformly stored in a designated local folder to form a closed collection of original corpora.

[0014] Step Two: Document Parsing and Two-Level Chunking

[0015] First-level segmentation: The original corpus is automatically segmented using a semantic coherence detection algorithm. When the semantic difference between adjacent segments exceeds a set threshold, a breakpoint is generated, thus obtaining a primary segment with a complete theme and logical coherence.

[0016] Second-level segmentation: Based on the fragment attributes, the segment is further refined. Code segments are divided into smaller segments to preserve contextual relationships, table areas are divided into medium segments to maintain the integrity of row and column information, and the main text is divided into larger segments to balance semantic density and retrieval efficiency, ultimately generating semantic units of appropriate size.

[0017] Step 3: Semantic Vectorization and Indexing

[0018] A deep neural network encoder f enc All semantic units are vectorized and mapped to a high-dimensional vector space v of uniform dimension. i The resulting vectors are then written in batches to a local vector database to create an offline index. This index supports both near nearest neighbor retrieval based on semantic distance and can be used in conjunction with keyword retrieval, providing underlying data support for "hybrid retrieval + reordering" in the online stage.

[0019] Furthermore, the online decision-making execution phase in step S1 is implemented in the following specific steps:

[0020] Step 1: Battlefield situation data and query vector generation

[0021] Step 1.1: Data Input

[0022] The system receives battlefield situation data in JSON format, which includes core elements such as troop deployment, equipment status, terrain coordinates, and sensor detection results of both sides, and uses this data as the input source for subsequent processing. The received JSON data is then converted into structured text content that conforms to the semantic logic of natural language, providing a unified text carrier for subsequent semantic encoding.

[0023] Output: Battlefield situation text data;

[0024] Step 1.2: Prompt Confirmation Construction

[0025] As an expert in the field of battlefield formation command, you need to make a deterministic selection of one of the two formation intelligent agents—air strike formation or reconnaissance-strike hybrid formation—based on the following battlefield situation text data, and provide key operational parameters, including the number of reconnaissance units, the configuration of strike units, and the flight altitude range; the specific content of the JSON data is: [embedded here 1) output—battlefield situation text data];

[0026] Output: Composite query results;

[0027] Step 1.3: Query Vector Generation

[0028] Reusing the deep neural network encoder f from the offline stage encThe composite query content constructed in step 1.2 is vectorized and mapped to a high-dimensional query vector v with the same dimension as the offline semantic unit. i This provides a directly accessible search vehicle for subsequent mixed searches;

[0029] Output: Query vector;

[0030] Step 2: Execute RAG Enhanced Search

[0031] Input: Query vector;

[0032] Step 2.1: Hybrid Similar Block Retrieval

[0033] The system employs a dual-track approach of "keyword retrieval + vector retrieval" to improve the comprehensiveness and efficiency of the search.

[0034] ①Based on the TF-IDF algorithm, keyword matching is performed on the core entities in the query to quickly filter out highly relevant text blocks;

[0035] ② Input the query vector into the vector database, and use the approximate nearest neighbor ANN search algorithm to calculate its cosine similarity with the situation text vector and tactical rule vector in the knowledge base, and return the top-k, that is, the top k semantically similar text blocks;

[0036] ③ Merge the two types of search results, remove duplicates, and form an initial set of search text blocks;

[0037] Step 2.2: Optimize Retrieval Results

[0038] Introducing a Cross-Encoder reordering model fine-tuned for the military field:

[0039] ① Using “query vector - text block vector” as input, the model captures the deep semantic relationship between the two in the military scenario through the attention mechanism inside the model, and finally outputs a fine relevance score on a scale of 0-10.

[0040] ② The initial set of text blocks is reordered from high to low scores, and text blocks with scores below 0.8 are removed. The top-5 high-value text blocks are retained to provide accurate knowledge support for subsequent large model generation.

[0041] Output: The retrieved text block;

[0042] Step 3: Integrate Input and Command Generation

[0043] The command decision prompt constructed in step 1 is concatenated with the text blocks retained after reordering in step 2 to form a comprehensive prompt containing a complete battlefield situation description and knowledge base reference text. This prompt is directly fed into the large language model, which generates the final combat command, realizing a closed-loop decision output with enhanced retrieval.

[0044] Furthermore, the specific implementation steps of step S2 are as follows:

[0045] A. Air Strike Group Command Intelligent Agent

[0046] Under the command and control of the formation commander, the air strike group command intelligence agent completes the strike on the opponent's long-range targets, forming a tracking-strike-target air-to-air kill chain, and intercepts the incoming targets at long distances.

[0047] 1. State space setting

[0048] The air strike group command agent can obtain the following status information:

[0049] Battlefield status acquired by manned / unmanned aircraft participating in air combat, including the observable situation of our side and the opponent; flight, platform and payload status of manned / unmanned aircraft;

[0050] 2. Motion space setting

[0051] Based on the air-to-air strike orders issued by the formation commander, the orders are further broken down into instructions for each firepower resource platform; the timing of the strike and the flight route plan are determined according to the current battlefield situation; and specific attack orders such as advancing, pausing, and striking are given to each formation.

[0052] 3. Setting reward and punishment functions

[0053] A reward of +5 is given for each enemy manned aircraft shot down; a reward of +3 is given for each enemy drone shot down; a penalty of -5 is given for each friendly manned aircraft shot down; a penalty of -3 is given for each friendly drone shot down.

[0054] B. Hybrid reconnaissance and strike command intelligent agent

[0055] Enemy aircraft that successfully penetrated the defenses attacked our nodes. The mixed command and control agent controlled each shipborne air defense missile and short-range air defense firepower to generate a tracking-strike-target air-sea kill chain, intercepted the incoming target at medium range, and completed the air defense and anti-missile mission.

[0056] 1. State space setting

[0057] The following status information can be obtained by investigating and attacking hybrid command agents:

[0058] The surrounding situation of the formation obtained through the formation commander; the situation of observable adversaries obtained through the formation commander; the position and navigation status of the friendly surface ships; and the status of the friendly surface ships' air defense and anti-missile firepower resources.

[0059] 2. Motion space setting

[0060] Based on the current situation of incoming enemy aircraft and weapons on the battlefield, target allocation and selection of air defense and anti-missile fire resources are carried out; decisions are made on when to launch air defense fire and the number to launch.

[0061] 3. Setting reward and punishment functions

[0062] For each enemy manned aircraft shot down, a reward of +5 is given; for each enemy drone shot down, a reward of +3 is given; for each enemy anti-ship weapon shot down, a reward of +2 is given; if a friendly ship is hit, a penalty of -10 is given; if a friendly aircraft carrier is hit, a penalty of -20 is given.

[0063] Furthermore, the multi-agent reinforcement learning architecture in step S3 is "centralized training + decentralized decision-making"; during training, a central controller is used to assist the agents in training; after training, the central controller is no longer needed, and each agent makes decisions independently based on local observations.

[0064] Furthermore, the centralized training system architecture involves deploying the value network on a central controller and the policy network on each agent; the decentralized decision-making system architecture involves agents no longer communicating after training, instead using their locally deployed policy networks to make decisions; the model training process is as follows:

[0065] Initialize the policy network and value function network for each agent;

[0066] (1) Intelligent agents collect experience data by interacting with the environment and store this data in a shared experience pool so that other intelligent agents can access and learn from it;

[0067] (2) Conduct centralized-distributed training; the agents perform training locally but share global information to promote better collaborative learning;

[0068] (3) Use value function networks to calculate the advantage function of each agent and measure the superiority of its chosen action relative to the average level;

[0069] (4) Perform PPO update: Update the policy network for each agent; use the PPO update rules to maximize the advantage function of experience and introduce a pruning term to limit the magnitude of a single update;

[0070] (5) Iteration: Repeat steps (2)-(5) until a satisfactory performance level or number of training sessions is achieved.

[0071] Furthermore, the specific algorithm flow is as follows:

[0072] 1) Initialize the evaluation network Policy Network π uThe neural network parameters ω and θ are hyperparameters, which are randomly initialized at the beginning and improved through training.

[0073] 2) Initialize the replay buffer D;

[0074] 3) Within a defined step size, each agent u executes the policy function. Generates an action, in which, For policy networks, For the action to be performed, The situation that each agent can obtain;

[0075] 4) Obtain the reward r based on the reward settings in step S2 and the simulation environment. t and the next state s t+1 ;

[0076] 5) Repeat steps 3) and 4) until the end of a game, obtaining the trajectory of one game for each agent.

[0077] Where T is the trajectory length;

[0078] 6) Calculate the Q-value and advantage: in, To estimate value,

[0079] Advantages;

[0080] 7) Storing data Go to the playback buffer D;

[0081] 8) Shuffle the data order in the replay buffer D and renumber them;

[0082] 9) Set the number of samples to j = 0;

[0083] 10) Sample a batch number B from the playback buffer D, which is D. j ,

[0084]

[0085] 11) Iterate through each agent and obtain the gradient. Update θ using Adam gradient descent u and ω u :

[0086]

[0087] Where, Δθ u For θ u Update amount, Δω u For ω uUpdate amount, f(r) i (θ u ) is the PPO algorithm function. To estimate value, To calculate value;

[0088] 12) Repeat steps 10) and 11) until... Otherwise, increment j by 1 and continue the loop through steps 11) and 12);

[0089] 13) Repeat steps 8)-12) until the set number of steps is reached;

[0090] 14) Repeat steps 3)-13) until the set number of rounds is reached;

[0091] 15) Save the policy model and complete the training.

[0092] The beneficial effects of this invention are as follows: Compared with existing formation command and control schemes, it fully leverages the advantages of large-scale models in knowledge retrieval and structured decision generation, and effectively enhances the model's generalization ability and adaptability in complex environments through hierarchical multi-agent reinforcement learning and alliance adversarial mechanisms. Specifically, this method uses RAG technology to inject exclusive knowledge support into the large-scale model, enabling it to output more rational and interpretable structured decision commands at the command level; at the lower level, the division of labor and cooperation among multiple agents improves the robustness and coordination of local task execution; at the same time, the hierarchical architecture can decompose large-scale formation decision problems into multiple manageable sub-tasks, reducing the overall decision complexity. This scheme not only solves the problem of insufficient information utilization in partially observable environments in traditional methods, but also enables flexible and efficient command and control in large-scale formation tasks, significantly improving the overall intelligence level and execution effect of the system. Attached Figure Description

[0093] Figure 1 This is a schematic diagram of an intelligent agent with a hierarchical structure in this invention.

[0094] Figure 2 This is a diagram of the large-scale intelligent formation command system based on RAG in this invention.

[0095] Figure 3 This is a diagram of the multi-agent reinforcement learning training architecture in this invention.

[0096] Figure 4 This is a flowchart of the multi-agent reinforcement learning training process in this invention. Detailed Implementation

[0097] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0098] The purpose of this invention is to provide a hierarchical command and control method that combines a large model with reinforcement learning, for use in multi-aircraft formation game scenarios. This method addresses scenarios involving the cooperation and decision-making of multiple formations by using a hierarchical approach to break down large-scale decision problems into sub-problems. By using a large model and reinforcement learning, the method helps the formation command agent make effective decisions while cooperating, thereby achieving optimal overall system performance.

[0099] At the formation command level, a linkage mechanism between a military knowledge vector base and a large model is constructed using RAG technology. This solves the problem of rigid decision-making caused by the inability of traditional expert rules to adapt to dynamic battlefield situations. Relying on the RAG military knowledge base for retrieval significantly improves the interpretability and executability of decisions. At the mission execution level, multi-agent reinforcement learning is used for training to obtain strategy models for specific tasks.

[0100] As shown in the figure, this invention provides a hierarchical command and control method combining large-scale models and reinforcement learning. Based on a hierarchical agent structure, the first-layer agent is a large-scale intelligent formation command agent based on RAG (Reinforcement Learning), responsible for mission planning and formation configuration. The second-layer agents consist of air strike formation and reconnaissance-strike hybrid formation agents, each executing tasks within their respective formations and cooperating to complete reconnaissance and strike missions. The second-layer agents determine which sub-task to execute based on instructions sent by the first-layer agents. The method includes the following steps:

[0101] S1. A large-scale intelligent formation command system based on RAG, including an offline knowledge preparation phase and an online decision-making and execution phase;

[0102] S2, define the state space, action space, and reward / punishment functions;

[0103] S3. Train the air strike formation and the reconnaissance-strike mixed formation agents separately, and build an intelligent game model based on the multi-agent reinforcement learning algorithm.

[0104] In this embodiment, in the offline knowledge preparation stage of step S1, in order to provide complete and readily accessible decision-making basis for the intelligent formation command system's online decision-making, it is necessary to first complete the localization construction of the military knowledge base; the specific implementation steps are as follows:

[0105] Step 1: Local Document Collection

[0106] Professional knowledge files, including historical battle examples, tactical regulations, sensor feature descriptions, and real-time intelligence reports, covering multiple formats such as PDF, Word, and Txt, are uniformly stored in a designated local folder to form a closed collection of original corpora.

[0107] Step Two: Document Parsing and Two-Level Chunking

[0108] First-level segmentation: The original corpus is automatically segmented using a semantic coherence detection algorithm. When the semantic difference between adjacent segments exceeds a set threshold, a breakpoint is generated, thus obtaining a primary segment with a complete theme and logical coherence.

[0109] Second-level segmentation: Based on the fragment attributes, the segment is further refined. Code segments are divided into smaller segments to preserve contextual relationships, table areas are divided into medium segments to maintain the integrity of row and column information, and the main text is divided into larger segments to balance semantic density and retrieval efficiency, ultimately generating semantic units of appropriate size.

[0110] Step 3: Semantic Vectorization and Indexing

[0111] A deep neural network encoder f enc All semantic units are vectorized and mapped to a high-dimensional vector space v of uniform dimension. i The resulting vectors are then written in batches to a local vector database to create an offline index. This index supports both near nearest neighbor retrieval based on semantic distance and can be used in conjunction with keyword retrieval, providing underlying data support for "hybrid retrieval + reordering" in the online stage.

[0112] In this embodiment, the online decision execution stage in step S1 is implemented in the following specific steps:

[0113] Step 1: Battlefield situation data and query vector generation

[0114] Step 1.1: Data Input

[0115] The system receives battlefield situation data in JSON format, which includes core elements such as troop deployment, equipment status, terrain coordinates, and sensor detection results of both sides, and uses this data as the input source for subsequent processing. The received JSON data is then converted into structured text content that conforms to the semantic logic of natural language, providing a unified text carrier for subsequent semantic encoding.

[0116] Output: Battlefield situation text data;

[0117] Step 1.2: Prompt Confirmation Construction

[0118] As an expert in the field of battlefield formation command, you need to make a deterministic selection of one of the two formation intelligent agents—air strike formation or reconnaissance-strike hybrid formation—based on the following battlefield situation text data, and provide key operational parameters, including the number of reconnaissance units, the configuration of strike units, and the flight altitude range; the specific content of the JSON data is: [embedded here 1) output—battlefield situation text data];

[0119] Output: Composite query results;

[0120] Step 1.3: Query Vector Generation

[0121] Reusing the deep neural network encoder f from the offline stage enc The composite query content constructed in step 1.2 is vectorized and mapped to a high-dimensional query vector v with the same dimension as the offline semantic unit. i This provides a directly accessible search vehicle for subsequent mixed searches;

[0122] Output: Query vector;

[0123] Step 2: Execute RAG Enhanced Search

[0124] Input: Query vector;

[0125] Step 2.1: Hybrid Similar Block Retrieval

[0126] The system employs a dual-track approach of "keyword retrieval + vector retrieval" to improve the comprehensiveness and efficiency of the search.

[0127] ①Based on the TF-IDF algorithm, keyword matching is performed on the core entities in the query to quickly filter out highly relevant text blocks;

[0128] ② Input the query vector into the vector database, and use the approximate nearest neighbor ANN search algorithm to calculate its cosine similarity with the situation text vector and tactical rule vector in the knowledge base, and return the top-k, that is, the top k semantically similar text blocks;

[0129] ③ Merge the two types of search results, remove duplicates, and form an initial set of search text blocks;

[0130] Step 2.2: Optimize Retrieval Results

[0131] Introducing a Cross-Encoder reordering model fine-tuned for the military field:

[0132] ① Using “query vector - text block vector” as input, the model captures the deep semantic relationship between the two in the military scenario through the attention mechanism inside the model, and finally outputs a fine relevance score on a scale of 0-10.

[0133] ② The initial set of text blocks is reordered from high to low scores, and text blocks with scores below 0.8 are removed. The top-5 high-value text blocks are retained to provide accurate knowledge support for subsequent large model generation.

[0134] Output: The retrieved text block;

[0135] Step 3: Integrate Input and Command Generation

[0136] The command decision prompt constructed in step 1 is concatenated with the text blocks retained after reordering in step 2 to form a comprehensive prompt containing a complete battlefield situation description and knowledge base reference text. This prompt is directly fed into the large language model, which generates the final combat command, realizing a closed-loop decision output with enhanced retrieval.

[0137] In this embodiment, the specific implementation steps of step S2 are as follows:

[0138] A. Air Strike Group Command Intelligent Agent

[0139] Under the command and control of the formation commander, the air strike group command intelligence agent completes the strike on the opponent's long-range targets, forming a tracking-strike-target air-to-air kill chain, and intercepts the incoming targets at long distances.

[0140] 1. State space setting

[0141] The air strike group command agent can obtain the following status information:

[0142] Battlefield status acquired by manned / unmanned aircraft participating in air combat, including the observable situation of our side and the opponent; flight, platform and payload status of manned / unmanned aircraft;

[0143] 2. Motion space setting

[0144] Based on the air-to-air strike orders issued by the formation commander, the orders are further broken down into instructions for each firepower resource platform; the timing of the strike and the flight route plan are determined according to the current battlefield situation; and specific attack orders such as advancing, pausing, and striking are given to each formation.

[0145] 3. Setting reward and punishment functions

[0146] A reward of +5 is given for each enemy manned aircraft shot down; a reward of +3 is given for each enemy drone shot down; a penalty of -5 is given for each friendly manned aircraft shot down; a penalty of -3 is given for each friendly drone shot down.

[0147] B. Hybrid reconnaissance and strike command intelligent agent

[0148] Enemy aircraft that successfully penetrated the defenses attacked our nodes. The mixed command and control agent controlled each shipborne air defense missile and short-range air defense firepower to generate a tracking-strike-target air-sea kill chain, intercepted the incoming target at medium range, and completed the air defense and anti-missile mission.

[0149] 1. State space setting

[0150] The following status information can be obtained by investigating and attacking hybrid command agents:

[0151] The surrounding situation of the formation obtained through the formation commander; the situation of observable adversaries obtained through the formation commander; the position and navigation status of the friendly surface ships; and the status of the friendly surface ships' air defense and anti-missile firepower resources.

[0152] 2. Motion space setting

[0153] Based on the current situation of incoming enemy aircraft and weapons on the battlefield, target allocation and selection of air defense and anti-missile fire resources are carried out; decisions are made on when to launch air defense fire and the number to launch.

[0154] 3. Setting reward and punishment functions

[0155] For each enemy manned aircraft shot down, a reward of +5 is given; for each enemy drone shot down, a reward of +3 is given; for each enemy anti-ship weapon shot down, a reward of +2 is given; if a friendly ship is hit, a penalty of -10 is given; if a friendly aircraft carrier is hit, a penalty of -20 is given.

[0156] In this embodiment, the multi-agent reinforcement learning architecture in step S3 is "centralized training + decentralized decision-making"; during training, a central controller is used to assist the agents in training; after training, the central controller is no longer needed, and each agent makes decisions independently based on its local observations. The centralized training system architecture: the value network is deployed on the central controller, and the policy network is deployed on each agent; the decentralized decision-making system architecture: after training is complete, the agents no longer communicate, and each agent makes decisions using its locally deployed policy network; the model training process is as follows:

[0157] Initialize the policy network and value function network for each agent;

[0158] (1) Intelligent agents collect experience data by interacting with the environment and store this data in a shared experience pool so that other intelligent agents can access and learn from it;

[0159] (2) Conduct centralized-distributed training; the agents perform training locally but share global information to promote better collaborative learning;

[0160] (3) Use value function networks to calculate the advantage function of each agent and measure the superiority of its chosen action relative to the average level;

[0161] (4) Perform PPO update: Update the policy network for each agent; use the PPO update rules to maximize the advantage function of experience and introduce a pruning term to limit the magnitude of a single update;

[0162] (5) Iteration: Repeat steps (2)-(5) until a satisfactory performance level or number of training sessions is achieved.

[0163] In this embodiment, the specific algorithm flow is as follows:

[0164] 1) Initialize the evaluation network Policy Network π u The neural network parameters ω and θ are hyperparameters, which are randomly initialized at the beginning and improved through training.

[0165] 2) Initialize the replay buffer D;

[0166] 3) Within a defined step size, each agent u executes the policy function. Generates an action, in which, For policy networks, For the action to be performed, The situation that each agent can obtain;

[0167] 4) Obtain the reward r based on the reward settings in step S2 and the simulation environment. t and the next state s t+1 ;

[0168] 5) Repeat steps 3) and 4) until the end of a game, obtaining the trajectory of one game for each agent.

[0169] Where T is the trajectory length;

[0170] 6) Calculate the Q-value and advantage: in, To estimate value,

[0171] Advantages;

[0172] 7) Storing data Go to the playback buffer D;

[0173] 8) Shuffle the data order in the replay buffer D and renumber them;

[0174] 9) Set the number of samples to j = 0;

[0175] 10) Sample a batch number B from the playback buffer D, which is D. j ,

[0176]

[0177] 11) Iterate through each agent and obtain the gradient. Update θ using Adam gradient descent u and ω u :

[0178]

[0179] Where, Δθ u For θ u Update amount, Δω u For ω u Update amount, f(r) i (θ u ) is the PPO algorithm function. To estimate value, To calculate value;

[0180] 12) Repeat steps 10) and 11) until... Otherwise, increment j by 1 and continue the loop through steps 11) and 12);

[0181] 13) Repeat steps 8)-12) until the set number of steps is reached;

[0182] 14) Repeat steps 3)-13) until the set number of rounds is reached;

[0183] 15) Save the policy model and complete the training.

[0184] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A hierarchical command and control method combining a large model and reinforcement learning, characterized in that, Based on a hierarchical intelligent agent structure, the first layer of intelligent agents is a large-scale intelligent formation command intelligent agent based on RAG, which is responsible for task planning and formation configuration; The second-layer intelligent agent consists of air strike groups and reconnaissance-strike hybrid groups, each executing tasks within its own group and cooperating with each other to complete reconnaissance and strike missions. The second-layer intelligent agent determines which sub-task to execute based on instructions sent by the first-layer intelligent agent; this includes the following steps: S1. A large-scale intelligent formation command system based on RAG, including an offline knowledge preparation phase and an online decision-making and execution phase; S2, define the state space, action space, and reward / punishment functions; S3. Train the air strike formation and the reconnaissance-strike mixed formation agents separately, and build an intelligent game model based on the multi-agent reinforcement learning algorithm.

2. The hierarchical command and control method combining a large model and reinforcement learning as described in claim 1, characterized in that, The offline knowledge preparation stage in step S1 is implemented in the following steps: Step 1: Local Document Collection Professional knowledge files are stored in a designated local folder to form a closed collection of original corpora. Step Two: Document Parsing and Two-Level Chunking First-level segmentation: The original corpus is automatically segmented using a semantic coherence detection algorithm. When the semantic difference between adjacent segments exceeds a set threshold, a breakpoint is generated, thus obtaining a primary segment with a complete theme and logical coherence. Second-level segmentation: Based on the fragment attributes, the segment is further refined. Code segments are divided into smaller segments to preserve contextual relationships, table areas are divided into medium segments to maintain the integrity of row and column information, and the main text is divided into larger segments to balance semantic density and retrieval efficiency, ultimately generating semantic units of appropriate size. Step 3: Semantic Vectorization and Indexing A deep neural network encoder f enc All semantic units are vectorized and mapped to a high-dimensional vector space v of uniform dimension. i The resulting vectors are then written in batches to a local vector database to create an offline index. This index supports both near nearest neighbor retrieval based on semantic distance and can be used in conjunction with keyword retrieval.

3. The hierarchical command and control method combining a large model and reinforcement learning according to claim 2, characterized in that, The online decision-making execution phase in step S1 is implemented in the following ways: Step 1: Battlefield situation data and query vector generation Step 1.1: Data Input The system receives battlefield situation data in JSON format, which includes core elements such as troop deployment, equipment status, terrain coordinates, and sensor detection results of both sides, and uses this data as the input source for subsequent processing. The received JSON data is then converted into structured text content that conforms to the semantic logic of natural language, providing a unified text carrier for subsequent semantic encoding. Output: Battlefield situation text data; Step 1.2: Prompt Confirmation Construction As an expert in the field of battlefield formation command, you need to make a deterministic selection of one of the two formation intelligent agents—air strike formation or reconnaissance-strike hybrid formation—based on the following battlefield situation text data, and provide key operational parameters, including the number of reconnaissance units, the configuration of strike units, and the flight altitude range; the specific content of the JSON data is: [embedded here 1) output—battlefield situation text data]; Output: Composite query results; Step 1.3: Query Vector Generation Reusing the deep neural network encoder f from the offline stage enc The composite query content constructed in step 1.2 is vectorized and mapped to a high-dimensional query vector v with the same dimension as the offline semantic unit. i This provides a directly accessible search vehicle for subsequent mixed searches; Output: Query vector; Step 2: Execute RAG Enhanced Search Input: Query vector; Step 2.1: Hybrid Similar Block Retrieval The system employs a dual-track approach of "keyword retrieval + vector retrieval" to improve the comprehensiveness and efficiency of the search. ①Based on the TF-IDF algorithm, keyword matching is performed on the core entities in the query to quickly filter out highly relevant text blocks; ② Input the query vector into the vector database, and use the approximate nearest neighbor ANN search algorithm to calculate its cosine similarity with the situation text vector and tactical rule vector in the knowledge base, and return the top-k, that is, the top k semantically similar text blocks; ③ Merge the two types of search results, remove duplicates, and form an initial set of search text blocks; Step 2.2: Optimize Retrieval Results Introducing a Cross-Encoder reordering model fine-tuned for the military field: ① Using "query vector - text block vector" as input, the model captures the deep semantic relationship between the two in the military scenario through the attention mechanism inside the model, and finally outputs a fine relevance score on a scale of 0-10. ② The initial set of text blocks is reordered from high to low scores, and text blocks with scores below 0.8 are removed. The top-5 high-value text blocks are retained to provide accurate knowledge support for subsequent large model generation. Output: The retrieved text block; Step 3: Integrate Input and Command Generation The command decision prompt constructed in step 1 is concatenated with the text blocks retained after reordering in step 2 to form a comprehensive prompt containing a complete battlefield situation description and knowledge base reference text. This prompt is directly fed into the large language model, which generates the final combat command, realizing a closed-loop decision output with enhanced retrieval.

4. The hierarchical command and control method combining a large model and reinforcement learning as described in claim 1, characterized in that, The specific implementation steps of step S2 are as follows: A. Air Strike Group Command Intelligent Agent Under the command and control of the formation commander, the air strike group command intelligence agent completes the strike on the opponent's long-range targets, forming a tracking-strike-target air-to-air kill chain, and intercepts the incoming targets at long distances.

1. State space setting The air strike group command agent can obtain the following status information: Battlefield status acquired by manned / unmanned aircraft participating in air combat, including the observable situation of our side and the opponent; flight, platform and payload status of manned / unmanned aircraft; 2. Motion space setting Based on the air-to-air strike orders issued by the formation commander, the orders are further broken down into instructions for each firepower resource platform; the timing of the strike and the flight route plan are determined according to the current battlefield situation; and specific attack orders such as advancing, pausing, and striking are given to each formation.

3. Setting reward and punishment functions A reward of +5 is given for each enemy manned aircraft shot down; a reward of +3 is given for each enemy drone shot down; a penalty of -5 is given for each friendly manned aircraft shot down; a penalty of -3 is given for each friendly drone shot down. B. Hybrid reconnaissance and strike command intelligent agent Enemy aircraft that successfully penetrated the defenses attacked our nodes. The mixed command and control agent controlled each shipborne air defense missile and short-range air defense firepower to generate a tracking-strike-target air-sea kill chain, intercepted the incoming target at medium range, and completed the air defense and anti-missile mission.

1. State space setting The following status information can be obtained by investigating and attacking hybrid command agents: The surrounding situation of the formation obtained through the formation commander; the situation of observable adversaries obtained through the formation commander; the position and navigation status of the friendly surface ships; and the status of the friendly surface ships' air defense and anti-missile firepower resources.

2. Motion space setting Based on the current situation of incoming enemy aircraft and weapons on the battlefield, target allocation and selection of air defense and anti-missile fire resources are carried out; decisions are made on when to launch air defense fire and the number to launch.

3. Setting reward and punishment functions For each enemy manned aircraft shot down, a reward of +5 is given; for each enemy drone shot down, a reward of +3 is given; for each enemy anti-ship weapon shot down, a reward of +2 is given; if a friendly ship is hit, a penalty of -10 is given; if a friendly aircraft carrier is hit, a penalty of -20 is given.

5. The hierarchical command and control method combining a large model and reinforcement learning according to claim 1, characterized in that, The multi-agent reinforcement learning architecture in step S3 is "centralized training + decentralized decision-making"; During training, a central controller is used to assist the agent in training. After training, a central controller is no longer needed, and each agent makes decisions independently based on local observations.

6. The hierarchical command and control method combining a large model and reinforcement learning according to claim 5, characterized in that, The system architecture for centralized training is as follows: the value network is deployed on a central controller, and the policy network is deployed on each agent. The system architecture for decentralized decision-making is as follows: after training is complete, agents no longer communicate; instead, they use their locally deployed policy networks to make decisions. The model training process is as follows: Initialize the policy network and value function network for each agent; (1) Intelligent agents collect experience data by interacting with the environment and store this data in a shared experience pool so that other intelligent agents can access and learn from it; (2) Conduct centralized-distributed training; the agents perform training locally but share global information to promote better collaborative learning; (3) Use value function networks to calculate the advantage function of each agent and measure the superiority of its chosen action relative to the average level; (4) Perform PPO update: Update the policy network for each agent; use the PPO update rules to maximize the advantage function of experience and introduce a pruning term to limit the magnitude of a single update; (5) Iteration: Repeat steps (2)-(5) until a satisfactory performance level or number of training sessions is achieved.

7. A hierarchical command and control method combining a large model and reinforcement learning according to claim 6, characterized in that, The specific algorithm flow is as follows: 1) Initialize the evaluation network Policy Network π u The neural network parameters ω and θ are hyperparameters, which are randomly initialized at the beginning and improved through training. 2) Initialize the replay buffer D; 3) Within a defined step size, each agent u executes the policy function. Generates an action, in which, For policy networks, For the action to be performed, The situation that each agent can obtain; 4) Obtain the reward r based on the reward settings in step S2 and the simulation environment. t and the next state s t+1 ; 5) Repeat steps 3) and 4) until the end of a game, obtaining the trajectory of one game for each agent. Where T is the trajectory length; 6) Calculate the Q-value and advantage: in, To estimate value, Advantages; 7) Storing data Go to the playback buffer D; 8) Shuffle the data order in the replay buffer D and renumber them; 9) Set the number of samples to j = 0; 10) Sample a batch number B from the playback buffer D, which is D. j , 11) Iterate through each agent and obtain the gradient. Update θ using Adam gradient descent u and ω u : Where, Δθ u For θ u Update amount, Δω u For ω u Update amount, f(r) i (θ u ) is the PPO algorithm function. To estimate value, To calculate value; 12) Repeat steps 10) and 11) until... Otherwise, increment j by 1 and continue the loop through steps 11) and 12); 13) Repeat steps 8)-12) until the set number of steps is reached; 14) Repeat steps 3)-13) until the set number of rounds is reached; 15) Save the policy model and complete the training.

Images