A command decision-making deep Q-network intelligent agent model system based on AOS framework

Abstract

The present invention discloses a command decision-making deep Q network intelligent agent model system based on the AOS framework. The system includes an AOS framework layer, an upper layer and a lower layer. The AOS framework layer isolates the upper layer from the lower layer. The upper layer includes a cognitive decision-making layer and a user interface layer, and the lower layer includes an information processing layer and a data access layer. The AOS framework layer calls basic data through the data access layer to provide an access interface for the cognitive decision-making layer. The information processing layer completes data processing by combining a military knowledge map and battlefield situation data through a data integration platform. The data access layer provides a data management interface and completes data processing according to the instructions of the AOS framework layer. The cognitive decision-making layer formulates command strategies and performs tactical deductions, and realizes situational analysis, strategy selection and action simulation through the AOS framework layer based on the instruction requests of the user interface layer. The user interface layer provides decision display and interactive operations for users, provides system analysis results, receives user instruction requests and passes them to the cognitive decision-making layer.

Description

Technical Field

[0001] The present invention relates to the field of decision-making intelligence and artificial intelligence applications, and in particular to a command decision-making deep Q network intelligent agent model system and method based on an AOS (Agent-Objective-Scene) framework. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] With the rapid development of artificial intelligence and big data technologies, the demand for intelligent and real-time response in the field of command decision-making is growing. At present, mainstream decision support systems are usually based on traditional rule models and statistical analysis methods, which have good performance in structured data processing. However, in complex and changing military scenarios, traditional systems are difficult to meet the needs of real-time processing and adaptive decision-making of dynamic data.

[0004] In order to improve decision-making efficiency, reinforcement learning has gradually been introduced into intelligent decision-making. Deep reinforcement learning algorithms such as DQN (Deep Q-Network) use neural networks to adaptively learn and generate strategies for large-scale data, and are suitable for intelligent agent decision-making in complex environments. However, in military command scenarios, simple algorithm models cannot effectively integrate multi-source heterogeneous data, and cannot provide efficient and reliable cognitive decision support in multi-target environments. In response to these needs, the AOS (Agent-Objective-Scene) framework was proposed to build a command agent model and realize target recognition and adaptive strategy generation in situational reasoning.

[0005] Currently, there are a variety of technologies available for intelligent decision-making systems, such as the combination of knowledge graphs and reinforcement learning models. Among them, knowledge graphs, as an effective data organization method, can use entity and relationship models to perform semantic association processing on unstructured and semi-structured data. Its typical applications include reasoning logic, scene understanding, and target recognition. However, the construction and management of knowledge graphs require high computing support and a rich data foundation. In practical applications, there is still a trade-off between efficiency and accuracy.

[0006] In addition, deep reinforcement learning algorithms such as DQN and its derivatives have emerged in recent years, which continuously optimize decision-making in uncertain situations by training intelligent agents to interact with the environment. However, traditional reinforcement learning algorithms face the problem of "state space explosion" in high-dimensional data and multi-objective environments, resulting in low learning efficiency. Summary of the invention

[0007] Purpose of the invention: In view of the shortcomings of the above-mentioned prior art, the present invention proposes a command decision model system and method based on the AOS (Agent-Objective-Scene) framework. The model combines deep reinforcement learning and knowledge graph technology to provide situational reasoning capabilities and real-time decision support for command agents, thereby adapting to command needs in a changing environment.

[0008] Technical solution: The present invention proposes a command decision-making intelligent agent model based on the AOS framework.

[0009] The system includes an information processing layer, a data access layer, an AOS framework layer, a cognitive decision layer and a user interface layer. The upper layer and the lower layer are isolated by the AOS framework layer. The upper layer includes the cognitive decision layer and the user interface layer, and the lower layer includes the information processing layer and the data access layer.

[0010] Information processing layer: Integrate multi-source battlefield data through the data integration platform to generate battlefield situation information, combine military knowledge graphs and battlefield situation data to complete data processing, including threat intelligence, resource status and target location. The data types include text, numerical values, geographic information and target images.

[0011] Data access layer: provides data access interface, calls data according to the instructions of AOS framework layer, including query, transmission and update. Data access layer supports multiple data sources and provides unified interface upward.

[0012] AOS framework layer: used to connect the data access layer and the cognitive decision layer. The AOS framework layer establishes a decision model through deep reinforcement learning models (such as DQN) combined with knowledge graphs. The AOS framework layer generates the strategy of the intelligent agent according to the environmental situation and provides a reasoning interface. It uses the knowledge graph to perform information association and situational reasoning to help the intelligent agent adapt to the dynamic battlefield environment. The AOS framework layer calls basic data through the data access layer and provides an access interface for the cognitive decision layer.

[0013] Cognitive decision-making layer: used to formulate command strategies and perform tactical deductions. According to the user interface layer or system requirements, the strategy generation and reasoning functions of the AOS framework layer are called to finally output the decision plan. The cognitive decision-making layer handles task allocation, threat assessment, and action recommendations, and returns the results to the user interface layer.

[0014] User interface layer: Provides decision display and interactive operation to users in a Web-based manner, and provides system analysis results. The user interface layer presents situational deduction results and decision suggestions, accepts user feedback requests, passes them to the cognitive decision layer, realizes interactive analysis and decision-making, and receives strategy suggestions returned by the cognitive decision layer.

[0015] Furthermore, the AOS framework layer includes two parts: a knowledge graph module and a decision model module.

[0016] The knowledge graph module constructs and stores a military data model, which includes an entity model, a relationship model and an attribute model. The knowledge graph module forms a knowledge graph from the military data model through relationship mapping and attribute constraints; generates a structured knowledge representation of the battlefield situation, describes entity information (such as targets, threats, tasks) through graph nodes, and provides capabilities such as situational association and causal reasoning.

[0017] The decision model module uses a deep reinforcement learning algorithm to build an intelligent agent, realizes situational awareness and decision optimization through strategy generation and environment interaction, and uses experience replay and strategy update mechanisms to improve the learning efficiency of the model; the combination of knowledge graph and DQN model enhances the system's situational understanding and strategy generation capabilities.

[0018] In a deep Q network (DQN), the model collects reward signals from the environment as feedback to guide the learning process through interaction with the environment. Through feedback, DQN uses a deep neural network to adjust the weights and parameters of the network to maximize the total reward, thereby optimizing the decision-making strategy.

[0019] The AOS framework layer deploys a deep Q network model, establishes a cognitive model and analyzes the input information based on the knowledge graph, and provides a decision-making interface for the intelligent agent model.

[0020] Furthermore, the data processing at the information processing layer includes collection and fusion, and the data includes geographic information, threat intelligence, resource information and task instructions.

[0021] Furthermore, the access interfaces provided by the AOS framework layer to the cognitive decision layer include data query, model training and status update.

[0022] Furthermore, the deep Q network DQN can be replaced by other deep reinforcement learning algorithms that meet the requirements.

[0023] A command decision method is applied to a command decision deep Q network intelligent agent model system based on an AOS framework.

[0024] Furthermore, the method comprises the following steps:

[0025] Step 1: Decision reasoning: The data access layer sends a decision instruction to the AOS framework layer, and the AOS framework layer generates a preliminary plan and performs optimization.

[0026] Step 2: Strategy optimization. The cognitive decision layer sends optimization instructions to the AOS framework layer. The AOS framework layer generates an optimization plan based on the existing state information. The cognitive decision layer selects the best plan and provides optimization suggestions to the user through the user interface layer.

[0027] Step 3: Scenario simulation. After the user interface layer triggers the task, the cognitive decision layer transmits the requirements to the AOS framework to generate the scenario environment. The decision model module establishes and monitors the simulation scenario and feeds back real-time data to the user.

[0028] Furthermore, the decision reasoning in step 1 includes the following steps:

[0029] Step 1-1: After receiving the decision request, the data access layer sends instructions to the AOS framework layer. The AOS framework layer obtains battlefield situation information through the knowledge graph module. The information includes threat factors, resource conditions, and combat environment.

[0030] Step 1-2: The decision model module extracts the characteristics of the combat situation based on the knowledge graph information and calls the deep reinforcement learning model to generate an action plan; if the situation data does not cover complete information, supplementary information is obtained through knowledge graph reasoning, and the weight of the decision model is adjusted according to different battlefield conditions.

[0031] Step 1-3: The model will simulate and deduce the generated preliminary plans, compare the effects of each plan through the decision-making module, and sort them by priority.

[0032] Furthermore, the step 2 strategy optimization includes the following steps:

[0033] Step 2-1: The cognitive decision-making layer sends optimization instructions to the AOS framework layer based on user command requests or task update requirements. The AOS framework layer extracts the current combat status characteristics through the DQN model. The status characteristics include unit location, resource quantity and action constraints.

[0034] Step 2-2: The decision model module generates an action plan based on the existing state information and reward function and updates the action priority, and pushes the plan to the cognitive decision layer according to the priority.

[0035] Step 2-3: The cognitive decision layer selects the best solution based on the optimization results and provides optimization suggestions to the user through the user interface layer.

[0036] Furthermore, the scenario simulation in step 3 includes the following steps:

[0037] Step 3-1: The user interface layer triggers the situational simulation task through user instructions. The cognitive decision-making layer transmits relevant requirements to the AOS framework layer according to the instruction content. The AOS framework layer uses the knowledge graph information to generate the situational environment.

[0038] Step 3-2: The decision model module establishes a simulation scenario based on the situational environment, monitors the simulation process, and outputs simulation feedback based on state changes.

[0039] Step 3-3: The user interface layer receives the situation simulation feedback results and presents the real-time data of the situation simulation to the user.

[0040] Beneficial effects: The present invention realizes dynamic intelligent command in complex situations by combining knowledge graphs and deep reinforcement learning technology. The introduction of knowledge graphs simplifies the expression of battlefield situations and improves the efficiency of situational reasoning. Combining the DQN model for strategy generation enables the intelligent agent to adapt to changing environments and effectively solves the decision-making problem in high-dimensional states. The system exhibits high response speed and stability in command decision-making. The present invention integrates the AOS framework with the deep reinforcement learning model, performs situational reasoning and strategy optimization based on the knowledge graph, and provides an intelligent command model that adapts to complex and changing environments, thereby effectively improving the real-time and reliability of the decision-making system in battlefield situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0042] Figure 1 It is a schematic diagram of the decision-making process of the RPD model of the present invention.

[0043] Figure 2 It is a schematic diagram of the cognitive judgment process of the command decision-making intelligent system performed by the AOS framework of the present invention.

[0044] Figure 3 It is a processing flow diagram of the present invention.

[0045] Figure 4 It is a schematic diagram of the basic idea of ​​the DQN algorithm of the present invention.

[0046] Figure 5 It is a schematic diagram of the deep learning state feature design of the present invention.

[0047] Figure 6 It is a schematic diagram of the battlefield feature extraction product neural network of the present invention.

[0048] Figure 7 It is a schematic diagram of the behavior decision model based on rule + deep reinforcement learning of the present invention.

[0049] Figure 8 It is a schematic diagram of the DQN algorithm network structure of the present invention.

[0050] Fig. 9 It is a schematic diagram of the winning rate and reward value of the models of different algorithms of the present invention.

[0051] Fig.10 It is a schematic diagram of the experimental results of the three AIs of the present invention at different difficulty levels.

[0052] Fig.11 It is a schematic diagram of the training time and test set accuracy of different algorithm models of the present invention.

[0053] Fig.12 It is a schematic diagram of the learning rate change of the present invention.

[0054] Fig.13 It is a schematic diagram of the change of the experience playback cache size of the present invention.

[0055] Fig.14 It is a schematic diagram of the target network update frequency change of the present invention.

[0056] Fig.15 It is a visual schematic diagram of the AI ​​model operation results of the present invention. DETAILED DESCRIPTION

[0057] Figure 1 Schematic diagram of the decision-making process of the RPD model of the present invention. The decision-making process steps described in the model are as follows:

[0058] Step S1: Decision makers obtain current situation information through various means.

[0059] Step S2, extracting situation features based on one's own experience.

[0060] Step S3, determine whether the current situation is familiar or typical, if "yes", go to step S4; otherwise re-evaluate the situation or collect more situation information and return to step S1.

[0061] Step S4, determine the current goal, situation characteristics, expected results and action methods based on experience. Quickly check whether the current preliminary plan can achieve the expected results based on experience. If "yes", extract the goal, situation characteristics and action plan from the existing situation memory in experience, match them with the current situation, and extract the action plan; otherwise, return to step S1.

[0062] Step S5, conduct mental simulation of the action plan, that is, use mental simulation to simulate the effect of the execution of the current action plan, and predict whether the plan can fully achieve the predetermined goal. If "yes", go to step S6; otherwise return to step S4; if it is partially feasible, make adjustments and then simulate again.

[0063] Step S6, execute the action plan, proceed to the next question decision or end.

[0064] Figure 2It is a schematic diagram of the cognitive judgment process of the command decision-making intelligent system performed by the AOS framework of the present invention. It mainly includes an intelligent situational cognition model, an intelligent decision-making planning model and an intelligent comprehensive control model. Among them, the intelligent situational cognition model is mainly used to abstract and process incomplete and imperfect complex situation information to form a scene, and to judge, identify and classify key scenes; the intelligent decision-making planning model mainly makes reasoning decisions based on scenes and knowledge to form action plans; the comprehensive control model performs global optimization control on the execution of the plan on the one hand, and controls the interaction between the intelligent agent and the superior commander (human or intelligent agent), the scene knowledge map and the combat simulation environment on the other hand.

[0065] Figure 3 It is a schematic diagram of the processing flow of the present invention. For the cleaned knowledge base data, the LLM model and the method of manually generating question-answer pairs are used to construct test data to improve the quality of the test data. By calling the LLM interface and providing a guiding context, the model is informed of the background and related information of the question to ensure that the generated questions are related to specific fields. Then, the prompt engineering method is used to provide example questions and continuously correct the prompt to guide the LLM to generate similar question-answer data. Knowledge base keywords are introduced into the prompt to ensure that the generated question-answer data contains key information related to the task. By adjusting the Temperature parameter, the diversity and accuracy of the generated questions are controlled. At the same time, questions are constructed using manual methods and randomly inserted into the machine-generated results to evaluate the generated answers.

[0066] Figure 4 It is a schematic diagram of the basic idea of ​​the DQN algorithm of the present invention. The DQN algorithm is a deep reinforcement learning algorithm based on the Q-learning algorithm, which uses a neural network to approximate the Q function. Specifically, the DQN algorithm uses a deep neural network to fit the Q function, and uses experience replay and target network for stable update. Among them, experience replay refers to storing the agent's historical experience in an experience pool, and then randomly sampling from it for training. The target network refers to using a fixed neural network to calculate the target Q value within a certain time interval, thereby stably updating the Q function.

[0067] Figure 5It is a schematic diagram of the deep learning state feature design of the present invention. The design extracts battlefield features from both single and global aspects. Global features include the overall force control situation, time process, operator differentiation, etc. BOP features include but are not limited to: attack: ammunition situation, etc.; defense: operator health, etc.; excitement: current speed; time: remaining time for weapon cooling, etc.; state: whether it is stationary, etc.; position: distance from the control point, etc. Map features include visual conditions, observation conditions, supply conditions, attack level, etc. Nearby features include terrain, elevation difference, cost, line of sight, attack level, etc.

[0068] Figure 6 This is a schematic diagram of the battlefield feature extraction neural network of the present invention. The input layer composed of feature vectors and matrices is alternately convolved and sub-sampled to achieve a high degree of abstraction of the feature vectors of the commander's behavior and the state of the evolution process of tactics and decision-making.

[0069] Figure 7 This is a schematic diagram of the behavior decision model based on rules + deep reinforcement learning of the present invention. The advantages of rule-based methods and learning-based methods are complementary. This decision method uses the DDPG decision module as the behavior node of the behavior tree, divides the task into multiple subtasks using the logical rules and prior knowledge of the behavior tree model, and uses the deep reinforcement learning method in a modular way.

[0070] Figure 8 Schematic diagram of the DQN algorithm network structure of the present invention. Figure 4 same.

[0071] A command decision-making agent model based on AOS framework.

[0072] The system includes an information processing layer, a data access layer, an AOS framework layer, a cognitive decision layer and a user interface layer. The upper layer and the lower layer are isolated by the AOS framework layer. The upper layer includes the cognitive decision layer and the user interface layer, and the lower layer includes the information processing layer and the data access layer.

[0073] Information processing layer: Integrate multi-source battlefield data through the data integration platform to generate battlefield situation information, combine military knowledge graphs and battlefield situation data to complete data processing, including threat intelligence, resource status and target location. The data types include text, numerical values, geographic information and target images.

[0074] Data access layer: provides data access interface, calls data according to the instructions of AOS framework layer, including query, transmission and update. Data access layer supports multiple data sources and provides unified interface upward.

[0075] AOS framework layer: used to connect the data access layer and the cognitive decision layer. The AOS framework layer establishes a decision model through deep reinforcement learning models (such as DQN) combined with knowledge graphs. The AOS framework layer generates the strategy of the intelligent agent according to the environmental situation and provides a reasoning interface. It uses the knowledge graph to perform information association and situational reasoning to help the intelligent agent adapt to the dynamic battlefield environment. The AOS framework layer calls basic data through the data access layer and provides an access interface for the cognitive decision layer.

[0076] Cognitive decision-making layer: used to formulate command strategies and perform tactical deductions. According to the user interface layer or system requirements, the strategy generation and reasoning functions of the AOS framework layer are called to finally output the decision plan. The cognitive decision-making layer handles task allocation, threat assessment, and action recommendations, and returns the results to the user interface layer.

[0077] User interface layer: Provides decision display and interactive operation to users in a Web-based manner, and provides system analysis results. The user interface layer presents situational deduction results and decision suggestions, accepts user feedback requests, passes them to the cognitive decision layer, realizes interactive analysis and decision-making, and receives strategy suggestions returned by the cognitive decision layer.

[0078] Furthermore, the AOS framework layer includes two parts: a knowledge graph module and a decision model module.

[0079] The knowledge graph module constructs and stores a military data model, which includes an entity model, a relationship model and an attribute model. The knowledge graph module forms a knowledge graph from the military data model through relationship mapping and attribute constraints; generates a structured knowledge representation of the battlefield situation, describes entity information (such as targets, threats, tasks) through graph nodes, and provides capabilities such as situational association and causal reasoning.

[0080] The decision model module uses a deep reinforcement learning algorithm to build an intelligent agent, realizes situational awareness and decision optimization through strategy generation and environment interaction, and uses experience replay and strategy update mechanisms to improve the learning efficiency of the model; the combination of knowledge graph and DQN model enhances the system's situational understanding and strategy generation capabilities.

[0081] The AOS framework layer deploys a deep Q network model, establishes a cognitive model and analyzes the input information based on the knowledge graph, and provides a decision-making interface for the intelligent agent model.

[0082] Furthermore, the data processing at the information processing layer includes collection and fusion, and the data includes geographic information, threat intelligence, resource information and task instructions.

[0083] Furthermore, the access interfaces provided by the AOS framework layer to the cognitive decision layer include data query, model training and status update.

[0084] Furthermore, the deep Q network DQN can be replaced by other deep reinforcement learning algorithms that meet the requirements.

[0085] A command decision method is applied to a command decision deep Q network intelligent agent model system based on an AOS framework.

[0086] Furthermore, the method comprises the following steps:

[0087] Step 1: Decision reasoning: The data access layer sends a decision instruction to the AOS framework layer, and the AOS framework layer generates a preliminary plan and performs optimization.

[0088] Step 2: Strategy optimization. The cognitive decision layer sends optimization instructions to the AOS framework layer. The AOS framework layer generates an optimization plan based on the existing state information. The cognitive decision layer selects the best plan and provides optimization suggestions to the user through the user interface layer.

[0089] Step 3: Scenario simulation. After the user interface layer triggers the task, the cognitive decision layer transmits the requirements to the AOS framework to generate the scenario environment. The decision model module establishes and monitors the simulation scenario and feeds back real-time data to the user.

[0090] Furthermore, the decision reasoning in step 1 includes the following steps:

[0091] Step 1-1: After receiving the decision request, the data access layer sends instructions to the AOS framework layer. The AOS framework layer obtains battlefield situation information through the knowledge graph module. The information includes threat factors, resource conditions, and combat environment.

[0092] Step 1-2: The decision model module extracts the characteristics of the combat situation based on the knowledge graph information and calls the deep reinforcement learning model to generate an action plan; if the situation data does not cover complete information, supplementary information is obtained through knowledge graph reasoning, and the weight of the decision model is adjusted according to different battlefield conditions.

[0093] Step 1-3: The model will simulate and deduce the generated preliminary plans, compare the effects of each plan through the decision-making module, and sort them by priority.

[0094] Furthermore, the step 2 strategy optimization includes the following steps:

[0095] Step 2-1: The cognitive decision-making layer sends optimization instructions to the AOS framework layer based on user command requests or task update requirements. The AOS framework layer extracts the current combat status characteristics through the DQN model. The status characteristics include unit location, resource quantity and action constraints.

[0096] Step 2-2: The decision model module generates an action plan based on the existing state information and reward function and updates the action priority, and pushes the plan to the cognitive decision layer according to the priority.

[0097] Step 2-3: The cognitive decision layer selects the best solution based on the optimization results and provides optimization suggestions to the user through the user interface layer.

[0098] Furthermore, the scenario simulation in step 3 includes the following steps:

[0099] Step 3-1: The user interface layer triggers the situational simulation task through user instructions. The cognitive decision-making layer transmits relevant requirements to the AOS framework layer according to the instruction content. The AOS framework layer uses the knowledge graph information to generate the situational environment.

[0100] Step 3-2: The decision model module establishes a simulation scenario based on the situational environment, monitors the simulation process, and outputs simulation feedback based on state changes.

[0101] Step 3-3: The user interface layer receives the situation simulation feedback results and presents the real-time data of the situation simulation to the user.

[0102] Based on the war chess AI model, design a reasonable experimental plan and evaluate the performance and effectiveness of the model through experimental results.

[0103] The performance of the model is evaluated by playing against humans. A dataset containing multiple games is used. For each game, the state and action sequence of the chess game are recorded, and the final winner and reward are calculated.

[0104] The following metrics are used in the experiments to evaluate the performance and effectiveness of the model:

[0105] ① Win rate: represents the proportion of the model winning in the battle.

[0106] ②Reward value: represents the average reward value obtained by the model in the battle.

[0107] ③ Game time: represents the average time the model plays against other algorithms or humans.

[0108] The following experimental results were obtained:

[0109] (1) Competing with other algorithm models

[0110] The model of this scheme is compared with other algorithms, including random strategy (Random), greedy strategy (Greedy) and other algorithms based on reinforcement learning. The experimental results show that Fig. 9 As shown in the figure, the model of this scheme can defeat other algorithms in most cases, and its winning rate and reward value are significantly better than other algorithms, showing good decision-making ability and stability.

[0111] (2) Experimental results of three AIs at different difficulty levels

[0112] from Fig.10 It can be seen that the winning rate of the AI ​​model of this solution is much higher than that of random AI and maximum attack power AI at all three difficulty levels. Especially at the difficult difficulty, the winning rate of the AI ​​model of this solution also reached a very high level, proving its performance in complex environments.

[0113] (3) Model generalization ability

[0114] Based on the algorithm comparison and analysis, we further compared and analyzed the learning efficiency and generalization ability of the model, and divided the model training data into a training group and a test group. The former is used to build the model, while the latter is used to examine the generalization ability of the model. The following are the results of the comparative analysis:

[0115] like Fig.11 As shown in the figure, the line graph represents the test set accuracy after training of different algorithms, and the bar graph represents the training time of different algorithm models. Compared with other algorithms, the chess AI model based on the DQN algorithm has higher test set accuracy and shorter training time. This shows that the DQN algorithm has higher learning efficiency and generalization ability, can learn and optimize decision-making strategies more quickly, and shows good generalization ability in new scenarios.

[0116] (5) Learning rate sensitivity analysis: The learning rate has a significant impact on model performance. Fig.12 It can be seen that when the learning rate is 0.01, the model performs best, with a win rate of 57.1% and an average reward value of 2.3.

[0117] (6) Experience replay cache size sensitivity analysis: Analysis Fig.13 ,The experience replay cache size has little effect on the model performance.

[0118] (7) Target network update frequency sensitivity analysis: Analysis Fig.14 , the target network update frequency has little effect on the model performance.

[0119] Fig.15 This is a visualization of the running results of the AI ​​model in a test set game. The red and blue sides in the figure represent the enemy and our side. A high score means that the decision was made according to the model at the specified position, and the victory was obtained with the corresponding score.

[0120] The experimental results show that the chess AI model based on the DQN algorithm can better solve the decision-making problem in the chess game, has good decision-making ability and stability, and is suitable for use in practical applications. At the same time, compared with other algorithms, the DQN algorithm can better solve the problem of continuous state and action space, and has better learning efficiency and generalization.

[0121] The present invention provides a thought and method for a command decision-making deep Q network agent model based on the AOS framework. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A command decision-making deep Q network agent model system based on the AOS framework, characterized by: The system includes an AOS framework layer, an upper layer and a lower layer, wherein the AOS framework layer isolates the upper layer from the lower layer, the upper layer includes a cognitive decision layer and a user interface layer, and the lower layer includes an information processing layer and a data access layer, wherein: The AOS framework layer is connected to the data access layer, and the AOS framework layer is connected to the cognitive decision layer. The AOS framework layer calls basic data through the data access layer and provides an access interface for the cognitive decision layer. The information processing layer completes data processing through the data integration platform, combined with military knowledge graphs and battlefield situation data; The data access layer provides a data management interface. According to the instructions of the AOS framework layer, it completes data processing through the data access interface and provides a data service interface for the AOS framework layer. The cognitive decision-making layer formulates command strategies and executes tactical deductions. Based on the command requests from the user interface layer, the cognitive decision-making layer implements situation analysis, strategy selection, and action simulation through the AOS framework layer, and returns the results to the user interface layer. The user interface layer provides users with decision display and interactive operations, provides system analysis results, receives user command requests and passes them to the cognitive decision layer, and receives policy recommendations returned by the cognitive decision layer.

2. According to the AOS framework-based command decision-making deep Q network agent model system of claim 1, it is characterized in that: The AOS framework layer includes two parts: the knowledge graph module and the decision model module. The knowledge graph module constructs and stores a military data model, which includes an entity model, a relationship model, and an attribute model. The knowledge graph module forms a knowledge graph from the military data model through relationship mapping and attribute constraints; The decision model module uses a deep reinforcement learning algorithm to build an intelligent agent, realizes situational awareness and decision optimization through strategy generation and environment interaction, and uses experience replay and strategy update mechanisms to improve the learning efficiency of the model; The AOS framework layer deploys a deep Q network model, establishes a cognitive model and analyzes the input information based on the knowledge graph, and provides a decision-making interface for the intelligent agent model.

3. According to the AOS framework-based command decision-making deep Q network agent model system of claim 1, it is characterized in that: Information processing layer data processing includes collection and fusion, and the data includes geographic information, threat intelligence, resource information and mission instructions.

4. According to the AOS framework-based command decision-making deep Q network agent model system of claim 1, it is characterized in that: The access interfaces provided by the AOS framework layer for the cognitive decision-making layer include data query, model training, and status update.

5. The command decision-making deep Q network intelligent agent model system based on the AOS framework according to claim 1 is characterized in that: The deep Q network DQN may be other deep reinforcement learning algorithms that meet the requirements.

6. A command decision method, characterized in that: The method is applied to a command decision-making deep Q network intelligent agent model system based on an AOS framework as claimed in any one of claims 1 to 5.

7. The command decision-making method according to claim 6, characterized in that: The method comprises the following steps: Step 1: Decision reasoning: the data access layer sends a decision instruction to the AOS framework layer, and the AOS framework layer generates a preliminary plan and performs the optimization; Step 2: Strategy optimization: the cognitive decision layer sends optimization instructions to the AOS framework layer. The AOS framework layer generates an optimization plan based on the existing state information. The cognitive decision layer selects the best plan and provides optimization suggestions to the user through the user interface layer. Step 3: Scenario simulation. After the user interface layer triggers the task, the cognitive decision layer transmits the requirements to the AOS framework to generate the scenario environment. The decision model module establishes and monitors the simulation scenario and feeds back real-time data to the user.

8. The command decision-making method according to claim 7, characterized in that: The decision reasoning in step 1 includes the following steps: Step 1-1: After receiving the decision request, the data access layer sends instructions to the AOS framework layer. The AOS framework layer obtains battlefield situation information through the knowledge graph module. The information includes threat factors, resource conditions, and combat environment; Step 1-2: The decision model module extracts the characteristics of the combat situation based on the knowledge graph information and calls the deep reinforcement learning model to generate an action plan. If the situation data does not cover complete information, it obtains supplementary information through knowledge graph reasoning and adjusts the weight of the decision model according to different battlefield conditions. Step 1-3: The model will simulate and deduce the generated preliminary plans, compare the effects of each plan through the decision-making module, and sort them by priority.

9. The command decision-making method according to claim 8, characterized in that: The step 2 strategy optimization includes the following steps: Step 2-1: The cognitive decision-making layer sends optimization instructions to the AOS framework layer according to the user's instruction request or task update requirements. The AOS framework layer extracts the current combat status characteristics through the DQN model. The status characteristics include unit location, resource quantity and action constraints; Step 2-2: The decision model module generates an action plan based on the existing state information and reward function and updates the action priority, and pushes the plan to the cognitive decision layer according to the priority; Step 2-3: The cognitive decision layer selects the best solution based on the optimization results and provides optimization suggestions to the user through the user interface layer.

10. The command decision method according to claim 9, characterized in that: The scenario simulation in step 3 includes the following steps: Step 3-1: The user interface layer triggers the situation simulation task through user instructions. The cognitive decision layer transmits relevant requirements to the AOS framework layer according to the instruction content. The AOS framework layer generates the situation environment using the knowledge graph information. Step 3-2: The decision model module establishes a simulation scenario based on the situational environment, monitors the simulation process, and outputs simulation feedback based on state changes; Step 3-3: The user interface layer receives the situation simulation feedback results and presents the real-time data of the situation simulation to the user.

Images