Knowledge evolution-oriented intelligent auxiliary decision-making system and construction method

Abstract

The invention relates to a knowledge evolution-oriented intelligent aid decision-making system and a construction method. The system comprises a simulation environment module, a reinforcement learning algorithm module and a knowledge evolution module. The reinforcement learning algorithm module comprises a data playback pool, a strategy network and a value network; the knowledge evolution module comprises a knowledge storage library, a knowledge updating model, a knowledge organization model and a knowledge enabling model; the simulation environment module, the reinforcement learning algorithm module and the knowledge evolution module are in communication connection with one another, and the simulation environment module virtualizes a confrontation scene to generate a current state, an award and a next state, so that the reinforcement learning algorithm module drives intelligent confrontation; the reinforcement learning algorithm module is used for driving intelligent confrontation and generating a large number of simulation samples; the knowledge evolution module is used for congealing and storing knowledge through sample data, providing powerful knowledge support for various complex decisions, and helping people make more intelligent decisions when facing constantly changing information and knowledge environments.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent auxiliary decision-making system oriented to knowledge evolution and a construction method thereof. Background Art

[0002] In today's rapidly developing information age, the decision-making process is becoming increasingly complex, requiring decision makers to not only process large amounts of information, but also understand and adapt to the evolving knowledge system. Traditional decision support systems often rely on static knowledge bases and fixed rules, which are difficult to adapt to the reality of frequent knowledge updates and changing needs. Especially in the fields of economy, medicine, environmental management, and military, a decision often requires the analysis of a large amount of dynamically changing data and the combination of the latest domain knowledge.

[0003] The development of artificial intelligence technology, especially breakthroughs in machine learning, natural language processing, knowledge graphs, etc., provides new possibilities for building more dynamic and intelligent decision support systems. These systems can continuously learn from new data, update and expand their knowledge base, and thus provide more accurate and timely decision support. However, a system that can effectively handle knowledge evolution, automatically update, and assist complex decision-making processes remains a huge challenge. Summary of the invention

[0004] Based on this, it is necessary to provide an intelligent decision-making support system and construction method for knowledge evolution in response to the above technical problems.

[0005] The present invention provides an intelligent auxiliary decision-making system for knowledge evolution, the system comprises a simulation environment module for providing environmental state data, a reinforcement learning algorithm module for executing a reinforcement learning algorithm, and a knowledge evolution module for providing rule data, wherein the simulation environment module, the reinforcement learning algorithm module and the knowledge evolution module are connected through an interface communication; wherein, The reinforcement learning algorithm module is used to obtain the environment state data from the simulation environment module, obtain the rule data from the knowledge evolution module, generate a decision instruction according to the environment state data and the rule data, and transmit the decision instruction back to the simulation environment module; The simulation environment module is used to execute the decision instruction in the current environment determined by the environmental state data, generate adversarial sample data, and send the adversarial sample data to the reinforcement learning algorithm module; The knowledge evolution module is used to obtain the adversarial sample data from the reinforcement learning algorithm module, and update and store the rule data according to the adversarial sample data until the rule data no longer changes; wherein the rule data is obtained based on the adversarial sample data.

[0006] Wherein, the reinforcement learning algorithm module includes a strategy network; The strategy network is used to receive environmental status data from the simulation environment module, calculate decision instructions under the current environment based on the received environmental status data, verify the decision instructions using rule data obtained from the knowledge evolution module, and output the verified decision instructions to the simulation environment module, so that the simulation environment module executes the verified decision instructions and obtains adversarial sample data.

[0007] Wherein, the reinforcement learning algorithm module also includes a data playback pool; the simulation environment module also includes a reward calculation function and a state transition probability function; The state transition probability function is used to calculate the next state according to the verified decision instruction and the environmental state data; the reward calculation function is used to calculate a reward for the verified decision instruction; The simulation environment module is also used to send the environment state data, the next state and the reward to the reinforcement learning algorithm module, and the data revisit pool of the reinforcement learning algorithm module is used to store the current environment state data, the reward and the next state.

[0008] Wherein, the strategy network is a network obtained through training; the reinforcement learning algorithm module also includes a value network; the reinforcement learning algorithm module is also used for: The strategy network receives the environment status data from the simulation environment module, and calculates the decision instructions under the current environment based on the received environment status data; The strategy network verifies the decision instruction using the initial rule data obtained from the knowledge evolution module, and outputs the verified decision instruction to the simulation environment module; The data revisit pool receives adversarial sample data transmitted from the simulation environment module, wherein the adversarial sample data is obtained after the simulation environment module executes the verified decision instruction; The value network obtains the environmental status data, the verified decision instructions and rewards from the data playback pool, and calculates and outputs the target value; The above steps are executed repeatedly until the policy network converges.

[0009] The knowledge evolution module is capable of storing and updating rule data according to the data stored in the data playback pool and predefined logical methods.

[0010] Wherein, the knowledge evolution module includes a knowledge repository and a knowledge update model; wherein, The knowledge repository is used to store the rule data; The knowledge updating model is used to calculate the similarity between the new rule data and the rule data in the knowledge repository. If the similarity is greater than a predetermined threshold, the rule data in the knowledge repository is replaced with new rule data with high similarity; otherwise, the new rule data is directly stored in the knowledge repository; wherein the new rule data is obtained based on adversarial sample data.

[0011] Wherein, the knowledge evolution module also includes a knowledge organization model; The knowledge organization model is used to establish association relationships for rule data in the knowledge repository through a graph network.

[0012] Wherein, the knowledge evolution module also includes a knowledge enabling model; the knowledge enabling model is used to classify and query the rule data in the knowledge repository.

[0013] Wherein, the strategy network and the value network both include a three-layer convolutional neural network and a two-layer fully connected network.

[0014] The present invention also provides a method for constructing an intelligent auxiliary decision-making system for knowledge evolution, the method being used to implement any of the above-mentioned intelligent auxiliary decision-making systems for knowledge evolution, the method comprising: The reinforcement learning algorithm module obtains the environment state data from the simulation environment module, obtains the rule data from the knowledge evolution module, generates a decision instruction according to the environment state data and the rule data, and transmits the decision instruction back to the simulation environment module; The simulation environment module executes the decision instruction in the current environment determined by the environmental state data to generate adversarial sample data; the simulation environment module sends the adversarial sample data to the reinforcement learning algorithm module; The knowledge evolution module obtains the adversarial sample data from the reinforcement learning algorithm module, and updates and stores the rule data according to the adversarial sample data until the rule data no longer changes; wherein the rule data is obtained according to the adversarial sample data; The above process is repeated until the rule data stored in the knowledge evolution module no longer changes.

[0015] Compared with the prior art, this application has the following beneficial effects: The present invention provides an intelligent auxiliary decision-making system for knowledge evolution and a construction method. The knowledge evolution module condenses new rule data according to historical adversarial sample data. The reinforcement learning algorithm module can obtain environmental state data from the simulation environment module and rule data from the knowledge evolution module, and generate decision instructions according to the environmental state data and rule data, so that the system can adapt to knowledge changes, provide efficient and personalized decision support, and help humans make more intelligent decisions when facing a constantly changing information and knowledge environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of the structure of an intelligent auxiliary decision-making system for knowledge evolution according to an embodiment; Figure 2 for Figure 1 Schematic diagram of updating the knowledge evolution module in the intelligent auxiliary decision-making system; Figure 3 for Figure 1 Schematic diagram of the interaction between the reinforcement learning algorithm module and the simulation environment module in the intelligent auxiliary decision-making system; Figure 4 The figure is a flowchart of an intelligent decision-making assistance method for knowledge evolution according to an embodiment. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] In today's rapidly developing information age, the decision-making process is becoming increasingly complex, requiring decision makers to not only process large amounts of information, but also understand and adapt to the evolving knowledge system. Traditional decision support systems often rely on static knowledge bases and fixed rules, which are difficult to adapt to the reality of frequent knowledge updates and changing needs. Especially in the fields of economy, medicine, environmental management, and military, a decision often requires the analysis of a large amount of dynamically changing data and the combination of the latest domain knowledge.

[0019] The development of artificial intelligence technology, especially breakthroughs in machine learning, natural language processing, knowledge graphs, etc., provides new possibilities for building more dynamic and intelligent decision support systems. These systems can continuously learn from new data, update and expand their knowledge base, and thus provide more accurate and timely decision support. However, a system that can effectively handle knowledge evolution, automatically update, and assist complex decision-making processes remains a huge challenge.

[0020] First, knowledge evolution involves the representation, condensation and updating of knowledge. An effective system needs to be able to identify and integrate new knowledge while updating or replacing outdated knowledge. In addition, knowledge evolution is not just an increase in quantity, but also involves changes in structure and relationships, which requires the system to understand and process complex knowledge structures. Secondly, intelligent decision-making support systems need to be able to understand user needs and provide personalized decision support. This requires not only that the system has powerful data processing capabilities, but also that it needs to be able to understand the user's context, preferences and decision-making style in order to provide more accurate and useful suggestions. Finally, with the continuous increase in the amount of knowledge and data, how to ensure the stability and scalability of the system while protecting user privacy and data security is also an important issue that must be considered when designing such a system.

[0021] To solve the above problems, the present invention proposes an intelligent decision-making support system and method for knowledge evolution, which is an intelligent assistance system that can adapt to knowledge changes, understand user needs, and provide efficient and personalized decision support. The system mainly includes a reinforcement learning algorithm module, a simulation environment module, and a knowledge evolution module. The reinforcement learning algorithm module is mainly used to drive intelligent confrontation to generate a large amount of simulated confrontation sample data; the knowledge evolution module condenses and stores knowledge by processing the confrontation sample data, providing strong knowledge support for various complex decisions, and helping humans make more intelligent decisions in the face of ever-changing information and knowledge environments.

[0022] like Figure 1 As shown, the system includes a simulation environment module for providing environmental state data, a reinforcement learning algorithm module for executing a reinforcement learning algorithm, and a knowledge evolution module for providing rule data. The simulation environment module, the reinforcement learning algorithm module, and the knowledge evolution module are connected through an interface communication; optionally, the system can standardize and format all data through the interface to ensure that the data can be transmitted and interacted between various modules in the system. Among them, The reinforcement learning algorithm module is used to obtain environmental state data from the simulation environment module, obtain rule data from the knowledge evolution module, generate decision instructions based on the environmental state data and rule data, and transmit the decision instructions back to the simulation environment module. The simulation environment module is used to execute decision instructions in the current environment determined by the environmental state data, generate adversarial sample data, and send the adversarial sample data to the reinforcement learning algorithm module; wherein the adversarial sample data may include environmental state data in the current environment as well as rewards, next environmental state, etc. Optionally, the system of the embodiment of the present invention may repeat the above-mentioned process of generating decision instructions and adversarial sample data multiple times within a predetermined time period to obtain a data sequence composed of adversarial sample data within the predetermined time period, and send the data sequence to the knowledge evolution module for knowledge condensation.

[0023] The knowledge evolution module is used to obtain adversarial sample data from the reinforcement learning algorithm module, and update and store rule data according to the adversarial sample data until the rule data in the knowledge evolution module no longer changes; wherein the rule data is obtained according to the adversarial sample data, and the rule data can be transmitted to the reinforcement learning algorithm module for the generation of the next decision instruction. The rule data can be new knowledge extracted from the adversarial sample data, and the new knowledge can be fed back to the next decision generation process of the reinforcement learning algorithm module, thereby realizing a dynamic learning and adaptive intelligent decision support system and improving the accuracy of decision making.

[0024] Optionally, in the scenario of confrontation between the red and blue sides, the environmental state data includes but is not limited to the duration of the confrontation, the area of ​​confrontation, the size and type of the red and blue forces, and the main actions of the blue side. Optionally, these rule data are abstractions and generalizations of the adversarial sample data, and the rule data captures the potential patterns, relationships, and laws in the original adversarial sample data. The original adversarial sample data is a specific observation value, containing a lot of details and possible noise. In contrast, the rule data is represented in a formalized way, such as logical rules, mathematical formulas, or decision trees, which has a higher level of abstraction and interpretability. Furthermore, the initial value of the rule data can be an empty set, that is, in the initial stage, the reinforcement learning algorithm module generates a decision instruction based on the environmental state data, and transmits the decision instruction back to the simulation environment module. Alternatively, the initial value of the rule data is a preset data set. For example, a human expert can condense relevant rule data from a confrontation scenario and represent the rule data as The multiple rules initially constructed are stored in the knowledge evolution module to reduce the search space of the reinforcement learning algorithm module and accelerate the convergence of the reinforcement learning algorithm.

[0025] In one embodiment, human experts can discover new knowledge from adversarial sample data, including but not limited to the domain knowledge of experts and the knowledge summarized in the learning process. The knowledge evolution module can obtain these new knowledge and condense more rules based on these new knowledge, that is, obtain rule data, and feed these rule data back to the reinforcement learning algorithm module for the next decision support, thereby realizing a dynamic learning and adaptive intelligent decision support system. Among them, human experts can use domain knowledge and experience, and use the IF-THEN logical rule method to filter knowledge from samples based on the principles of consistency, effectiveness and interpretability. IF-THEN rules can represent basic knowledge structures, which are suitable for describing the behavior or decision-making process of intelligent agents under specific conditions. IF-THEN rules have the characteristics of clear logic, simple operation, and easy expansion, and can efficiently describe the correspondence between conditions and behaviors. In addition, experts also need to monitor sample data and dynamically adjust the conditions and behavior content of IF-THEN rules in real time according to needs. For example, adjust the condition range according to the latest observation results to ensure that the rules adapt to changing scenarios.

[0026] In one embodiment, human experts can use domain knowledge and experience, and use the IF-THEN logic rule method to filter out knowledge from samples based on the principles of consistency, effectiveness, and explainability. IF-THEN rules can represent basic knowledge structures, which are suitable for describing the behavior or decision-making process of intelligent agents under specific conditions. IF-THEN rules have the characteristics of clear logic, simple operation, and easy expansion, and can efficiently describe the correspondence between conditions and behaviors. In addition, experts also need to monitor sample data and dynamically adjust the conditions and behavior content of IF-THEN rules in real time according to needs. For example, adjust the condition range according to the latest observation results to ensure that the rules adapt to the changing scenarios. This optimization can make the rules more accurate and improve the response effect of the intelligent agent. Afterwards, human experts can store the above-mentioned logical method of discovering new knowledge in the knowledge evolution module, and the knowledge evolution module can generate and store the new rule data according to the predefined logical method and historical adversarial sample data.

[0027] The intelligent decision-making support system for knowledge evolution of the embodiment of the present invention can effectively represent the connotation of the adversarial scenario by continuously extracting new knowledge from adversarial samples and using this changing knowledge. At the same time, the system of the present invention effectively reduces the search space of the reinforcement learning algorithm by combining rule data with the reinforcement learning algorithm, speeds up the reinforcement learning speed, and improves the response speed in actual decision-making. In addition, the intelligent decision-making support system for knowledge evolution of the embodiment of the present invention can effectively realize the growth and update of knowledge through the knowledge evolution module, and can effectively improve the comprehensibility of decisions by using new knowledge, making it easy for humans to understand, and can provide certain auxiliary significance for human decision-making.

[0028] Continue to see Figure 1 The reinforcement learning algorithm module includes a policy network; wherein the policy network is mainly used to generate decision instructions, and may be a neural network including a three-layer convolutional neural network and a two-layer fully connected network, which is not specifically limited in the embodiments of the present invention; the policy network may be a neural network obtained after training, and the training process of the policy network may be described below. In the embodiments of the present invention, the policy network is used to receive environmental status data from the simulation environment module. , based on the received environmental status data Computing Agent Action Strategies , and according to the action strategy of the computing agent The probability size determines the decision-making instructions of the agent in the current environment Afterwards, the policy network can use the rule data obtained from the knowledge evolution module to make decisions. Verify and send the verified decision instructions Output to the simulation environment module so that the simulation environment module executes the decision instruction And obtain adversarial sample data. Among them, in the scenario of confrontation between the red and blue sides, the environmental status data includes but is not limited to the duration of the confrontation, the confrontation area, the size and type of the red and blue forces, and the main actions of the blue side. The intelligent agent is the physical unit of the red side controlled by the reinforcement learning algorithm.

[0029] The simulation environment module also includes a reward calculation function And the state transition probability function , the simulation environment module is used to execute decision instructions , and the reward calculation function And the state transition probability function Calculate and obtain adversarial sample data. The adversarial sample data may include the decision instructions and their rewards under the current environment state, as well as the next environment state. Specifically, The state transition probability function is used to determine the decision instruction and current environmental status data , calculate the next state ; Reward calculation function For decision making instructions Calculate a reward Afterwards, the simulation environment module can transmit the current environment status data through the communication interface. , next state And rewards The reinforcement learning module can also store and evaluate the adversarial sample data it receives.

[0030] Optionally, a reward calculation function Instructions for decision making Calculate a reward score , used to evaluate the decision instruction In this embodiment of the present invention, the reward calculation function can be ,in represents the Euclidean distance from the agent to the target point, In the embodiment of the present invention, the reward calculation function depends on the task goal of the red team and the expected effect. The task goal comes from the assumption data of the knowledge evolution module. The assumption data is transmitted to the simulation environment module interface through the knowledge evolution module interface. The current state generated by the simulation environment module ,award and the next state , and is passed to the reinforcement learning algorithm module interface through the simulation environment module interface, and finally stored in the reinforcement learning algorithm module, so that the reinforcement learning module stores and evaluates the adversarial sample data it receives.

[0031] like Figure 1 As shown, the reinforcement learning algorithm module may also include a value network and a data revisit pool, wherein the data revisit pool of the reinforcement learning algorithm module is used to store the sample data transmitted by the simulation environment module, that is, the data revisit pool can be used to store the current environment state data ,award and the next state The value network is mainly used to guide the learning of the policy network, and may also include a three-layer convolutional neural network and a two-layer fully connected network, which is not specifically limited in the embodiment of the present invention. The value network is used to obtain environmental status data from the data playback pool. , Decision Instructions and rewards The output target value is calculated based on the adversarial sample data. , in order to evaluate the adversarial samples obtained in the current environment, and then determine whether the policy network in the reinforcement learning algorithm module converges.

[0032] The knowledge evolution module can be a knowledge repository, a knowledge update model, a knowledge organization model, and a knowledge enablement model; wherein the knowledge repository is used to store rule data, which can be new knowledge extracted from adversarial sample data in the data recovery pool. This new knowledge can be fed back to the reinforcement learning algorithm module for the next decision support of the policy network. For example, human experts extract relevant knowledge from the virtual confrontation scene of drones. This knowledge can be the combat rules of drones in the confrontation scene. For example, this new knowledge can be expressed as rules ,These rules can be the tactical actions of the UAV such as the “scissor maneuver”, and then the ,preliminary constructed multiple rules are stored in the knowledge repository.

[0033] The knowledge update model is used to update the knowledge in the knowledge repository. Specifically, the knowledge update model is used to calculate the similarity between the new rule data and the rule data in the knowledge repository. If the similarity is greater than a predetermined threshold, the rule data in the knowledge repository is replaced with new rule data with high similarity; otherwise, the new rule data is directly stored in the knowledge repository, thereby dynamically updating the rule data in the knowledge repository. The knowledge organization model is mainly used to associate knowledge and reference knowledge. The knowledge organization model is used to establish association relationships for rule data in the knowledge repository through a graph network. The knowledge enablement model mainly classifies and queries knowledge, and can recommend appropriate knowledge based on the needs of the scenario or task.

[0034] The auxiliary decision system in the embodiment of the present invention can train the reinforcement learning algorithm model through the reinforcement learning algorithm module, and use the trained reinforcement learning algorithm model to assist decision making. Figure 2 and Figure 3 As shown, the intelligent decision-making support system for knowledge evolution of the embodiment of the present invention can perform multiple rounds of iterative operations through the policy network and the value network, and update and store the rule data in the knowledge repository until the policy network converges to obtain a trained policy network, which can be used for decision-making support. Specifically, In the first round of iteration, the reinforcement learning algorithm module can interact with the simulation environment module to obtain the current environment state data from the simulation environment module, and generate decision instructions based on the current environment state data, and then send the decision instructions to the simulation environment module for execution to obtain adversarial sample data. Human experts or knowledge evolution modules can obtain adversarial sample data from the data recovery pool and extract new knowledge (i.e., rule data) from the adversarial sample data. The new knowledge can be stored in the knowledge repository of the knowledge evolution module for the next decision support of the policy network of the reinforcement learning algorithm module.

[0035] In the second round of iteration, the reinforcement learning algorithm module can interact with the simulation environment module and the knowledge evolution module to obtain the current environment state data from the simulation environment module and the rule data from the knowledge repository of the knowledge evolution module, and generate a decision instruction based on the current environment state data and the rule data, and then send the decision instruction to the simulation environment module for execution to obtain new adversarial sample data (i.e., the above-mentioned historical adversarial sample data). Human experts or knowledge evolution modules can obtain this adversarial sample data from the data recovery pool and extract new knowledge (i.e., rule data) from the adversarial sample data. The new knowledge can be stored in the knowledge repository of the knowledge evolution module for the next decision support of the policy network of the reinforcement learning algorithm module.

[0036] This iterative operation is repeated until the After a round of iterations, the preset conditions are met, the above calculation process is terminated, and the trained reinforcement learning algorithm model is obtained. The system can store the reinforcement learning algorithm model for auxiliary decision-making. Figure 3 As shown, in each round of iteration, the reinforcement learning algorithm model of the reinforcement learning algorithm module can be used to perform the following steps: the reinforcement learning agent can obtain rule data from the knowledge repository and generate decision instructions (i.e., Figure 3 Specifically, the reinforcement learning agent first generates a decision instruction based on the current environment state data, and then verifies the decision instruction according to the received rule data to obtain the verified decision instruction (i.e. Figure 3 The decision instruction after verification is sent to the simulation environment module. The simulation environment module can execute the action in the simulation environment determined by the current environment state data, determine the adversarial sample data such as the reward of the action and the next environment state, and return the adversarial sample data to the reinforcement learning algorithm module.

[0037] Furthermore, the intelligent decision-making support system of the embodiment of the present invention can use the trained reinforcement learning algorithm model to perform knowledge evolution. For example, the system can set the simulation step size to , and from From the moment until At the end of the moment, the trained policy network is used to perform multiple simulation experiments to obtain multiple adversarial sample data, and these adversarial sample data are stored in the data recycling pool. The process of obtaining adversarial sample data can be found in the above Figure 2 and Figure 3 . Afterwards, human experts or knowledge evolution modules can obtain these adversarial sample data from the data recovery pool, and obtain new rule data based on these adversarial sample data. These new rule data can be fed back to the reinforcement learning algorithm module (i.e., the reinforcement learning agent in the figure) to support the next decision support. At the same time, the knowledge evolution module can store these new rule data in the knowledge repository, and the knowledge update model can compare the new rule data with the original rule data in the knowledge repository to update the knowledge in the knowledge repository. The knowledge organization model can also associate the new rule data with the knowledge in the knowledge repository. The system of an embodiment of the present invention executes the above-mentioned knowledge update process multiple times until the rule data in the knowledge repository no longer changes, thereby ensuring that the system can provide reliable and effective decision support in a variety of battlefield environments.

[0038] In one embodiment, Figure 4 As shown, an embodiment of the present invention further provides an intelligent auxiliary decision-making method for knowledge evolution, which is used to implement the above-mentioned intelligent auxiliary decision-making system for knowledge evolution, and the method includes: S401, the reinforcement learning algorithm module obtains the environment state data from the simulation environment module, obtains the rule data from the knowledge evolution module, generates a decision instruction according to the environment state data and the rule data, and transmits the decision instruction back to the simulation environment module; wherein, the reinforcement learning algorithm module includes a policy network, the policy network generates a decision instruction according to the environment state data and the rule data, and transmits the decision instruction back to the simulation environment module.

[0039] S402, the simulation environment module executes the decision instruction in the current environment determined by the environment state data to generate adversarial sample data; wherein the reward calculation function of the simulation environment module And the state transition probability function Calculate and obtain adversarial sample data.

[0040] S403, the simulation environment module sends the adversarial sample data to the reinforcement learning algorithm module; wherein the simulation environment module sends the adversarial sample data to the reinforcement learning algorithm module through the communication interface.

[0041] S404, the knowledge evolution module obtains the adversarial sample data from the reinforcement learning algorithm module, and updates and stores the rule data according to the adversarial sample data until the rule data no longer changes; wherein the rule data is obtained according to the adversarial sample data; S405: Determine whether the rule data in the knowledge repository no longer changes; if not, return to the above step S401 and repeat the above process until the rule data stored in the knowledge evolution module no longer changes. If yes, the system saves the knowledge update model and the deep learning algorithm model, and ends the above knowledge update process.

[0042] The implementation steps of the method in the embodiment of the present invention are basically consistent with the principles of the above system. For details, please refer to the above description and will not be repeated here.

[0043] The embodiment of the present invention can further illustrate the method of the embodiment of the present invention by instantiating a simulated confrontation scene between red and blue drones. The method of the embodiment of the present invention may include the following steps: Step 1: Initialize the random strategy and data playback pool. The decision instructions of the red drone generated by the random strategy are executed in the simulation environment module. The simulation environment module receives the decision instructions. After that, the state transition occurs to a new state and a reward is calculated Specifically, the initialization random strategy includes: setting various parameters of the red and blue drones in the simulation environment according to the mission requirements, including the flight speed of the drones, the duration of the confrontation, the confrontation area, the number and type of ammunition carried, and the evaluation and expected effect of the mission. Among them, the confrontation area, the location of the red drone and the information of the blue drone constitute the environmental state data. , the red drone action decision instruction is expressed as , the expected effect of the red team is presented through rewards, which are expressed as .

[0044] Step 2: The policy network interacts with the simulation environment module under the joint drive of the initialized random strategy and the knowledge repository, and The reinforcement learning algorithm module generates decision instructions based on the state information of the environment and issues decision instructions to the Red Army UAV in the virtual confrontation scene to control the movement of the Red Army UAV in the simulated confrontation scene. , using the current strategy Decision Instructions The state information and decision instructions are simultaneously input into the reinforcement learning algorithm module, which outputs the verified decision instructions according to the new rule data. The verified decision instructions are then sent to the simulation environment model to control the movement of the Red Team UAV in the virtual environment.

[0045] The knowledge repository stores multiple rules data that are initially constructed. The multiple rule data may be relevant knowledge condensed by human experts from virtual confrontation scenarios, and the rule data stored in the knowledge repository may be used to reduce the exploration space of the intelligent agent.

[0046] Step 3: Repeat step 2 above and continue to train the reinforcement learning algorithm model (specifically the policy network) until the model achieves the expected effect of the Red Team UAV. At this time, the model is considered to have initially converged and the reinforcement learning algorithm model is saved. .

[0047] Step 4: Set the maximum simulation step size , to strengthen the learning model Test in confrontation scenarios. In this way, through extensive testing of the system, it can be ensured that the system can provide reliable and effective decision support in a variety of battlefield environments. The test process is as follows.

[0048] Step 5, from From the moment until At the end of the moment, the execution strategy collects the sequence data of the entire game And store these adversarial data in the data replay pool.

[0049] Step 6: Human experts or knowledge evolution modules use adversarial sample data in the data replay pool , extract new relevant knowledge to form corresponding new rule data.

[0050] Step 7: The new rule data is stored in the knowledge repository through the knowledge evolution model, the rule data in the knowledge repository is updated using the knowledge update module, and the rule data in the knowledge repository are associated with each other using the knowledge organization model. The rule data in the knowledge evolution module can be provided to humans for reference, and humans can query the required rule data from the knowledge repository using the knowledge enablement model according to their needs.

[0051] In the embodiment of the present invention, the reinforcement learning algorithm module in the system obtains the state data and reward data from the environment through its interface with the simulation environment, and obtains the rule knowledge in the knowledge repository through its interface with the knowledge evolution module, and makes decisions driven by knowledge and strategy, and the decision results are passed to the simulation environment module to generate adversarial sample data. The knowledge evolution module condenses new rules from the continuously generated adversarial data, and updates the new rules to the knowledge repository; this iteration is repeated until the knowledge update model converges.

[0052] The above-described embodiment only expresses one implementation method of the present application, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of this application shall be based on the attached claims.

Claims

1. An intelligent decision-making support system for knowledge evolution, characterized in that: The system includes a simulation environment module for providing environment state data, a reinforcement learning algorithm module for executing a reinforcement learning algorithm, and a knowledge evolution module for providing rule data. The simulation environment module, the reinforcement learning algorithm module, and the knowledge evolution module are connected through an interface communication; wherein, The reinforcement learning algorithm module is used to obtain the environment state data from the simulation environment module, obtain the rule data from the knowledge evolution module, generate a decision instruction according to the environment state data and the rule data, and transmit the decision instruction back to the simulation environment module; The simulation environment module is used to execute the decision instruction in the current environment determined by the environmental state data, generate adversarial sample data, and send the adversarial sample data to the reinforcement learning algorithm module; The knowledge evolution module is used to obtain the adversarial sample data from the reinforcement learning algorithm module, and update and store the rule data according to the adversarial sample data until the rule data no longer changes; wherein the rule data is obtained based on the adversarial sample data.

2. The system according to claim 1, characterized in that The reinforcement learning algorithm module includes a strategy network; The strategy network is used to receive environmental status data from the simulation environment module, calculate decision instructions under the current environment based on the received environmental status data, verify the decision instructions using rule data obtained from the knowledge evolution module, and output the verified decision instructions to the simulation environment module, so that the simulation environment module executes the verified decision instructions and obtains adversarial sample data.

3. The system according to claim 2, characterized in that The reinforcement learning algorithm module also includes a data playback pool; the simulation environment module also includes a reward calculation function and a state transition probability function; The state transition probability function is used to calculate the next state according to the verified decision instruction and the environmental state data; the reward calculation function is used to calculate a reward for the verified decision instruction; The simulation environment module is also used to send the environment state data, the next state and the reward to the reinforcement learning algorithm module, and the data revisit pool of the reinforcement learning algorithm module is used to store the current environment state data, the reward and the next state.

4. The system according to claim 3, characterized in that The strategy network is a network obtained through training; the reinforcement learning algorithm module also includes a value network; the reinforcement learning algorithm module is also used for: The strategy network receives the environment status data from the simulation environment module, and calculates the decision instructions under the current environment based on the received environment status data; The strategy network verifies the decision instruction using the initial rule data obtained from the knowledge evolution module, and outputs the verified decision instruction to the simulation environment module; The data revisit pool receives adversarial sample data transmitted from the simulation environment module, wherein the adversarial sample data is obtained after the simulation environment module executes the verified decision instruction; The value network obtains the environmental status data, the verified decision instructions and rewards from the data playback pool, and calculates and outputs the target value; The above steps are executed repeatedly until the policy network converges.

5. The system according to claim 3, characterized in that The knowledge evolution module can store and update rule data according to the data stored in the data playback pool and the predefined logical method.

6. The system according to any one of claims 1 to 5, characterized in that: The knowledge evolution module includes a knowledge repository and a knowledge update model; wherein, The knowledge repository is used to store the rule data; The knowledge updating model is used to calculate the similarity between the new rule data and the rule data in the knowledge repository. If the similarity is greater than a predetermined threshold, the rule data in the knowledge repository is replaced with new rule data with high similarity; otherwise, the new rule data is directly stored in the knowledge repository; wherein the new rule data is obtained based on adversarial sample data.

7. The system according to claim 6, characterized in that The knowledge evolution module also includes a knowledge organization model; The knowledge organization model is used to establish association relationships for rule data in the knowledge repository through a graph network.

8. The system according to claim 7, characterized in that The knowledge evolution module also includes a knowledge enabling model; the knowledge enabling model is used to classify and query the rule data in the knowledge repository.

9. The system according to claim 4, characterized in that The strategy network and the value network both include a three-layer convolutional neural network and a two-layer fully connected network.

10. A method for constructing intelligent auxiliary decision-making for knowledge evolution, the method being used to implement the intelligent auxiliary decision-making system for knowledge evolution as claimed in any one of claims 1 to 8, characterized in that: The method comprises: The reinforcement learning algorithm module obtains the environment state data from the simulation environment module, obtains the rule data from the knowledge evolution module, generates a decision instruction according to the environment state data and the rule data, and transmits the decision instruction back to the simulation environment module; The simulation environment module executes the decision instruction in the current environment determined by the environmental state data to generate adversarial sample data; the simulation environment module sends the adversarial sample data to the reinforcement learning algorithm module; The knowledge evolution module obtains the adversarial sample data from the reinforcement learning algorithm module, and updates and stores the rule data according to the adversarial sample data until the rule data no longer changes; wherein the rule data is obtained according to the adversarial sample data; The above process is repeated until the rule data stored in the knowledge evolution module no longer changes.

Images