A method and apparatus for generating adversarial behavior trees based on key window states in games.

Abstract

This invention relates to the fields of game theory, behavior trees, and deep reinforcement learning. It discloses a method for generating game theory behavior trees based on key window states. The method includes: adding noise to the agent's state at each decision moment in an intelligent decision-making model; calculating the difference between the initial policy evaluation and the perturbed policy evaluation at each decision moment; selecting a preset number of agent states as key window states based on this difference; generating condition nodes and action nodes for the key window using these key window states; connecting the condition nodes and action nodes with the key window as the root node to form a spanning tree for the key window; and concatenating the spanning trees of each key window in chronological order to obtain the spanning tree of the intelligent decision-making model. This invention also proposes a device, equipment, and medium for generating game theory behavior trees based on key window states. This invention can solve the problem that behavior trees are difficult to directly apply to explain the complex decision-making processes of deep reinforcement learning-based game theory.

Description

Technical Field

[0001] This invention relates to the fields of game theory, behavior trees, and deep reinforcement learning, and particularly to a method, apparatus, electronic device, and computer-readable storage medium for generating game theory behavior trees based on key window states. Background Technology

[0002] Artificial intelligence technology is playing an increasingly important role in planning and decision-making. However, the complexity, high dimensionality, and non-causality of intelligent decision-making models, such as deep reinforcement learning models, make the decisions they generate incomprehensible to humans, exhibiting a "black box" nature. If interpretability analysis can be conducted on intelligent decision-making models, dismantling the "black box," it will help to control the intermediate decision-making processes and decision-making basis of intelligent decision-making models. Otherwise, the scalability and security of intelligent decision-making models in application cannot be guaranteed.

[0003] Traditional interpretability methods for intelligent decision-making models can be categorized into manual analysis and intelligent methods. Manual analysis leverages human experience to analyze the decision-making process of intelligent decision-making models based on case studies, but it falls short in terms of accuracy and efficiency. Intelligent methods include model performance analysis, model theory analysis, and model approximation analysis. However, model performance analysis cannot explain the basis of the model's decisions and lacks causality; model theory analysis cannot keep pace with the development of intelligent decision-making models; and in model approximation analysis, behavior trees are a highly interpretable hierarchical tree-structured algorithm that effectively organizes large-scale decision-making tasks through hierarchy and modularity, addressing the non-causal nature of interpretability methods for intelligent decision-making models.

[0004] However, behavior trees are currently mainly used in the field of intelligent planning and have not been applied to the research of interpretable methods for game-based adversarial intelligent decision-making models based on deep reinforcement learning. Furthermore, the structure of behavior trees is relatively simple and fixed, lacking the node foundation for extending interpretable methods for intelligent decision-making models. To maintain the causality and interpretability of the behavior tree structure, the number of nodes is limited. Therefore, existing behavior trees are difficult to directly apply to explaining the complex decision-making processes of game-based adversarial games based on deep reinforcement learning. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and computer-readable storage medium for generating game adversarial behavior trees based on key window states. Its main purpose is to solve the problem that behavior trees are difficult to directly apply to explain the complex decision-making process of game adversarial based on deep reinforcement learning.

[0006] To achieve the above objectives, this invention provides a method for generating adversarial behavior trees based on key window states, comprising:

[0007] Obtain the decision sequence of the preset intelligent decision model, wherein the decision sequence includes the initial state and initial policy evaluation of the agent controlled by the preset intelligent decision model at each decision time.

[0008] Noise is added to the initial state of the agent at each decision moment to obtain the perturbed state, and the perturbed policy evaluation corresponding to each perturbed state is generated using the preset intelligent decision model;

[0009] The difference between the initial policy evaluation and the policy evaluation after perturbation at each decision moment is calculated sequentially. Based on the difference, a preset number of decision moments are selected as the initial state of the agent as the key window state, and the decision moment corresponding to the key window state is taken as the key window.

[0010] Obtain attribute information corresponding to the state of each key window, and convert the attribute information into a condition node of the first preset format for the corresponding key window. Use the preset intelligent decision model to generate a predictive decision action for the corresponding key window based on the condition node, and convert the predictive decision action of the key window into an action node of the second preset format.

[0011] Using each key window as a control node, and the condition nodes and action nodes of the key window as branches of the corresponding control node, a behavior tree corresponding to each key window is obtained.

[0012] Based on the chronological order of each key window, the control nodes in the behavior tree of each key window are connected in series to obtain the behavior tree of the preset intelligent decision-making model.

[0013] Optionally, the step of obtaining the attribute information corresponding to each key window state and converting it into a condition node of a first preset format for the corresponding key window includes:

[0014] The attribute information corresponding to each key window state is cleaned, and the cleaned attribute information is segmented into words to obtain a word segmentation set.

[0015] Each word in the word segmentation set is sequentially matched with preset conditional keywords using fuzzy matching.

[0016] The successfully matched word segments are converted according to the first preset format, and the converted word segments are filled into the condition nodes of the corresponding key windows.

[0017] Optionally, the step of generating a predictive decision action for the corresponding key window based on the condition node using the preset intelligent decision model includes:

[0018] Using the preset intelligent decision-making model, the probability value of each preset decision action is calculated based on the condition nodes of each key window;

[0019] The preset decision action with the highest probability value is selected as the predictive decision action for the corresponding key window.

[0020] Optionally, the step of converting the prediction decision action of the key window into an action node in a second preset format includes:

[0021] According to the preset syntax rules, the predictive decision actions of the key window are split into two parts: action instructions and control data.

[0022] Obtain the preset instruction code corresponding to the action instruction;

[0023] The corresponding instruction codes and control data are assembled according to the second preset format to obtain the action nodes of the corresponding key windows.

[0024] To address the aforementioned problems, the present invention also provides a device for generating a game adversarial behavior tree based on key window states, the device comprising:

[0025] The decision sequence acquisition module is used to acquire the decision sequence of a preset intelligent decision model, wherein the decision sequence includes the initial state and initial policy evaluation of the agent controlled by the preset intelligent decision model at each decision time.

[0026] The noise addition module is used to add noise to the initial state of the agent at each decision moment to obtain the perturbed state, and to generate the perturbed policy evaluation corresponding to each perturbed state using the preset intelligent decision model.

[0027] The key window state identification module is used to sequentially calculate the difference between the initial policy evaluation and the perturbation policy evaluation at each decision moment, select the state of the agent corresponding to the maximum difference of a preset number as the key window state, and take the decision moment corresponding to the key window state as the key window.

[0028] The behavior tree node generation module is used to obtain attribute information corresponding to the state of each key window, convert the attribute information into a condition node of the first preset format corresponding to the key window, use the preset intelligent decision model to generate the predictive decision action of the corresponding key window according to the condition node, and convert the predictive decision action of the key window into an action node of the second preset format.

[0029] The key window behavior tree construction module is used to obtain the behavior tree corresponding to each key window by taking each key window as a control node and the condition nodes and action nodes of the same key window as branches of the corresponding control node.

[0030] The model behavior tree assembly module is used to connect the control nodes in the behavior tree of each key window according to the time sequence of each key window to obtain the behavior tree of the preset intelligent decision model.

[0031] Optionally, the behavior tree node generation module converts the attribute information into condition nodes in a first preset format corresponding to the key window using the following method:

[0032] The attribute information corresponding to each key window state is cleaned, and the cleaned attribute information is segmented into words to obtain a word segmentation set.

[0033] Each word in the word segmentation set is sequentially matched with preset conditional keywords using fuzzy matching.

[0034] The successfully matched word segments are converted according to the first preset format, and the converted word segments are filled into the condition nodes of the corresponding key windows.

[0035] Optionally, the behavior tree node generation module utilizes the preset intelligent decision-making model to generate predictive decision actions for corresponding key windows based on the condition nodes using the following method:

[0036] Using the preset intelligent decision-making model, the probability value of each preset decision action is calculated based on the condition nodes of each key window;

[0037] The preset decision action with the highest probability value is selected as the predictive decision action for the corresponding key window.

[0038] Optionally, the behavior tree node generation module converts the prediction decision action of the key window into action nodes in a second preset format using the following method:

[0039] According to the preset syntax rules, the predictive decision actions of the key window are split into two parts: action instructions and control data.

[0040] Obtain the preset instruction code corresponding to the action instruction;

[0041] The corresponding instruction codes and control data are assembled according to the second preset format to obtain the action nodes of the corresponding key windows.

[0042] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0043] Memory, storing at least one computer program; and

[0044] The processor executes the program stored in the memory to implement the game adversarial behavior tree generation method based on key window states described above.

[0045] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned method for generating a game adversarial behavior tree based on key window states.

[0046] This application filters out key window states from the decision sequence by adding noise to the initial state of the agent at each decision moment and calculating the difference between the initial policy evaluation and the perturbed policy evaluation at each decision moment. Then, based on the attribute information and predicted decision actions corresponding to the key window states, it constructs the condition nodes and action nodes of the key window behavior tree. Finally, according to the time sequence of each key window, it connects the control nodes in the behavior tree of each key window to obtain the behavior tree of the preset intelligent decision model. By constructing the behavior tree of the intelligent decision model based on the key window states, the complex decision-making process of the intelligent decision model is focused on a limited number of key windows that play a key role. The behavior tree is used to show the causal relationship of the decision of the key windows, so that the behavior tree can be directly applied to explain the complex decision-making process of game adversarial based on deep reinforcement learning. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating a method for generating a game adversarial behavior tree based on key window states, as provided in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the behavior tree structure in a game adversarial behavior tree generation method based on key window states provided in an embodiment of the present invention.

[0049] Figure 3 A functional block diagram of a game adversarial behavior tree generation device based on key window states provided in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the game adversarial behavior tree generation method based on key window states, according to an embodiment of the present invention.

[0051] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] This application provides a method for generating a game-theoretic adversarial behavior tree based on key window states. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method for generating a game-theoretic adversarial behavior tree based on key window states can be executed by software or hardware installed on a terminal device or a server device. The software can be a blockchain platform. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0054] Reference Figure 1 The diagram shown is a flowchart illustrating a method for generating a game-theoretic behavior tree based on key window states according to an embodiment of the present invention. In this embodiment, the method for generating a game-theoretic behavior tree based on key window states includes:

[0055] S1. Obtain the decision sequence of the preset intelligent decision model, wherein the decision sequence includes the initial state and initial policy evaluation of the agent controlled by the preset intelligent decision model at each decision time.

[0056] Intelligent decision-making models are models that use artificial intelligence and machine learning techniques to assist the decision-making process. These models use large amounts of data and algorithms to analyze, predict, and optimize decision-making problems, helping decision-makers make more informed decisions.

[0057] This invention uses a drone aerial game simulation scenario as an example to describe the process of transforming the preset intelligent decision-making model into an interpretable behavior tree structure. In this embodiment, the preset intelligent decision-making model refers to a model that makes corresponding game decisions based on different drone states. For example, in a 3v3 drone intelligent game simulation scenario, the preset intelligent decision-making model can determine whether to attack the enemy drone based on its coordinates.

[0058] In this embodiment of the invention, the decision sequence generated by the preset intelligent decision-making model can be represented as:

[0059] {(s t ,a t ,q t |t={1,2,…,T},

[0060] Where t∈[1,T], T is a natural number, representing each decision moment, and the decision moment refers to the moment in the game generated by the preset intelligent decision-making model when a decision needs to be made; s t This represents the initial state of the agent controlled by the preset intelligent decision-making model at each decision moment. In this embodiment of the invention, the agent is a drone, for example, s t This includes, but is not limited to, the location coordinates of the area where the drone is located, or the location coordinates of allies or enemy aircraft that it can observe; a t This indicates that the preset intelligent decision-making model is based on each s t Make the corresponding initial decision action, for example, the a t This includes, but is not limited to, traveling to different areas of space or attacking enemy drones; q t This represents the initial strategy evaluation made by the preset intelligent decision-making model for each decision-making moment, based on the initial decision action.

[0061] In this embodiment of the invention, the decision-making process executed by the preset intelligent decision-making model is broken down into multiple decision moments s. t And obtain each decision time s. t The corresponding initial state s of the agent t Initial decision action a t and initial strategy evaluation q t This prepares for converting the decision-making process of intelligent decision-making models into interpretable behavior trees.

[0062] S2. Add noise to the initial state of the agent at each decision moment to obtain the perturbed state. Use the preset intelligent decision model to generate the perturbed policy evaluation corresponding to each perturbed state. Calculate the difference between the initial policy evaluation and the perturbed policy evaluation at each decision moment in sequence. Select a preset number of initial states of the agent at decision moments as key window states based on the difference. Use the decision moments corresponding to the key window states as key windows.

[0063] In this embodiment of the invention, the key window state refers to the drone state that plays an important role in the decision-making process throughout the entire drone intelligent game simulation scenario; it is the window state during which the preset intelligent decision-making model makes an effective decision. For example, the key window state can refer to the drone's area coordinates, enemy aircraft coordinates, etc.

[0064] In this embodiment of the invention, key window states are identified from the decision sequence. The identified key window states can be further used to construct an interpretable behavior tree for the preset intelligent decision model. Compared with the method of constructing a behavior tree using all agent states in the decision sequence, the complex decision-making process of the intelligent decision model is focused on a limited number of key windows that play a key role, which can reduce the interpretability complexity of the intelligent decision model.

[0065] Furthermore, in this embodiment of the invention, Gaussian noise can be generated using the Gaussian formula, and then the generated Gaussian noise can be added sequentially to the initial state of the agent corresponding to each decision time to obtain the perturbed state corresponding to each decision time.

[0066] In this embodiment of the invention, the post-disturbance state can be represented as: Among them, the The s represents the perturbation state corresponding to the decision time t. t Let t represent the initial state of the agent at decision time t, and τ represent the noise added to the state of the agent at decision time t.

[0067] In this embodiment of the invention, a Gaussian noise-based perturbation method is adopted, which, compared with the conventional numerical perturbation method, can more flexibly add adaptive perturbation values ​​to the state features and improve the efficiency of key window state recognition.

[0068] In this embodiment of the invention, the policy evaluation network in the preset intelligent decision-making model can be used to evaluate the post-perturbation state. Decision action following the disturbance at decision time t Calculate the strategy evaluation after the perturbation

[0069] In this embodiment of the invention, the difference corresponding to each decision moment is sorted from largest to smallest, and the initial state of the agent corresponding to a preset number of decision moments is selected as the key window state according to the order of the difference from largest to smallest. The preset number can be set according to actual business needs. For example, if the spanning tree to be constructed includes a 5-layer structure, the preset number can be set to 5.

[0070] It is understandable that the initial policy evaluation q corresponds to each of the aforementioned decision moments. t Evaluation of post-disturbance strategies The difference between them reflects the impact of noise disturbance on the agent's decision-making. The larger the difference, the greater the impact, indicating that the agent's state at the decision moment is a critical window state.

[0071] S3. Obtain the attribute information corresponding to the state of each key window, and convert the attribute information into a condition node of the first preset format for the corresponding key window. Use the preset intelligent decision model to generate a predictive decision action for the corresponding key window based on the condition node, and convert the predictive decision action of the key window into an action node of the second preset format.

[0072] In this embodiment of the invention, in the drone aerial game simulation scenario, the attribute information corresponding to each key window state includes, but is not limited to, the position (coordinates) of the agent and the enemy units (several drones, or specific drone number information) and the enemy aircraft positions (coordinates) appearing in the agent's field of vision.

[0073] In this embodiment of the invention, the first preset format and the second preset format refer to information such as the storage format and data encoding rules of the data in the condition node and the action node, respectively. It can be understood that the condition node and action node corresponding to the key window can be set with specific data storage format or data encoding rules according to actual business needs, so as to standardize the expression format of the final generated behavior tree and improve the readability of the behavior tree.

[0074] In an optional embodiment of the present invention, the attribute information can be converted into a condition node of a first preset format corresponding to the key window using the following method:

[0075] The attribute information corresponding to each key window state is cleaned, and the cleaned attribute information is segmented into words to obtain a word segmentation set.

[0076] Each word in the word segmentation set is sequentially matched with preset conditional keywords using fuzzy matching.

[0077] The successfully matched word segments are converted according to the first preset format, and the converted word segments are filled into the condition nodes of the corresponding key windows.

[0078] In this embodiment of the invention, the attribute information can be cleaned by removing stop words and useless punctuation marks, thereby improving the usability of the attribute information.

[0079] In this embodiment of the invention, the preset conditional keywords can be set according to actual business needs, including but not limited to text keywords used to represent location coordinates, area labels, our identification or enemy aircraft identification.

[0080] It is understandable that in the application of the preset intelligent decision-making model in the scenario of drone aerial game confrontation, the state of the intelligent agent corresponding to different condition nodes will be analyzed and judged, and the decision action to be executed will be determined in the end.

[0081] In detail, the step of generating predictive decision actions for corresponding key windows based on the condition nodes using the preset intelligent decision-making model includes:

[0082] Using the preset intelligent decision-making model, the probability value of each preset decision action is calculated based on the condition nodes of each key window;

[0083] The preset decision action with the highest probability value is selected as the predictive decision action for the corresponding key window.

[0084] In this embodiment of the invention, the preset decision action refers to all possible decision actions predetermined for a specific actual application scenario, such as attacking an enemy aircraft with a certain number or heading to a designated area.

[0085] In another optional embodiment of the present invention, the prediction decision action of the key window can be converted into an action node in a second preset format by the following method:

[0086] According to the preset syntax rules, the predictive decision actions of the key window are split into two parts: action instructions and control data.

[0087] Obtain the preset instruction code corresponding to the action instruction;

[0088] The corresponding instruction codes and control data are assembled according to the second preset format to obtain the action nodes of the corresponding key windows.

[0089] In this embodiment of the invention, the preset syntax rules refer to the syntax rules corresponding to the instruction language that the actual intelligent agent can recognize. For example, if the decision action of a certain key window is to attack enemy aircraft number 1, according to the preset syntax rules, the attack is determined as the action command, and enemy aircraft number 1 is determined as control data; if the decision action is to go to area B, then going is determined as the action command, and area B is determined as control data.

[0090] The preset instruction code refers to the unified encoding of all preset decision actions according to actual business needs. For example, the preset instruction code for "attack" is 00, the preset instruction code for "go to" is 01, and the preset instruction code for "follow" is 02.

[0091] In this embodiment of the invention, the second preset format can be set according to actual business needs. For example, the action node uses eight bytes to store the decision action, the first four bytes to store the instruction code, and the last four bytes to store the control data.

[0092] S4. Using each key window as a control node, and the condition nodes and action nodes of the key window as branches of the corresponding control node, a behavior tree corresponding to each key window is obtained. According to the time sequence of each key window, the control nodes in the behavior tree of each key window are connected in series to obtain the behavior tree of the preset intelligent decision model.

[0093] Reference Figure 2 The diagram shown illustrates the structure of the behavior tree in a game-theoretic behavior tree generation method based on key window states according to an embodiment of the present invention. In this embodiment, rectangle 21 represents a control node, i.e., the root node of the behavior tree corresponding to each key window; diamond 22 represents a condition node; and circular 23 represents an action node. The control nodes of each layer are connected according to the chronological order of the key windows to form a multi-layered behavior tree.

[0094] In this embodiment of the invention, the decision-making process of the preset intelligent decision-making model in drone game confrontation is sorted out by constructing a behavior tree. In the behavior tree of the preset intelligent decision-making model, each layer corresponds to a key decision moment. Through condition nodes and action nodes, the origin of the final decision action executed at the key decision moment can be clearly explained. Then, the causal relationship of the core key decision of the preset intelligent decision-making model and the sequential relationship of the execution of decision actions are completely displayed according to the order of decision moments.

[0095] This invention proposes a method to filter key window states from the decision sequence by adding noise to the initial state of the agent at each decision moment, and then calculating the difference between the evaluation of the publishing strategy and the evaluation of the strategy after perturbation at each decision moment. Based on the attribute information and decision actions corresponding to the key window states, condition nodes and action nodes of the key window behavior tree are constructed. Finally, according to the temporal order of each key window, the control nodes in the behavior tree of each key window are connected in series to obtain the behavior tree of the preset intelligent decision model. By constructing the behavior tree of the intelligent decision model based on the key window states, the complex decision-making process of the intelligent decision model is focused on a limited number of key windows that play a crucial role. The behavior tree is used to demonstrate the causal relationship of the decisions of the key windows, allowing the behavior tree to be directly applied to explain the complex decision-making process of game adversarial based on deep reinforcement learning.

[0096] like Figure 3 The diagram shown is a functional block diagram of a game adversarial behavior tree generation device based on key window states provided in an embodiment of the present invention.

[0097] The game-theoretic behavior tree generation device 100 based on key window states described in this invention can be installed in an electronic device. According to the functions implemented, the game-theoretic behavior tree generation device 100 includes a decision sequence acquisition module 101, a noise addition module 102, a key window state recognition module 103, a behavior tree node generation module 104, a key window behavior tree construction module 105, and a model behavior tree assembly module 106. The modules described in this invention can also be called units, referring to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0098] In this embodiment, the functions of each module / unit are as follows:

[0099] The decision sequence acquisition module 101 is used to acquire the decision sequence of the preset intelligent decision model, wherein the decision sequence includes the initial state and initial policy evaluation of the agent controlled by the preset intelligent decision model at each decision time.

[0100] The noise addition module 102 is used to add noise to the initial state of the agent at each decision moment to obtain the perturbed state, and use the preset intelligent decision model to generate the perturbed policy evaluation corresponding to each perturbed state.

[0101] The key window state identification module 103 is used to sequentially calculate the difference between the initial policy evaluation and the perturbation policy evaluation at each decision moment, select the state of the agent corresponding to the maximum difference of a preset number as the key window state, and take the decision moment corresponding to the key window state as the key window.

[0102] The behavior tree node generation module 104 is used to obtain attribute information corresponding to each key window state, convert the attribute information into a first preset format condition node for the corresponding key window, use the preset intelligent decision model to generate a predictive decision action for the corresponding key window based on the condition node, and convert the predictive decision action for the key window into an action node in a second preset format.

[0103] The key window behavior tree construction module 105 is used to obtain a behavior tree corresponding to each key window by taking each key window as a control node and the condition node and action node of the same key window as branches of the corresponding control node.

[0104] The model behavior tree assembly module 106 is used to connect the control nodes in the behavior tree of each key window according to the time sequence of each key window to obtain the behavior tree of the preset intelligent decision model.

[0105] In detail, the specific implementation methods of each module of the game adversarial behavior tree generation device 100 based on key window states are as follows:

[0106] Step 1: Obtain the decision sequence of the preset intelligent decision model, wherein the decision sequence includes the initial state and initial policy evaluation of the agent controlled by the preset intelligent decision model at each decision time.

[0107] Intelligent decision-making models are models that use artificial intelligence and machine learning techniques to assist the decision-making process. These models use large amounts of data and algorithms to analyze, predict, and optimize decision-making problems, helping decision-makers make more informed decisions.

[0108] This invention uses a drone aerial game simulation scenario as an example to describe the process of transforming the preset intelligent decision-making model into an interpretable behavior tree structure. In this embodiment, the preset intelligent decision-making model refers to a model that makes corresponding game decisions based on different drone states. For example, in a 3v3 drone intelligent game simulation scenario, the preset intelligent decision-making model can determine whether to attack the enemy drone based on its coordinates.

[0109] In this embodiment of the invention, the decision sequence generated by the preset intelligent decision-making model can be represented as:

[0110] {(s t ,a t ,q t |t={1,2,…,T},

[0111] Where t∈[1,T], T is a natural number, representing each decision moment, and the decision moment refers to the moment in the game generated by the preset intelligent decision-making model when a decision needs to be made; s t This represents the initial state of the agent controlled by the preset intelligent decision-making model at each decision moment. In this embodiment of the invention, the agent is a drone, for example, s t This includes, but is not limited to, the location coordinates of the area where the drone is located, or the location coordinates of allies or enemy aircraft that it can observe; a t This indicates that the preset intelligent decision-making model is based on each s t Make the corresponding initial decision action, for example, the a t This includes, but is not limited to, traveling to different areas of space or attacking enemy drones; q t This represents the initial strategy evaluation made by the preset intelligent decision-making model for each decision-making moment, based on the initial decision action.

[0112] In this embodiment of the invention, the decision-making process executed by the preset intelligent decision-making model is broken down into multiple decision moments s. tAnd obtain each decision time s. t The corresponding initial state s of the agent t Initial decision action a t and initial strategy evaluation q t This prepares for converting the decision-making process of intelligent decision-making models into interpretable behavior trees.

[0113] Step 2: Add noise to the initial state of the agent at each decision moment to obtain the perturbed state. Use the preset intelligent decision model to generate the perturbed policy evaluation corresponding to each perturbed state. Calculate the difference between the initial policy evaluation and the perturbed policy evaluation at each decision moment in sequence. Select a preset number of initial states of the agent at decision moments as key window states based on the difference, and use the decision moments corresponding to the key window states as key windows.

[0114] In this embodiment of the invention, the key window state refers to the drone state that plays an important role in the decision-making process throughout the entire drone intelligent game simulation scenario; it is the window state during which the preset intelligent decision-making model makes an effective decision. For example, the key window state can refer to the drone's area coordinates, enemy aircraft coordinates, etc.

[0115] In this embodiment of the invention, key window states are identified from the decision sequence. The identified key window states can be further used to construct an interpretable behavior tree for the preset intelligent decision model. Compared with the method of constructing a behavior tree using all agent states in the decision sequence, the complex decision-making process of the intelligent decision model is focused on a limited number of key windows that play a key role, which can reduce the interpretability complexity of the intelligent decision model.

[0116] Furthermore, in this embodiment of the invention, Gaussian noise can be generated using the Gaussian formula, and then the generated Gaussian noise can be added sequentially to the initial state of the agent corresponding to each decision time to obtain the perturbed state corresponding to each decision time.

[0117] In this embodiment of the invention, the post-disturbance state can be represented as: Among them, the The s represents the perturbation state corresponding to the decision time t. t Let t represent the initial state of the agent at decision time t, and τ represent the noise added to the state of the agent at decision time t.

[0118] In this embodiment of the invention, a Gaussian noise-based perturbation method is adopted, which, compared with the conventional numerical perturbation method, can more flexibly add adaptive perturbation values ​​to the state features and improve the efficiency of key window state recognition.

[0119] In this embodiment of the invention, the policy evaluation network in the preset intelligent decision-making model can be used to evaluate the post-perturbation state. Decision action following the disturbance at decision time t Calculate the strategy evaluation after the perturbation

[0120] In this embodiment of the invention, the difference corresponding to each decision moment is sorted from largest to smallest, and the initial state of the agent corresponding to a preset number of decision moments is selected as the key window state according to the order of the difference from largest to smallest. The preset number can be set according to actual business needs. For example, if the spanning tree to be constructed includes a 5-layer structure, the preset number can be set to 5.

[0121] It is understandable that the initial policy evaluation q corresponds to each of the aforementioned decision moments. t Evaluation of post-disturbance strategies The difference between them reflects the impact of noise disturbance on the agent's decision-making. The larger the difference, the greater the impact, indicating that the agent's state at the decision moment is a critical window state.

[0122] Step 3: Obtain the attribute information corresponding to each key window state, convert the attribute information into a first preset format condition node for the corresponding key window, use the preset intelligent decision model to generate a predictive decision action for the corresponding key window based on the condition node, and convert the predictive decision action for the key window into an action node in a second preset format.

[0123] In this embodiment of the invention, in the drone aerial game simulation scenario, the attribute information corresponding to each key window state includes, but is not limited to, the position (coordinates) of the agent and the enemy units (several drones, or specific drone number information) and the enemy aircraft positions (coordinates) appearing in the agent's field of vision.

[0124] In this embodiment of the invention, the first preset format and the second preset format refer to information such as the storage format and data encoding rules of the data in the condition node and the action node, respectively. It can be understood that the condition node and action node corresponding to the key window can be set with specific data storage format or data encoding rules according to actual business needs, so as to standardize the expression format of the final generated behavior tree and improve the readability of the behavior tree.

[0125] In an optional embodiment of the present invention, the attribute information can be converted into a condition node of a first preset format corresponding to the key window using the following method:

[0126] The attribute information corresponding to each key window state is cleaned, and the cleaned attribute information is segmented into words to obtain a word segmentation set.

[0127] Each word in the word segmentation set is sequentially matched with preset conditional keywords using fuzzy matching.

[0128] The successfully matched word segments are converted according to the first preset format, and the converted word segments are filled into the condition nodes of the corresponding key windows.

[0129] In this embodiment of the invention, the attribute information can be cleaned by removing stop words and useless punctuation marks, thereby improving the usability of the attribute information.

[0130] In this embodiment of the invention, the preset conditional keywords can be set according to actual business needs, including but not limited to text keywords used to represent location coordinates, area labels, our identification or enemy aircraft identification.

[0131] It is understandable that in the application of the preset intelligent decision-making model in the scenario of drone aerial game confrontation, the state of the intelligent agent corresponding to different condition nodes will be analyzed and judged, and the decision action to be executed will be determined in the end.

[0132] In detail, the step of generating predictive decision actions for corresponding key windows based on the condition nodes using the preset intelligent decision-making model includes:

[0133] Using the preset intelligent decision-making model, the probability value of each preset decision action is calculated based on the condition nodes of each key window;

[0134] The preset decision action with the highest probability value is selected as the predictive decision action for the corresponding key window.

[0135] In this embodiment of the invention, the preset decision action refers to all possible decision actions predetermined for a specific actual application scenario, such as attacking an enemy aircraft with a certain number or heading to a designated area.

[0136] In another optional embodiment of the present invention, the prediction decision action of the key window can be converted into an action node in a second preset format by the following method:

[0137] According to the preset syntax rules, the predictive decision actions of the key window are split into two parts: action instructions and control data.

[0138] Obtain the preset instruction code corresponding to the action instruction;

[0139] The corresponding instruction codes and control data are assembled according to the second preset format to obtain the action nodes of the corresponding key windows.

[0140] In this embodiment of the invention, the preset syntax rules refer to the syntax rules corresponding to the instruction language that the actual intelligent agent can recognize. For example, if the decision action of a certain key window is to attack enemy aircraft number 1, according to the preset syntax rules, the attack is determined as the action command, and enemy aircraft number 1 is determined as control data; if the decision action is to go to area B, then going is determined as the action command, and area B is determined as control data.

[0141] The preset instruction code refers to the unified encoding of all preset decision actions according to actual business needs. For example, the preset instruction code for "attack" is 00, the preset instruction code for "go to" is 01, and the preset instruction code for "follow" is 02.

[0142] In this embodiment of the invention, the second preset format can be set according to actual business needs. For example, the action node uses eight bytes to store the decision action, the first four bytes to store the instruction code, and the last four bytes to store the control data.

[0143] Step 4: Using each key window as a control node, and the condition nodes and action nodes of the key window as branches of the corresponding control node, obtain the behavior tree corresponding to each key window. According to the time sequence of each key window, connect the control nodes in the behavior tree of each key window to obtain the behavior tree of the preset intelligent decision model.

[0144] Reference Figure 2 The diagram shown illustrates the structure of the behavior tree in a game-theoretic behavior tree generation method based on key window states according to an embodiment of the present invention. In this embodiment, rectangle 21 represents a control node, i.e., the root node of the behavior tree corresponding to each key window; diamond 22 represents a condition node; and circular 23 represents an action node. The control nodes of each layer are connected according to the chronological order of the key windows to form a multi-layered behavior tree.

[0145] In this embodiment of the invention, the decision-making process of the preset intelligent decision-making model in drone game confrontation is sorted out by constructing a behavior tree. In the behavior tree of the preset intelligent decision-making model, each layer corresponds to a key decision moment. Through condition nodes and action nodes, the origin of the final decision action executed at the key decision moment can be clearly explained. Then, the causal relationship of the core key decision of the preset intelligent decision-making model and the sequential relationship of the execution of decision actions are completely displayed according to the order of decision moments.

[0146] The device of this invention filters out key window states from the decision sequence by adding noise to the initial state of the agent at each decision moment, and then calculating the difference between the evaluation of the publishing strategy and the evaluation of the strategy after perturbation at each decision moment. Based on the attribute information and decision actions corresponding to the key window states, it constructs condition nodes and action nodes of the key window behavior tree. Finally, according to the temporal order of each key window, it connects the control nodes in the behavior tree of each key window to obtain the behavior tree of the preset intelligent decision model. By constructing the behavior tree of the intelligent decision model based on the key window states, the complex decision-making process of the intelligent decision model is focused on a limited number of key windows that play a crucial role. The behavior tree displays the causal relationship of the decisions of the key windows, allowing the behavior tree to be directly applied to explain the complex decision-making process of game adversarial competition based on deep reinforcement learning.

[0147] like Figure 4 The diagram shown is a structural schematic of an electronic device that implements a method for generating a game adversarial behavior tree based on key window states, according to an embodiment of the present invention.

[0148] The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a game adversarial behavior tree generation based on key window states.

[0149] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as code generated based on a game-theoretic behavior tree of key window states, but also to temporarily store data that has been output or will be output.

[0150] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a game-theoretic behavior tree generation program based on key window states) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0151] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0152] Figure 4 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 4 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0153] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0154] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0155] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0156] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0157] The generation of the game adversarial behavior tree based on key window states, stored in the memory 11 of the electronic device 1, is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0158] Obtain the decision sequence of the preset intelligent decision model, wherein the decision sequence includes the initial state and initial policy evaluation of the agent controlled by the preset intelligent decision model at each decision time.

[0159] Noise is added to the initial state of the agent at each decision moment to obtain the perturbed state, and the perturbed policy evaluation corresponding to each perturbed state is generated using the preset intelligent decision model;

[0160] The difference between the initial policy evaluation and the policy evaluation after perturbation at each decision moment is calculated sequentially. Based on the difference, a preset number of decision moments are selected as the initial state of the agent as the key window state, and the decision moment corresponding to the key window state is taken as the key window.

[0161] Obtain attribute information corresponding to the state of each key window, and convert the attribute information into a condition node of the first preset format for the corresponding key window. Use the preset intelligent decision model to generate a predictive decision action for the corresponding key window based on the condition node, and convert the predictive decision action of the key window into an action node of the second preset format.

[0162] Using each key window as a control node, and the condition nodes and action nodes of the key window as branches of the corresponding control node, a behavior tree corresponding to each key window is obtained.

[0163] Based on the chronological order of each key window, the control nodes in the behavior tree of each key window are connected in series to obtain the behavior tree of the preset intelligent decision-making model.

[0164] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0165] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0166] Obtain the decision sequence of the preset intelligent decision model, wherein the decision sequence includes the initial state and initial policy evaluation of the agent controlled by the preset intelligent decision model at each decision time.

[0167] Noise is added to the initial state of the agent at each decision moment to obtain the perturbed state, and the perturbed policy evaluation corresponding to each perturbed state is generated using the preset intelligent decision model;

[0168] The difference between the initial policy evaluation and the policy evaluation after perturbation at each decision moment is calculated sequentially. Based on the difference, a preset number of decision moments are selected as the initial state of the agent as the key window state, and the decision moment corresponding to the key window state is taken as the key window.

[0169] Obtain attribute information corresponding to the state of each key window, and convert the attribute information into a condition node of the first preset format for the corresponding key window. Use the preset intelligent decision model to generate a predictive decision action for the corresponding key window based on the condition node, and convert the predictive decision action of the key window into an action node of the second preset format.

[0170] Using each key window as a control node, and the condition nodes and action nodes of the key window as branches of the corresponding control node, a behavior tree corresponding to each key window is obtained.

[0171] Based on the chronological order of each key window, the control nodes in the behavior tree of each key window are connected in series to obtain the behavior tree of the preset intelligent decision-making model.

[0172] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0173] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0174] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0175] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0176] The embodiments of this application can acquire and process relevant data based on holographic projection technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0177] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating a game-theoretic adversarial behavior tree based on key window states, characterized in that, The method includes: Obtain the decision sequence of the preset intelligent decision model, wherein the decision sequence includes the initial state and initial policy evaluation of the agent controlled by the preset intelligent decision model at each decision time. Noise is added to the initial state of the agent at each decision moment to obtain the perturbed state, and the perturbed policy evaluation corresponding to each perturbed state is generated using the preset intelligent decision model; The difference between the initial policy evaluation and the policy evaluation after perturbation at each decision moment is calculated sequentially. Based on the difference, a preset number of decision moments are selected as the initial state of the agent as the key window state, and the decision moment corresponding to the key window state is taken as the key window. Obtain attribute information corresponding to the state of each key window, and convert the attribute information into a condition node of the first preset format for the corresponding key window. Use the preset intelligent decision model to generate a predictive decision action for the corresponding key window based on the condition node, and convert the predictive decision action of the key window into an action node of the second preset format. Using each key window as a control node, and the condition nodes and action nodes of the key window as branches of the corresponding control node, a behavior tree corresponding to each key window is obtained. Based on the chronological order of each key window, the control nodes in the behavior tree of each key window are connected in series to obtain the behavior tree of the preset intelligent decision-making model.

2. The method for generating a game adversarial behavior tree based on key window states as described in claim 1, characterized in that, The condition node for obtaining the attribute information corresponding to each key window state and converting it into a first preset format for the corresponding key window includes: The attribute information corresponding to each key window state is cleaned, and the cleaned attribute information is segmented into words to obtain a word segmentation set. Each word in the word segmentation set is sequentially matched with preset conditional keywords using fuzzy matching. The successfully matched word segments are converted according to the first preset format, and the converted word segments are filled into the condition nodes of the corresponding key windows.

3. The method for generating a game adversarial behavior tree based on key window states as described in claim 1, characterized in that, The step of generating a predictive decision action for a corresponding key window based on the condition nodes using the preset intelligent decision model includes: Using the preset intelligent decision-making model, the probability value of each preset decision action is calculated based on the condition nodes of each key window; The preset decision action with the highest probability value is selected as the predictive decision action for the corresponding key window.

4. The method for generating a game adversarial behavior tree based on key window states as described in claim 1, characterized in that, The step of converting the predictive decision action of the key window into an action node in a second preset format includes: According to the preset syntax rules, the predictive decision actions of the key window are split into two parts: action instructions and control data. Obtain the preset instruction code corresponding to the action instruction; The corresponding instruction codes and control data are assembled according to the second preset format to obtain the action nodes of the corresponding key windows.

5. A device for generating a game-theoretic adversarial behavior tree based on key window states, characterized in that, The device includes: The decision sequence acquisition module is used to acquire the decision sequence of a preset intelligent decision model, wherein the decision sequence includes the initial state and initial policy evaluation of the agent controlled by the preset intelligent decision model at each decision time. The noise addition module is used to add noise to the initial state of the agent at each decision moment to obtain the perturbed state, and to generate the perturbed policy evaluation corresponding to each perturbed state using the preset intelligent decision model. The key window state identification module is used to sequentially calculate the difference between the initial policy evaluation and the perturbation policy evaluation at each decision moment, select the state of the agent corresponding to the maximum difference of a preset number as the key window state, and take the decision moment corresponding to the key window state as the key window. The behavior tree node generation module is used to obtain attribute information corresponding to the state of each key window, convert the attribute information into a condition node of the first preset format corresponding to the key window, use the preset intelligent decision model to generate the predictive decision action of the corresponding key window according to the condition node, and convert the predictive decision action of the key window into an action node of the second preset format. The key window behavior tree construction module is used to obtain the behavior tree corresponding to each key window by taking each key window as a control node and the condition nodes and action nodes of the same key window as branches of the corresponding control node. The model behavior tree assembly module is used to connect the control nodes in the behavior tree of each key window according to the time sequence of each key window to obtain the behavior tree of the preset intelligent decision model.

6. The game adversarial behavior tree generation device based on key window states as described in claim 5, characterized in that, The behavior tree node generation module converts the attribute information into condition nodes of a first preset format corresponding to the key window using the following method: The attribute information corresponding to each key window state is cleaned, and the cleaned attribute information is segmented into words to obtain a word segmentation set. Each word in the word segmentation set is sequentially matched with preset conditional keywords using fuzzy matching. The successfully matched word segments are converted according to the first preset format, and the converted word segments are filled into the condition nodes of the corresponding key windows.

7. The game adversarial behavior tree generation device based on key window states as described in claim 5, characterized in that, The behavior tree node generation module uses the preset intelligent decision-making model to generate predictive decision actions for corresponding key windows based on the condition nodes through the following method: Using the preset intelligent decision-making model, the probability value of each preset decision action is calculated based on the condition nodes of each key window; The preset decision action with the highest probability value is selected as the predictive decision action for the corresponding key window.

8. The game adversarial behavior tree generation device based on key window states as described in claim 5, characterized in that, The behavior tree node generation module converts the prediction decision actions of key windows into action nodes in a second preset format using the following method: According to the preset syntax rules, the predictive decision actions of the key window are split into two parts: action instructions and control data. Obtain the preset instruction code corresponding to the action instruction; The corresponding instruction codes and control data are assembled according to the second preset format to obtain the action nodes of the corresponding key windows.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the game adversarial behavior tree generation method based on any one of claims 1 to 4.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the game adversarial behavior tree generation method based on any one of claims 1 to 4.

Images