Wargame agent training method, prediction method and corresponding system
By combining imitation learning with reinforcement learning of generated expert demonstration data sets and multi-task networks, the problems of sparse rewards and complex decision-making in the construction of agents in wargame deduction are solved, and efficient acquisition of high-performance agents is achieved, and training efficiency and decision-making ability are improved. .
Patent Information
- Application Number
- CN202410302417.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-15
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-03-15
AI Technical Summary
In wargame deduction, the construction of an agent faces characteristics such as sparse rewards, complex decision-making factors, and random state transfers. It is difficult for the existing technology to efficiently obtain an agent with good decision-making level.
By generating expert demonstration data sets, combining imitation learning and reinforcement learning, a multi-task network is built, and the structure of branches and operator branches is used to process imitation learning and reinforcement learning training is carried out to optimize the decision strategy of the agent.
It realizes efficient acquisition of agents with higher performance in the wargame deduction environment, improves training efficiency and decision-making capabilities of agents, reduces dependence on data and manual participation, and improves scalability and strategy exploration capabilities.
Smart Images

Figure CN118045360B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to the field of computer technology, and more specifically, to a training method, a prediction method and a corresponding system for a war game agent. Background Art
[0002] Wargame is a game that simulates real war. The complexity of its state-action space is much higher than that of chess and card games. It also has characteristics such as imperfect information and long-term decision-making. It can provide a strategy verification environment for the research of more complex intelligent decision-making technologies.
[0003] The existing methods of constructing intelligent agents can be divided into three categories: knowledge planning-based, supervised learning-based, and reinforcement learning-based. The intelligent agent based on knowledge planning encodes the high-level knowledge of human experts through technologies such as finite state automata, rule systems, conditional trees, and parameterized scripts to construct intelligent agent decision-making strategies. It is good at handling relatively fixed common battlefield situations and has excellent interpretability and good performance. The intelligent agent based on supervised learning uses the difference between the strategy generated by the calculation model and the replay data under the same state as a supervisory signal to guide the model to fit the data. This enables the intelligent agent to get rid of the complex artificial rule design while gaining the ability to cope with high-dimensional states and action spaces. Compared with more complex reinforcement learning methods, it has the advantages of being significantly simple, lightweight, and easy to train. The intelligent agent based on reinforcement learning continuously adjusts its own strategy according to the rewards fed back by the environment through exploration and trial and error in the environment to achieve the goal of maximizing the cumulative reward. This mechanism is consistent with the human learning process and has the potential to surpass the human level. In recent years, it has achieved great success in the field of game gambling.
[0004] The construction of intelligent agents in war game simulations faces many challenges. However, there is currently a lack of systems and methods that are adapted to the characteristics of war game simulations and can efficiently obtain intelligent agents with good decision-making capabilities. Summary of the invention
[0005] The exemplary embodiments of the present disclosure provide a training method, a prediction method and a corresponding system for a war game agent. In view of the characteristics of war game simulation, such as sparse rewards, complex decision factors and high randomness of state transitions, an agent construction system and method that are more suitable for the characteristics of war game simulation are proposed, and efficient acquisition of agents with higher performance is achieved through innovation of system structure and corresponding method flow.
[0006] According to a first aspect of an embodiment of the present disclosure, a method for training a wargame agent is provided, the training method comprising: generating an expert demonstration data set based on a wargame confrontation environment; performing imitation learning training on a multi-task network in the wargame agent for selecting maneuvering positions for each operator of our side based on the expert demonstration data set, so as to obtain a preliminarily trained multi-task network; performing reinforcement learning training on the preliminarily trained multi-task network based on the wargame confrontation environment, so as to obtain an optimized multi-task network; wherein the expert demonstration data set comprises: situation features required for decision-making of the multi-task network, and maneuvering position labels of N operators of our side, where N is an integer greater than 1.
[0007] Optionally, the multi-task network includes: a shared feature processing branch and N operator branches, and the N operator branches correspond one-to-one to the N operators; wherein the input of the shared feature processing branch includes: the overall situation characteristics of the war game confrontation, and the output of the shared feature processing branch includes: shared features; wherein the input of the i-th operator branch includes: the shared features and the situation characteristics of the i-th operator itself, and the output of the i-th operator branch includes: the maneuvering direction selected for the i-th operator, i is an integer greater than 0 and less than or equal to N.
[0008] Optionally, the i-th operator branch includes: a first convolutional neural network module, a first attention mechanism module, a second attention mechanism module, a third attention mechanism module, a splicing module, a deep learning module based on a self-attention mechanism, and an output module; the shared feature processing branch includes: a first multi-layer perceptron module, a second multi-layer perceptron module, and a second convolutional neural network module; wherein the input of the first convolutional neural network module includes the position feature of the i-th operator itself, the input of the first attention mechanism module includes the output of the first convolutional neural network module and the output of the second convolutional neural network module, the input of the second attention mechanism module includes the output of the second multi-layer perceptron module, and the third attention mechanism module includes the output of the second multi-layer perceptron module. The input of the module includes the output of the first multi-layer perceptron module, the input of the splicing module includes the output of the first attention mechanism module, the output of the second attention mechanism module and the output of the third attention mechanism module, the input of the deep learning module includes the output of the splicing module, the input of the output module includes the output of the deep learning module, and the output of the output module includes the maneuvering position selected for the i-th operator; wherein, the input of the first multi-layer perceptron module includes the overall statistical characteristics of the war chess confrontation, the input of the second multi-layer perceptron module includes the attribute state characteristics of each operator of all participants, and the input of the second convolutional neural network module includes the overall spatial characteristics of the war chess confrontation.
[0009] Optionally, the step of generating an expert demonstration data set based on a war game confrontation environment includes: obtaining replay data generated by a user using the war game confrontation environment; performing quality screening on the replay data; and generating the expert demonstration data set based on the screened replay data.
[0010] Optionally, based on the expert demonstration data set, the step of performing imitation learning training on the multi-task network includes: using a behavioral cloning algorithm to train the multi-task network to fit the expert demonstration data set, so that the data distribution of the maneuvering positions of each operator on our side output by the multi-task network is close to the data distribution of the maneuvering position labels in the expert demonstration data set.
[0011] Optionally, based on the war game confrontation environment, the step of performing reinforcement learning training on the preliminarily trained multi-task network includes: using a reinforcement learning algorithm to update the parameters of the preliminarily trained multi-task network by interacting with the war game confrontation environment.
[0012] Optionally, the step of updating the parameters of the preliminarily trained multi-task network by interacting with the war game confrontation environment using a reinforcement learning algorithm includes: in each round of iteration of the reinforcement learning training, obtaining the situation characteristics required for the decision-making of the multi-task network based on the current war game confrontation environment and inputting them into the multi-task network, and controlling the actions of each of our operators in the war game confrontation environment according to the maneuvering positions selected by the multi-task network for each of our operators to update the war game confrontation environment; and updating the parameters of the multi-task network with the goal of accumulating and maximizing the rewards of the war game confrontation environment for the actions.
[0013] According to a second aspect of an embodiment of the present disclosure, a prediction method for a war game agent is provided, the prediction method comprising: based on a current war game confrontation environment, obtaining situation features required for decision-making of a multi-task network in the war game agent; inputting the obtained situation features into the multi-task network to obtain the maneuvering positions selected by the multi-task network for each of our operators; controlling the actions of each of our operators in the war game confrontation environment according to the maneuvering positions selected by the multi-task network for each of our operators; wherein the war game agent is trained by executing the training method as described above.
[0014] According to a third aspect of an embodiment of the present disclosure, a training system for a wargame agent is provided, the training system comprising: an expert data generating device, configured to generate an expert demonstration data set based on a wargame confrontation environment; an imitation learning training device, configured to perform imitation learning training on a multi-task network in the wargame agent for selecting maneuvering positions for each operator of our side, based on the expert demonstration data set, so as to obtain a preliminarily trained multi-task network; a reinforcement learning training device, configured to perform reinforcement learning training on the preliminarily trained multi-task network, based on the wargame confrontation environment, so as to obtain an optimized multi-task network; wherein the expert demonstration data set comprises: situation features required for decision-making of the multi-task network, and maneuvering position labels of N operators of our side, where N is an integer greater than 1.
[0015] According to a fourth aspect of an embodiment of the present disclosure, a prediction system for a wargame agent is provided, the prediction system comprising: a situation representation device, configured to obtain situation features required for decision-making of a multi-task network in the wargame agent based on a current wargame confrontation environment; a prediction device, configured to input the obtained situation features into the multi-task network to obtain the maneuvering positions selected by the multi-task network for each of our operators; an action control device, configured to control the actions of each of our operators in the wargame confrontation environment according to the maneuvering positions selected by the multi-task network for each of our operators; wherein the wargame agent is trained by executing the training method as described above.
[0016] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided. When the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the training method and / or prediction method as described above.
[0017] According to a sixth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to execute the training method and / or prediction method as described above.
[0018] According to a seventh aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising computer executable instructions, which, when executed by at least one processor, implement the training method and / or prediction method as described above.
[0019] According to the training method, prediction method and corresponding system of the wargame agent of the exemplary embodiment of the present disclosure, a system structure and training process are provided that systematically combine the training data set with the reinforcement learning training environment, support the agent training that integrates imitation learning and reinforcement learning, can improve the training efficiency, and obtain a higher performance agent.
[0020] In the following description, some aspects and / or advantages of the general inventive concept will be set forth, and some other aspects and / or advantages will be known through the following description or the implementation of the general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] These and / or other aspects and advantages of the present application will become clearer and easier to understand from the following detailed description of the embodiments of the present application in conjunction with the accompanying drawings, in which:
[0022] Figure 1 A flowchart showing a method for training a war game agent according to an exemplary embodiment of the present disclosure;
[0023] Figure 2 A structural block diagram showing a multi-task network according to an exemplary embodiment of the present disclosure;
[0024] Figure 3 A flowchart showing a method for generating an expert demonstration data set according to an exemplary embodiment of the present disclosure;
[0025] Figure 4 A flowchart showing a method for generating a maneuvering position tag according to an exemplary embodiment of the present disclosure is shown;
[0026] Figure 5 An example of a method for training a war game agent according to an exemplary embodiment of the present disclosure is shown;
[0027] Figure 6 An example of situation feature definition according to an exemplary embodiment of the present disclosure is shown;
[0028] Figure 7 A flowchart showing a prediction method of a war game agent according to an exemplary embodiment of the present disclosure;
[0029] Figure 8 A structural block diagram showing a training system for a war game agent according to an exemplary embodiment of the present disclosure is shown;
[0030] Fig. 9 A structural block diagram showing a training system for a war game agent according to another exemplary embodiment of the present disclosure is shown;
[0031] Fig.10 A structural block diagram showing a prediction system of a war game agent according to an exemplary embodiment of the present disclosure;
[0032] Fig.11 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0033] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments will be described below with reference to the drawings in order to explain the present disclosure.
[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0035] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0036] The present disclosure takes into account the following characteristics of existing agent construction methods, which make it difficult for them to meet the requirements of war game simulation environments for agents:
[0037] Intelligent agents based on knowledge planning cannot actively learn and improve strategies. The construction and expansion of various knowledge bases require a lot of manual participation, which is difficult to implement and lacks scalability. In addition, due to the large scale of the search space, it is difficult for the knowledge base to cover the entire search space. When faced with situations that are not logged in in the knowledge base, it is difficult for the intelligent agent to effectively handle them and it is difficult to cope with high-dimensional state and action spaces.
[0038] The supervised learning-based intelligent agents are highly dependent on data. The differences in the levels of players lead to uneven quality of replay data. Although high-quality data can be screened by certain standards, the definition and implementation of the screening standards and the practical limitations of the amount of data are the first difficulties faced by the application of supervised methods to train intelligent agents. At the same time, under ideal circumstances, the strategies obtained by supervised learning can only approximate human strategies, lack the ability to actively explore new strategies, and are difficult to surpass human levels.
[0039] Reinforcement learning-based agents have relatively low data utilization rates and usually require massive amounts of data to be collected online. In addition, slight changes in hyperparameters may lead to huge differences in results. This instability makes training difficult and places high demands on resources such as computing power. In addition, rewards in reinforcement learning play a vital role in model training. However, rewards in many environments are relatively sparse, resulting in most samples obtained by the agent not containing effective reward signals, which in turn makes the neural network parameters update slowly or even difficult to converge, greatly limiting the application of reinforcement learning methods.
[0040] Therefore, in view of the characteristics of war game simulation, such as sparse rewards, complex decision factors, and high randomness of state transition, this paper proposes an intelligent agent construction system and method that is more suitable for the characteristics of war game simulation, which combines the training data set with the reinforcement learning training environment in an orderly manner, supports the integration of imitation learning and reinforcement learning, and realizes the efficient acquisition of intelligent agents with higher performance through the innovation of system structure and corresponding method flow. Figures 1 to 11 Provide detailed explanation.
[0041] Figure 1 A flowchart of a method for training a war game agent according to an exemplary embodiment of the present disclosure is shown.
[0042] Reference Figure 1 In step S101, an expert demonstration dataset (hereinafter also referred to as expert dataset) is generated based on the war game confrontation environment.
[0043] The war game agent includes, but is not limited to: a multi-task network (i.e., a multi-task learning network) for selecting maneuvering positions for each operator of our side. The input of the multi-task network includes: the situation features required for its decision-making; the output includes: the maneuvering positions selected by it for N operators of our side, where N is an integer greater than 1.
[0044] Accordingly, the expert demonstration data set may include, but is not limited to: situation features required for multi-task network decision-making, and maneuvering position labels of our N operators.
[0045] In war games, "maneuver position" usually refers to the direction or position of a counter on the board. The importance of maneuver position is that it determines how counters move on the board and how they interact with other counters or obstacles. The correct maneuver position is crucial to developing effective tactics and strategies because it directly affects the deployment, mobility, and firepower of counters. In war games, the agent needs to choose the appropriate maneuver position for our counters based on the situation on the board, the attributes of the counters, and the battle situation in order to achieve the ultimate goal.
[0046] As an exemplary embodiment, the situation features input to the multi-task network may include not only the situation features of our N operators themselves, but also the situation features of the entire wargame confrontation.
[0047] As an exemplary embodiment, the situational features of the operator itself may include, but are not limited to, the positional features of the operator itself.
[0048] As an exemplary embodiment, the situational characteristics of the overall wargame confrontation may include but are not limited to at least one of the following items: overall statistical characteristics of the wargame confrontation, attribute state characteristics of all operators of all participants, and overall spatial characteristics of the wargame confrontation. Figure 6 The overall statistical characteristics of the war game confrontation are shown (i.e., Figure 6 The game statistics in the game), the attribute state characteristics of all operators of all participants (i.e., Figure 6 operator characteristics in the game), the overall spatial characteristics of the game (i.e., Figure 6 An example of the definition of spatial features in .
[0049] Regarding the multi-task network, the multi-task network may include: a shared feature processing branch and N operator branches, and the N operator branches correspond one-to-one to the N operators.
[0050] As an exemplary embodiment, the input of the shared feature processing branch may include: situational features of the overall wargame confrontation, and the output of the shared feature processing branch may include: shared features.
[0051] As an exemplary embodiment, the input of the i-th operator branch among N operator branches may include: the situation characteristics of the i-th operator itself and the shared characteristics output by the shared feature processing branch; the output of the i-th operator branch may include: the maneuvering direction selected for the i-th operator, where i is an integer greater than 0 and less than or equal to N.
[0052] For multi-task scenarios composed of multiple agents, the present invention introduces multi-task learning to simultaneously model the decisions of multiple operators on our side, which further improves the performance and reasoning efficiency of the agent compared with ordinary single-task learning, and has the advantages of simplifying and stabilizing the system.
[0053] The following will be combined Figure 2 To describe the specific structure of the multi-task network, we will not expand it here. Figure 3 and Figure 4 An exemplary embodiment of step S101 is described below, which will not be expanded here.
[0054] return Figure 1 In step S102, based on the expert demonstration data set, the multi-task network is trained by imitation learning to obtain a preliminarily trained multi-task network.
[0055] Specifically, based on the expert demonstration dataset, an imitation learning algorithm is used to adjust the parameters of the multi-task network to complete the preliminary training of the multi-task network.
[0056] As an exemplary embodiment, the imitation learning algorithm used in step S102 may include, but is not limited to, a behavior cloning algorithm. For example, an inverse reinforcement learning algorithm may also be used.
[0057] As an exemplary embodiment, a behavior cloning algorithm can be used to train a multi-task network to fit the expert demonstration data set, so that the data distribution of the maneuvering positions of each operator on our side output by the multi-task network is close to the data distribution of the maneuvering position labels in the expert demonstration data set, thereby achieving the preliminary construction of the parameters of the multi-task network. For example, the data distribution may refer to the proportion of different maneuvering positions.
[0058] In view of the characteristics of sparse rewards and complex decision-making factors in war game simulations, this paper constructs high-quality expert demonstration data and realizes the efficient construction of intelligent agents by fitting expert data through imitation learning.
[0059] In step S103, based on the war game confrontation environment, reinforcement learning training is performed on the preliminarily trained multi-task network to obtain an optimized multi-task network.
[0060] As an exemplary embodiment, a reinforcement learning algorithm can be used to update the parameters of the preliminarily trained multi-task network by interacting with the war game environment (i.e., optimizing the parameters of the multi-task network constructed using the imitation learning algorithm).
[0061] Specifically, as a preferred exemplary embodiment, in each round of iteration of reinforcement learning training, the situation features required for multi-task network decision-making can be obtained based on the current war game confrontation environment and input into the multi-task network, and the actions of each operator on our side can be controlled according to the maneuvering direction selected by the multi-task network for each operator on our side in the war game confrontation environment to update the war game confrontation environment; and, with the goal of accumulating and maximizing the rewards for actions in the war game confrontation environment, the parameters of the multi-task network are updated, so that the war game confrontation environment provides an interactive environment for reinforcement learning training, and the intelligent agent improves its strategy level through continuous exploration and trial and error in the war game confrontation environment.
[0062] In view of the characteristics of war game simulation, such as high randomness of state transition, large state space and limited data set coverage, this paper uses reinforcement learning to perform online tuning of the intelligent agent.
[0063] Figure 2 A structural block diagram of a multi-task network according to an exemplary embodiment of the present disclosure is shown.
[0064] Reference Figure 2, the i-th operator branch of the N operator branches of the multi-task network includes: the first convolutional neural network module (i.e., CNN1), the first attention mechanism module (i.e., att1), the second attention mechanism module (i.e., att2), the third attention mechanism module (i.e., att3), the concatenation module (i.e., concat), the deep learning module based on the self-attention mechanism (i.e., Transformer), and the output module. It should be understood that Figure 2 In the description, only the first operator branch (ie, operator 1 branch) is taken as an example for detailed description.
[0065] The shared feature processing branch of the multi-task network includes: a first multi-layer perceptron module (ie, MLP1), a second multi-layer perceptron module (ie, MLP2), and a second convolutional neural network module (ie, CNN2).
[0066] As an exemplary embodiment, for the i-th operator branch, the input of the first convolutional neural network module may include the positional features of the i-th operator itself; the input of the first attention mechanism module may include the output of the first convolutional neural network module and the output of the second convolutional neural network module; the input of the second attention mechanism module may include the output of the second multi-layer perceptron module; the input of the third attention mechanism module may include the output of the first multi-layer perceptron module; the input of the splicing module may include the output of the first attention mechanism module, the output of the second attention mechanism module, and the output of the third attention mechanism module; the input of the deep learning module may include the output of the splicing module; the input of the output module may include the output of the deep learning module; and the output of the output module may include the maneuvering position selected for the i-th operator.
[0067] As an exemplary embodiment, for the shared feature processing branch, the input of the first multi-layer perceptron module may include the overall statistical characteristics of the war game confrontation; the input of the second multi-layer perceptron module may include the attribute state characteristics of each operator of all participants; and the input of the second convolutional neural network module may include the overall spatial characteristics of the war game confrontation.
[0068] As an exemplary embodiment, the output of the splicing module may include: t 、f t-1 ,……,f t-m There are m+1 features in total (m is an integer greater than 0), among which f t represents the feature corresponding to the current moment, f t-1 represents the features corresponding to the moment before the current moment, and so on. Accordingly, the input of the multi-task network may include: the situation features of the current moment and the situation features of m moments before the current moment. The output of the deep learning module may include: f final .
[0069] It should be understood that the structures of the N operator branches are the same (i.e., they all include a first convolutional neural network module, a first attention mechanism module, a second attention mechanism module, a third attention mechanism module, a splicing module, a deep learning module based on a self-attention mechanism, and an output module), but after imitation learning training and reinforcement learning training, the specific module parameters of different operator branches may be different.
[0070] Based on the characteristics of both shared and independent features of operators in war game simulation, this paper proposes a method such as Figure 2 The multi-task attention network structure shown in the figure has a network input that is the situation features listed in the text on the far left, and a network output that is the maneuvering position predicted by each of our operators on the far right. The main feature is that all operators share a shared feature pool, while each operator has relatively independent branches and will learn features that are useful to itself from the shared feature pool.
[0071] Figure 3 A flowchart of a method for generating an expert demonstration dataset according to an exemplary embodiment of the present disclosure is shown.
[0072] Reference Figure 3 In step S201, the replay data generated by the user using the war game confrontation environment is obtained.
[0073] In step S202, the obtained replay data is quality screened.
[0074] As an exemplary embodiment, high-quality sample data that can represent expert behavior (i.e., can reflect the expert's high-level performance) can be screened out from the review data according to certain rules for use in imitation learning.
[0075] In step S203, an expert demonstration data set is generated based on the screened review data.
[0076] As an exemplary embodiment, situational features can be extracted from the filtered replay data, and corresponding maneuvering position labels can be generated for the filtered replay data, thereby obtaining an expert demonstration data set.
[0077] Figure 4 A flow chart of a method for generating a maneuvering position label according to an exemplary embodiment of the present disclosure is shown.
[0078] Figure 5 An example of a method for training a war game agent according to an exemplary embodiment of the present disclosure is shown.
[0079] Reference Figure 5The wargame agent learning and training method provided by the present disclosure is divided into two stages: imitation learning and reinforcement learning. The first stage is the stage of building a wargame agent based on imitation learning. First, an expert demonstration data set is built based on the wargame deduction and replay data, and then the imitation learning algorithm is used to train the multi-task network; the second stage is the stage of optimizing the wargame agent based on reinforcement learning. First, the reinforcement learning algorithm is implemented and connected to the wargame confrontation environment. The network parameters obtained in the imitation learning stage are used as the initialization parameters of the multi-task network in the reinforcement learning training, and then the reinforcement learning algorithm is used to make the agent iteratively update the strategy in the interaction with the environment.
[0080] In view of the characteristics of sparse rewards and complex decision-making factors in war game simulation, the present invention constructs high-quality expert demonstration data and realizes efficient construction of intelligent agents by fitting expert data through imitation learning; in view of the characteristics of high randomness of state transitions, large state space and limited coverage of data sets in war game simulation, the present invention performs online tuning of intelligent agents through reinforcement learning; in view of multi-task scenarios composed of multiple intelligent agents, the present invention introduces multi-task learning to simultaneously model the decisions of our multiple operators, which further improves the performance and reasoning efficiency of the intelligent agent compared with ordinary single-task learning, and has the advantages of simplifying and stabilizing the system.
[0081] Compared with the knowledge planning method, the intelligent agent disclosed in the present invention improves strategies through learning, reduces manual participation, and improves scalability; compared with the supervised method, the present invention uses reinforcement learning for online tuning based on imitation learning, reduces the degree of dependence on the data set, and improves the upper limit of the intelligent agent level; compared with the reinforcement learning method, the present invention provides a higher starting point for reinforcement learning through imitation learning, which is conducive to the exploration and improvement of strategies and improves training efficiency. Therefore, the war game intelligent agent construction system and method proposed in the present invention, through system structure innovation and comprehensive application of cutting-edge technologies, meets the requirements of war game deduction characteristics for intelligent agent construction, alleviates the shortcomings of existing technologies, and realizes the efficient acquisition of intelligent agents with higher performance, which provides a certain reference value for the exploration and research of cognitive intelligence in complex game environments.
[0082] Figure 7 A flow chart of a prediction method of a war game agent according to an exemplary embodiment of the present disclosure is shown. The war game agent is trained using the training method described in the exemplary embodiment above.
[0083] Reference Figure 7 In step S301, based on the current war game confrontation environment, the situation features required for the multi-task network decision in the war game agent are obtained.
[0084] In step S302, the acquired situation features are input into the multi-task network to obtain the maneuvering positions selected by the multi-task network for each operator of our side.
[0085] In step S303, in the war game confrontation environment, the actions of each operator on the side are controlled according to the maneuvering position selected for each operator on the side by the multi-task network.
[0086] Figure 8 A structural block diagram of a war game agent training system according to an exemplary embodiment of the present disclosure is shown.
[0087] Reference Figure 8 According to an exemplary embodiment of the present disclosure, a training system for a wargame agent includes: an expert data generating device 100, an imitation learning training device 200, and a reinforcement learning training device 300.
[0088] Specifically, the expert data generating device 100 is configured to generate an expert demonstration data set based on a war game confrontation environment.
[0089] The imitation learning training device 200 is configured to perform imitation learning training on the multi-task network used to select maneuvering positions for each operator of our side in the war game agent based on the expert demonstration data set, so as to obtain a preliminarily trained multi-task network.
[0090] The reinforcement learning training device 300 is configured to perform reinforcement learning training on the preliminarily trained multi-task network based on a war game confrontation environment to obtain an optimized multi-task network.
[0091] The expert demonstration dataset includes: situation features required for multi-task network decision-making and maneuver position labels of our N operators, where N is an integer greater than 1.
[0092] As an exemplary embodiment, the multi-task network may include: a shared feature processing branch and N operator branches, and the N operator branches correspond one-to-one to the N operators.
[0093] As an exemplary embodiment, the input of the shared feature processing branch may include: situational features of the overall wargame confrontation, and the output of the shared feature processing branch may include: shared features.
[0094] As an exemplary embodiment, the input of the i-th operator branch among N operator branches may include: the situation characteristics of the i-th operator itself and the shared characteristics output by the shared feature processing branch; the output of the i-th operator branch may include: the maneuvering direction selected for the i-th operator, where i is an integer greater than 0 and less than or equal to N.
[0095] As an exemplary embodiment, the i-th operator branch may include: a first convolutional neural network module, a first attention mechanism module, a second attention mechanism module, a third attention mechanism module, a splicing module, a deep learning module based on a self-attention mechanism, and an output module.
[0096] As an exemplary embodiment, the shared feature processing branch may include: a first multi-layer perceptron module, a second multi-layer perceptron module, and a second convolutional neural network module.
[0097] As an exemplary embodiment, for the i-th operator branch, the input of the first convolutional neural network module may include the positional features of the i-th operator itself; the input of the first attention mechanism module may include the output of the first convolutional neural network module and the output of the second convolutional neural network module; the input of the second attention mechanism module may include the output of the second multi-layer perceptron module; the input of the third attention mechanism module may include the output of the first multi-layer perceptron module; the input of the splicing module may include the output of the first attention mechanism module, the output of the second attention mechanism module, and the output of the third attention mechanism module; the input of the deep learning module may include the output of the splicing module; the input of the output module may include the output of the deep learning module; and the output of the output module may include the maneuvering position selected for the i-th operator.
[0098] As an exemplary embodiment, for the shared feature processing branch, the input of the first multi-layer perceptron module may include the overall statistical characteristics of the war game confrontation; the input of the second multi-layer perceptron module may include the attribute state characteristics of each operator of all participants; and the input of the second convolutional neural network module may include the overall spatial characteristics of the war game confrontation.
[0099] As an exemplary embodiment, the expert data generating device 100 may be configured to: obtain replay data generated by a user using a wargame confrontation environment; perform quality screening on the replay data; and generate an expert demonstration data set based on the screened replay data.
[0100] As an exemplary embodiment, the imitation learning training device 200 can be configured to: use a behavioral cloning algorithm to train a multi-task network to fit the expert demonstration data set, so that the data distribution of the maneuvering positions of each operator on our side output by the multi-task network is close to the data distribution of the maneuvering position labels in the expert demonstration data set.
[0101] As an exemplary embodiment, the reinforcement learning training device 300 may be configured to use a reinforcement learning algorithm to update the parameters of a preliminarily trained multi-task network by interacting with a war game confrontation environment.
[0102] As an exemplary embodiment, the reinforcement learning training device 300 can be configured as follows: in each round of iteration of the reinforcement learning training, the situation features required for the multi-task network decision-making obtained based on the current war game confrontation environment are input into the multi-task network, and in the war game confrontation environment, the actions of each of our operators are controlled according to the maneuvering directions selected by the multi-task network for each of our operators to update the war game confrontation environment; the parameters of the multi-task network are updated with the goal of accumulating and maximizing the rewards for actions in the war game confrontation environment.
[0103] Fig. 9 A structural block diagram of a war game agent training system according to another exemplary embodiment of the present disclosure is shown.
[0104] Reference Fig. 9 In the training system, the expert data generation device 100 obtains the replay data generated by the user using the wargame confrontation environment, and after quality screening and other operations on the replay data, the situation representation device 400 extracts situation features from the replay data to form samples to constitute the expert demonstration data set. The imitation learning training device 200 uses the behavior cloning algorithm to train the multi-task network 1 to fit the expert demonstration data set. The reinforcement learning training device 300 uses the network parameters of the multi-task network 1 trained by the imitation learning training device 200 as the initialization parameters of the multi-task network 1', and uses the reinforcement learning algorithm to update the multi-task network 1' through interaction with the wargame confrontation environment.
[0105] Specifically, the situation representation device 400 is used to extract situation features required for agent decision-making from the underlying attributes of the war game. In the imitation learning stage, the device is responsible for extracting situation features from the replay data in the process of building the expert demonstration data set; in the reinforcement learning stage, the device is responsible for extracting situation features from the current war game confrontation environment.
[0106] The expert demonstration data set is used for training in the imitation learning phase. The agent learns the selection of maneuvering positions in war game simulation by fitting the data distribution of the expert demonstration data set. The data in the data set is in the form of (situation features, maneuvering position labels), where the situation features are obtained by the situation representation device 400 from the war game simulation replay data, and the maneuvering position labels are generated according to pre-set rules.
[0107] The wargame confrontation environment is used to provide an interactive environment for the reinforcement learning training device 300. The intelligent agent continuously explores and tries in the wargame confrontation environment to improve the strategy level. In each iteration of the reinforcement learning training, the situation representation device 400 obtains the situation features from the wargame confrontation environment as the input of the reinforcement learning algorithm, and the predicted action output by the reinforcement learning algorithm is returned to the wargame confrontation environment for execution, thereby updating the environment state.
[0108] Multi-task network 1 is used to predict the maneuvering positions selected by the agent for each of our operators under the current situation characteristics in the imitation learning stage. The learning goal is to approximate the data distribution of the expert demonstration data set.
[0109] Multi-task network 1' is used to predict the maneuvering position selected by the agent for each of our operators under the current situation characteristics in the reinforcement learning stage. The structure is the same as that of multi-task network 1. The network parameters of multi-task network 1 obtained through training are used as initialization parameters. The learning goal is to maximize the cumulative reward sum of the reinforcement learning algorithm.
[0110] The imitation learning training device 200 uses the multi-task network 1 to fit the expert demonstration data set, and uses the behavior cloning algorithm to update the multi-task network 1 so that the output of the network approaches the data distribution of the expert demonstration data set, thereby obtaining a war game intelligent agent with a certain decision-making level.
[0111] The reinforcement learning training device 300 uses the reinforcement learning algorithm to perform online tuning on the intelligent agent to achieve optimization and improvement of the strategy. The device uses the situation characteristics obtained from the current state of the war game confrontation environment as the algorithm input, uses the multi-task network 1' as the policy network in the reinforcement learning algorithm and initializes it using the network parameters obtained in the imitation learning stage, and then uses the reinforcement learning algorithm to update and optimize the intelligent agent in the trial and error interaction with the environment.
[0112] Fig.10 The structural block diagram of the prediction system of the war game agent according to the exemplary embodiment of the present disclosure is shown. The war game agent is trained using the training method described in the exemplary embodiment above.
[0113] Reference Fig.10 According to an exemplary embodiment of the present disclosure, the prediction system of a war game agent includes: a situation representation device 500, a prediction device 600, and an action control device 700.
[0114] Specifically, the situation representation device 500 is configured to obtain situation features required for multi-task network decision-making in the war game agent based on the current war game confrontation environment.
[0115] The prediction device 600 is configured to input the acquired situation features into the multi-task network to obtain the maneuvering direction selected by the multi-task network for each of our operators.
[0116] The action control device 700 is configured to control the actions of each operator of our side according to the maneuvering direction selected for each operator of our side by the multi-task network in a war game confrontation environment.
[0117] It should be understood that the specific processing performed by the training system and prediction system of the war game agent according to the exemplary embodiment of the present disclosure has been referred to. Figures 1 to 7The details have been described in detail and will not be repeated here.
[0118] It should be understood that each device in the training system and prediction system of the war game agent according to the exemplary embodiments of the present disclosure may be implemented as a hardware component and / or a software component.
[0119] Fig.11 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.
[0120] Reference Fig.11 The electronic device includes: at least one memory 800 and at least one processor 900, wherein the at least one memory 800 stores a set of computer executable instructions. When the set of computer executable instructions is executed by the at least one processor 900, the training method and / or prediction method of the war game agent as described in the above exemplary embodiments is executed.
[0121] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
[0122] In an electronic device, the processor 900 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 900 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0123] The processor 900 may execute instructions or codes stored in the memory 800, wherein the memory 800 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.
[0124] The memory 800 may be integrated with the processor 900, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 800 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 800 and the processor 900 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 900 can read files stored in the memory.
[0125] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.
[0126] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the training method and / or prediction method of the war game agent as described in the above exemplary embodiments. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.
[0127] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by at least one processor to complete the training method and / or prediction method of the war game agent as described in the above exemplary embodiments.
[0128] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0129] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for training a war game agent, characterized in that: The training method comprises: Generate expert demonstration datasets based on wargame confrontation environments; Based on the expert demonstration data set, imitation learning training is performed on the multi-task network used for selecting maneuvering positions for each operator of our side in the war game agent to obtain a preliminarily trained multi-task network; Based on the war game confrontation environment, reinforcement learning training is performed on the preliminarily trained multi-task network to obtain an optimized multi-task network; The expert demonstration data set includes: the situation features required for the multi-task network decision-making, the maneuvering position labels of our N operators, where N is an integer greater than 1; The multi-task network includes: a shared feature processing branch and N operator branches, and the N operator branches correspond one-to-one to the N operators; The input of the shared feature processing branch includes: the overall situation characteristics of the wargame confrontation, and the output of the shared feature processing branch includes: shared features; The input of the i-th operator branch includes: the shared feature and the situation feature of the i-th operator itself, and the output of the i-th operator branch includes: the maneuvering position selected for the i-th operator, where i is an integer greater than 0 and less than or equal to N; The overall situation characteristics of the wargame confrontation include at least one of the following items: overall statistical characteristics of the wargame confrontation, attribute state characteristics of all operators of all participating parties, and overall spatial characteristics of the wargame confrontation; The step of performing imitation learning training on the multi-task network based on the expert demonstration data set includes: using a behavior cloning algorithm to train the multi-task network to fit the expert demonstration data set, so that the data distribution of the maneuvering positions of each operator of our side output by the multi-task network is close to the data distribution of the maneuvering position labels in the expert demonstration data set; Among them, based on the war game confrontation environment, the step of performing reinforcement learning training on the preliminarily trained multi-task network includes: using a reinforcement learning algorithm to update the parameters of the preliminarily trained multi-task network by interacting with the war game confrontation environment.
2. The training method according to claim 1, characterized in that: The i-th operator branch includes: a first convolutional neural network module, a first attention mechanism module, a second attention mechanism module, a third attention mechanism module, a splicing module, a deep learning module based on a self-attention mechanism, and an output module; the shared feature processing branch includes: a first multi-layer perceptron module, a second multi-layer perceptron module, and a second convolutional neural network module; Among them, the input of the first convolutional neural network module includes the position feature of the i-th operator itself, the input of the first attention mechanism module includes the output of the first convolutional neural network module and the output of the second convolutional neural network module, the input of the second attention mechanism module includes the output of the second multi-layer perceptron module, the input of the third attention mechanism module includes the output of the first multi-layer perceptron module, the input of the splicing module includes the output of the first attention mechanism module, the output of the second attention mechanism module and the output of the third attention mechanism module, the input of the deep learning module includes the output of the splicing module, the input of the output module includes the output of the deep learning module, and the output of the output module includes the maneuvering position selected for the i-th operator; Among them, the input of the first multi-layer perceptron module includes the overall statistical characteristics of the war game confrontation, the input of the second multi-layer perceptron module includes the attribute state characteristics of each operator of all participants, and the input of the second convolutional neural network module includes the overall spatial characteristics of the war game confrontation.
3. The training method according to claim 1, characterized in that: The steps to generate an expert demonstration dataset based on a wargame adversarial environment include: Obtaining replay data generated by the user using the wargame confrontation environment; Performing quality screening on the replay data; Based on the screened review data, the expert demonstration data set is generated.
4. The training method according to claim 1, characterized in that: The steps of using a reinforcement learning algorithm to update the parameters of the preliminarily trained multi-task network by interacting with the war game adversarial environment include: In each round of iteration of reinforcement learning training, based on the current war game confrontation environment, the situation features required for the decision-making of the multi-task network are obtained and input into the multi-task network, and the actions of each operator of our side are controlled according to the maneuvering orientation selected by the multi-task network for each operator of our side in the war game confrontation environment, so as to update the war game confrontation environment; The parameters of the multi-task network are updated with the goal of accumulating and maximizing the rewards of the war game confrontation environment for actions.
5. A prediction method for a war game agent, characterized in that: The prediction method comprises: Based on the current war game confrontation environment, obtaining situational features required for multi-task network decision-making in the war game agent; Inputting the acquired situation features into the multi-task network to obtain the maneuvering positions selected by the multi-task network for each operator of our side; Controlling the actions of each operator of our side according to the maneuvering position selected for each operator of our side by the multi-task network in the war game confrontation environment; Wherein, the war game agent is trained by executing the training method described in any one of claims 1 to 4.
6. A war game agent training system, characterized in that: The training system comprises: An expert data generating device is configured to generate an expert demonstration data set based on a war game confrontation environment; An imitation learning training device is configured to perform imitation learning training on the multi-task network used for selecting maneuvering positions for each operator of our side in the war game agent based on the expert demonstration data set, so as to obtain a preliminarily trained multi-task network; A reinforcement learning training device is configured to perform reinforcement learning training on the preliminarily trained multi-task network based on the war game confrontation environment to obtain an optimized multi-task network; The expert demonstration data set includes: the situation features required for the multi-task network decision-making, the maneuvering position labels of our N operators, where N is an integer greater than 1, The multi-task network includes: a shared feature processing branch and N operator branches, and the N operator branches correspond one-to-one to the N operators; The input of the shared feature processing branch includes: the overall situation characteristics of the wargame confrontation, and the output of the shared feature processing branch includes: shared features; The input of the i-th operator branch includes: the shared feature and the situation feature of the i-th operator itself, and the output of the i-th operator branch includes: the maneuvering position selected for the i-th operator, where i is an integer greater than 0 and less than or equal to N; The overall situation characteristics of the wargame confrontation include at least one of the following items: overall statistical characteristics of the wargame confrontation, attribute state characteristics of all operators of all participating parties, and overall spatial characteristics of the wargame confrontation; The imitation learning training device is configured to use a behavior cloning algorithm to train the multi-task network to fit the expert demonstration data set, so that the data distribution of the maneuvering positions of each operator of our side output by the multi-task network is close to the data distribution of the maneuvering position labels in the expert demonstration data set; Among them, the reinforcement learning training device is configured to use a reinforcement learning algorithm to update the parameters of the preliminarily trained multi-task network by interacting with the war game confrontation environment.
7. A prediction system for a war game agent, characterized in that: The prediction system comprises: A situation representation device is configured to obtain situation features required for multi-task network decision-making in the war game agent based on the current war game confrontation environment; A prediction device is configured to input the acquired situation features into the multi-task network to obtain the maneuvering position selected by the multi-task network for each operator of our side; An action control device is configured to control the actions of each operator of our side according to the maneuvering position selected for each operator of our side by the multi-task network in the war game confrontation environment; Wherein, the war game agent is trained by executing the training method described in any one of claims 1 to 4.
8. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by a processor of an electronic device, the electronic device is enabled to execute the training method according to any one of claims 1 to 4 and / or the prediction method according to claim 5.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the training method according to any one of claims 1 to 4 and / or the prediction method according to claim 5.
10. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by at least one processor, the training method according to any one of claims 1 to 4 and / or the prediction method according to claim 5 are implemented.
Citation Information
Patent Citations
Intelligent agent task allocation training method based on imitation learning and safety reinforcement learning
CN117291254A
