Strategic scene scheduling navigation method based on hierarchical reinforcement learning and tactical analysis

Abstract

This invention provides a strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis, belonging to the field of multi-agent path navigation and planning technology. By utilizing related technologies in hierarchical reinforcement learning, tactical analysis, and multi-agent path navigation and planning, this method focuses on the effectiveness and efficiency of battlefield corps scheduling in complex hierarchical motion spaces. Facing more complex environments and more dynamic scenarios, it proposes a superior scheduling and navigation method. This method adjusts the upper-level strategy by keeping the lower-level strategy unchanged. The upper-level strategy trained based on tactical analysis methods has significant advantages compared to upper-level strategies using random target selection methods, which is of great significance for the construction of digital and automated armies.

Description

Technical Field

[0001] This invention relates to the field of multi-agent path navigation planning technology, and in particular to a strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis. Background Technology

[0002] In strategic simulation environments, the strategic behavior of multi-agent agents is highly complex, operating at the legion level, and primarily involves strategies such as pursuit, encirclement, ambush, and breakout. During legion deployment, the choice of different strategies for each agent or group of agents has a vast scope, and dynamic adjustments are required based on environmental factors. Multi-agent actions in legion deployment environments are required to simulate the specific situations of deploying two, three, or even multiple armies in real-world scenarios, ultimately achieving a high standard of anthropomorphism. Currently, the construction of anthropomorphic models for complex multi-agent strategies is still in a state of immaturity and imperfection, with many shortcomings remaining in the technological field.

[0003] The commonly used algorithm for multi-agent corps-style scheduling and navigation is reinforcement learning, which uses random agents and LSTM for policy space initialization and linking. This algorithmic framework mostly operates in simplified, abstract scenarios, achieving, on average, the performance of a novice strategist. It can simply combine strategic and micro-level individual policies and perform a certain degree of intensity classification, exhibiting clear objectives. However, it has a long learning time, poor learning rate, poor adaptability to different environments, and poor model reusability.

[0004] Another more advanced approach is to use behavior trees for dynamic reorganization of strategic tasks. Faced with complex battlefield environments, to enable multi-agent systems to have task planning and reorganization capabilities, thereby influencing strategic decision-making, behavior trees are constructed by decomposing strategic tasks. This dynamically balances the complexity and priority of different task sequences and combinations, solving the learning problems related to task arrangement during AI learning. This algorithm simplifies the game theory difficulty in the strategic scheduling process and achieves good computational results with limited computing resources. However, its computational performance remains poor for high real-time problems. Furthermore, the lightweight approach struggles to achieve Nash equilibrium in unstable strategic environments. While its adaptability is somewhat enhanced, maintainability is poor, and experience utilization is low.

[0005] In general, most current algorithmic models for strategic deployment are only effective in specific wartime environments. They fail to respond well to complex battlefield environments and the demands of rapid, real-time strategy implementation. Furthermore, in an effort to simplify the deployment problem, most algorithmic models suffer from poor reusability, maintainability, and portability. Consequently, they do not achieve practical results in more realistic battlefield environments and are unlikely to be considered successful engineering achievements. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis, so as to realize efficient combat target selection and macro-group control of the legion.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis includes the following steps:

[0009] Step 1: Construct a Legion environment model;

[0010] Mini Warcraft, a virtual combat environment built using the Unity3D engine;

[0011] Step 2: Construct a hierarchical model of formation agent policies:

[0012] The agent policy hierarchical model includes an upper-level policy model and a lower-level policy model. Each policy model includes a behavior action model, a state influence model, and a decision model.

[0013] Based on the different core tasks of the agents, the strategy in the Mini Warcraft environment is divided into two layers: upper strategy and lower strategy. In the upper strategy model, tactical attack targets are assigned to the formation, while in the lower strategy model, the movement of agents under the entire formation is controlled by the unit.

[0014] Step 3: Construct the reward function for legion dispatch navigation;

[0015] By combining influence maps and prior knowledge, a complete reward function is designed to assist the agent in learning to avoid dangerous military locations and learn advanced strategies.

[0016] Step 4: Select a training algorithm;

[0017] Both the upper-level and lower-level policies are trained using the PPO algorithm;

[0018] Step 5: Design the model architecture;

[0019] The network architectures of both the lower-level policy model and the upper-level policy model are based on the standard Actor-Critic network architecture design; a state value network is used to estimate the state value to determine the quality of the current policy network's learning; a policy network is used to approximate the agent's policy function, and the policy network parameters are updated based on the state value network's estimate;

[0020] Step 6: Train the agent models corresponding to the upper-layer and lower-layer policies based on the model architecture constructed in Step 5;

[0021] Upper-level and lower-level strategies cannot be trained simultaneously; they should be trained one by one in a bottom-up order.

[0022] The beneficial effects of adopting the above technical solution are as follows: The strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis provided by this invention proposes a hierarchical strategy. It adjusts the upper-level strategy by keeping the lower-level strategy unchanged, and compares the upper-level strategy with two methods: proximity target selection and random target selection. The upper-level strategy trained based on the tactical analysis method has significant advantages compared to the upper-level strategy using the random target selection method. Furthermore, the upper-level strategy trained based on the tactical analysis method can select targets for the formation in real time according to the battlefield situation, resulting in more diverse strategic styles and providing human players with a richer combat experience. Attached Figure Description

[0023] Figure 1 A schematic diagram of the strategy layering model provided in the embodiments of the present invention;

[0024] Figure 2 These are specific environmental scene renderings provided for embodiments of the present invention;

[0025] Figure 3 This is a schematic diagram illustrating the selection of tactical offensive targets using a higher-level strategy, provided in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the lower-level strategy formation's eight-directional movement provided in an embodiment of the present invention;

[0027] Figure 5 An example diagram of the impact map provided in the embodiments of the present invention;

[0028] Figure 6 This is a network architecture diagram of upper-layer and lower-layer policies provided in an embodiment of the present invention. Detailed Implementation

[0029] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0030] The strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis provided in this embodiment is as follows.

[0031] Step 1: Construct a Legion environment model;

[0032] Virtual environments serve as testing grounds for deep reinforcement learning algorithms. To verify the effectiveness of relational multi-agent deep reinforcement learning methods based on mean-field theory, this embodiment constructs a virtual combat environment, MiniWarcraft, using the Unity3D engine. The specific method is as follows:

[0033] Step 1.1: Construct a coordinate unit model;

[0034] Mini Warcraft uses a flat terrain environment with the center of the scene at coordinate zero (0, 0, 0). A standard unit is defined as the area occupied by one soldier unit, and the entire environment is 100 x 100 standard units in size. Soldiers move at a speed of 5 units per second.

[0035] Step 1.2: Construct a battlefield model;

[0036] The Mini Warcraft environment contains two armies, with isomorphic agents distinguished only by color. The armies are grouped into four squads, each containing 72 soldier units arranged in 6 rows and 12 columns. Squads and individual soldiers move at the same speed of 5 units per second. However, when squads are grouped together, the movement speed of the soldiers increases to 8 units per second.

[0037] Step 2: Construct a hierarchical model of formation agent strategies;

[0038] The hierarchical policy model for intelligent agents consists of two layers: an upper-layer policy model and a lower-layer policy model. Each layer comprises three parts: a behavior action model, a state influence model, and a decision model. The specific construction method is as follows:

[0039] Step 2.1: Construct a two-layer strategy model;

[0040] Based on the different core tasks of the agents, the strategy in the Mini Warcraft environment is divided into two layers: an upper-layer strategy and a lower-layer strategy. The upper-layer strategy model assigns tactical attack targets to the formation, while the lower-layer strategy model controls the movement of agents within the entire formation.

[0041] Step 2.2: Construct a behavioral action model;

[0042] Step 2.2.1: Construct the behavioral action model of the upper-level strategy model;

[0043] The upper-level strategy model is designated as the tactical target strategy layer. Assuming the enemy initially has N formations, the upper-level strategy has N+1 selectable targets, including at least one empty target. After selecting an empty target, the formations will regroup on the spot. In a large-scale battle scenario in MiniWarcraft, each side's army initially has 4 formations, meaning the upper-level strategy has 5 selectable actions. During training, the formations of both sides are labeled and distinguished, and the actions of the upper-level strategy are converted into one-dimensional one-hot vectors, which are then input into the neural network as key feature information for the middle-level strategy.

[0044] Step 2.2.2: Construct the behavioral action model of the lower-level strategy model;

[0045] The lower-level strategy model is designated as the formation movement layer. Formation movement is discretized in 8 directions, and stationary actions are added, resulting in a total of 9 actions. When the formation moves, the overall orientation of the formation will remain consistent with the direction of movement. That is, when the direction of movement changes, the entire formation will adjust its orientation around the formation's central axis.

[0046] Step 2.3: Construct a state influence model;

[0047] In the hierarchical reinforcement learning strategy training method based on tactical analysis, the influence map constructed by the tactical analysis method is crucial state information. This step will elaborate on the process of constructing the influence map and the construction of the state influence model, including other key state features, as follows:

[0048] Step 2.3.1: Construct an impact map;

[0049] A 100x100 grid is constructed to cover the entire map for rendering. Each soldier unit is assigned an influence of size 1. The influence of military units decreases with increasing distance; this embodiment uses the most common linear decrease equation. To calculate the decline in military influence. Among them, I d I0 is the influence of a military unit at a given distance d; I0 is the influence of a military unit at a distance of 0, which is equivalent to the inherent military strength of a military unit.

[0050] Step 2.3.2: Optimize the impact map;

[0051] Dividing a game level into m tiles and assuming there are n game objects, the time complexity for calculating influence graphs is O(nm), and the space complexity is O(m). However, current real-time strategy games have a large number of game units and tiles, making conventional influence graph calculations very expensive. Therefore, it is necessary to optimize the influence graph calculation process, as follows:

[0052] Step 2.3.2.1: Finite radius determination;

[0053] Set a maximum influence radius for each unit, and let k be the number of blocks within the maximum influence radius of a unit, where k is much smaller than m.

[0054] Step 2.3.2.2: Apply a convolution filter;

[0055] After digitizing the unit influence, it is displayed in the influence map. A Gaussian filter (convolution kernel) is used to blur the initial influence value and form a gradual spread, diffusing the blurred value around the hierarchy.

[0056] Step 2.3.2.3: Perform map overlay;

[0057] Let the influence of each position be equal to the maximum influence contributed by any unit in that position. If a game object A with high influence and several game objects B with lower influence are located in the same area, then the influence of that area is only affected by game object A.

[0058] Step 2.3.3: Introduce other features;

[0059] The influence map is a key feature of both upper-level and lower-level policies. In addition, upper-level and middle-level policies require other features to assist in learning, as follows:

[0060] Step 2.3.3.1: Introduce other features to the upper-level policy model:

[0061] It incorporates status information such as the formation positions of both sides' armies in the training environment, formation direction of movement, formation size, and the attack targets of friendly formations.

[0062] Step 2.3.3.2: Introduce other features to the lower-level policy model:

[0063] The output vector of the upper-level strategy, i.e. the attack target of the current formation, will be added to the upper-level strategy features as an auxiliary feature.

[0064] Step 2.4: Construct a decision model;

[0065] The Mini Warcraft environment has a maximum running step size, meaning it will reset after the environment runs for more than 40,000 steps. The agents corresponding to the upper and lower policy models can take one action every 5 game steps, meaning a single agent can take a maximum of 8,000 effective actions during a single training session.

[0066] Step 3: Construct the reward function for legion dispatch navigation;

[0067] The hierarchical reinforcement learning strategy training method based on tactical analysis relies heavily on the influence map in tactical analysis to design a reward function that allows the formation to learn efficient and reasonable attack routes. The influence map comprehensively represents the size of military forces and the proximity of military units at various locations on the map. Therefore, a complete reward function can be designed by combining the influence map and prior knowledge to assist the agent in learning to avoid dangerous military positions and to learn advanced strategies such as piecemeal attacks or encirclement. The specific method is as follows:

[0068] Step 3.1, affecting map normalization;

[0069] The influence value in each grid is scaled by dividing it by the largest influence value in the entire influence map. In the scaled influence map, influence values ​​range from 0 to 1.

[0070] Step 3.2: Construction of the reward function;

[0071] In the large-scale battle scenarios of Mini Warcraft, the map can be divided into two parts according to the faction. Considering that the tasks of the upper and lower layer strategy models are different, the methods for constructing their reward functions are also different, as detailed below:

[0072] Step 3.2.1: Constructing the reward function for the upper-level policy model;

[0073] An auxiliary reward is constructed based on the difference between the influence value of the current formation center position and the influence value of the center position of the selected attack target. When the difference is positive, the agent corresponding to the upper-level strategy receives a positive reward; otherwise, it receives a negative reward. That is, the upper-level strategy should favor selecting relatively weaker targets when choosing attack targets.

[0074] Step 3.2.2: Constructing the reward function for the lower-level policy model;

[0075] Its reward function is constructed similarly to the upper-level strategy, primarily utilizing the difference between the current formation's center position's influence value and the enemy's maximum influence value in the grid near the movement path to build an auxiliary reward. When this difference is positive, the agent corresponding to the lower-level strategy receives a positive reward; otherwise, it receives a negative reward. In other words, the lower-level strategy should avoid areas with strong enemy forces when planning movement routes to prevent surprise attacks. In Mini Warcraft's large-scale battle scenarios, the maximum enemy influence value within a radius 1.5 times the formation's influence radius is calculated, and this value is used as the enemy's maximum influence value in the grid near the formation's movement path.

[0076] Step 4: Select a training algorithm;

[0077] As can be seen from the policy hierarchy, an overall policy comprises several upper-level and lower-level policy models, with the number of upper-level and lower-level policy models matching the number of formations at the start of training. Since the custom training environment features flat terrain and collision-free formations, to accelerate policy training and quickly validate the policy hierarchy framework, this embodiment uses the PPO algorithm for training both upper-level and lower-level policies. The policy gradient loss function of the PPO algorithm in this embodiment is...

[0078]

[0079] Where, p θ (a t |s t ) is in policy θ, state s t Next action a t The probability distribution of A; θ′ (a t |s t ) is in policy θ′, state s t The action advantage estimation is as follows; ∈ is a hyperparameter, which is generally taken as a very small value, such as 0.1. When A θ′ (a t |s t When the value is greater than 0, the agent will take action a towards increasing its size. t The network parameters are updated in the direction of the probability, i.e., p is increased. θ (a t |s t However, the increase cannot exceed 1 + ∈ A. θ′ (a t |s t When the value is less than 0, the agent will take action a towards reducing the value. t The network parameters are updated in the direction of the probability, i.e., p is decreased. θ (a t |s t However, the reduction cannot be less than 1 - ∈. This ensures that the difference between policy θ and policy θ′ is not too large, thus making the agent's behavior policy update process more stable.

[0080] Step 5: Design the model architecture;

[0081] The hierarchical policy model reveals that lower-level policy models are directly influenced by upper-level policy models; that is, the output of the upper-level policy model serves as the input to the lower-level policy model. Therefore, both can be designed together when designing the model architecture. Since both the upper-level and lower-level policy models employ the PPO algorithm, their network architectures are based on the standard Actor-Critic network architecture. A state value network is used to estimate the state value, determining the quality of the current policy network's learning. A policy network approximates the agent's policy function, and the policy network parameters are updated based on the state value network's estimate.

[0082] Step 6: Train the agent models corresponding to the upper-layer and lower-layer policies based on the model architecture constructed in Step 5;

[0083] The strategic scenario corps dispatching and navigation method based on hierarchical reinforcement learning and tactical analysis must first clarify one point when implementing it: upper-level and lower-level strategies cannot be trained simultaneously. They should be trained one by one in a bottom-up training order to prevent environmental instability caused by perturbations between strategies. The specific method is as follows:

[0084] Step 6.1, Agent training for the lower-level policy model:

[0085] The task objective of the lower-level strategy is to control the formation to reach the attack target location, and it is trained using the PPO algorithm.

[0086] Step 6.1.1, Training process of the lower-level policy model:

[0087] Since the upper-level strategy is not yet trained when the lower-level strategy is trained, it is impossible to assign attack targets to the lower-level strategy in real time based on the situation. Therefore, each time the lower-level strategy training begins, an attack target must be manually assigned to the lower-level strategy, and the mission ends when the formation reaches the attack target. To increase the generalization ability of the lower-level strategy, the positions of both formations on the map are randomly initialized before training begins, and an attack target is randomly assigned to each formation.

[0088] Step 6.1.2, Training and optimization of the lower-level policy model:

[0089] To further reduce the learning difficulty of lower-level strategies, this embodiment employs a course-based learning method to gradually increase the strength of the lower-level strategies. In the initial training phase, the positions of one group of units are fixed; that is, the units in fixed positions will not move during training. As the strength of the lower-level strategies increases, the units in fixed positions are released, and their movements are controlled using a random strategy. This gradually increasing training difficulty enhances the strength and robustness of the lower-level strategies.

[0090] Step 6.2, Agent training for the upper-level policy model:

[0091] The upper-level strategy's objective is to schedule formations, assigning attack targets to them in the hope of achieving victory, and it is also trained using the PPO algorithm.

[0092] Step 6.2.1, Training process of the upper-level policy model:

[0093] While training the upper-level strategy, the lower-level strategies have already been trained. At this point, a unified framework can be used to train the upper-level strategy. During upper-level strategy training, a naive self-play method is used to control the battle between the two sides to improve the strategy. The battle flow during training is as follows: the upper-level strategy assigns attack targets to the lower-level strategy in real time based on the battle situation; the trained lower-level strategy then controls the formation to move towards the attack target.

[0094] This embodiment aims to analyze the core task objective and how to decompose it without violating it before using hierarchical reinforcement learning to classify the strategy for the task objective. For situations with a large number of agents in a game-theoretic scenario, a general strategy layering scheme is proposed. First, the agents are grouped into formations, and a formation movement strategy is designed to control the movement of agents within each formation. Then, a top-level tactical objective strategy is designed to assign tactical attack objectives to the formations. The overall strategy layering is as follows: Figure 1 As shown.

[0095] The purpose of this embodiment is to construct a legion simulation virtual scene. Figure 2 The entire environment can be clearly seen from there. According to... Figure 1 The strategy hierarchy method in Mini Warcraft groups and organizes armies in large-scale battle scenarios. Each army is divided into 4 squads, with each squad containing 72 soldier units, arranged in a 6-row, 12-column distribution. Similar to small-scale battle scenarios, strategies within the same hierarchy are isomorphic across different armies. A maximum game step size of 40,000 steps is also set in this scenario; exceeding this maximum step size will reset the environment to prevent agents from falling into strategy traps during training.

[0096] The purpose of this embodiment is to construct an intelligent agent model for the upper-level and lower-level strategies of a legion. For example... Figure 3 As shown, in a large-scale battle scenario in MiniWarcraft, each side's army initially contains 4 formations, meaning the upper-level strategy has 5 selectable actions. During the experiment, the formations of both sides are labeled and distinguished, and the actions of the upper-level strategy are converted into one-dimensional one-hot vectors, which are then input into the neural network as key feature information for the lower-level strategy. For example... Figure 4As shown, the formation movement of the lower-level strategy is discretized in 8 directions. When the formation moves, the overall orientation of the formation will be consistent with the direction of movement. That is, when the direction of movement changes, the entire formation will adjust its orientation around the formation's central axis.

[0097] In the large-scale battle scenarios of Mini Warcraft, the map can be divided into two parts according to the faction. For the upper-level strategy, its task is to select attack targets for the formation. When constructing the reward function, the auxiliary reward is mainly based on the difference between the influence value of the current formation's center position and the influence value of the center position of the selected attack target. When the difference is positive, the agent corresponding to the upper-level strategy receives a positive reward; otherwise, it receives a negative reward. That is, the upper-level strategy should favor selecting relatively weaker targets when choosing attack targets. The specific reward function settings are shown in Table 1.

[0098] Table 1 Main reward functions of the upper-level strategy

[0099] Reward Conditions Reward Value Select empty target -0.01 Select enemy formation (Current formation center influence - Selected enemy formation center influence) Formation battle victory +10 Formation battle failed -10

[0100] For the lower-level strategy, its task is to control the formation to move towards the attack target along a safe and efficient path. Its reward function is constructed similarly to the upper-level strategy, primarily using the difference between the current formation's center position's influence value and the enemy's maximum influence value in the grid near the movement path to build an auxiliary reward. When the difference is positive, the agent corresponding to the lower-level strategy receives a positive reward; otherwise, it receives a negative reward. That is, the lower-level strategy should avoid passing through areas with strong enemy forces when planning its movement route to prevent being ambushed. In the large-scale battle scenario of Mini Warcraft, the enemy's maximum influence value within a radius 1.5 times the formation's influence radius is calculated and used as the enemy's maximum influence value in the grid near the formation's movement path. The specific reward function settings are shown in Table 2.

[0101] Table 2 Main reward functions of the lower-level strategy

[0102] Reward Conditions Reward Value Enemy military unit proximity Min[(Current group center influence - maximum influence of nearby enemies), 0] Proximity of friendly military units (Current formation center influence - average enemy formation center influence) Distance from target formation Min[-(distance from target formation - 10) * 0.001, 0]

[0103] The purpose of this embodiment is to construct an influence map in tactical analysis methods to assist agents in decision-making and accelerate the convergence speed and learning effect of agent strategies. Before constructing the influence map, military units must be assigned influence. In MiniWarcraft, each soldier unit is assigned an influence of size 1. The influence of military units decreases with increasing distance. This embodiment uses the most commonly used linear descent equation to calculate military influence. To reduce the computational cost of the influence map, a finite influence radius method is used to limit the range of military unit influence. In large-scale battle scenarios in Mini Warcraft, to simplify the calculation of the influence map, the influence of soldiers in a formation is accumulated, and the influence map is calculated uniformly on a formation basis. The formation influence radius is dynamically set according to the number of soldiers in the formation. A full formation has a maximum of 72 soldier units, at which point its influence radius is 40. Afterward, the influence radius decreases by 5 for every 10 soldiers lost.

[0104] To illustrate the process of constructing the influence map in detail, a 20×20 area is proposed for its construction. Nine soldier units with an influence level of 1 and an influence radius of 3 are placed on this map. Visualizing the influence map will then yield the following results: Figure 5 The influence map shown. Darker colors represent greater influence and stronger military power in that region. The influence map is then edited... Figure 5 The map is divided into two parts by a diagonal line. In the upper left part, the four soldiers are evenly positioned with no weak points. In the lower right part, one of the five soldiers is scattered and could be a target for attack.

[0105] This embodiment mainly introduces the model architecture of a strategic scenario corps dispatch and navigation method based on hierarchical reinforcement learning and tactical analysis. Figure 1 It is known that the lower-level policy is directly influenced by the upper-level policy; that is, the output of the upper-level policy serves as the input of the lower-level policy. Therefore, the two can be designed together when designing the model architecture. Since both the upper-level and lower-level policies use the PPO algorithm, their network architectures are based on the standard Actor-Critic network architecture. A state value network is used to estimate the state value to determine the quality of the current policy network's learning. A policy network is used to approximate the agent's policy function, and the policy network parameters are updated based on the state value network's estimate. The upper-level and lower-level policy network architectures are as follows: Figure 6 As shown.

[0106] To verify the effectiveness of the proposed method, comparative experiments were conducted on both the upper-level and lower-level policies, comparing them with manual methods based on prior knowledge. When validating the upper-level policy, the lower-level policy was kept constant, and two methods—proximity target selection and random target selection—were compared with the upper-level policy. When validating the lower-level policy, the upper-level policy was kept constant, and formation movement was controlled using the shortest-distance straight-line movement method, compared with the lower-level policy.

[0107] The advantages and disadvantages of different strategies are comprehensively analyzed by comparing win rates. Through rigorous comparative experiments, the win rates of different strategies after 100 battles were statistically analyzed. For ease of representation, FHRL (Fully Hierarchical Reinforcement Learning) is used to represent the hierarchical reinforcement learning strategy training method based on tactical analysis. RUP (Random Upper-level Policy) is used to represent the hierarchical reinforcement learning method with a random upper-level policy. PUP (Proximity Upper-level Policy) is used to represent the hierarchical reinforcement learning method with a proximity upper-level policy. PLP (Proximity Lower-level Policy) is used to represent the hierarchical reinforcement learning method with a proximity lower-level policy. The win rate data between different strategies and methods are shown in Table 3.

[0108] Table 3 Win rate data for different strategies

[0109]

[0110]

[0111] Table 3 shows that the upper-level strategy trained based on tactical analysis methods has a significant advantage over the upper-level strategy using random target selection, demonstrating the effectiveness of deep reinforcement learning. However, the advantage of the upper-level strategy trained based on tactical analysis methods is significantly reduced compared to the upper-level strategy using proximity target selection. This is because the proximity target selection strategy, based on human prior knowledge, has considerable strength, but suffers from a lack of tactical variety, making it easily detectable and countered by human players. In contrast, the upper-level strategy trained based on tactical analysis methods can select targets for the formation in real-time based on the battlefield situation, offering more diverse strategic styles and a richer combat experience for human players. The lower-level strategy trained based on tactical analysis methods has a relatively significant advantage compared to the lower-level strategy using the shortest distance straight-line movement method. This is because the agent strategy using the shortest distance straight-line movement method is too simplistic and more prone to being surrounded by the enemy. If the battlefield environment has more complex terrain, such as slopes, obstacles, or lakes, the advantage of the lower-level strategy trained based on tactical analysis methods will be even more pronounced.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis, characterized in that: Includes the following steps: Step 1: Construct a Legion environment model; Mini Warcraft, a virtual combat environment built using the Unity3D engine, is as follows: Step 1.1: Construct a coordinate unit model; Mini Warcraft uses a flat terrain environment with the center of the scene at coordinate zero (0, 0, 0). The area occupied by one soldier unit is 1 standard unit, and the size of the entire environment is 100×100 standard units. The soldier moves at a speed of 5 units per second. Step 1.2: Construct a battlefield model; The Mini Warcraft environment includes two armies, whose agents are isomorphic and distinguished only by color. The armies are grouped into four squads, each containing 72 soldier units arranged in 6 rows and 12 columns. Squads and individual soldiers move at the same speed of 5 units per second. However, when squads are grouped together, the movement speed of individual soldiers increases to 8 units per second. Step 2: Construct a hierarchical model of formation agent policies: The agent policy hierarchical model includes an upper-level policy model and a lower-level policy model. Each policy model includes a behavior action model, a state influence model, and a decision model. Based on the different core tasks of the agents, the strategy in the Mini Warcraft environment is divided into two layers: upper strategy and lower strategy. In the upper strategy model, tactical attack targets are assigned to the formation, while in the lower strategy model, the movement of agents under the entire formation is controlled by the unit. Step 3: Construct the reward function for legion dispatch navigation; By combining influence maps and prior knowledge, a complete reward function is designed to assist the agent in learning to avoid dangerous military locations and learn advanced strategies. Step 4: Select a training algorithm; Both the upper-level and lower-level policies are trained using the PPO algorithm; Step 5: Design the model architecture; Both the lower-level policy model and the upper-level policy model are based on the standard Actor-Critic network architecture design; a state value network is used to estimate the state value to determine how well the current policy network has learned. The agent's policy function is approximated using a policy network, and the policy network parameters are updated based on the estimate from the state value network. Step 6: Train the agent models corresponding to the upper-layer and lower-layer policies based on the model architecture constructed in Step 5; Upper-level and lower-level policies cannot be trained simultaneously; they should be trained one after the other in a bottom-up order. The specific method is as follows: Step 6.1: Agent training for the lower-level policy model; The task objective of the lower-level strategy is to control the formation to reach the attack target location. It is trained using the PPO algorithm, as detailed below: Step 6.1.1, Training process of the lower-level policy model: Since the upper-level strategy has not been trained when the lower-level strategy is trained, it is impossible to assign attack targets to the lower-level strategy in real time according to the situation. Therefore, each time the lower-level strategy training starts, the attack target must be manually assigned to the lower-level strategy. The mission ends when the formation reaches the attack target. In order to increase the generalization ability of the lower-level strategy, the positions of both formations on the map are randomly initialized before the training starts, and an attack target is randomly assigned to each formation. Step 6.1.2, Training and optimization of the lower-level policy model: To further reduce the learning difficulty of lower-level strategies, a course-based learning method is adopted to gradually increase the strength of the lower-level strategies. In the early stage of training, the position of one side's formation will be fixed, that is, the formation with the fixed position will not move during training. As the strength of the lower-level strategy increases, the formation with the fixed position will be released and its movement will be controlled by a random strategy. Step 6.2, Agent training for the upper-level policy model: The upper-level strategy's objective is to schedule formations, assigning attack targets to them in hopes of achieving victory. It is also trained using the PPO algorithm, as detailed below: Step 6.2.1, Training process of the upper-level policy model: When training the upper-level strategy, the lower-level strategies have already been trained. At this point, a unified framework is used to train the upper-level strategy. When training the upper-level strategy, a naive self-play method is used to control the battle between the two sides to improve the strategy. The battle flow during training is that the upper-level strategy allocates attack targets to the lower-level strategy in real time according to the battle situation, and the trained lower-level strategy controls the formation to move towards the attack target. 2.The strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis according to claim 1, characterized in that: In step 2, the method for constructing the behavior action model of the upper-level strategy model is as follows: The upper-level strategy model is the tactical objective strategy layer. Assume the enemy initially has... If the formation is divided into groups, then the optional objectives of the upper-level strategy are: There are 1,000 formations, including one empty target. After selecting an empty target, the formation will reorganize and regroup on the spot. In the large-scale battle scenario of MiniWarcraft, each side's army initially has 4 formations, meaning that the upper-level strategy has 5 selectable actions. During training, the formations of both sides will be labeled and distinguished, and the actions of the upper-level strategy will be converted into one-dimensional one-hot vectors, which will be used as key feature information of the middle-level strategy and input into the neural network.

3. The strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis according to claim 2, characterized in that: In step 2, the method for constructing the behavior action model of the lower-level strategy model is as follows: The lower-level strategy model is designated as the formation movement layer. Formation movement is discretized in 8 directions, and stationary actions are added, resulting in a total of 9 actions. When the formation moves, the overall orientation of the formation will be consistent with the direction of movement. That is, when the direction of movement changes, the entire formation will adjust its orientation around the formation's central axis.

4. The strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis according to claim 3, characterized in that: In step 2, the specific method for constructing the state influence model is as follows: Step 2.3.1: Construct an impact map; Construct a 100x100 grid to cover the entire map for rendering; assign each soldier unit an influence of size 1; the influence of military units decreases with increasing distance, using a linear decrease equation. To calculate the decline in military influence; among them, It is a military unit given a distance The impact; It represents the influence of a military unit when the distance is 0, which is equivalent to the inherent military strength of the military unit; Step 2.3.2: Optimize the impact map; Divide a game level into There are 4 tiles, assuming there are 10 tiles. If there are 1 game object, then the time complexity for calculating influence is O(n). The space complexity is Currently, real-time strategy games have a large number of game units and tiles, making conventional influence graph calculations very expensive. Therefore, it is necessary to optimize the influence graph calculation process, as follows: Step 2.3.2.1: Finite radius determination; Set a maximum influence radius for each unit, and set the number of tiles within the maximum influence radius to be [value missing]. , much smaller ; Step 2.3.2.2: Apply a convolution filter; After digitizing the unit influence, it is displayed in the influence map. A Gaussian filter, i.e. a convolution kernel, is used to blur the initial influence value and form a gradual extension, spreading the blurred value around the hierarchy. Step 2.3.2.3: Perform map overlay; Let the influence of each position be equal to the maximum influence contributed by any unit in that position; if a game object A with a large influence and several game objects B with smaller influence are in the same area, then the influence of that area is only affected by game object A. Step 2.3.3: Introduce other features; The influence map is a key feature of both upper-level and lower-level policies. In addition, upper-level and middle-level policies require other features to assist in learning, as follows: Step 2.3.3.1: Introduce other features to the upper-level strategy model; The system incorporates information on the formation positions of both sides' forces in the training environment, their direction of movement, formation size, and the attack target status of friendly formations. Step 2.3.3.2: Introduce other features to the lower-level policy model: The output vector of the upper-level strategy, i.e. the attack target of the current formation, is added to the upper-level strategy features as an auxiliary feature.

5. The strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis according to claim 4, characterized in that: In step 2, the specific method for constructing the decision model is as follows: The Mini Warcraft environment has a maximum running step size, meaning that the environment will be reset after the running time exceeds the maximum game step size, which is 40,000 steps. The agents corresponding to the upper-level policy model and the lower-level policy model take one action every 5 game steps, meaning that an agent can take a maximum of 8,000 effective actions in one training session.

6. The strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis according to claim 5, characterized in that: The specific method for step 3 is as follows: Step 3.1, affecting map normalization; The influence value in each grid is scaled by dividing it by the largest influence value in the entire influence map; in the scaled influence map, the influence values ​​are between 0 and 1. Step 3.2: Construction of the reward function; In the large-scale battle scenarios of Mini Warcraft, the map is divided into two parts according to the faction, and considering that the tasks of the upper and lower strategy models are different, the methods for constructing their reward functions are also different, as detailed below: Step 3.2.1: Constructing the reward function for the upper-level policy model; The auxiliary reward for the upper-level strategy model is constructed based on the difference between the influence value of the current formation center position and the influence value of the center position of the selected attack target. When the difference in the auxiliary reward for constructing the upper-level strategy model is positive, the agent corresponding to the upper-level strategy receives a positive reward; otherwise, it receives a negative reward. That is, the upper-level strategy should favor selecting relatively weaker targets when choosing attack targets. Step 3.2.2: Constructing the reward function for the lower-level policy model: The auxiliary reward for the lower-level strategy model is constructed by using the difference between the influence value of the current formation center position and the maximum influence value of the enemy in the grid near the movement path. When the difference in the auxiliary reward for constructing the lower-level strategy model is positive, the agent corresponding to the lower-level strategy receives a positive reward; otherwise, it receives a negative reward. That is, when planning the movement route, the lower-level strategy should avoid passing through areas where the enemy is strong to prevent being ambushed. In the large-scale battle scenario of Mini Warcraft, the maximum influence value of the enemy within a radius of 1.5 times the formation's influence radius is calculated, and this value is used as the maximum influence value of the enemy in the grid near the formation's movement path.

7. The strategic scenario scheduling and navigation method based on hierarchical reinforcement learning and tactical analysis according to claim 6, characterized in that: In step 4, the policy gradient loss function of the PPO algorithm used for both the upper-layer policy and the lower-layer policy is: ； in, In strategy ,state Next action The probability distribution; In strategy ,state Estimation of the action advantage under the following circumstances; It is a hyperparameter; when When the value is greater than 0, actions will be taken to increase the size of the agent. Update network parameters in the direction of probability, i.e., increase... However, the increase cannot exceed ;when When the value is less than 0, the agent will take action to reduce the value. Update network parameters in the direction of probability, i.e., decrease However, the reduction cannot be less than This ensures the strategy With strategy The difference will not be too large, thus making the process of updating the agent's behavior strategy more stable.

Images