Agent behavior tree generation method and system based on reinforcement learning and genetic algorithm

Abstract

The application provides an agent behavior tree generation method and system based on reinforcement learning and a genetic algorithm, and belongs to the technical field of artificial intelligence.The offspring behavior trees generated by the genetic algorithm are added to the action space of the reinforcement learning.The generated behavior trees are called by the reinforcement learning to drive the agent, and the value estimation of the behavior trees by the reinforcement learning is used as a reference for measuring the fitness.The genetic algorithm screens, recombines and mutates the offspring of the behavior trees according to the fitness, and iterates the process until the behavior trees capable of driving the agent to complete the task target are obtained.Thus, the behavior trees capable of driving the agent to complete the task target can be generated in an experimental environment providing only delayed rewards.The application provides a self-generated control strategy expressed by a behavior tree for various applications such as game agents, robots and unmanned vehicles, and reduces the cost.

Description

TECHNICAL FIELD

[0001] The application relates to an agent behavior tree generation method and system, in particular to an agent behavior tree generation method and system based on reinforcement learning and a genetic algorithm, and belongs to the technical field of artificial intelligence. BACKGROUND

[0002] With the progress of technology, more and more application scenarios need to realize artificial intelligence behavior. In the field of artificial intelligence, behavior trees are widely used to describe the behavior of agents, drive agents to complete a certain task goal, and are often applied to virtual simulation, games, robots, unmanned vehicles, etc. For example, in many virtual reality development projects, a behavior tree is needed to manage the development of the entire program plot. The behavior tree is a tree structure containing logic nodes and behavior nodes. Each time a behavior needs to be found, the root node of the tree is started, each node is traversed, and the first behavior that meets the current data is found.

[0003] The behavior tree is usually artificially designed and edited by a designer, but the artificial construction of the behavior tree is time-consuming and requires high experience of the designer. Moreover, the constructed behavior tree is usually tested and optimized for a certain type of scene, and has poor adaptability to new environments. A new behavior tree needs to be tailored for each environment. Therefore, a method for automatically generating a behavior tree is needed to save the cost of manually editing the behavior tree.

[0004] The existing solution is to use a genetic algorithm to generate a behavior tree, combine the conditions, actions and logic nodes of the behavior tree, generate different behavior trees as the initial generation, test the effect of the behavior tree on the agent in the game, calculate the fitness function, and select the behavior trees with low fitness according to the fitness function. The behavior trees with high fitness are left as offspring, and they are recombined and mutated to generate new behavior trees. The above process is iterated until a behavior tree with a fitness reaching a threshold is obtained. However, when calculating the fitness, a reward information needs to be provided after each behavior tree is executed, but many applications have delayed rewards, which cannot provide an effective fitness function and cannot be applied to this method. SUMMARY

[0005] The purpose of the present application is to solve the technical problem that the agent requires a reward information to be fed back after each behavior tree execution when generating a behavior tree using a genetic algorithm, which cannot be applied to applications with delayed rewards. The application creatively proposes an agent behavior tree generation method and system based on reinforcement learning and a genetic algorithm.

[0006] The innovation of the present application is that the offspring behavior tree generated by the genetic algorithm is added to the action space of reinforcement learning. The generated behavior tree is used to drive the agent by using reinforcement learning, and the value estimation of the behavior tree by reinforcement learning is used as a reference to measure the fitness. The genetic algorithm screens, recombines and mutates the offspring of the behavior tree according to the fitness, and iterates the process repeatedly until the behavior tree capable of driving the agent to complete the task goal is obtained. Thus, the behavior tree capable of driving the agent to complete the task goal can be generated in an experimental environment providing only delayed rewards.

[0007] The present application adopts the following technical solutions.

[0008] An agent behavior tree generation method based on reinforcement learning and genetic algorithm, comprising the following steps:

[0009] Step 1: initialize the behavior tree library.

[0010] Specifically, a set of behavior trees is designed to cover all possible logic, conditions and action nodes, serving as the behavior tree library.

[0011] Step 2: initialize the action space.

[0012] Specifically, a set of behavior trees is designed as the initial action space of reinforcement learning.

[0013] Step 3: reset the accumulated value estimation.

[0014] A table equal in size to the action space is set and reset to 0.

[0015] Step 4: perform reinforcement learning training.

[0016] One training iteration is performed using reinforcement learning. The current environment information is used as input, and the value estimation of the behavior trees in the action space is used as output. The behavior tree with the highest value is selected for experiment, and then the value estimation of the behavior tree is updated according to the reward value given by the environment.

[0017] Step 5: determine whether the behavior tree can complete the goal.

[0018] Determine whether the behavior tree selected by reinforcement learning can independently complete the task, i.e., determine whether the current behavior tree can drive the agent to complete all task goals from the initial state. If yes, it means that the current behavior tree has reached the algorithm goal, and the process ends. If no, return to step 4.

[0019] Step 6: accumulate the value estimation of the output layer of the neural network.

[0020] The value estimation of each behavior tree by the current reinforcement learning is accumulated.

[0021] Step 7: Determine whether the number of iterations exceeds a threshold value.

[0022] Determine whether the number of iterations of reinforcement learning exceeds a predetermined threshold value. If yes, go to step 7, if no, return to step 3 and perform a new round of iteration.

[0023] Step 8: Calculate the fitness of the behavior tree.

[0024] According to the accumulated behavior tree value estimate, calculate the fitness of the behavior tree.

[0025] Step 9: Run genetic algorithm to generate new behavior tree.

[0026] According to the fitness of the behavior tree, retain the top N behavior trees with the highest fitness from the parent generation, generate offspring behavior trees, and replace the remaining behavior trees, keeping the total number of behavior trees unchanged.

[0027] Among them, the process of generating offspring behavior trees includes recombination and mutation. The recombination is a random recombination of the retained parent behavior tree nodes to generate new offspring behavior trees. The mutation is to take nodes / subtrees from the behavior tree library in step 1 and randomly replace the nodes / subtrees of the offspring behavior trees.

[0028] Step 10: Update the action space.

[0029] Use the retained parent behavior trees and generated offspring behavior trees to update the action space, the position of the parent behavior trees remains unchanged, and the offspring behavior trees fill the vacant positions.

[0030] Further, based on the above method, the present application proposes an agent behavior tree generation system based on reinforcement learning and genetic algorithm, including a behavior tree library module, an action space management module, a reinforcement learning module, a fitness calculation module and a genetic algorithm module.

[0031] Among them, the behavior tree library module is used to initialize and store the behavior tree library, that is, to realize step 1 of the above method.

[0032] The action space management module is used to initialize and update the action space, that is, to realize steps 2 and 10 of the above method.

[0033] The reinforcement learning module is used to reset the accumulated value estimate, perform reinforcement learning training, determine whether the behavior tree can complete the target, accumulate the value estimate of the output layer of the neural network, and determine whether the number of iterations exceeds a threshold value, that is, to realize steps 3, 4, 5, 6 and 7 of the above method.

[0034] The fitness calculation module is used to calculate the fitness of the behavior tree, that is, to realize step 8 of the above method.

[0035] The genetic algorithm module is configured to run a genetic algorithm to generate a new behavior tree, i.e., to implement step 9 of the above method.

[0036] The connection relationship between the above modules is shown in Figure 3 and is as follows:

[0037] The output end of the behavior tree library module is connected to the input end of the action space management module and the input end of the genetic algorithm module, the output end of the action space management module is connected to the input end of the reinforcement learning module, the output end of the reinforcement learning module is connected to the input end of the fitness calculation module, the output end of the fitness calculation module is connected to the input end of the genetic algorithm module, and the output end of the genetic algorithm module is connected to the input end of the action space management module.

[0038] Advantages

[0039] Compared with the prior art, when the genetic algorithm is used to generate a behavior tree, in the case that the scenario has a delayed reward and cannot provide input for the fitness calculation of each behavior tree, the reinforcement learning algorithm is used to evaluate the value of the behavior tree to provide input for the fitness calculation, so that the genetic algorithm can be effectively used for behavior tree generation in a scenario with a delayed reward. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a schematic diagram of a land simulation environment.

[0041] Figure 2 is a schematic diagram of the flow of the method.

[0042] Figure 3 is a schematic diagram of the structure of the system of the application. DETAILED DESCRIPTION

[0043] The application will be described in detail below with reference to the accompanying drawings and examples. It should be noted that the described examples are only intended to facilitate the understanding of the application and do not limit it in any way.

[0044] Examples

[0045] This example is based on Unity to build a virtual simulation system, as shown in Figure 1 The system simulates an environment covering various terrain elements, including mountains, sand, rivers, bridges, jungles, etc. There is a small car that can move freely, which can find target objects within a certain range through a simulated radar, and can rotate the barrel and fire cannonballs to attack the target. The task of the car is to start from a certain point on the map, move to find the target object, and hit it with the cannon. The simulation environment is responsible for determining whether it is hit or timed out (60s), and resetting the game environment when the task is successful or failed. The flow of the application is as shown in Figure 2As shown, an agent behavior tree generation method based on reinforcement learning and genetic algorithm includes the following steps:

[0046] Step 1: Initialize the behavior tree library.

[0047] Specifically, 8 behavior trees are designed, each of which can drive the car to complete basic actions (such as moving, finding targets, and striking), covering all possible logic, conditions, and action nodes used in the simulation environment, as a behavior tree library.

[0048] Step 2: Initialize the action space.

[0049] Using the nodes in the behavior tree library, 20 behavior trees are randomly generated as the initial action space of reinforcement learning.

[0050] Step 3: Reset the accumulated value estimate.

[0051] Initialize a list with a size equal to the size of the action space, and the values are initialized to 0.

[0052] Step 4: Reinforcement learning training.

[0053] Specifically, DQN is used as the reinforcement learning algorithm for one iteration. In each iteration, the environment information dimension is 3, which is the current car coordinate value (x, y, z). The output value estimate dimension is 20, which is the same size as the action space, corresponding to each behavior tree, the behavior tree with the highest value is selected for testing. Whenever a behavior tree ends execution, the reward value is obtained from the environment, the task is completed for 20, the timeout is not completed for -20, and other cases are -1. According to the reward value, DQN updates the value estimate of the behavior tree.

[0054] Step 5: Determine whether the behavior tree can complete the target.

[0055] Determine whether the behavior tree selected by reinforcement learning can independently complete the task. That is, from the initial state, whether the current behavior tree can drive the agent to complete all task goals. If yes, the current behavior tree has reached the goal, and the process ends. If not, return to step 4.

[0056] Step 6: Accumulate the value estimate of the neural network output layer.

[0057] Specifically, update the list established in step 3, add the value estimate of each behavior tree in step 4 to the value recorded in the corresponding position in the list, and store it in the list.

[0058] Step 7: Determine whether the iteration number exceeds the threshold value.

[0059] It is determined whether the number of iterations of the reinforcement learning exceeds a predetermined threshold, for example, the threshold is set to 20. If yes, go to step 7; if no, return to step 3 to perform a new round of iteration.

[0060] Step 8: Calculate the fitness of the behavior tree.

[0061] According to the accumulated value estimate of the behavior tree, the fitness of the behavior tree is calculated. Here, the value estimate of the behavior tree is directly taken as the fitness.

[0062] Step 9: Run the genetic algorithm to generate a new behavior tree.

[0063] Specifically, according to the fitness of the behavior tree, the top 10 behavior trees with the highest fitness in the parent generation are retained, 10 new child behavior trees are generated to replace the remaining 10 behavior trees, and the total number of behavior trees is kept unchanged at 20.

[0064] Wherein, the process of generating child behavior trees is divided into two steps: recombination and mutation.

[0065] When recombination, first copy the 10 retained parent behavior trees to generate 10 child behavior trees. For each child behavior tree, randomly select a "sequential execution" node or "loop until" node from the retained parent behavior trees to replace a random node in the subtree.

[0066] When mutation, for each child behavior tree, randomly take a "sequential execution" node or "loop until" node from the behavior tree library in step 1 to replace a random node in the subtree.

[0067] It should be noted that when replacing the nodes of the behavior tree, the subtree below the node should be replaced together.

[0068] Step 10: Update the action space.

[0069] Specifically, the action space is updated using the 10 retained parent behavior trees and the 10 generated child behavior trees, the positions of the 10 parent behavior trees remain unchanged, and the 10 child behavior trees are used to fill the vacant positions in the action space in turn.

[0070] Experimental comparison and verification

[0071] Because the scene has a delayed reward, it is not possible to set a reasonable fitness function for the behavior tree generation method based on the genetic algorithm, which makes it impossible to use this method. Therefore, the control group of the present embodiment is a method of generating a behavior tree completely randomly. In order to compare more fairly, the algorithm architecture and parameters of step 1 are adopted, except that the fitness value is randomly generated according to the normal distribution.

[0072] Under the same condition, through 5 experiments, the method proposed in the present application can find the expected behavior tree within the maximum number of iterations (25 generations) in each experiment, with an average of 7.8 generations, while the random method cannot find the expected behavior tree within the maximum number of iterations in 2 experiments, and the average is 9 generations. Therefore, the method of the present application can significantly improve the efficiency of obtaining the target behavior tree (i.e. the behavior tree that can independently drive the intelligent agent to complete the strike task), and reduce the number of iterations.

[0073] Through the above steps, the behavior tree capable of completing the task target is effectively generated in the environment with delayed rewards. The method proposed in the present application can be used to automatically generate the control strategy expressed by the behavior tree for game intelligent agents, robots, unmanned vehicles and other applications, reducing the cost of manual design. The generated behavior tree can also be combined with reinforcement learning as a form of macro action to improve the performance of reinforcement learning. Since the behavior tree has interpretability and is easy for humans to understand, the generated behavior tree can also provide a reference for designers to create new control strategies.

[0074] The specific description described above further details the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating agent behavior trees based on reinforcement learning and genetic algorithms, characterized in that, The virtual simulation system covers various terrain features, including mountains, sand dunes, rivers, bridges, and jungles. It includes a freely moving vehicle that uses simulated radar to locate targets and can rotate its cannon to fire projectiles. The vehicle's task is to start from a point on the map, move to find the target, and fire to hit it. The simulation environment is responsible for determining whether a hit is successful or if a timeout occurs, and resets the game environment upon success or failure. Includes the following steps: Step 1: Initialize the behavior tree library; Design a set of behavior trees, each of which can drive the car to complete basic actions, covering all possible logic, condition and action nodes used in the simulation environment, as a behavior tree library; Step 2: Initialize the motion space; Using nodes from the behavior tree library, several behavior trees are randomly generated as the initial action space for reinforcement learning; Step 3: Reset the accumulated value estimate by setting up a table with the same size as the action space and resetting it to 0; Step 4: Conduct reinforcement learning training; A training iteration is performed using reinforcement learning, with environmental information as input and the value estimate of the behavior tree in the action space as output. The behavior tree with the highest value is selected for experimentation, and the value estimate of the behavior tree is updated based on the reward value given by the environment. In each iteration, the environmental information dimension is 3, which is the current coordinate value (x, y, z) of the car; the output value estimate dimension is the same as the action space size, corresponding to each behavior tree, and the behavior tree with the highest value is selected for testing; whenever a behavior tree finishes execution, the reward value is obtained from the environment; the value estimate of the behavior tree is updated based on the reward value. Step 5: Determine if the behavior tree can achieve the goal; Determine whether the behavior tree selected by reinforcement learning can complete the task independently. That is, determine whether the current behavior tree can drive the agent to complete all task objectives from the initial state. If yes, it means that the current behavior tree has reached the algorithm objective and the process ends. If no, return to step 4. Step 6: Accumulate the value estimates of the neural network output layer; The current reinforcement learning value estimates for each behavior tree are summed up. Step 7: Determine if the number of iterations exceeds the threshold; Determine whether the number of iterations of reinforcement learning exceeds a predetermined threshold; if yes, proceed to step 7; if no, return to step 3 and start a new round of iterations. Step 8: Calculate the fitness of the behavior tree; Calculate the behavior tree fitness based on the accumulated behavior tree value estimate; Step 9: Run the genetic algorithm to generate a new behavior tree; Based on the fitness of the behavior trees, retain the N behavior trees with the highest fitness in the parent generation, generate child behavior trees, and replace the remaining behavior trees, keeping the total number of behavior trees unchanged; The process of generating a child behavior tree includes recombination and mutation. Recombination involves randomly recombinating the retained parent behavior tree nodes to generate a new child behavior tree. Mutation involves taking nodes / subtrees from the behavior tree library in step 1 and randomly replacing the nodes / subtrees of the child behavior tree. Step 10: Update the action space; The action space is updated using the preserved parent behavior tree and the generated child behavior tree. The position of the parent behavior tree remains unchanged, and the child behavior tree fills the empty positions.

2. A system for generating agent behavior trees based on reinforcement learning and genetic algorithms, as described in claim 1, characterized in that, It includes a behavior tree database module, an action space management module, a reinforcement learning module, a fitness calculation module, and a genetic algorithm module; Among them, the behavior tree library module is used to initialize and store the behavior tree library; The action space management module is used to initialize and update the action space. The reinforcement learning module is used to reset the accumulated value estimate, perform reinforcement learning training, determine whether the behavior tree can achieve the goal, accumulate the value estimate of the neural network output layer, and determine whether the number of iterations exceeds the threshold. The fitness calculation module is used to calculate the fitness of the behavior tree; The genetic algorithm module is used to run genetic algorithms and generate new behavior trees; The connection relationships between the above modules are as follows: The output of the behavior tree library module is connected to the input of the action space management module and the genetic algorithm module, respectively. The output of the action space management module is connected to the input of the reinforcement learning module. The output of the reinforcement learning module is connected to the input of the fitness calculation module. The output of the fitness calculation module is connected to the input of the genetic algorithm module. The output of the genetic algorithm module is connected to the input of the action space management module.

Images