A method for improving the rationality of agent actions based on explicit integrals

Abstract

This invention discloses a method for improving the rationality of agent actions based on explicit integration. The invention includes: 1. Creating an environment state file and providing a globally accessible interface for the environment state data. An environment-aware state data file is created separately for each agent, along with a data access interface. 2. Classifying the attributes of the real environment state, adding multiple thresholds for each attribute. Different combinations of state attribute levels correspond to different real environment states. 3. Creating a Q-table to store state-action values. 4. Setting initial real environment states and rewards, initiating reinforcement learning, and obtaining the learned Q-table. 5. The agent retrieves the corresponding action from the Q-table based on the current environment-aware state. This action causes a change in the real environment state, which in turn causes a change in the agent's environment-aware state. This invention is simple and efficient, solving the problem of agents being "too sensitive" to environmental reactions.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically game artificial intelligence based on reinforcement learning. It provides a method for improving the rationality of agent actions based on explicit integrals. Background Technology

[0002] Game playability is a crucial aspect of game design, and it's central to the game's "artificial intelligence." Currently, most games utilize AI based on finite state machines, behavior trees, and reinforcement learning. A finite state machine is a directed graph where each node represents a state. An intelligent agent can only be in one state at a time; changes in the environment alter the state, leading to different actions. Behavior tree-based game AI uses a search tree constructed according to expert rules. This tree search process represents the agent's decision-making, with the leaf nodes representing specific actions. Reinforcement learning-based game AI continuously allows the agent to "trial and error" in the environment, constantly learning and updating the state-action value. When the estimated value stabilizes after multiple iterations, it can be considered the actual value of the action. The agent then decides its next action based on this value.

[0003] However, current game AI agents all suffer from a problem: their environmental perception is "too sensitive." For example, when a player enters an enemy's field of vision, the enemy will immediately attack. This is unreasonable because a normal player, after seeing a target, will have a confirmation time before taking further action, only after fully confirming it is an enemy. Another example is behavior tree-based game AI, which retrieves leaf nodes of the tree based on the environmental state. Once the environment changes, the agent immediately retrieves the corresponding behavior to be performed in that new environment.

[0004] So what kind of artificial intelligence can be considered good artificial intelligence? (This depends on the player's personal experience; if the player feels like they are playing against a real person, then it is a good AI model. Game AI should not perform actions that are beyond human comprehension.)

[0005] Status attribute level combinations:

[0006] In most games, both players and enemies have specific numerical attributes, such as whether they are dead, health, field of vision, movement speed, attack speed, attack power, and defense.

[0007] We divide each attribute into different levels. For example, we divide the bool value into two levels: 0 and 1. We divide the int or float numeric attributes into several custom levels based on different thresholds. For example, we set two thresholds for health, x1 and x2. Then health is divided into 3 levels: Level 0: [0, x1), Level 1: [x1, x2), Level 2: [x2, ∞).

[0008] A state is a combination of all state attribute levels. When any attribute level changes, it means that a new state has been entered.

[0009] Proximity:

[0010] The proximity of two different combinations of state attribute levels can be measured by their step size. For example, if the state attribute level combination of state 1 is [1,0,3,3] and the state attribute level combination of state 2 is [1,0,2,3], then the step size between state 1 and state 2 should be 1. That is, [1,0,3,3] - [1,0,2,3] = [0,0,1,0]. Then, the sum of the absolute values ​​of each element is the step size between the two states, which is the proximity.

[0011] Reinforcement learning:

[0012] The basic elements of reinforcement learning include: agent, environment, and goal.

[0013] Reinforcement learning is the learning process undertaken by an agent in the course of interaction with the environment in order to achieve a goal.

[0014] The main elements of reinforcement learning are: state, action, and reward.

[0015] Specifically, agents take actions based on environmental conditions, which bring direct or indirect rewards. These rewards are determined by the ultimate goal. For example, in a Go game, the ultimate goal is to occupy territory, not just capture the opponent's stones; therefore, the reward should be set around territory occupation. We consider the expected value of an action's reward in a given state to be the actual value of that action to the goal. Rewards are immediate feedback, while goals are long-term outcomes.

[0016] The core elements of reinforcement learning: strategy and value

[0017] An action may not have an immediate reward upon completion, but it must have value to the ultimate goal. Its specific value cannot be assessed after the action ends; only after the game concludes can the estimated value be updated based on the rewards obtained in that game. After multiple game iterations, the estimated value will converge to the expected value of the reward. We consider the expected value of the reward for that action to be the value of that action. The value function is the function that evaluates the value of an action.

[0018] A policy function is a function that determines what action to take based on the current environment. There are generally two types of policy functions: greedy and ε-greedy.

[0019] The core issue in reinforcement learning is the trade-off between exploration and exploitation.

[0020] Exploitation is the use of something, specifically the value function learned through reinforcement learning. Generally, we would use the action with the highest value.

[0021] Exploration means not only utilizing actions with high estimated value that we have already learned, but also exploring and uncovering actions with low estimated value but high actual value.

[0022] Q Learning:

[0023] Q-learning is a hetero-policy temporal difference learning model in reinforcement learning. The estimated value of an action in a given state can be obtained by the following formula:

[0024] Q(S t A t )←Q(S t A t )+α*Error

[0025] Where Q(S) t A t ) represents the state S at time t. t Execution of Action A t The estimated value of the final goal, where α represents the learning rate and Error represents the error.

[0026]

[0027] The TD Target part can be written as:

[0028]

[0029] Generally, actions from a more distant past have less impact on the present. For example, drinking coffee ten years ago will have less impact on me than drinking coffee ten minutes ago. To reflect this, we introduce a discount factor γ∈[0,1] in TD Target. The discount factor, as the name suggests, applies a discount to the value of the next action. n As n increases, the value of the action decreases continuously, and the discount on the corresponding action value also decreases continuously.

[0030] TD Target = R t +γ*Q(S t+1 At+1 )

[0031] Therefore, the formula for updating state-action value is:

[0032] Q(S t A t )←Q(S t A t )+α*[R t +γ*Q(S t+1 A t+1 )-Q(S t A t )]

[0033] Since each value update requires knowing S t A t ,R t ,S t+1 A t+1 Therefore, this learning model is called SARSA.

[0034] SARSA is a temporal difference learning method based on the same policy. "Same policy" means that the action performed in the next state will also be used to update the action value of the current state. "Different policy" means that the action performed in the next state may not necessarily be used to update the action value of the current state. For example, in SARSA, the value update uses the action value Q(S) of the next state. t+1 A t+1 ).

[0035] Q-learning is a heterogeneous temporal difference learning method. The heterogeneous strategy is reflected in the fact that the action selection strategy used in the next execution and the action selection strategy used during value update are two different strategies. In Q-learning, the action selected during value update adopts a greedy strategy, that is, the action with the highest estimated value.

[0036]

[0037]

[0038] Therefore, the formula for updating the state-value of Q-learning is:

[0039]

[0040] The next action will be executed using an ε-greedy strategy, which means randomly selecting an action with a probability of ε and adopting a greedy strategy with a probability of (1-ε).

[0041] Q-learning first constructs a Q-table, which is essentially a state-action value table:

[0042] <![CDATA[A1]]> <![CDATA[A2]]> <![CDATA[A3]]> <![CDATA[S1]]> <![CDATA[Q(S1,A1)]]> <![CDATA[Q(S1,A2)]]> <![CDATA[Q(S1,A3)]]> <![CDATA[S2]]> <![CDATA[Q(S2,A1)]]> <![CDATA[Q(S2,A2)]]> <![CDATA[Q(S2,A3)]]>

[0043] The values ​​in the table are called Q values.

[0044] Q-learning involves writing the estimated value of all actions into a Q-table, which serves as a reference for the agent's final actions.

[0045] The interaction between agent actions and the environment:

[0046] The world as perceived by humans is not necessarily the true reality, and the same applies to agents in games: the environmental state variable represents the real world, but there may be a certain margin of error between the agent's perception of the real world and the actual world. When an agent performs an action, it causes a change in the environmental state; for example, placing a stone on a square in Go changes the state of the board. Similarly, when an agent moves to the player, the distance between the agent and the player changes. However, the environment perceived by the agent is not the same as the real environment, so the environmental state should not be directly used as the agent's perceived environmental state.

[0047] Changes in environmental state attribute values ​​will cause changes in the corresponding agent's perceived state attribute values, which can be considered as applying a pulling force to the environmental perception state. The magnitude of the pulling force is proportional to the error between them. For example, if the environmental state attribute "distance from player" is 100, and the current agent's environmental perception state attribute "distance from player" is 80, then the simplest error between the two is 100 - 80 = 20, which will apply a positive pulling force to the agent's environmental perception state attribute "distance from player" until the error is 0. We define the pulling force...

[0048] f 拉 =η*Error

[0049] Where η represents environmental perception sensitivity, and Error is the error. Let:

[0050] Error = tanh(e - e')

[0051] Where e represents the environmental state, and e' represents the perceived environmental state. tanh represents the hyperbolic tangent function.

[0052] so:

[0053] f 拉 =η*tanh(e-e')

[0054] Explicit integration algorithm:

[0055] Numerical integration algorithms are an effective method for solving time-discrete differential equations of motion in structural dynamics. Given the position x of a particle of mass m at time t... t and speed v tOur goal is to determine the particle's position x at the next moment. t+1 x can be calculated using explicit integration. t+1 .

[0056]

[0057] Where F t Indicates the resultant force acting on the body:

[0058] F t =f 拉 -f 阻

[0059] When the particle is not moving, f 阻 =f 拉 .

[0060] When the particle moves, f 阻 =μmg, where the direction of drag is opposite to the direction of motion. Here, μ is the drag coefficient, m represents mass, and g represents gravitational acceleration.

[0061] Actions and basic movements:

[0062] An action is a sequence of several basic movements.

[0063] Agent's action queue and execution unit:

[0064] In the game, each agent maintains a queue called the agent action queue. The agent action queue is a priority queue, implemented using a max-heap. The head of the queue contains the actions with the highest estimated value.

[0065] The execution unit always selects the action with the highest estimated value from the agent's action queue as the current action to execute. When a higher-value action appears in the queue, it immediately preempts the execution unit and inserts the action from the current execution unit back into the action queue. Summary of the Invention

[0066] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for improving the rationality of agent actions based on explicit integration.

[0067] The technical solution adopted by this invention to solve its technical problem is as follows:

[0068] Step 1: Create an environment status file and provide a globally accessible interface for the environment status data. Create a separate environment awareness status data file for each agent and provide a data access interface.

[0069] Step 2: Classify the real-world environment state attributes by adding several thresholds for each attribute. Different combinations of attribute levels correspond to different real-world environment states.

[0070] Step 3: Create a Q table to store state-action values.

[0071] Step 4: Set the initial real-world environment and rewards, begin reinforcement learning, and obtain the Q-table after learning.

[0072] Step 5: The agent retrieves the corresponding action from the Q-table based on the current environmental perception state. This action causes a change in the actual environmental state, which in turn causes a change in the agent's environmental perception state. In other words, explicit integration is used to continuously "approximate" the agent's environmental perception state to the actual environmental state.

[0073] Furthermore, step 1 specifically includes the following steps:

[0074] 1-1. Create a Scriptable Object class file named EnvironmentState_SO. Add environment state property variables.

[0075] 1-2. Create a Scriptable Object file named EnvironmentState_Data based on the Scriptable Object class file.

[0076] 1-3. Create a data access interface class for EnvironmentState_Data, named EnvironmentState_Dao. Then, set EnvironmentState_Data as a property of this data access interface class.

[0077] 1-4. Create an environment-aware state for each different agent.

[0078] Furthermore, in step 2, each real-world environmental state attribute is graded, i.e., a threshold is set. The number of grades for each attribute is equal to the number of thresholds plus 1. For example, if the number of thresholds is set to 5, then the attribute will have 6 grades, ranging from 0 to 5.

[0079] The specific description of step 2 is as follows:

[0080] 2-1. Add a threshold list for each attribute value in EnvironmentState_SO.

[0081] 2-2. Set thresholds for the threshold list in step 2-1. All environment attribute values ​​and thresholds are stored in the data file EnvironmentState_Data.

[0082] 2-3. EnvironmentState_Dao provides interface methods for calculating the current state of the real environment.

[0083] Furthermore, the real-world environment state attribute is a Boolean or numeric type.

[0084] Furthermore, the Q table created in step 3 is a persistent file stored on disk. When needed, a copy is copied from the disk into memory. After a game ends, the Q values ​​learned in that game are written back to the disk.

[0085] Furthermore, step 4 specifically includes the following steps:

[0086] 4-1. Set the initial real-world environment state, including the number of agents, their spawn locations, shelter locations, etc. All data is written to EnvironmentState_Data via EnvironmentState_Dao; set the target reward value.

[0087] 4-2. Read the Q table from disk into memory.

[0088] 4-3. The agent retrieves the Q-table based on the current real-world state. If the state does not exist in the Q-table, a Q-table entry is created, and all Q-values ​​are initialized to a large value. Otherwise, the action with the largest Q-value from the current real-world state is selected for execution (if multiple actions have the same maximum value, the action with the first maximum value is selected).

[0089] 4-4. When an action is executed, it will change the current state of the real environment, and may therefore enter a new state, updating the Q value of the action taken in the previous state of the real environment.

[0090]

[0091] Among them, S t Indicates the current state, A t R represents the action to be taken in the current state, α represents the learning rate, and R represents the learning rate. t Let represent the reward value obtained at time t. γ represents the discount factor, and S... t+1 Indicates the next state.

[0092] When the agent achieves the preset goal, R will be transferred in the final success state. t Set it to the reward value pre-set in step 4-1.

[0093] 4-5. Repeat steps 4-3 to 4-4 until the game ends.

[0094] 4-6. Write the Q-table values ​​back to disk. Continue iterative training from step 4-3. End training when the Q-values ​​in the Q-table converge to a specific value.

[0095] The Q-table is the agent's action guide. Obtaining the Q-table means that the agent knows what action should be taken in what situation.

[0096] Furthermore, step 5 aims to make the agent's behavior "more reasonable." If the next action is decided directly based on the environmental state, there will be a problem that the agent is "too sensitive" to the environment.

[0097] More specifically, the specific steps of step 5 are as follows:

[0098] 5-1. Initially, set the agent's environment perception state to the real environment state.

[0099] 5-2. Select an action from the Q table based on the environmental perception status and add it to the action queue.

[0100] 5-3. If the agent's current environmental perception state is not found in the Q table, the nearest real environmental state will be used as the current state for execution.

[0101] 5-4. Select the action with the highest value from the action queue and enter the execution unit. If an action in the execution unit is preempted by a higher priority action before it is completed, then the action is added back to the action queue.

[0102] 5-5. Actions taken during execution alter the actual state of the environment, and the agent makes decisions based on the perceived state of the environment. Therefore, the error between the actual state of the environment and the perceived state is proportional to the tension. Let the tension be f. 拉 =η*tanh(e-e'), where e represents the actual environmental state, e' represents the perceived environmental state, and η represents the parameter. The magnitude of the drag f 阻 =min(f 拉 Let μ be the drag coefficient, m be the mass, and g be the acceleration due to gravity. The direction of drag is opposite to the direction of motion. The net force F = f 拉 -f 阻 .

[0103] 5-6. Continuously update the environmental perception state according to the explicit integration formula:

[0104]

[0105] Where, x t+1 The value of the state attribute at the next moment, x t This represents the current state attribute value, Δt represents the change over time, and v t+1v represents the rate of change of the state attribute value at the next moment. t F represents the rate of change of the current state attribute value. t This represents the force acting on the object at the current moment, and m represents the mass.

[0106] 5-7. Return to step 5-2 until the game ends.

[0107] The beneficial effects of this invention are as follows:

[0108] The advantages of this invention are its simplicity and efficiency. It solves the problem of agents being "too sensitive" to their environment. Currently, game AI agents act based on the actual state of the environment, resulting in immediate reactions that are not realistic. To make the agent's actions more reasonable and improve the player's gaming experience, this invention proposes a method of applying explicit integrals to the agent's perceived state. The agent's actions change the actual environmental state, but the agent's understanding of the world is based on its perceived environmental state. The process of the perceived environmental state approximating the actual environmental state is the agent's process of understanding the world. Because this process involves a certain reaction time, the agent is no longer reacting instantly. Attached Figure Description

[0109] Figure 1 This is a time series diagram of the learning process.

[0110] Figure 2 This refers to the relationship between actions and basic movements.

[0111] Figure 3 This relates to the relationship between the ready queue and the execution unit.

[0112] Figure 4 Set the threshold for the status attribute. Detailed Implementation

[0113] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0114] This invention is a method for improving the rationality of agent actions based on explicit integration. First, environmental state attributes are categorized, with different combinations of levels representing different states. During reinforcement learning, actions are selected based on these states. Actions change the environmental state, leading to an error between the actual environmental state and the agent's perceived environmental state. The magnitude of this error determines the magnitude of the pull force, which is the force that approximates the agent's perceived environmental state to the true environmental state. By utilizing this pull force, the values ​​in the agent's perceived environmental state can be continuously updated, thereby making the agent's perceived world closer to the real world.

[0115] A Q-table is obtained through Q-learning. Then, based on the combination of different state attribute levels perceived by the agent's environment, the corresponding state in the Q-table is searched. If the agent's perceived environment state does not exist in the Q-table at a certain moment, it means that the corresponding real environment state was not entered during reinforcement learning. The action to be taken in this real environment state should be related to the nearest real environment state, and the vacant state needs to be set to the nearest real environment state. The action with the highest value in the agent's current perceived environment state is selected and added to the action queue, and the action with the highest value is selected from the action queue as the current action to be executed.

[0116] Example 1:

[0117] The experimental content of this invention is described as follows: Several sheep and several wolves are placed in a scene. Initially, the positions of the sheep and the shelter are randomly placed. The agent AI of the wolves is controlled by behavior trees, while the agent AI of the sheep will go through a learning stage and an application stage.

[0118] The learning phase utilizes the Q-table obtained from Q-learning. Specifically, if a wolf catches a sheep, it will eat it. If the sheep runs into the shelter, it has successfully escaped and will receive the reward value set in step 4-1. Sheep that are eaten receive no reward, i.e., their reward value is set to 0. Through multiple iterations, the agent's action reward will approach the expected value of the reward; this expected value can be considered the value of the agent's action. The action value in any state is stored in the Q-table.

[0119] In the application phase, the sheep agent's state is divided into the real state and the environment-aware state. Initially, the environment-aware state and the real state are the same. The sheep agent chooses the action with the highest value in the initial state to execute. The execution of the sheep agent's action may change the real environment state. Since there is an error between the real environment state and the environment-aware state, the magnitude of the error determines the magnitude of the force applied to the environment-aware state. The environment-aware state at the next moment is calculated using an explicit integral formula. When the environment-aware state changes, it prompts the sheep agent to choose its highest-value action in the new state.

[0120] By repeatedly entering new environmental perception states to choose actions, the sheep agent no longer acts immediately based on the actual state of the environment. This makes the sheep agent's actions more realistic, thereby improving the user experience.

Claims

1. A method for improving the rationality of agent actions based on explicit integrals, characterized in that... Includes the following steps: Step 1: Create an environment status file and provide a globally accessible interface for the environment status data; create a separate environment awareness status data file for each agent and provide a data access interface. Step 1 is implemented as follows: 1-1. Create a Scriptable Object class file named EnvironmentState_SO; and add environment state property variables; 1-2. Create a Scriptable Object file named EnvironmentState_Data based on the Scriptable Object class file; 1-3. Create a data access interface class for EnvironmentState_Data, named EnvironmentState_Dao; and set EnvironmentState_Data as an attribute of the data access interface class; 1-4. Create an environment-aware state for each different agent; Step 2: Classify the real environment state attributes and add several thresholds for each real environment state attribute; different combinations of state attribute levels correspond to different real environment states. Step 2 is implemented as follows: 2-1. Add a threshold list for each attribute value in EnvironmentState_SO; 2-2. Set thresholds for the threshold list in step 2-1. All environment attribute values ​​and thresholds are stored in the data file EnvironmentState_Data. 2-3. The EnvironmentState_Dao provides interface methods for calculating the current state of the real environment; Step 3: Create a Q table to store state-action value; Step 4: Set the initial real-world environment and rewards, begin reinforcement learning, and obtain the Q-table after learning; Step 5: The agent retrieves the corresponding action from the Q table based on the current environmental perception state. This action will cause a change in the real environmental state; in turn, it will cause a change in the agent's environmental perception state. That is, the agent's environmental perception state is continuously "approached" to the real environmental state by using explicit integration. Step 5 is implemented as follows: 5-1. Initially, set the agent's environmental perception state to the real environmental state; 5-2. Select an action from the Q table based on the environmental perception status and add it to the action queue; 5-3. If the agent's current environmental perception state is not found in the Q table, the nearest real environmental state will be used as the current state for execution; 5-4. Select the action with the highest value from the action queue and enter the execution unit. If an action in the execution unit is preempted by a higher priority action before it is completed, then the action is added back to the action queue. 5-5. Actions taken during execution alter the actual state of the environment, and the agent makes decisions based on the perceived state of the environment. Therefore, the error between the actual state of the environment and the perceived state is proportional to the tension; let the tension f 拉 =η*tanh(e-e'), where e represents the actual environmental state, e' represents the perceived environmental state, and η represents the parameter; the magnitude of the drag f 阻 =min(f 拉 The force F = f(μmg) represents the drag coefficient, m represents the mass, and g represents the gravitational acceleration; the direction of drag is opposite to the direction of motion; the net force F = f(μmg). 拉 -f 阻 ; 5-6. Continuously update the environmental perception state according to the explicit integration formula: Where, x t+1 The value of the state attribute at the next moment, x t This represents the current state attribute value, Δt represents the change over time, and v t+1 v represents the rate of change of the state attribute value at the next moment. t F represents the rate of change of the current state attribute value. t This represents the force acting on the object at the current moment, where m represents the mass. 5-7. Return to step 5-2 until the game ends.

2. The method for improving the rationality of agent actions based on explicit integration according to claim 1, characterized in that... In step 2, each real-world environmental state attribute is classified into levels, i.e., a threshold is set, and the number of levels for each attribute = the number of thresholds + 1.

3. A method for improving the rationality of agent actions based on explicit integration according to claim 1 or 2, characterized in that... The Q-table created in step 3 is a persistent file stored on disk. When needed, a copy is copied from the disk to memory. After a game ends, the Q-values ​​learned in that game are written back to disk. The Q-table is obtained through Q-learning. Then, the agent searches for the corresponding state in the Q-table based on the combination of different state attribute levels perceived by the environment. If the agent's environment perception state does not exist in the Q-table at a certain moment, it means that the agent did not enter the corresponding real environment state during reinforcement learning. The actions taken in the real environment state should be related to the nearest real environment state. The missing state needs to be set to the nearest real environment state. The agent selects the action with the highest value under the current environmental awareness state and adds it to the action queue, and then selects the action with the highest value from the action queue as the current action to be executed.

4. A method for improving the rationality of agent actions based on explicit integration according to claim 1 or 2, characterized in that... Step 4 is implemented as follows: 4-1. Set the initial real-world environment state, including the number of agents, their spawn locations, and shelter locations. All data is written to EnvironmentState_Data via EnvironmentState_Dao; set the target reward value. 4-2. Read the Q table from disk into memory; 4-3. The agent retrieves the Q-table based on the current real environment state. If the state does not exist in the Q-table, a Q-table entry is created and all Q values ​​are initialized to a large value; otherwise, the action with the largest Q value is selected from the real environment state for execution. If there are multiple actions with the same maximum value, the action with the first maximum value is selected. 4-4. When an action is executed, it will change the current state of the real environment, and may therefore enter a new state, updating the Q value of the action taken in the previous state of the real environment; Among them, S t Indicates the current state, A t R represents the action to be taken in the current state, α represents the learning rate, and R represents the learning rate. t S represents the reward value obtained at time t; γ represents the discount factor, and S... t+1 Indicates the next state; When the agent achieves the preset goal, R will be transferred in the final success state. t Set it to the reward value pre-set in step 4-1; 4-5. Repeat steps 4-3 to 4-4 continuously until the game ends; 4-6. Write the Q-table values ​​back to disk; continue iterative training from step 4-3; end training if the Q-values ​​in the Q-table converge to a specific value; The Q-table is the agent's action guide. Obtaining the Q-table means that the agent knows what action should be taken in what situation.

Images