Agent strategy coordination method in unstable environment based on game theory

Abstract

The present invention provides a method for the strategic coordination of intelligent agents in an unstable environment based on game theory, comprising the following steps: step 1, obtaining the state value and benefit of the intelligent agent; step 2, obtaining the expected benefit of the intelligent agent according to the dynamic differential programming method, and obtaining the optimal benefit of the intelligent agent according to the Markov formula; step 3, obtaining the most expected action selection of the intelligent agent according to the state value at the current moment and the expected benefit of the intelligent agent; step 4, when the intelligent agent is in a dynamic environment, obtaining the change of the expected benefit of the intelligent agent over time, and obtaining the optimal action selection according to the expected benefit after the change. The method of the present invention overcomes the limitations of traditional game theory and reinforcement learning methods in dynamic unstable environments, and provides a new and efficient solution for multi-agent collaborative decision-making.

Description

Technical Field

[0001] The present invention belongs to the field of multi-agent reinforcement learning, and specifically relates to an agent strategy coordination method in an unstable environment based on game theory. Background Art

[0002] The combination of agent collaborative decision-making and game theory has become a research hotspot in the field of multi-agent systems and adaptive control. In these fields, multiple agents need to cooperate or compete under limited resources to achieve the optimal collective goal. In practical applications, these systems often face complex problems such as unstable environment and incomplete information. For example, in intelligent transportation systems, each autonomous driving vehicle needs to process a large amount of uncertain information in real time and make quick decisions to ensure the smooth operation of the entire system. Game theory, as a mathematical tool for studying the interaction of multiple agents, can effectively analyze and predict the strategy choices of each agent and its game equilibrium state.

[0003] In order to deal with these complex multi-agent system decision-making problems, dynamic differential programming (DDP) and Markov decision process (MDP) are widely used. The DDP method can effectively optimize the decision-making process of the agent by establishing a recursive relationship between the state space and the action space. MDP can accurately simulate the interaction between the agent's behavior and the environment by using state transition probabilities and reward functions. Combining these methods, the agent can make adaptive decisions in complex, dynamic, and uncertain environments.

[0004] Traditional multi-agent collaborative decision-making methods mainly rely on classic game theory models and reinforcement learning methods. The application of game theory in multi-agent environments is usually based on static games or repeated games, assuming that each agent makes independent decisions and tries to optimize its own benefits. However, in the real world, the environment is often dynamically changing, and the information between agents is often incomplete, which makes traditional game theory methods less effective in dynamic environments.

[0005] On the other hand, traditional reinforcement learning methods learn how to take the best action in each state through the interaction between the agent and the environment. However, the application of reinforcement learning in multi-agent systems faces some challenges, such as how to deal with the competition and cooperation between agents, and how to effectively solve the problem of strategy selection under incomplete information. In addition, reinforcement learning usually requires a large amount of training data and may face computational bottlenecks and efficiency issues in large-scale systems. Therefore, although traditional technologies have achieved certain results in some scenarios, their limitations are becoming increasingly apparent in complex scenarios of dynamic environments and multi-agent collaboration.

[0006] Although existing technical methods (such as traditional game theory and reinforcement learning) can effectively handle the decision-making problems of intelligent agents in some simple scenarios, they have obvious shortcomings when dealing with dynamic and unstable environments. First, traditional game theory methods usually assume that the environment is static and the information is complete, which is inconsistent with the dynamic environment and incomplete information conditions in reality. In practical applications, the state of the environment changes frequently, and the interaction between intelligent agents is also highly dynamic. Traditional game theory methods are difficult to adapt to these complex situations.

[0007] Secondly, although traditional reinforcement learning methods perform well in the decision-making of a single agent, in a multi-agent system, the strategy selection and benefit evaluation between agents become more complicated. In the context of incomplete information and environmental changes, pure reinforcement learning may not be able to achieve effective collaboration between agents and maximize long-term benefits. In addition, the "exploration and utilization" dilemma and computational overhead problems in reinforcement learning also limit its application in large-scale, multi-agent systems. Summary of the invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to provide a strategy coordination method of intelligent agents in an unstable environment based on game theory in view of the shortcomings of the prior art, including the following steps:

[0009] Step 1, obtain the agent state value and benefit;

[0010] Step 2: According to the dynamic difference programming method, the expected return of the intelligent agent is obtained, and according to the Markov formula, the optimal return of the intelligent agent is obtained;

[0011] Step 3: According to the current state value and the expected benefit of the agent, the most desired action choice of the agent is obtained;

[0012] Step 4: When the agent is in a dynamic environment, obtain the change of the agent's expected return over time, and obtain the optimal action selection based on the changed expected return.

[0013] Step 1 includes: setting the state of the agent to , the initial action value is , the initialization gain is .

[0014] In step 1, the expected return is calculated using the following formula:

[0015] ,

[0016] in, For expected return, is the number of agents, is the probability of the ith agent adopting a strategy in a certain state (such as the direction, speed, and road section of a vehicle in a traffic scene), is the profit of the ith agent in a certain state.

[0017] In step 1, the agent expects the return With the initialization return If the expected return is With the initialization return The difference between them is less than or equal to the set optimal threshold (Generally, the value is 0.01~0.1), the state of the agent remains unchanged, that is, it is determined that the current state value is close to the optimal state. At this time, the state value of the agent is not updated and is maintained as the current state value. The state update formula is expressed as:

[0018] ,

[0019] in Indicates the current state;

[0020] If the expected return With the initialization return The difference between them is greater than the optimal threshold , indicating that the current state has not reached the optimal state and needs to be updated. At this time, the agent updates the state value according to the expected return. The new state value is the state corresponding to the expected return. The updated state value for:

[0021] .

[0022] Step 2 includes: The expected return of each agent is calculated as follows:

[0023] ,

[0024] in, represents the expected return of the ith agent; , Respectively represent the state and action of the i-th agent; , Respectively represent the state and action of the j-th agent; Indicates that the jth agent is in state and actions The joint probability under represents the state set of the jth agent, represents the action set of the jth agent; is the state of the ith agent and actions Next, based on the state of the jth agent and actions The income obtained.

[0025] Step 2 also includes: setting the current state set to , the state set at the next moment is , the state set of the i-th agent at the current moment is , the action set is , the expected return of the agent in dynamic programming is calculated using the following formula:

[0026] ,

[0027] in, is the state of the ith agent The value function under is the state of the ith agent Next action The immediate benefit obtained; γ is the discount factor; Indicates that from the current state and actions Transition to next state probability.

[0028] Step 2 also includes: calculating the optimal benefit of the i-th agent using the following formula :

[0029] .

[0030] Step 3 includes: setting the current state of the i-th agent to , the action is , transfer to the next state The income is , according to the Markov formula, the i-th agent moves from the current state Transition to next state Expected return It is expressed as:

[0031] ,

[0032] in Indicates that the i-th agent changes from the current state Transition to next state The conditional probability of

[0033] The optimal benefit selection formula of the agent is:

[0034] ,

[0035] in, For optimal profit.

[0036] Step 4 includes: setting the optimal action choice of the i-th agent at time t to , the corresponding expected return is , the expected return of the i-th agent at time t is calculated by the following formula :

[0037] ,

[0038] in, The agent is in state Next action Immediate benefits obtained, Indicates from the state Execute an action Transfer to state The probability of Status The value under , S represents the state set.

[0039] Step 4 also includes: when the time process moves to t+1, the agent uses the Markov property formula to predict the benefits brought by future state transfer based on the expected benefits at the current moment, thereby updating the expected benefits. According to the Markov property, the agent changes from the current state to Transition to next state Expected return for:

[0040] ,

[0041] Set the value of the value function Q of the i-th agent at time t to , the expected payoff table of the ith agent Shown as:

[0042] ,

[0043] Optimal benefit update in a dynamic environment: After each state transition, the agent will calculate the expected benefit at the current moment and update its optimal benefit based on the expected benefit at the current moment. The update formula for the optimal benefit is:

[0044] ,

[0045] in, Indicates that the i-th agent is in state The agent compares the benefits of different action options and chooses the action that maximizes the optimal benefit under the condition.

[0046] State transfer and behavior decision: Assume that the state transfer matrix of the i-th agent is , then the behavior decision of the ith agent in different states is expressed as:

[0047] ,

[0048] in, is the state of the ith agent Next select action The probability of Indicates that the agent is in state In the case of probability.

[0049] Beneficial effects: The present invention solves the defects of the prior art by introducing a dynamic differential programming method combined with game theory. In a dynamic environment, the agent can continuously adjust its strategy based on real-time status and benefit information, thereby effectively responding to environmental changes and optimizing long-term benefits. In addition, by combining the Markov decision process and Q-value update, the present invention can achieve strategy coordination in a multi-agent environment, improving the stability and collaboration of the system. This method overcomes the limitations of traditional game theory and reinforcement learning methods in dynamic unstable environments, and provides a new and efficient solution for multi-agent collaborative decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of the method of the present invention.

[0051] Figure 2 It is a schematic diagram of the agent state update process.

[0052] Figure 3 It is a schematic diagram of the agent’s optimal strategy and optimal action selection process.

[0053] Figure 4 It is a schematic diagram of the decision-making and expected benefit update process of the intelligent agent in a dynamic environment. DETAILED DESCRIPTION

[0054] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0055] like Figure 1 As shown, the embodiment of the present invention provides an agent strategy coordination method in an unstable environment based on game theory, including:

[0056] Step 1, obtain the agent state value and benefit;

[0057] Step 2: According to the dynamic difference programming method, the expected return of the intelligent agent is obtained, and according to the Markov formula, the optimal return of the intelligent agent is obtained;

[0058] Step 3: According to the current state value and the expected benefit of the agent, the most desired action choice of the agent is obtained;

[0059] Step 4: When the agent is in a dynamic environment, obtain the change of the agent's expected return over time, and obtain the optimal action selection based on the changed expected return.

[0060] The state update process of the agent adjusts and optimizes its decision-making strategy through the profit function in game theory. First, in step 1, the agent is initialized and the initialization state profit value is calculated. This profit value reflects the profit level of the agent in the current environment state. Figure 2 As shown in Figure 1, it is a schematic diagram of the agent state update process. Specifically, the state of the agent is set to , the initial action value is , the state benefit obtained by a certain calculation method (such as a rule-based calculation method, a calculation method based on environmental feedback, or a calculation method based on probability distribution) is . Define the profit value at this time as the profit after initialization.

[0061] Next, we calculate the expected return of the agent based on the return function in game theory. The expected return takes into account the interaction between the agent's strategy and the opponent's strategy. Therefore, in the game, the expected return depends not only on the state of the agent itself, but also on the strategy choices of other agents. The expected return formula is set as follows:

[0062] ,

[0063] in, For expected return, is the number of agents, is the probability of each agent taking a certain strategy in a specific state, The expected return is calculated for each agent in this state and used as the basis for the agent's decision.

[0064] When the expected return is calculated, the agent compares its expected return with the initial return. If the difference between the two is less than or equal to the optimal threshold (set as ), the state of the agent remains unchanged, that is, it is considered that the current state value is close to the optimal state. At this time, the state value of the agent is not updated and remains the current state value. The state update formula can be expressed as:

[0065] ,

[0066] If the difference between the expected return and the initial return is greater than the optimal threshold, it means that the current state has not reached the optimal state and needs to be updated. At this time, the agent updates the state value according to the expected return, and the new state value is the state corresponding to the expected return. Therefore, the updated state value is:

[0067] ,

[0068] This update mechanism is based on the calculation of expected returns in game theory. By comparing with the initial returns, the agent can flexibly adjust its strategy to cope with changes in unstable environments. Through the dynamic adjustment mechanism, the agent can gradually optimize its decision-making based on the current state and return information.

[0069] Take a specific example to illustrate. Set the initial state of the agent to The initial action is , and its initialization yield is , the expected return is calculated through game theory , the optimal threshold is At this point, the difference between the calculated expected return and the initial return is , which is exactly equal to the optimal threshold. Therefore, the agent will update its state value to the state corresponding to the expected return, that is, the state value is updated to 7.

[0070] This method can ensure that in a dynamic environment, the intelligent agent gradually adjusts its strategy and state according to the changes in benefits, avoids falling into the local optimal solution in an unstable environment, and realizes the coordination and optimization of strategies during the game.

[0071] In the agent strategy coordination method in an unstable environment based on game theory, the core task of step 2 is to calculate the expected return and optimal return of the agent. This process involves multi-agent strategies and dynamic state transitions in the game environment. In order to elaborate on this process, it is first necessary to clarify how the agent calculates its expected return based on the probability of state and action in a game environment with incomplete information, and optimizes its strategy through dynamic differential programming. The following is a more detailed description of the steps:

[0072] Step 2 includes: calculation of expected return and optimal return:

[0073] In a game environment, the agent faces not only its own decisions, but also the behavior of its opponent. Therefore, the expected return of the agent is determined by the joint probability distribution of its own and its opponent's strategies. The expected return calculation formula for each agent is as follows:

[0074] ,

[0075] in, represents the expected return of the ith agent under a specific strategy; , Respectively represent the state and action of the i-th agent; , Respectively represent the state and action of the j-th agent; Indicates that the jth agent is in state and actions The joint probability under is the state of the ith agent and actions Next, based on the state of the jth agent and actions The income obtained.

[0076] In the game, the agent needs to consider the mutual influence of the joint probability with other agents, so the expected benefit of each agent is not only affected by its own state, but also by the actions and states chosen by other agents. In this way, the agent dynamically calculates and adjusts its benefits.

[0077] State transition and strategy selection: In order to optimize the agent's strategy according to the state transition of the game environment, the agent needs to calculate the benefits and optimal strategies under different states according to the dynamic differential programming method. Set the current state set to , the state set at the next moment is , the state set of the i-th agent is , the action set is , the expected return of the agent in dynamic programming is calculated using the following formula:

[0078] ,

[0079] in, is the state of the ith agent The value function under , represents the maximum expected benefit of the i-th agent; is the state of the ith agent Next action The immediate income obtained; γ is the discount factor, which is used to balance the weight of immediate income and future income; is the state transition probability, indicating that and actions Transition to next state probability.

[0080] Update the strategy and get the optimal return: In this process, the agent continuously updates its strategy based on the expected return calculated by the dynamic programming method to pursue higher expected returns. The agent compares the returns under different strategies, selects the action that maximizes the expected return, and adjusts its behavior strategy based on this. Assume that under a certain strategy, the expected return of the ith agent is , and its optimal benefit is:

[0081] ,

[0082] After the agent calculates the expected benefits under different states, it will choose the strategy that maximizes the expected benefits and update its current decision-making strategy based on the optimal benefits. In actual games, the strategic coordination and information sharing among multiple agents will affect the final distribution of benefits, especially in an environment with incomplete information, where agents need to predict and adjust based on the possible actions of the other party.

[0083] Set the state set of the i-th agent to , the action set is , the state set of the jth agent is , the action set is The agent's payoff function is given by the game matrix, as shown in Table 1 below.

[0084] Table 1

[0085]

[0086] Through the joint probability in the game and , the i-th agent will calculate the expected return and obtain the expected return through dynamic programming .if If is greater than a certain threshold, the agent will select the corresponding action and update its strategy. The next agent i executes The gain is 5, and if the other party executes , then the benefit of the i-th agent is 2. The agent will compare the expected benefits of all strategies and choose the optimal strategy for decision-making.

[0087] In step 3, the agent determines the optimal strategy and the optimal action through game theory and dynamic differential programming methods. Specifically, when the agent determines that the current strategy can obtain the optimal benefit, it will execute the optimal action associated with the strategy. This process does not only rely on a single benefit calculation, but also involves the comprehensive application of state transfer and dynamic programming to ensure that the strategy selected by the agent can remain optimal in a constantly changing environment. Figure 3As shown, this is a schematic diagram of the optimal strategy and optimal action selection process of the intelligent agent.

[0088] Determination of optimal benefits and selection of optimal strategies: First, the agent calculates the benefits based on the current state and strategy. When a certain action is selected, the agent calculates the benefits of transferring from the current state to the next state using the dynamic differential programming method. Set the current state of the i-th agent to , the action is , transfer to the next state The income is Through dynamic programming, the agent will calculate all possible benefits based on different state transitions and action choices, and choose the strategy that maximizes the benefit.

[0089] Combining the Markov formula with the optimal strategy selection: The agent also needs to consider the influence of the Markov formula and calculate the expected return after each decision. The Markov formula is used to describe how the agent predicts the return of the future state based on the current state and action under environmental changes. According to the Markov formula, the i-th agent changes from the current state Transition to next state Expected return It is expressed as:

[0090] ,

[0091] This formula reflects the agent's change from the current state Execute an action After, the future state In this way, the agent is able to maximize future benefits by choosing the current state and action.

[0092] Update of optimal return: When the agent obtains the expected return through the above calculation method, it will compare it with the current optimal return. If the expected return is greater than the current optimal return, the agent will update its optimal return and select the corresponding strategy based on the new optimal return. The agent's optimal return selection formula is:

[0093] ,

[0094] in, The agent continuously adjusts its strategy to maximize the benefits in each state and finally reaches the optimal strategy.

[0095] Set the agent to state Next select action , its immediate benefit is 5. If the action selected by the agent can transfer to the state , and from the state Transfer to The expected return is 7, then according to dynamic programming, the intelligent experience will calculate the state To status The total expected return of Next, transfer to The probability of transferring to The probability is 0.2, then the expected return of the agent is calculated as:

[0096] ,

[0097] According to the principle of maximizing expected benefits, the agent will choose a strategy that can make the expected benefit reach 7.4. If the current optimal benefit is less than 7.4, the agent will update its strategy to choose action , and calculate the subsequent state transitions and benefits based on the new optimal strategy.

[0098] By combining the dynamic differential programming method with the Markov formula, the agent can dynamically adjust its strategy according to state transition and action selection in a constantly changing environment, ensuring that the optimal action that maximizes the expected return is selected. By continuously updating its optimal return, the agent can not only make optimal decisions under static conditions, but also cope with environmental uncertainty in dynamic games to achieve strategy coordination and optimization.

[0099] In step 4, when facing a dynamic environment, the agent needs to continuously adjust its strategy to optimize the expected return. The key to this process is how to use the dynamic difference programming method and the Markov decision process (MDP) model, combined with the basic principles of game theory, to determine the optimal action and update the expected return according to the changes in state transitions. The following is a detailed description of this part.

[0100] Calculation and update of expected returns in a dynamic environment: In a dynamic environment, the agent will calculate the expected return based on the state and action selection at each moment. When the state value of the agent changes at different moments, the expected return will be dynamically adjusted over time. The expected return at each moment can be gradually calculated through the dynamic differential programming method combined with the return model in game theory, and the decision-making process of the agent can be optimized. Figure 4 The figure shows a schematic diagram of the decision-making and expected benefit update process of the intelligent agent in a dynamic environment.

[0101] Assume that at time t, the optimal action choice of the i-th agent is , the corresponding expected return is , the expected return of the i-th agent at time t is calculated by the following formula :

[0102] ,

[0103] in, The agent is in state Next action Immediate benefits obtained, Indicates from the state Execute an action Transfer to state The probability of Status The value below.

[0104] When the time progresses to t+1, the agent uses the Markov property to predict the benefits of future state transfers based on the expected benefits at the current moment, thereby updating the expected benefits. Transition to next state Expected return for:

[0105] ,

[0106] Revenue calculation based on Q value: In a dynamic environment, agents often use Q value to represent the expected return of a state-action pair. Q value reflects the expected return of performing an action in a given state. Therefore, the expected return can be calculated by Q value. Assume that the Q value of the i-th agent at time t is , then the expected return of the ith agent is expressed as:

[0107] ,

[0108] This method of calculating Q values ​​allows the agent to more intuitively evaluate the expected benefits of different actions in the current state and make the best decision.

[0109] Optimal return update in a dynamic environment: As time goes by, the agent needs to constantly adjust its optimal return to cope with changes in the dynamic environment. After each state transition, the agent calculates the expected return at the current moment and updates its optimal return based on this result. The update formula for the optimal return is:

[0110] ,

[0111] in, Indicates that the i-th agent is in state The agent compares the benefits of different action options and selects the action that maximizes the optimal benefit under the condition.

[0112] State transition and behavior decision: The decision of the agent in a dynamic environment is not only affected by the current state and action selection, but also by the state transition caused by different actions. The probability distribution of state transition reflects the stability of the agent's behavior and the uncertainty of the environment. Assume that the state transition matrix of the agent is , then the agent's behavior decision in different states is expressed as:

[0113] ,

[0114] in, For the agent in state Next select action The agent adjusts its strategy by maximizing the expected reward, that is, choosing the action with the largest expected reward and updating the state transition matrix according to the action.

[0115] Assume that at time t, the agent is in state Next select action , and the expected return is calculated based on the dynamic difference programming method: The agent is expected to transfer to the state at the next time t+1 The probability of transferring to state The probability of is 0.3. Suppose that from the state To status The income is 7, the status The expected return is 3, then according to the Markov formula, the agent calculates the expected return at time t+1 as:

[0116] ,

[0117] Therefore, the agent will update its optimal strategy based on this expected benefit and choose to Next execution If the new strategy leads to a higher expected reward, the agent will adjust its behavior.

[0118] In a dynamic environment, the agent continuously updates its strategy based on state transitions and benefits to maximize expected benefits through dynamic differential programming methods and Markov decision processes. Combining Q-values ​​and Markov properties, the agent can flexibly adjust its decisions, adapt to dynamically changing environments, and ensure the best results in the game. The key to this process is how to effectively handle the dynamic changes in state transition probabilities, strategy updates, and expected benefits to ensure that the agent's strategy always tends to be optimal.

[0119] The method of this embodiment can be used in the following fields:

[0120] 1. Autonomous driving and intelligent transportation systems;

[0121] In autonomous vehicles and intelligent traffic management systems, multiple autonomous vehicles and traffic facilities (such as smart traffic lights, driverless buses, etc.) need to collaborate and make decisions in real time to improve traffic flow and safety. Each autonomous vehicle is an independent agent, facing an unstable environment with incomplete information. For example, factors such as traffic conditions, road conditions, and the behavior of other vehicles will affect the vehicle's decision-making. In this dynamic and complex environment, traditional single-agent decision-making methods are difficult to cope with, and collaboration between agents is particularly critical. Through the game theory and dynamic differential programming methods in this solution, autonomous vehicles can adjust their strategies according to real-time traffic data while ensuring safety, thereby achieving collaborative optimization, avoiding traffic jams, and improving traffic efficiency and safety.

[0122] 2. Drone swarms and collaborative tasks;

[0123] In unmanned aerial vehicle (UAV) swarm missions, multiple drones need to work together to complete complex tasks, such as search and rescue, environmental monitoring, agricultural spraying, etc. These tasks often require drones to make real-time decisions in dynamic environments, such as choosing the optimal action strategy under changing wind speeds, target locations, equipment status, etc. Through game theory models and dynamic differential programming, the system can dynamically adjust the behavior of each drone, so that each drone in the swarm can maximize its individual benefits while ensuring the synergy of the entire swarm. For example, in a search mission, each drone can collaboratively allocate the search area based on mission requirements and activity information of other drones to avoid duplication of work and improve mission efficiency.

[0124] 3. Smart grid and energy management;

[0125] In smart grids and energy management systems, multiple energy supply and consumption units (such as solar power generation, wind power generation, electric vehicle charging stations, etc.) need to be scheduled and coordinated according to real-time power demand and supply. Due to the spatiotemporal fluctuations in power demand and production, smart grids must be effectively dispatched in an uncertain and dynamically changing environment to avoid energy waste or insufficient supply. In this scenario, the agent strategy coordination method based on game theory can help each power device adjust its power supply or consumption strategy in real time while ensuring the stability and energy efficiency of the overall power grid. For example, in wind power and solar power generation systems, multiple power generation units can make collaborative decisions based on weather forecasts and demand forecasts to optimize power generation and energy storage arrangements and avoid excessive waste or insufficiency.

[0126] 4. Unmanned ships and intelligent shipping;

[0127] The application of unmanned ships in smart shipping is also becoming increasingly important. Ships need to coordinate routes, avoid obstacles, and deal with sudden weather changes. In the shipping system, multiple ships operate in a dynamic marine environment, and each ship, as an intelligent agent, needs to make decisions in an environment with incomplete information and uncertainty. Using game theory and dynamic differential programming methods, ships can choose the optimal navigation strategy based on real-time environmental conditions and the behavior of other ships while complying with traffic rules. This can not only improve navigation efficiency, but also effectively reduce collision risks and energy consumption. For example, in a busy port area, multiple ships can dynamically adjust their navigation paths and speeds based on each other's status and destination to avoid congestion.

[0128] 5. Financial markets and smart investment;

[0129] In the financial market, investors (agents) make investment decisions by selecting financial instruments such as stocks and bonds. The uncertainty of the market environment, the volatility of investment instruments, and the behavior of other investors will affect the decisions of individual investors. In this environment, through the model of game theory, investors can make predictions based on historical data and market trends, while considering the behavior of other investors, so as to optimize their investment strategies. Dynamic differential programming methods can help investors adjust their investment portfolios under different market conditions to maximize returns. For example, in stock market transactions, investors can adjust their buying and selling operations in real time according to market fluctuations and changes in strategies of other investors, reduce risks and improve investment returns.

[0130] In this embodiment, a method for intelligent agent strategy coordination in an unstable environment based on game theory is provided, including the following steps:

[0131] Step 1, agent initialization and state definition;

[0132] During the implementation process, each agent in the system needs to be initialized first. Each agent needs to define its state space, action space, benefit function, and other possible parameters. The state space represents all possible states of the agent in the environment, the action space represents all actions that the agent can choose, and the benefit function describes the benefit obtained by the agent after choosing a certain action in a certain state.

[0133] Initially, the system needs to obtain the initial state of each agent through measurement or perception, and calculate the initial benefit based on the state and action value. The state and action value of the agent should be updated in real time based on the dynamic changes of the system to ensure that the decision of each agent always reflects the actual situation of the current environment.

[0134] Implementation method: Obtain state information in the environment in real time through sensors or data interfaces, and calculate the initial benefit based on the set benefit function. For example, in autonomous driving, the agent (vehicle) initializes its state value by perceiving the traffic conditions, vehicle position, speed and other information of the surrounding environment, and evaluates the initial driving benefit based on this information.

[0135] Step 2: Construction of gaming environment and calculation of expected returns;

[0136] In step 2, the expected benefits are first calculated based on the states and actions of all agents in the game environment. Since the behavior of an agent is affected by the strategies of other agents, the benefits of each agent not only depend on its own actions, but are also closely related to the behavior of other agents.

[0137] Implementation steps:

[0138] Define the state space, action space, and strategy set of each agent in the game environment.

[0139] Using game theory methods, the expected return of each agent is calculated. The expected return reflects the expected benefit of the agent choosing a certain action in a specific state by integrating the rewards and returns under different states and actions.

[0140] The dynamic difference programming method (DDP) is used to calculate the benefits of the agent in different states, and the state transfer matrix and reward function are used to estimate the benefits under different strategies in the current state.

[0141] Using the Markov assumption, a dynamic transfer model is constructed to predict the possibility of future states and calculate the expected return based on this.

[0142] For example: In the scenario of autonomous driving, each vehicle can choose to accelerate, decelerate, turn, etc. Assuming the vehicle's current state is s, and it chooses the acceleration action a, the vehicle's expected benefit will be calculated based on the current traffic conditions and the behavior of other vehicles, taking into account factors such as traffic flow, driving speed, and expected collision risk.

[0143] Step 3, optimal strategy selection and profit update;

[0144] In step 3, the agent selects the optimal strategy based on the expected return and the current state. At this point, the agent will select its strategy based on the dynamic differential programming method and maximize the return by updating the strategy.

[0145] Implementation steps: For each agent, its strategy update is based on the current state value, expected benefit, and the strategies of other agents. By choosing the action corresponding to the optimal benefit, the agent can choose the optimal behavior path.

[0146] According to the dynamic difference programming method, the current strategy is evaluated through the state transfer matrix and reward function, and the expected return corresponding to the optimal strategy is calculated.

[0147] Update the state value of the agent. If the difference between the optimal benefit and the initialization benefit is less than a certain threshold, keep the current state; otherwise, update the state value according to the expected benefit.

[0148] For example, in a UAV collaborative mission, suppose a UAV has multiple possible actions during flight (such as changing the flight path, lowering the flight altitude, etc.). Each action has a different expected benefit. Based on the calculation of game theory, each UAV will choose an optimal strategy to minimize energy consumption and maximize the efficiency of mission completion.

[0149] Step 4, dynamic environmental adaptation and processing of benefits changing over time;

[0150] In step 4, the agent will continuously update its expected return and strategy selection in a dynamic environment. Whenever the state of the environment changes, the agent needs to re-evaluate and select the optimal strategy. This process relies on the Markov decision process (MDP) and dynamic differential programming methods.

[0151] Implementation steps: At each time step t, calculate the immediate benefit of the agent based on its current state and selected action.

[0152] According to the assumption of Markov decision process, the state at the next time t+1 is predicted and the state transfer matrix is ​​updated.

[0153] Recalculate the expected return of the agent and update the strategy. The agent's return will be adjusted over time to ensure that its decision remains optimal at different time points.

[0154] For example, in smart grid dispatch, each agent (such as a power generation unit or battery) will choose a power supply strategy based on load demand and grid status. As the grid status changes (such as load fluctuations), the agent will adjust its power supply strategy and dynamically update its benefits and optimal strategy.

[0155] Step 5: System integration and optimization;

[0156] Finally, through system integration and optimization, we ensure that all agents can work together. System integration includes communication, data sharing and information fusion between agents, ensuring that each agent can make decisions based on shared environmental information. The optimization process ensures that the entire system reaches the global optimum under the collaborative work of multiple agents, rather than just optimizing the benefits of a single agent.

[0157] Implementation steps: Realize real-time communication and information sharing between agents to ensure that each agent can obtain global information of the current environment.

[0158] Through a centralized decision-making system or distributed optimization algorithm, the decisions of each intelligent agent are integrated to achieve the global optimum.

[0159] The present invention provides a method for intelligent agent strategy coordination in an unstable environment based on game theory. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A strategy coordination method for intelligent agents in an unstable environment based on game theory, characterized by: The following steps are involved: Step 1, obtain the agent state value and benefit; Step 2: According to the dynamic difference programming method, the expected return of the intelligent agent is obtained, and according to the Markov formula, the optimal return of the intelligent agent is obtained; Step 3: According to the current state value and the expected benefit of the agent, the most desired action choice of the agent is obtained; Step 4: When the agent is in a dynamic environment, obtain the change of the agent's expected return over time, and obtain the optimal action selection based on the changed expected return; Step 1 includes: setting the state of the agent to s0, the initial action value to a0, and obtaining the initialization payoff to R0; In step 1, the expected return is calculated using the following formula: Where E(R) is the expected return, N is the number of agents, and p i is the probability that the ith agent adopts a strategy in a state, R i is the profit of the ith agent in a state; In step 1, the agent compares the expected return E(R) with the initialized return R0. If the difference between the expected return E(R) and the initialized return R0 is less than or equal to the set optimal threshold Δ th , the state of the agent remains unchanged. At this time, the state value of the agent is not updated and is maintained as the current state value. The state update formula is expressed as: |E(R)-R0|≤Δ th Then current =s0, where s current Indicates the current state; If the difference between the expected return E(R) and the initialized return R0 is greater than the optimal threshold Δ th , indicating that the current state has not reached the optimal state and needs to be updated. At this time, the agent updates the state value according to the expected return. The new state value is the state corresponding to the expected return. The updated state value s new for: s new =E(R); Step 2 includes: The expected return of each agent is calculated as follows: Among them, E i (R) represents the expected return of the ith agent; S i ,a i Respectively represent the state and action of the ith agent; s j ,a j Respectively represent the state and action of the jth agent; P(s j ,a j ) indicates that the jth agent is in state s j and action a j The joint probability under S j represents the state set of the jth agent, A j represents the action set of the jth agent; R i (s i ,a i ,s j ,a j ) is the i-th agent in state s i and action a i Next, based on the state s of the jth agent j and action a j the benefits gained; Step 2 also includes: setting the current state set to S t , the state set at the next moment is S t+1 , the state set of the i-th agent at the current moment is S i , the action set is A i , the expected return of the agent in dynamic programming is calculated using the following formula: Among them, V i (s i ) is the state of the ith agent in s i The value function under R i (s i ,a i ) is the state of the ith agent in s i Next, perform action a i The immediate benefit obtained; γ is the discount factor; P(s i+1 |s i ,a i ) indicates that from the current state s i and action a i Transition to next state S i+1 The probability of Step 2 also includes: using the following formula to calculate the optimal return R of the i-th agent * : R * =maxE i (R); Step 3 includes: setting the current state of the i-th agent to s i , action is a i , transfer to the next state s i+1 The revenue is R i (s i ,a i ,s i+1 ), according to the Markov formula, the i-th agent moves from the current state s i Transition to next state s i+1 The expected return E i (R) is expressed as: Where P(s i+1 |s i ) indicates that the i-th agent changes from the current state s i Transition to next state s i+1 The conditional probability of The optimal benefit selection formula of the agent is: Among them, R * (s i ) is the optimal return; Step 4 includes: setting the optimal action choice of the i-th agent at time t to be a t , the corresponding expected return is The expected return of the i-th agent at time t is calculated by the following formula: Among them, R i (s t ,a t ) is the agent in state s t Next, perform action a t The immediate benefit obtained, P(s t+1 |s t ,a t ) indicates that from state s t Execute action a t Transfer to state S t+1 The probability of V i (s t+1 ) is state S t+1 The value under, S represents the state set; Step 4 also includes: when the time process moves to t+1, the agent uses the Markov property formula to predict the benefits brought by future state transfer based on the expected benefits at the current moment, thereby updating the expected benefits. According to the Markov property, the agent changes from the current state s t Transition to next state s t+1 Expected return for: Set the value of the value function Q of the i-th agent at time t to The expected payoff table of the i-th agent Shown as: Optimal benefit update in a dynamic environment: After each state transition, the agent will calculate the expected benefit at the current moment and update its optimal benefit based on the expected benefit at the current moment. The update formula for the optimal benefit is: in, Indicates that the i-th agent is in state s t The agent compares the benefits of different action options and chooses the action that maximizes the optimal benefit under the condition. State transfer and behavior decision: Assume that the state transfer matrix of the i-th agent is P(s t+1 |s t ), then the behavior decision of the ith agent in different states is expressed as: Among them, π(a t |s t ) is the state of the ith agent in s t Next select action a t The probability of |, P(s t+1 |s t ) indicates that the agent is in state s t In the case of t+1 probability.

Images