Method, system, device and medium for analyzing performance limit of agent game strategy

Abstract

This invention discloses a method, system, device, and medium for performance bound analysis of intelligent agent game strategies. It models the unobservable portion of the opponent's state as a bounded parameter Markov decision process, which more realistically reflects the uncertainty in the game process than a typical Markov decision process, helping the agent make more informed decisions. The upper and lower bounds of performance are calculated using an iterative algorithm, providing a concrete analytical object for performance analysis.

Description

Technical Field

[0001] This invention relates to the field of performance limit analysis technology for intelligent agent game strategies, and in particular to a method, system, device and medium for performance limit analysis of intelligent agent game strategies. Background Technology

[0002] Markov Decision Processes (MDPs) are a mathematical framework used to describe reinforcement learning problems. In an MDP, an agent interacts with the environment to receive rewards and attempts to maximize its understanding of the environment to obtain a higher total reward.

[0003] In a game-like contest between agents, one agent is the attacker, and the other is the defender. The attacker is considered the subject (actor), and the defender's state is considered the environment. The four-tuple in a Markov decision process is (the defender's current state, the attacker's action, the defender's next state, and the immediate reward for whether the attack is effective). The attacker decides its action based on the defender's current state, employing its own strategy.

[0004] Performance bounds analysis is a method for determining the optimal solution to a Multivariate Problem (MDP). This method provides a way to compute the optimal performance bounds, which reflect the maximum possible total reward—the best performance an agent can achieve. This method helps an agent determine the best policy and evaluate the performance of any other policy relative to the optimal policy.

[0005] Currently, research on the performance limit analysis of strategies mainly focuses on the following aspects:

[0006] 1. Boundary analysis method based on optimal control theory. This method is based on optimal control theory. By solving the Bellman equation and dynamic programming, the corresponding policy value function is obtained, and then the performance bounds of the policy are evaluated. This method is relatively accurate, but it is computationally intensive and can only handle small-scale problems.

[0007] 2. Boundary Analysis Methods Based on Reinforcement Learning. This method utilizes reinforcement learning algorithms, such as Q-learning and policy evaluation, to estimate the value function of a given policy and then calculate its upper bound or confidence interval, thereby evaluating the policy's performance limitations. This method is computationally fast but has relatively low accuracy.

[0008] 3. Boundary Analysis Method Based on Hamilton-Jacobi-Bellman Equations. This method, based on the Hamilton-Jacobi-Bellman equations, obtains the upper and lower boundaries of the policy value function by solving explicit or implicit optimal control equations. This method boasts high accuracy and fast computation speed, making it suitable for handling large-scale problems.

[0009] However, in actual game competition, the above performance limit analysis schemes are somewhat lacking in analytical capabilities and cannot effectively help agents choose more reasonable actions, resulting in poor game performance for the attacker. The level of intelligence in game applications also needs to be improved. Summary of the Invention

[0010] The purpose of this invention is to provide a method, system, device, and medium for performance boundary analysis of intelligent agent game strategies. This method can evaluate the performance of strategies through performance boundary analysis, thereby helping intelligent agents make better choices, enabling attackers to choose attack actions more rationally, and improving game performance and the intelligence level of game applications.

[0011] The objective of this invention is achieved through the following technical solution:

[0012] A method for performance boundary analysis of agent game strategies includes:

[0013] Obtain the real historical game sequence and the attacking side's strategy to be analyzed; each item in the real historical game sequence includes: the defender's current state, the attacker's action, the defender's next state, and the immediate reward for whether the attack is effective.

[0014] Based on real historical game sequences, when the defender is in different observation states, the strategy to be analyzed is adopted to obtain the state transition probability and state value function under the strategy to be analyzed. Then, the bounded parameter Markov decision process model is obtained through state aggregation.

[0015] In the bounded parameter Markov decision process model, the upper and lower bounds of each state value function interval are updated according to the upper and lower bound iteration algorithm, which serve as the upper and lower bounds of the overall performance of the strategy to be analyzed.

[0016] Output the upper and lower bounds of the overall performance of the strategy to be analyzed.

[0017] A performance boundary analysis system for agent game strategies includes:

[0018] The data receiving module is used to acquire real historical game sequences and the attacking side's strategy to be analyzed. Each entry in the real historical game sequence includes: the defender's current state, the attacker's action, the defender's next state, and the immediate reward for whether the attack is effective.

[0019] The BMDP model building module is used to adopt the strategy to be analyzed based on real historical game sequences when the defender is in different observation states, obtain the state transition probability and state value function under the strategy to be analyzed, and then obtain the bounded parameter Markov decision process model through state aggregation.

[0020] The performance bounds calculation module is used to update the upper and lower bounds of each state value function interval in a bounded parameter Markov decision process model according to the upper and lower bounds iteration algorithm, which serve as the overall performance upper and lower bounds of the strategy to be analyzed.

[0021] The data sending module is used to output the upper and lower bounds of the overall performance of the strategy to be analyzed.

[0022] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0024] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0025] As can be seen from the technical solution provided by the present invention, in the process of intelligent agent game confrontation, the performance of a given strategy is analyzed, and the upper and lower bounds of the performance are given through the upper and lower bound iteration algorithm. The performance of a single strategy can be evaluated and its performance can be quantified. If multiple strategies are obtained, the upper and lower bounds of the performance calculated for different strategies can be evaluated, and the strategy with better performance can be selected. This helps the attacker to choose attack actions more reasonably, thereby improving the game performance and the intelligence level of game application. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a performance limit analysis method for an agent game strategy provided in an embodiment of the present invention;

[0028] Figure 2 A flowchart of the calculation of Update(v,π,o) is provided for an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of a performance limit analysis system for intelligent agent game strategies provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0032] First, the following explanations are provided for the terms that may be used in this article:

[0033] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0034] The following provides a detailed description of the performance limit analysis method, system, device, and medium for intelligent agent game strategies provided by this invention. Contents not described in detail in the embodiments of this invention belong to prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0035] Example 1

[0036] This invention provides a method for performance bounds analysis of agent-based game strategies. It models the uncertainty arising from the incomplete observation of the opponent's state as a bounded parameter Markov decision process (BMDP). Then, iteratively obtaining the upper and lower bounds of a specific strategy's performance, the BMDP is a decision model used to handle decision problems with stochasticity and uncertainty. BMDP is an extension of Markov decision processes, assuming that environmental parameters are bounded. In the BMDP model, environmental parameters are assumed to belong to a known finite interval, making BMDP more suitable for practical problems than MDP and helping people make more rational decisions. BMDP models are typically used to find optimal strategies that minimize or maximize the corresponding decision process's objective function. Applying BMDP models to performance bounds analysis of agent-based game strategies can help attackers choose attack actions more rationally, improving game performance and the intelligence level of game applications.

[0037] like Figure 1 As shown, the method of the present invention mainly includes the following steps:

[0038] Step 1: Obtain the actual historical game sequences and the strategies to be analyzed.

[0039] In this embodiment of the invention, the two opposing sides in the game are the attacker and the defender, each represented by a single agent. For the attacker, due to environmental influences, the state observed by the attacker is generally inaccurate. In other words, because the defender's state is not fully observable, taking the same action under the same observed state may result in the defender transitioning to a different state. Therefore, it is necessary to first obtain the actual historical game sequence. The actual historical game sequence is the data generated during the game between the attacker and the defender. Each side corresponds to an agent, and each entry in the actual historical game sequence is a quadruple in the Markov decision process described earlier (the defender's current state, the attacker's action, the defender's next state, and the immediate reward for whether the attack is effective).

[0040] Furthermore, the strategies to be analyzed can be directly read from the attacker.

[0041] In this embodiment of the invention, game-based competition is a practical application field encompassing various scenarios, such as game-based competition in gaming scenarios. Taking a gaming scenario as an example: the current state of the defender refers to the current state of the defender's controlled character in the game scenario; the attacker's action refers to the attack action output by the attacker (which corresponds to the actual control command in the game scenario); the defender's next state refers to the next state of the defender's controlled character in the game scenario; and the immediate reward for whether the attack is effective mainly assesses whether the attacker's action is reasonable.

[0042] Step 2: Based on the real historical game sequence, when the defender is in different observation states, adopt the strategy to be analyzed, obtain the state transition probability and state value function under the strategy to be analyzed, and obtain the bounded parameter Markov decision process model through state aggregation.

[0043] In this embodiment of the invention, before performing upper and lower bound iteration calculations, a BMDP model (Bounded Parameter Markov Decision Process Model) needs to be constructed based on the strategy and previous historical game sequences (real historical game sequences). Specifically: the action taken by the defender in a certain state is determined by the strategy to be analyzed, and thus the state transition probability p(s′|s,a) is calculated in the real historical game sequence, expressed as:

[0044]

[0045] Where p(s′|s,a) represents the probability that the defender's state will change to state s′ in the next moment after the attacker takes action a in the defender's current state s. Action a is the action decided by the attacker using the strategy to be analyzed in the defender's current state s. Num represents the number of times the same transition rule occurs in the historical game sequence. Num(s′|s,a) represents the number of times the defender's state will change to state s′ in the next moment after the attacker takes action a in the defender's current state s. Num(s,a) represents the number of times the attacker takes action a in the historical game sequence in the defender's current state s.

[0046] In this embodiment of the invention, the state value function can be calculated based on the state transition probability and the immediate reward for whether an attack is effective in the actual historical game sequence.

[0047] Then, the states are approximated by partially observable parameters, and similar states are aggregated into a state set q: {s1, s2, ..., s...} n Let q be the set of states, and n be the number of states in the state set. Similar states refer to states whose corresponding parameters belong to a set range. The state transition probabilities of each state in the state set q become an interval, and the corresponding state value functions also become an interval, forming a bounded parameter Markov decision process model. After state aggregation, the maximum value of the state value function in the state set is used as the upper bound of the state value function, and the minimum value is used as the lower bound, forming the state value function interval. Similarly, the maximum value of the state transition probability in the state set is used as the upper bound, and the minimum value is used as the lower bound, forming the state transition probability interval.

[0048] Those skilled in the art will understand that the state of an agent can be characterized by multiple parameters. When the attacker observes the defender, the parameters obtained are usually inaccurate, hence they are called partially observable parameters.

[0049] Those skilled in the art will understand that there are multiple ways to solve the state value function. For example, dynamic programming can be used to solve the state value function.

[0050] Step 3: In the bounded parameter Markov decision process model, update the upper and lower bounds of each state value function interval according to the upper and lower bound iteration algorithm, and use them as the upper and lower bounds of the overall performance of the strategy to be analyzed.

[0051] The preferred implementation method for this step is as follows:

[0052] 1) Construct a container O containing the set of all states (q1, ..., q) that the defender can reach in the next time step. n′The initial state sequence is obtained by arranging the lower bounds V of each state value function interval from smallest to largest; where (q1,…,q…) n′ Each item in ) represents a set of states, and n′ is the number of all state sets that the defender can reach in the next moment.

[0053] 2) If a set of states can transition to a set of states whose lower bound on the value function interval is smaller, then the minimization operation is used. Construct an MDP transition matrix with this transition, where the lower bound of the state-value function interval of this MDP transition matrix will be smaller than the lower bound of the original state-value function interval.

[0054] Those skilled in the art will understand that there is a concept of residual probability in BMDP. The probability of a state set transitioning to another state set is bounded. If it is possible to transition to three state sets, and the lower bounds of the corresponding three state value function intervals are different, then by allocating residual transition probabilities, the probability of transitioning to the state set with the smaller lower bound of the value function is increased, thereby constructing a new MDP transition matrix.

[0055] 3) By iterating through step 2), we will eventually obtain an MDP transition matrix with a lower bound of the minimum function interval. In essence, the remaining probability mass is distributed to the front state of the sorting, thereby obtaining the first state sequence.

[0056] After the above process, the first state in the updated first state sequence is the set of states with the smallest lower bound of the value function interval, which is the lower bound of the total performance.

[0057] In this embodiment of the invention, the remaining probability probability quality is an industry term. For example, the transition probability range of one state to another is [0.7, 0.9]. When considering the lower bound, the remaining probability 1 - 0.7 = 0.3 represents the probability of it transitioning to another state. When considering the upper bound, the remaining probability 1 - 0.9 = 0.1 represents the probability of it transitioning to another state.

[0058] 4) Using a method similar to steps 1) to 3) above, based on maximization operation The MDP transition matrix with the maximum function value is obtained through iteration. Finally, the front end of the second state sequence obtained by updating is the set of states with the largest upper bound of the state value function interval, which gives the upper bound of the total performance.

[0059] Step 4: Output the upper and lower bounds of the overall performance of the strategy to be analyzed.

[0060] The above method yields the upper and lower bounds of all state value functions, which can be considered as the overall performance upper and lower bounds of the strategies to be analyzed. For multiple strategies to be analyzed, the corresponding overall performance upper and lower bounds can be obtained using the same method. Based on the overall performance upper and lower bounds of each strategy, the optimal strategy can be evaluated. Assigning the optimal strategy to the attacker allows the attacker to choose attack actions more rationally, thereby improving game performance and the intelligence level of game applications. Considering the large number of states (i.e., a large value of n′), strategy evaluation can be performed using the average value or the upper and lower bounds of the value functions of a specified number of states. The specific method can be determined based on the actual situation and is not limited here.

[0061] The algorithm flow is shown in Table 1.

[0062] Table 1: Algorithm Flow

[0063]

[0064] In Table 1: The interval strategy evaluation algorithm corresponds to one upper and lower bound iteration process. Multiple iterations are required to obtain the final result, which will eventually converge. Rows 2 and 4 refer to sorting by the lower bound and upper bound, respectively. Whether the sorting is ascending or descending is determined by increasing and decreasing. V ↓ It is a vector containing n real numbers, giving states q1 to q... n′ The lower bound of the value function, V ↑ It is a vector containing n real numbers, giving states q1 to q... n The upper bound of the value function interval, Update(v,π,o) updates v using the maximum order MDP of o, the specific calculation steps are as follows: Figure 2 As shown, o is the state arrangement, a state vector (q1,…,q n′ (the arrangement), v refers to V ↓ With V ↑ A real vector of length n. This refers to the upper and lower bounds of the output state-value function, where π represents the strategy.

[0065] In this embodiment of the invention, the upper and lower bounds of the value function for each state are calculated and then summed to obtain the upper and lower bounds of the total performance.

[0066] Figure 2 The process of updating the state value function through the initial state value function, policy, and state ordering is presented. The main steps are described as follows:

[0067] Step 1) Create an n×n real matrix F′.

[0068] Step 2) For each state p in defender A, calculate the remaining probability mass remaining = 1 - used, where used = ∑ state q F ↓p,q (π(p)), where state represents traversing all states, π(p) represents the action taken by the attacker in state p, and F ↓p,q It represents the lower bound of the state transition probability interval from state p to state set q for the defender under action π(p).

[0069] Step 3) Update the remaining state for each state.

[0070] Step 3.1) For O, from q1 to q n For each state, calculate:

[0071] min = F ↓p,o(i) (π(p)), desired=F ↑p,o(i) (π(p)), where F ↓p,o(i) This represents the lower bound of the state transition probability from state p to the i-th state in the state set o under action π(p). Similarly, F ↑p,o(i) "Lì" represents the upper bound of the corresponding state transition probability, "min" represents the initial lower bound, and "desired" represents the initial upper bound. Here, the lower and upper bounds of the state transition probability refer to the lower and upper bounds of the state transition probability of the state set o. The state set o is all the states after aggregation, that is, the several states of the defender.

[0072] Step 3.2) If desired≦remaining, then F′(p,o(i))=min+desired, where F′(p,o(i)) is the redistributed state transition probability matrix, representing the state transition probability of the defender from state p to the i-th state in the state set o.

[0073] Step 3.3) If desired>remaining, then Fv(p,o(i))=min+remaining.

[0074] Step 3.4) remaining=max(0,remaining-desired)}.

[0075] Step 3.5) V ↓ (p)=R(p)+γ∑ state q F′(p,q)v(q), V ↓R(p), γ, and v(q) represent the lower bound of the state value function of state p, the immediate reward of state p, and the discount factor, respectively. F′(p,q) is the state transition probability of the defender from state p to state q after redistribution. v(q) represents the upper or lower bound of the state value function of the initial input state q. The upper bound is input when calculating the performance upper bound, and the lower bound is input when calculating the performance lower bound.

[0076] Based on the above introduction, an example is provided for illustration.

[0077] In game theory, performance bounds analysis of the obtained strategy is a crucial process. Assume two players, A and B (two agents), are competing against each other, with player A being the attacker and player B the defender, and player A's state is partially observable over player B's. This invention uses the following steps to obtain the upper and lower bounds of the performance of strategy A in this game.

[0078] 1) Obtain the real historical game sequence and player A's strategy π (the strategy to be analyzed).

[0079] 2) Obtain the BMDP model based on the method described in step 2 above.

[0080] 3) Perform the calculation based on the method described in step 3 above.

[0081] 4) Output the upper and lower bounds of the performance of each state value function after iteration.

[0082] A mapping relationship was established between the opponent behavior estimation bias and the upper and lower bounds of the adversarial strategy value function.

[0083]

[0084] Here, boundanalysis refers to performance limit analysis. R represents the partially observable state of the agent, and R represents the immediate reward value.

[0085] In multi-agent game systems, using these performance upper and lower bounds to evaluate strategies can help agents make better choices.

[0086] The solution provided in this invention modeles the unobservable portion of the opponent's state as a bounded parameter Markov decision process, which more realistically reflects the uncertainty in the game process than a typical Markov decision process, helping the agent make more informed decisions. By calculating the upper and lower bounds of performance using an iterative algorithm, a concrete analytical object is provided for performance analysis, allowing for better strategy analysis and selection of higher-performing strategies. This helps the attacker choose attack actions more rationally, improving game performance and the intelligence of game applications.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0088] Example 2

[0089] This invention also provides a performance limit analysis system for intelligent agent game strategies, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 3 As shown, the system mainly includes:

[0090] A performance boundary analysis system for agent game strategies includes:

[0091] The data receiving module is used to acquire real historical game sequences and the attacking side's strategy to be analyzed. Each entry in the real historical game sequence includes: the defender's current state, the attacker's action, the defender's next state, and the immediate reward for whether the attack is effective.

[0092] The BMDP model construction module is used to adopt the strategy to be analyzed based on real historical game sequences when the defender is in different observation states, obtain the state transition probabilities and state value functions under the strategy to be analyzed, and then obtain the bounded parameter Markov decision process model through state aggregation; the state transition probability interval construction unit and the state interval value function construction unit are used to construct the state transition probability interval and the state value function interval, respectively.

[0093] The performance bounds calculation module is used to update the upper and lower bounds of each state value function interval in a bounded parameter Markov decision process model according to the upper and lower bounds iteration algorithm (executed by the upper and lower bounds iteration algorithm unit), as the overall performance upper and lower bounds of the strategy to be analyzed.

[0094] The data sending module is used to output the upper and lower bounds of the overall performance of the strategy to be analyzed.

[0095] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0096] The technical details involved in each module of the system have been described in detail in the previous method embodiments, so they will not be repeated here.

[0097] Example 3

[0098] The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0099] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0100] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0101] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0102] The output device can be a display terminal;

[0103] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0104] Example 4

[0105] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0106] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0107] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for performance boundary analysis of intelligent agent game strategies, characterized in that, include: Obtain authentic historical game sequences and the attacker's strategy to be analyzed. Each entry in the authentic historical game sequence includes: the defender's current state, the attacker's action, the defender's next state, and the immediate reward for whether the attack is effective. When the game confrontation is a game confrontation in a game scenario, the defender's current state refers to the current state of the defender's controlled character in the game scenario, the attacker's action refers to the attack action output by the attacker, the defender's next state refers to the next state of the defender's controlled character in the game scenario, and the immediate reward for whether the attack is effective is used to evaluate whether the attacker's action is reasonable. Based on real historical game sequences, when the defender is in different observation states, the strategy to be analyzed is adopted to obtain the state transition probability and state value function under the strategy to be analyzed. Then, the bounded parameter Markov decision process model is obtained through state aggregation. In the bounded parameter Markov decision process model, the upper and lower bounds of each state value function interval are updated according to the upper and lower bound iteration algorithm, which serve as the upper and lower bounds of the overall performance of the strategy to be analyzed. Output the upper and lower bounds of the total performance of the strategy to be analyzed; when there are multiple strategies to be analyzed, perform performance boundary analysis on each strategy to obtain the upper and lower bounds of the total performance of each strategy to be analyzed; evaluate the best strategy based on the upper and lower bounds of the total performance of each strategy to be analyzed, and configure the best strategy on the attacker. Methods for obtaining the state transition probabilities under the policy to be analyzed include: By determining the opponent's action in a certain state through the strategy to be analyzed, the state transition probability p(s′|s,a) can be calculated in the actual historical game sequence, expressed as: Where p(s′|s,a) represents the probability that the defender's state will change to state s′ in the next moment after the attacker takes action a in the defender's current state s, and action a is the action decided by the attacker using the strategy to be analyzed in the defender's current state s. Num represents the number of times the same transition rule occurs in the historical game sequence. Num(s′|s,a) represents the number of times the defender's state will change to state s′ in the next moment after the attacker takes action a in the defender's current state s. Num(s,a) represents the number of times the attacker takes action a in the historical game sequence in the defender's current state s. One way to obtain the state-value function under the strategy to be analyzed is to calculate the state-value function based on the state transition probability and the immediate reward for whether an attack is effective in the actual historical game sequence.

2. The performance limit analysis method for an agent game strategy according to claim 1, characterized in that, The bounded parameter Markov decision process model obtained through state aggregation includes: Group similar states into a state set q: {s1, s2, ..., s} n Let q be the number of states in the state set, and n be the number of states in the state set. Similar states refer to states whose corresponding parameters belong to a set range. The state transition probabilities and state value functions of each state in the state set q each become an interval, forming a bounded parameter Markov decision process model. After state aggregation, the maximum value of the state value function in the state set is used as the upper bound of the state value function, and the minimum value is used as the lower bound, forming the state value function interval. The maximum value of the state transition probability in the state set is used as the upper bound, and the minimum value is used as the lower bound, forming the state transition probability interval.

3. The performance limit analysis method for an intelligent agent game strategy according to claim 1, characterized in that, In a bounded parameter Markov decision process model, the upper and lower bounds of each state value function are determined using an iterative algorithm, including: Construct a container O containing the set of all states that the defender can reach in the next time step (q1,...,q...). n′ The initial state sequence is obtained by arranging the lower bounds V of each state value function interval from smallest to largest; where (q1,…,q…) n′ Each item in ) represents a set of states, and n′ is the number of all state sets that the defender can reach in the next moment; If a set of states can transition to a set of states whose lower bound on the interval of the value function is smaller, then the minimization operation is performed. Construct an MDP transition matrix with this transition; through continuous iteration, an MDP transition matrix with a lower bound of the minimum value function interval will eventually be obtained. In essence, the remaining probability mass is distributed to the front state of the sorted state, thereby obtaining the first state sequence. The first state is the set of states with the minimum lower bound of the value function interval, thus obtaining the lower bound of the total performance. Based on maximization operation The MDP transition matrix with the maximum function value is obtained through iteration. Finally, the front end of the second state sequence obtained by updating is the set of states with the largest upper bound of the state value function interval, thus obtaining the upper bound of the total performance.

4. A performance limit analysis system for intelligent agent game strategies, characterized in that, To implement the method according to any one of claims 1 to 3, comprising: The data receiving module is used to acquire real historical game sequences and the attacking side's strategy to be analyzed. Each entry in the real historical game sequence includes: the defender's current state, the attacker's action, the defender's next state, and the immediate reward for whether the attack is effective. The BMDP model building module is used to adopt the strategy to be analyzed based on real historical game sequences when the defender is in different observation states, obtain the state transition probability and state value function under the strategy to be analyzed, and then obtain the bounded parameter Markov decision process model through state aggregation. The performance bounds calculation module is used to update the upper and lower bounds of each state value function interval in a bounded parameter Markov decision process model according to the upper and lower bounds iteration algorithm, which serve as the overall performance upper and lower bounds of the strategy to be analyzed. The data sending module is used to output the upper and lower bounds of the overall performance of the strategy to be analyzed.

5. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 3.

6. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 3.

Images