Large-Scale Policy Evaluation in Multi-Agent Systems
Through iterative evaluation and Markov chain optimization methods, the complexity of strategy evaluation in multi-agent environments is solved, and efficient strategy evaluation and ranking in large-scale systems are realized. It is suitable for applications such as autonomous vehicles, reducing computing and Storage requirements.
Patent Information
- Application Number
- CN201980098743.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-03
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-09-03
AI Technical Summary
The prior art is difficult to effectively evaluate the effectiveness of a single agent or overall joint strategy in a multiagent environment, especially in large-scale systems, where traditional methods such as Elo rating and Nash equilibrium have limitations and cannot handle non-transmission behaviors. and expand to large-scale populations.
By iteratively executing the steps in the policy evaluation system, selecting operational policy combinations, receiving value vectors represent the benefits of the corresponding combinations, relying on these values to estimate the ranking of the strategy combinations, and converge to the best ranking in the iterative update, using Markov chains and Augmented Lagrang Japanese stochastic optimization method, accessing only one row or column of the transfer matrix, reducing computational complexity and storage requirements.
It has realized efficient evaluation and ranking of agent strategies in large-scale multi-agent systems, significantly reducing time and space complexity, suitable for complex environments such as autonomous vehicles, and improving computing efficiency and memory usage efficiency.
Smart Images

Figure CN114207539B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to multi-agent systems and, in particular, to policy evaluation in such systems. Background Art
[0002] Many real-world applications involve modeling multi-agent systems, from selecting the strongest AI robot to compete against humans in video games to developing a population of safe self-driving cars that will never collide with each other.
[0003] These tasks usually involve different types of interactions between multiple agents involved. As multi-agent learning techniques become increasingly sophisticated, how to effectively evaluate the effectiveness of individual agents' strategies or overall joint strategies in such multi-agent environments becomes a key issue.
[0004] As described in the book "Ranking Chess Players, Past and Present" by Arpad E. Elo, Arco Pub., 1978, Elo rating is a common strategy evaluation method widely used to rank players in competitive games such as football, chess, and Go. Players with higher Elo ratings have a higher probability of winning a game than players with lower Elo ratings. If a player with a higher Elo rating wins, only a few points are taken from the players with lower ratings; however, if the player with lower ratings wins, many more points are taken from the players with higher ratings. However, the disadvantage of the Elo rating method is that it cannot handle non-transitive behavior in the game. For example, the periodic best response in the game of rock, paper, and scissors. In addition, in this game, if more agents playing scissors are added to the game, the agent playing rock will get a high ELO score; however, in reality, the probability of rock, paper, and scissors is the same no matter how many times rock is played.
[0005] Computing Nash equilibria is a classic solution concept for strategic evaluation in game theory, as described in “Re-evaluating Evaluation” by David Balduzzi, Karl Tuyls, Julien Perolat, and Thore Graepel, Advances in Neural Information Processing Systems, 2018, pp. 3268-3279, and in “Equilibria in n-person games” by John F. Nash et al., 1950, Proceedings of the National Academy of Sciences, vol. 36, no. 1, pp. 48-49. However, targeting Nash equilibria is problematic in many ways. For example, there is no guarantee that any dynamic game will converge to a Nash equilibrium. Furthermore, it is well known that solving Nash equilibria in general and randomized games is a PPAD-complete problem, which severely limits the complexity of multi-agent systems that can be evaluated. It is not clear how to select an equilibrium among the many Nash equilibria of a game.
[0006] Reproduction dynamics from evolutionary game theory are often deployed to capture the microdynamics of interacting agents in multi-agent systems, as described by Peter Schuster and Karl Sigmund in "Reproduction Dynamics" published in Journal of Theoretical Biology, Vol. 100, No. 3, pp. 533-538 in 1983. These approaches focus on studying basins of attraction and equilibria where the strategies of the agents are evaluated. However, these approaches are limited in that they can only be feasibly applied to games involving a small number of agents. Furthermore, the approaches cannot handle games with asymmetric payoffs.
[0007] As described in “α-Rank: Evolutionary multi-agent evaluation” by Shayegan Omidshafiei, Christos Papadimitriou, Georgios Piliouras, Karl Tuyls, Mark Rowland, Jean-Baptiste Lespiau, Wojciech M Czarnecki, Marc Lanctot, Julien Perolat, and Remi Munos, published in Nature Scientific Reports in 2019, α-Rank enables evolutionary analysis of multi-agent interactions in asymmetric games by introducing Markov chains. However, the method runs in polynomial time and space with respect to the total number of pure strategy combinations, which makes it impractical for real-world applications. When designing a self-driving car solution for a community of ten cars, each of which has only two strategies to choose from, the time complexity of finding the best joint strategy using α-Rank would be This is intolerable. Therefore, the method cannot be extended to solve multi-agent problems with large population sizes.
[0008] Whether in a simulation engine or in a real environment, the inherent dynamic behavior of the multi-agent policy evaluation problem needs to be modeled in a scalable way to solve large-scale policy evaluation problems. Summary of the invention
[0009] According to a first aspect, a computer-implemented strategy evaluation system is provided for identifying an optimal combination of operational strategies implemented by multiple strategy-driven participants, each participant being able to adopt any one of multiple operational strategies, wherein the strategy evaluation system is used to iteratively perform the following steps: (i) selecting a first combination of operational strategies, the first combination defining a strategy for each participant; (ii) receiving a value vector, each value vector representing the benefit of a corresponding second combination, wherein all of the participants except one participant adopt the strategy defined for them in the first combination, and the participants adopt different strategies; and (iii) estimating a ranking of the participants' strategy combinations based on these values, as well as a previously estimated ranking of the participants' strategy combinations.
[0010] The value set may be a row or column of a transition matrix of the joint strategy combinations of all the participants. This may allow the system to iteratively converge to a ranking of all possible strategy combinations that is optimal within a predetermined threshold without listing all of these possible combinations.
[0011] The iteratively performed steps may include: (iv) evaluating whether the rankings of the strategy combinations of the participants have converged to a predetermined degree; if the evaluation is positive, assigning to each of the participants the strategy defined for it in the highest ranked combination in the ranking. Once the system has iteratively converged on an appropriate solution, the solution may be advantageously adopted by each participant, each participant implementing the corresponding strategy indicated for it in the highest ranked combination of the solution.
[0012] The step of estimating the ranking of the strategy combinations of the participants may include: estimating an adjustment from the previously estimated ranking of the strategy combinations of the participants; and applying the adjustment to the previously estimated ranking of the strategy combinations of the participants. This may help the system to obtain a final solution in an iterative manner, thereby potentially improving efficiency.
[0013] The system may be used to calculate each of the values as:
[0014]
[0015] in:
[0016]
[0017] and:
[0018] α is the ranking strength;
[0019] k is the index of the first agent;
[0020] M (k) (si k ,s i -k ) is the use strategy s i k The fitness / reward of agent k, while other agents use strategy s i -k ;
[0021] m is a hyperparameter representing the population size;
[0022] S k is the kth joint combination.
[0023] The system may be used to estimate the ranking of the strategy combinations of the participants as:
[0024]
[0025]
[0026] in:
[0027] b is the value vector;
[0028] 1 is a vector full of 1s;
[0029] x is the ranking to be output;
[0030] eta is the learning rate;
[0031] Epsilon is the precision parameter.
[0032] The system can be used to store the set of values in a compressed form, where: (i) the number of the most common ones of the values; (ii) the index and number of each of the values that is not equal to the most common one of the values. This can reduce memory consumption.
[0033] The number of each of the values that is not equal to the most common one of the values may be assigned as the difference between the number and the number of the most common one of the values. This may reduce memory consumption.
[0034] In each iteration, it will be appreciated that the only values estimated depending on the preferred strategy combinations of the participants are the value sets, each value set representing the benefit of a corresponding second combination, where all but one of the participants adopt the strategy defined for them in the first combination, and where the participants adopt different strategies. This can simplify the calculation process.
[0035] In each iteration, it can be understood that the only value estimated depending on the preferred strategy combination of the participant is the value of the corresponding row or column of the transition matrix of the joint strategy combination of the participant. This can simplify the calculation process.
[0036] The participants may be policy-driven participants. The participants may be at least partially autonomous vehicles, and the policy may be a driving policy. M may indicate a representation of the fitness of a combined driving policy combination. The participants may be able to interact with each other when executing the policy. In this way, the top-ranked combination may enable the participants to operate relatively efficiently as a group.
[0037] The step of selecting the participants can be performed randomly. This can effectively find the best combination.
[0038] According to a second aspect, a computer-implemented method is provided for identifying an optimal combination of operating strategies implemented by multiple participants, each participant being able to adopt any one of a plurality of operating strategies, wherein the method comprises iteratively performing the following steps: (i) selecting a first combination of operating strategies, the first combination defining a strategy for each participant; (ii) receiving a value vector, each value vector representing the benefit of a corresponding second combination, wherein all of the participants except one participant adopt the strategy defined for them in the first combination, and the participants adopt different strategies; and (iii) estimating a ranking of the participants' strategy combinations based on these values, as well as a previously estimated ranking of the participants' strategy combinations.
[0039] According to a third aspect, there is provided a data carrier storing a set of instructions in a non-transitory form, wherein the set of instructions is used to cause a computer to execute the method according to the preceding paragraph. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The present invention will now be described by way of example with reference to the accompanying drawings.
[0041] In the attached picture:
[0042] Figure 1 An overview of the inventive method for determining the ranking of joint strategies compared to conventional methods is shown;
[0043] Figure 2 The random optimization process (Algorithm 1) is shown;
[0044] Figure 3 An example of an algorithm summarizing the entire evolutionary strategy evaluation method of the present invention is shown (Algorithm 2);
[0045] Figure 4An example of a computer-implemented method for identifying an optimal combination of operating strategies implemented by a plurality of participants is shown;
[0046] Figures 5 to 8 An exemplary flow chart illustrating the application of the methods described herein;
[0047] Fig. 9 The calculation based on the starting joint strategy combination and the ending joint strategy combination is shown. Figure 7 701 and Figure 8 A more detailed outline of the steps performed to fix the probability shown at 801 in FIG.
[0048] Fig.10 It summarizes the steps that are typically performed when running a simulator;
[0049] Fig.11 A data structure for storing a sparse vector utilizing the sparse property of a transition probability matrix C is shown;
[0050] Fig.12 shows a comparison of time and space complexity for an example of a population of autonomous vehicles, where N = number of cars, K = number of strategies per car,
[0051] Fig.13 The results of comparative experiments showing the time complexity of the method of the present invention are shown;
[0052] Fig.14 The results of comparative experiments showing the spatial complexity of the method of the present invention are shown;
[0053] Fig.15 An example of a policy evaluation system is shown. DETAILED DESCRIPTION
[0054] The present invention relates to an evolutionary principled method for evaluating and ranking the strategies of agents in a large-scale multi-agent setting. The method ranks the joint strategies of agents supported by evolutionary dynamics by screening out transient strategies that will disappear in long-term evolutionary interactions.
[0055] Figure 1 An overview of the method of the present invention, shown generally at 101, is shown compared to a conventional method, shown generally at 102. The method of the present invention and how it differs from conventional methods will now be described.
[0056] Assume there is a community of agents And each agent i has a set of strategies / policies that it can apply For example, in the case of self-driving cars, each driving strategy can be thought of as a policy, defining the agent’s actions on the road.
[0057] Without losing generality, assume that the total number of driving strategies of different agents is K i Can be different. For convenience, K i =K j ,(i,j∈{1,…,N}), assuming they are the same.
[0058] Each agent has a payoff function In other words, the payoff function for each agent i depends not only on the strategy chosen for agent i, but also on the strategies of all other agents.
[0059] Assume that each agent i chooses strategy s iq ∈S i ,Will Recorded as a joint strategy combination, and using represents the corresponding payment, where is the strategic combination of all agents except agent i.
[0060] The goal is to provide a ranking algorithm that can rank the set of all possible joint strategy combinations under evaluation and provide insights into these strategies, including their strengths, weaknesses, and, in an evolutionary sense, their survival dynamics.
[0061] For each agent i, create a population It contains m copies of agent i. The strategy evaluation game can basically be expressed as a Markov chain, where the state space is given as the set of all joint strategy combinations, that is, Transition probability matrix The definition is as follows, which is applicable to any s i ,s j ∈S MC :
[0062]
[0063] in:
[0064]
[0065] is the competitive strategy of the intelligent agent k Other intelligent agents compete Fitness / reward of . In equation (2), α is defined as the ranking strength, which is a hyperparameter of the algorithm.
[0066] The constructed Markov chain is irreducible and ergodic, so in the joint strategy combination set There is a limiting distribution on :
[0067] v=lim t→∞ v0C t (3)
[0068] Does not depend on the initial distribution v0. v is also a stationary distribution.
[0069] The stationary distribution v of each agent shows the average time the Markov chain spends in the corresponding state. Therefore, in order to select the most desirable joint strategy combination, s * The definition is as follows:
[0070]
[0071] Furthermore, the stationary distribution π allows for a single strategy s for each agent i iq The ranking is as follows. For each strategy s iq ∈S i , calculate the values of:
[0072]
[0073] Based on this value, all strategies s of agent i can be iq ∈S i Sort.
[0074] Calculating the corresponding stationary distribution v of the transition probability matrix C can be transformed into a convex constrained optimization problem:
[0075]
[0076] in,
[0077] Figure 1 102, conventional methods require pre-calculation and storage of the entire transfer matrix, which may be very large. Figure 1 It is generally shown at 101 that by adopting the augmented Lagrangian stochastic optimization method to calculate the stationary distribution, only one column (or row) of the transfer matrix is accessed in each iteration, and the number of iterations required is much less than that of the traditional linear algebra method or the power method.
[0078] Specifically, the above problem is reformulated as an unconstrained problem:
[0079]
[0080] Here, the following relaxation of the equality constraints is used:
[0081] x T 1-1+∈≥0&1+∈-x T 1≥0 (8)
[0082] For some precision parameters, ∈ > 0. In equation (7), α is a weighting parameter.
[0083] Then, an iterative update is performed. During the iterative update, the parameters ∈ and α are gradually reduced.
[0084] The above problem can be further transformed into a finite sum minimization problem. i Represents the matrix C T -i's i-th line, then:
[0085]
[0086] like Figure 1 As shown at 103 in FIG. 1 , a random optimization process is used as an iterative method to solve the above optimization problem. The process is summarized by Algorithm 1, as shown in FIG. Figure 2 shown.
[0087] Thus, a first combination of strategies is selected, the first combination defining a strategy for each participant (e.g., each car in a population of autonomous vehicles). Value vectors are received, each value vector representing a benefit of a corresponding second combination of operating strategies, wherein all but one of the participants adopt the strategies defined for them in the first combination, and the participants adopt different strategies. A ranking of the strategy combinations of the participants is estimated in dependence on the values, as well as a ranking of the strategy combinations of the participants that was previously estimated. The ranking is iteratively updated by estimating an adjustment from the ranking of the previously estimated strategy combination; and the adjustment is applied to the ranking of the previously estimated strategy combination.
[0088] However, in previous methods, the entire transition matrix needs to be pre-computed and stored; in the current method, the value set is the row or column of the transition matrix of the joint strategy combination, that is, only one row or column of the matrix is needed in each iteration.
[0089] In a preferred embodiment, at each iteration, it is understood that the only value estimated depending on the preferred strategy combination of the participants is the set of values, each value set representing the benefit of a corresponding second combination, wherein all of the participants except one adopt the strategy defined for them in the first combination, and the participants adopt different strategies. Thus, the algorithm can select all possible joint strategy combinations different from one agent.
[0090] The result of this method is the optimal stationary distribution π * , that is, the optimal ranking of joint strategies, such as Figure 1 It is generally shown at 104 .
[0091] The entire evolution strategy evaluation method is summarized by Algorithm 2, as Figure 3 shown.
[0092] Instead of determining each agent's strategy at the start of the game, each agent's strategy can be augmented and learned during the game. The entire transition matrix does not need to be precomputed and stored. The method can use rows or columns of the transition matrix in each iteration. This can allow the system to iteratively converge on the best ranking of possible strategy combinations.
[0093] When it is detected that the ranking of the strategy combinations of the participants has converged to a predetermined degree, that is, π=π * , each of the participants may be assigned the strategy defined for it in the highest ranking combination in the ranking.
[0094] Figure 4 A computer-implemented method is summarized for identifying an optimal combination of operating strategies implemented by a plurality of participants, each participant being able to adopt any one of a plurality of operating strategies. The method comprises iteratively performing the following steps. In step 401, the method comprises: selecting a first combination of operating strategies, the first combination defining a strategy for each participant. In step 402, the method comprises: receiving a value vector, each value vector representing a benefit of a corresponding second strategy combination, wherein all of the participants except one participant adopt the strategy defined for them in the first combination, and the participants adopt different strategies. In step 403, the method comprises: estimating a ranking of the participants' strategy combinations in dependence on the values, as well as a previously estimated ranking of the participants' strategy combinations.
[0095] Figures 5 to 8 A detailed system diagram showing how the proposed method can be applied.
[0096] Figure 5 The system input is shown at 501. At 502 an initial value of π is set. Figures 6 to 8 The iterative steps performed as described above are shown.
[0097] Fig. 9 The calculation based on the starting joint strategy combination and the ending joint strategy combination is shown. Figure 7 701 and Figure 8 A more detailed outline of the steps performed to fix the probability shown at 801 in FIG.
[0098] The general framework can solve the policy evaluation problem for any multi-agent system. Therefore, the method can be deployed in any accessible multi-agent environment or equivalent simulation engine that can take as input the joint strategy of the agents and output the fitness / reward value of each agent. Given such an environment, by following Figures 5 to 8 The calculation process of the system diagram shown in can obtain the final ranking of all joint strategy combinations as output. Therefore, the algorithm (Algorithm 2, as shown in Figure 3 As shown) the stationary distribution π can be calculated * , that is, the ranking of joint strategies, such as Figure 1 Shown generally.
[0099] Fig.10 The steps that are usually performed when the simulator is running are summarized. At 1001, a joint strategy combination is obtained from the agents. At 1002, the above method is executed. At 1003, a reward for each agent is obtained.
[0100] The transition probability matrix is usually sparse, that is, it contains many identical values. In the present invention, the implementation of the algorithm also introduces a data structure for storing sparse vectors that fully utilizes the sparse properties of the transition probability matrix C. Fig.11 In this data structure, vectors (i.e., rows or columns of a matrix) are stored in a compressed form, which specifies: (i) the number of most common values in the vector (in Fig.11 (ii) the index and number of each value in the vector that is not equal to the most common value. The number of each of the values that is not equal to the most common value can be specified as the difference between the number and the number of the most common value. For example, Fig.11 In , for values of 0.7 and 0.4, the differences are 0.6 and 0.3 respectively. Representing sparse vectors in this way can reduce memory consumption.
[0101] Therefore, in calculating the stationary distribution π * This policy evaluation method can significantly reduce the amount of computation in time and space when the stationary distribution is essentially the desired ranking. In particular, the method described in this paper does not require the pre-computation and storage of the entire transition matrix, which can be very large. By adopting an augmented Lagrangian stochastic optimization method, only one column of the transition matrix is accessed in each iteration. The number of iterations required is also much smaller than traditional linear algebra methods or power methods. Therefore, the method is directly applicable to policy evaluation of large-scale multi-agent systems.
[0102] As mentioned above, in one example, the participants are self-driving cars and the strategies are driving strategies. In this example, M in equation (2) indicates a representation of the fitness of the combined driving strategy combination. The cars may be able to interact with each other when executing the strategies. Thus, the highest ranked strategy combination may enable the cars to operate relatively efficiently and safely as a group.
[0103] Fig.12 shows a comparison of time and space complexity for an example of self-driving cars, where N = number of cars, K = number of strategies per car,
[0104] Fig.13 and Fig.14 The results of comparative experiments on the time and space complexity of the method of the present invention are shown respectively.
[0105] Fig.13 It is shown that compared to traditional linear algebra methods using the numerical algorithm libraries Python shown at 1302, Numpy shown at 1303, and Scipy shown at 1304, the method described in this article (as shown by the line shown at 1301) can achieve much faster speeds (up to about 1000 times).
[0106] Fig.14 It is shown that the method described herein (as shown by the line shown at 1401) can significantly reduce memory consumption compared to the traditional linear algebra method using the numerical algorithm library Python shown at 1402, Numpy shown at 1403, and Scipy shown at 1404. Memory consumption is reduced because repeated values are not saved.
[0107] Fig.15 A schematic diagram of a policy evaluation system 1500 is shown, which is used to implement the above method and its associated components. The system may include a processor 1501 and a non-volatile memory 1502. The system may include multiple processors and multiple memories. The memory may store data that can be executed by the processor. The processor may be used to operate according to a computer program stored in a non-transient form on a machine-readable storage medium. The computer program may store instructions for causing the processor to perform its method in the manner described herein.
[0108] In summary, this paper describes an evolutionary dynamics approach for evaluating the policies of agents in large-scale multi-agent interactions. The approach is able to model the inherent dynamical behavior of multi-agent policy evaluation problems, such as a population of self-driving cars. Unlike traditional methods that suffer from conditional issues when the total number of joint strategy combinations is large, the proposed algorithm can be extended to solve large-scale multi-agent policy evaluation problems.
[0109] The method randomly samples the transfer matrix to fully utilize the sparsity of the ranking algorithm. Therefore, the method can be applied to large-scale intelligent agent populations, shorten the calculation time, and improve the efficiency of memory usage. Compared with traditional numerical algorithm libraries such as Python, Numpy and Scipy and traditional methods such as the power method in PageRank, the implementation of the present invention can increase the speed by 1000 times and significantly reduce memory consumption.
[0110] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features. With the common knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, without considering whether such features or combinations of features can solve any problem disclosed herein; and without causing any impact on the scope of the claims. This application shows that various aspects of the present invention can be composed of any such individual features or combinations of features. In view of the foregoing description, it will be obvious to those skilled in the art that various modifications can be made within the scope of the present invention.
Claims
1. A computer-implemented strategy evaluation system for identifying an optimal combination of operational strategies implemented by a plurality of participants, each participant being able to adopt any one of a plurality of operational strategies, characterized in that: The policy evaluation system is used to iteratively perform the following steps: (i) selecting a first combination of operating strategies, the first combination defining a strategy for each participant; (ii) calculating value vectors based on the strategies of the participants, each value vector representing the payoff of the corresponding second combination, wherein all the participants except one participant adopt the strategy defined for them in the first combination, and the participants adopt different strategies, and the value vectors satisfy: in: and: α is the ranking strength; k is the index of the first participant; M (k) (s i k ,s i -k ) is the strategy s used by participant k i k , while other participants use strategy s i -k Fitness / reward at time m is a hyperparameter representing the population size; S k is the kth joint combination; b i is a vector of values; s j k 、s i k is the strategy used by player k s i -k is the strategy used by other participants other than k; s i ,s j ∈S MC ,in, M (k) (s j k ,s i -k ) is the strategy s used by participant k j k , while other participants use strategy s i -k Fitness / reward at time (iii) estimating the ranking of the strategy combinations of the participants depending on the value vector; (iv) updating the ranking of the strategy combinations of the participants according to the value vector and the ranking of the strategy combinations of the participants estimated previously; (v) evaluating whether the rankings of the strategy combinations of the participants have converged to a predetermined degree; If the evaluation is positive, each of the participants is assigned the strategy defined for it in the combination that ranks highest in the ranking.
2. The computer-implemented policy evaluation system of claim 1, wherein: The value vector is a row or a column of a transition matrix of the joint strategy combination of all the participants.
3. A computer-implemented policy evaluation system according to any one of the preceding claims, characterized in that: The step of estimating the ranking of the combination of strategies of the participants includes: estimating an adjustment from the previously estimated ranking of the combination of strategies of the participants; and applying the adjustment to the previously estimated ranking of the combination of strategies of the participants.
4. A computer-implemented policy evaluation system according to claim 1 or 2, characterized in that: The ranking of the strategy combinations of the participants is updated according to the value vector and the previously estimated ranking of the strategy combinations of the participants, satisfying: in: and: is the value vector in the t-th estimation calculation; 1 is a vector full of 1s; x t is the ranking of the output of the t-th estimation calculation; x t+1 is the ranking of the output of the t+1th estimation calculation; For x t The transposed matrix of η t is the learning rate for the t-th estimation calculation; ∈ t is the precision parameter of the t-th estimation calculation; α t is the ranking strength calculated by the tth estimation.
5. The computer-implemented policy evaluation system of claim 1, wherein: Used to store the value vector in a compressed form, wherein: (i) the number of the most common ones in the value vector; (ii) the index and number of each of the value vectors that are not equal to the most common one in the value vector are specified.
6. The computer-implemented policy evaluation system of claim 5, wherein: The number of each of the value vectors that is not equal to the most common one of the value vectors is assigned as the difference between the number and the number of the most common one of the value vectors.
7. The computer-implemented policy evaluation system of claim 2, wherein: In each iteration, the only value estimated depending on the preferred strategy combination of the participants is the value vector, each value vector representing the payoff of a corresponding second combination, wherein all but one of the participants adopt the strategy defined for them in the first combination and wherein the participants adopt different strategies.
8. The computer-implemented policy evaluation system of claim 7, wherein: In each iteration, the only value that depends on the estimation of the preferred strategy combination of the participant is the value of the corresponding row or column of the transition matrix of the joint strategy combination of the participant.
9. The computer-implemented policy evaluation system of claim 1, wherein: The participant is an at least partially autonomous vehicle and the policy is a driving policy.
10. The computer-implemented policy evaluation system of claim 4, wherein: The M (k) (s j k ,s i -k ) indicates the fitness of the combined driving strategy.
11. The computer-implemented policy evaluation system of claim 1, wherein: The step of selecting participants was performed randomly.
12. A computer-implemented method for identifying an optimal combination of operating strategies implemented by a plurality of participants, each participant being able to adopt any one of a plurality of operating strategies, characterized in that: The method comprises iteratively performing the following steps: (i) selecting a first combination of operating strategies, the first combination defining a strategy for each participant; (ii) calculating value vectors based on the strategies of the participants, each value vector representing the payoff of the corresponding second combination, wherein all the participants except one participant adopt the strategy defined for them in the first combination, and the participants adopt different strategies, and the value vectors satisfy: in: and: α is the ranking strength; k is the index of the first participant; M (k) (s i k ,s i -k ) is the strategy s used by participant k i k , while other participants use strategy s i -k Fitness / reward at time m is a hyperparameter representing the population size; S k is the kth joint combination; b i is a vector of values; s j k 、s i k is the strategy used by player k s i -k is the strategy used by other participants other than k; s i ,s j ∈S MC ,in, M (k) (s j k ,s i -k ) is the strategy s used by participant k j k , while other participants use strategy s i -k Fitness / reward at time (iii) estimating the ranking of the strategy combinations of the participants depending on the value vector; (iv) updating the ranking of the strategy combinations of the participants according to the value vector and the ranking of the strategy combinations of the participants estimated previously; (v) evaluating whether the rankings of the strategy combinations of the participants have converged to a predetermined degree; If the evaluation is positive, each of the participants is assigned the strategy defined for it in the combination that ranks highest in the ranking.
13. A data carrier storing a set of instructions in a non-transient form, characterized in that: The set of instructions is used to cause a computer to perform the method according to claim 12.
