Strategy search method, device and equipment for multi-participant resource allocation and medium

Abstract

The invention belongs to the technical field of strategy search, and relates to a strategy search method and device for multi-participant resource allocation, equipment and a medium. The method comprises the following steps: acquiring a multi-participant resource allocation scene and modeling a Markov model to obtain an expected function value vector of a state; constructing approximate Nash equilibrium to serve as a constraint condition of an expected function value vector of the state; establishing an updating rule of the strategy in a time step, and establishing a utility function of a participant in the strategy to obtain an iteration mode of the strategy; an iteration mode of the strategy is simplified and solved, and an equilibrium solution is obtained; according to the equilibrium solution, calculating a target value function of each participant, and calculating a response of approximate Nash equilibrium; obtaining a state value function by combining the expected function value vector of the state; and taking minimization of the loss function as a target, carrying out iterative search on the strategy until stopping, and outputting the current strategy as an optimal solution of the Markov model. According to the invention, the strategy can be accurately and quickly searched.

Description

Technical Field

[0001] The present application relates to the field of strategy search technology, and in particular to a strategy search method, device, equipment and medium for multi-participant resource allocation. Background Art

[0002] In multiplayer game scenarios, due to the non-stationarity of the environment (opponents), the equilibrium selection problem faced by multiple equilibrium solutions makes the solution concept and the game learning process show a state of "offline coupling and online decoupling". According to the stage of strategy solution, in the offline training process, a simulation environment can be used to simulate the interaction with the opponent, and the sampled data contains the coupling state of the enemy and us. The "blueprint" strategy is obtained by large-scale computing or the pre-training model is obtained by reinforcement learning. In the online game process, since only one's own strategy can be controlled, the generation of the strategy is in a decoupled state, and an adaptive counter-strategy needs to be adopted to deal with the opponent.

[0003] In the existing technology, faced with the challenge of incomplete (imperfect) information, the existing technology suppresses the non-transitive nature of strategies, which leads to difficulties in exploration and takes a long time to solve. The relevant technologies that use optimization theory to solve the exact solution have high computational complexity and poor effectiveness. The approximate solution obtained by the reinforcement learning method cannot adapt well to the generation of online countermeasure strategies. Summary of the invention

[0004] Based on this, it is necessary to provide a multi-participant resource allocation strategy search method, device, equipment and medium to address the above technical problems, which can accurately and quickly search for multi-participant resource allocation strategies.

[0005] A strategy search method for multi-participant resource allocation, including: Obtain multi-participant resource allocation scenarios, including: competitive resource allocation scenarios of multiple participants around equipment capital investment, multiple merchants investing in multiple types of products to occupy the market, or multiple campaign teams investing in elections in multiple regions to win public support, and model the multi-participant resource allocation scenarios to obtain a Markov model; according to the Markov model, obtain the expected function value vector of the state; Construct a Nash equilibrium strategy and use a neural network to optimize it to obtain an approximate Nash equilibrium as a constraint on the expected function value vector of the state; The regularized leader-follower method is used to establish the update rules of the strategy at the time step; the imitation learning method based on historical interaction data is used to generate the anchor strategy and establish the utility function of the participants in the strategy; according to the update rules and utility function, the iteration method of the strategy is obtained; the iteration method of the strategy is simplified and solved by the convex conjugate method to obtain the equilibrium solution; According to the equilibrium solution, the objective value function of each participant is calculated; according to the objective value function, the response of the approximate Nash equilibrium is calculated; according to the response of the approximate Nash equilibrium and the expected function value vector of the state, the state value function is obtained; according to the state value function, a loss function is established, and with the goal of minimizing the loss function, the strategy is iteratively searched until the preset conditions are met, and the iteration is stopped, and the current strategy is output as the optimal solution of the Markov model.

[0006] In one embodiment, a regularized leader-following method is used to establish an update rule for the strategy at a time step, including:

[0007] In the formula, for time step variable, is a first-order strong regularization function, is the learning rate for online convex optimization, for Time step loss.

[0008] In one embodiment, an imitation learning method based on historical interaction data is used to generate an anchoring strategy and establish a utility function of the participant in the strategy, including:

[0009] In the formula, For participants In Strategy The utility function of is the original reward function, is the regulating factor, is the divergence, For participants strategy, For the exclusion of participants Other people’s strategies, Anchoring strategy.

[0010] In one embodiment, according to the update rule and the utility function, an iterative method of the strategy is obtained, including:

[0011] In the formula, For participants exist Time step strategy, is the strategy space, For strategy Information entropy.

[0012] In one embodiment, the objective value function of each participant is calculated according to the equilibrium solution, including:

[0013] In the formula, is the target value function, is the learning rate of reinforcement learning, Status Take action The rewards you get when is the discount factor, To take action The probability of Status Take action The corresponding state value function. In one embodiment, according to the objective value function, calculating the response of the approximate Nash equilibrium includes:

[0014] In the formula, is the response approximating the Nash equilibrium, An action taken by a participant, is the action vector taken by the participant, The state-action value function corresponding to the action taken by the participant, For the exclusion of participants The combined strategy of all others.

[0015] In one embodiment, establishing a loss function includes:

[0016] In the formula, is the state loss function, Status Value function, For status rewards, To use hyperparameters Status The value function of is the strategy loss function, For participants Actions, For participants Adopt an action The probability of For participants In Status Take action strategy.

[0017] A strategy search device for multi-participant resource allocation, comprising: The acquisition module is used to acquire multi-participant resource allocation scenarios, which include: competitive resource allocation scenarios of multiple participants focusing on equipment capital investment, multiple merchants investing capital around multiple types of products to occupy the market, or multiple campaign teams investing election funds in multiple regions to win public support, and modeling the multi-participant resource allocation scenarios to obtain a Markov model; according to the Markov model, the expected function value vector of the state is obtained; The equilibrium module is used to construct the Nash equilibrium strategy and optimize it using a neural network to obtain an approximate Nash equilibrium as a constraint condition for the expected function value vector of the state; The solution module is used to establish the update rules of the strategy at the time step by using the regularized leader-follower method; to generate the anchor strategy by using the imitation learning method based on the historical interaction data, and to establish the utility function of the participants in the strategy; to obtain the iteration method of the strategy according to the update rules and the utility function; to simplify the iteration method of the strategy, and to solve it by using the convex conjugate method to obtain the equilibrium solution; The output module is used to calculate the objective value function of each participant according to the equilibrium solution; calculate the approximate Nash equilibrium response according to the objective value function; obtain the state value function according to the approximate Nash equilibrium response and the expected function value vector of the state; establish the loss function according to the state value function, and iteratively search the strategy with the goal of minimizing the loss function until the iteration is stopped when the preset conditions are met, and output the current strategy as the optimal solution of the Markov model.

[0018] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented: Obtain multi-participant resource allocation scenarios, including: competitive resource allocation scenarios of multiple participants around equipment capital investment, multiple merchants investing in multiple types of products to occupy the market, or multiple campaign teams investing in elections in multiple regions to win public support, and model the multi-participant resource allocation scenarios to obtain a Markov model; according to the Markov model, obtain the expected function value vector of the state; Construct a Nash equilibrium strategy and use a neural network to optimize it to obtain an approximate Nash equilibrium as a constraint on the expected function value vector of the state; The regularized leader-follower method is used to establish the update rules of the strategy at the time step; the imitation learning method based on historical interaction data is used to generate the anchor strategy and establish the utility function of the participants in the strategy; according to the update rules and utility function, the iteration method of the strategy is obtained; the iteration method of the strategy is simplified and solved by the convex conjugate method to obtain the equilibrium solution; According to the equilibrium solution, the objective value function of each participant is calculated; according to the objective value function, the response of the approximate Nash equilibrium is calculated; according to the response of the approximate Nash equilibrium and the expected function value vector of the state, the state value function is obtained; according to the state value function, a loss function is established, and with the goal of minimizing the loss function, the strategy is iteratively searched until the preset conditions are met, and the iteration is stopped, and the current strategy is output as the optimal solution of the Markov model.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps: Obtain multi-participant resource allocation scenarios, including: competitive resource allocation scenarios of multiple participants around equipment capital investment, multiple merchants investing in multiple types of products to occupy the market, or multiple campaign teams investing in elections in multiple regions to win public support, and model the multi-participant resource allocation scenarios to obtain a Markov model; according to the Markov model, obtain the expected function value vector of the state; Construct a Nash equilibrium strategy and use a neural network to optimize it to obtain an approximate Nash equilibrium as a constraint on the expected function value vector of the state; The regularized leader-follower method is used to establish the update rules of the strategy at the time step; the imitation learning method based on historical interaction data is used to generate the anchor strategy and establish the utility function of the participants in the strategy; according to the update rules and utility function, the iteration method of the strategy is obtained; the iteration method of the strategy is simplified and solved by the convex conjugate method to obtain the equilibrium solution; According to the equilibrium solution, the objective value function of each participant is calculated; according to the objective value function, the response of the approximate Nash equilibrium is calculated; according to the response of the approximate Nash equilibrium and the expected function value vector of the state, the state value function is obtained; according to the state value function, a loss function is established, and with the goal of minimizing the loss function, the strategy is iteratively searched until the preset conditions are met, and the iteration is stopped, and the current strategy is output as the optimal solution of the Markov model.

[0020] The above-mentioned multi-participant resource allocation strategy search method, device, equipment and medium focus on offline blueprint strategy learning and training and online adaptation and counter-strategy search, and provide a multi-participant resource allocation game strategy search framework based on maximum and minimum entropy. In the face of the challenge of incompletely knowable (imperfect) information, in order to solve the exploration problem caused by the non-transitive suppression of strategies, based on the regularization theory in online convex optimization, and focusing on the needs of iterative strategy rapid solution, a Nash equilibrium strategy iterative solution method based on maximum and minimum entropy is proposed; first, based on the anchoring strategy, a Nash equilibrium strategy solution method guided by maximum and minimum entropy regularization is designed; then, based on the blueprint strategy training, based on Nash Q learning and dual oracles, a maximum and minimum Nash oracle strategy learning framework is designed using Nash value iteration, which effectively handles the uncertainty challenge brought by the opponent with imperfect information. In addition, the effectiveness of the proposed method is verified on the "Diplomacy" board game platform, and the practicality of the method is empirically analyzed using the "Brotto Game". BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 An application scenario diagram of a strategy search method for multi-participant resource allocation in one embodiment; Figure 2 A schematic diagram of a flow chart of a strategy search method for multi-participant resource allocation in one embodiment; Figure 3 A framework diagram of a strategy search method for multi-participant resource allocation in one embodiment; Figure 4 This is a performance comparison diagram of algorithms for a 5-resource 3-partition scenario when MME-PSRO solves the Broto game in one embodiment; Figure 5 This is a performance comparison diagram of algorithms in a 5-resource 4-partition scenario when MME-PSRO solves the Broto game in one embodiment; Figure 6 This is a performance comparison diagram of algorithms for a 5-resource 5-partition scenario when MME-PSRO solves the Broto game in one embodiment; Figure 7 An example diagram of a typical mission map of a diplomatic board game in one embodiment; Figure 8 It is a diagram of the coding module of the MME-NVO network architecture in one embodiment; Fig. 9 It is a diagram of the SoS reward information of the overall MME-NVO model in one embodiment; Fig.10 is a diagram of SoS reward information for zone A in the MME-NVO model in one embodiment; Fig.11 is a diagram of SoS reward information for zone B in the MME-NVO model in one embodiment; Fig.12 A diagram of SoS reward information for zone C in the MME-NVO model in one embodiment; Fig.13 It is a diagram of the SoS reward information of the D zone in the MME-NVO model in one embodiment; Fig.14 It is a diagram of SoS reward information of zone E in the MME-NVO model in one embodiment; Fig.15 A diagram of SoS reward information for zone F in the MME-NVO model in one embodiment; Fig.16 A diagram of SoS reward information for zone G in the MME-NVO model in one embodiment; Fig.17 A structural block diagram of a strategy search device for multi-participant resource allocation in one embodiment; Fig.18 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0023] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.

[0024] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0025] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0026] The multi-participant resource allocation strategy search method provided in this application can be applied to Figure 1 In the application scenario diagram shown, the terminal 102 communicates with the server 104 through the network, and the terminal 102 may include but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 may be various portal websites, servers corresponding to the backend of the work system, etc.

[0027] This application provides a strategy search method for multi-participant resource allocation, such as Figure 2 The flowchart shown in FIG. 1 shows a method for applying the method to Figure 1 The following is an example of a terminal in the example, including: Step 202: Obtain a multi-participant resource allocation scenario and build a model to obtain a Markov model; and obtain an expected function value vector of the state based on the Markov model.

[0028] Specifically: The stakeholders involved in the competition are modeled as multiple game participants, who invest in equipment in multiple regions, and use the equipment to seize advantages and launch multi-stage competition. Multi-participant resource allocation scenarios are obtained, including: competitive resource allocation scenarios where multiple participants invest in equipment, multiple merchants invest in multiple types of products to occupy the market, or multiple campaign teams invest in elections in multiple regions to win public support, and multi-participant resource allocation scenarios are modeled to obtain a multi-stage Markov model: Multi-player competitive resource allocation problems can usually be modeled using a multi-stage Markov game, which can be represented by a 6-tuple ,in represents the set of participants, represents the global state set of the game, Indicates participants The action space of is the product of the action spaces of all participants, is the state transfer function, where for A collection of probability distributions in space; For participants The reward function of the game starts from the initial state At the beginning, the game trajectory is recorded as , given the strategy combinations of all game participants , then the probability of the game trajectory is ; According to the Markov model, From the game state The game trajectory at the beginning ends in the state, and the expected function value vector of the state is obtained:

[0029] In the formula, is the expected function value vector of the state, is the game trajectory, To adopt a strategy Get the track The probability of is the action pair on the trajectory, is the discount factor, For reward.

[0030] Step 204, constructing a Nash equilibrium strategy and optimizing it with a neural network to obtain an approximate Nash equilibrium. The approximate Nash equilibrium is an approximation of the best benefit obtained by the participants in the worst case in the equipment resource allocation process, which is used as a constraint condition for the expected function value vector of the state.

[0031] Specifically: Game participants Coping strategies The best response corresponds to the strategy with the largest game state value function, that is:

[0032] In the formula, The optimal strategy The corresponding joint strategy value, For any strategy The corresponding joint strategy value; For any game participant ,make is the value function vector components, if the strategy A strategy is called a Nash equilibrium strategy if the following conditions are met:

[0033] In the formula, For participants Any strategy, is the joint strategy value function; For an approximate Nash equilibrium , usually referred to as its exploitability .

[0034] Since this application is a multi-stage game, the Nash equilibrium for a single stage is represented by express.

[0035] Step 206, adopt the regularized leader-follower method to establish the update rules of the strategy at the time step; adopt the imitation learning method based on historical interaction data to generate the anchor strategy and establish the utility function of the participants in the strategy; according to the update rules and utility function, obtain the iteration method of the strategy; simplify the iteration method of the strategy, and use the convex conjugate method to solve it to obtain the equilibrium solution.

[0036] Specifically: Follow the Regularized Leader is a typical online convex optimization method to assist in learning game strategies. For the first-order strongly convex regularization function , online convex optimization learning rate Loss (utility) function for feedback at different time steps from the environment ; Using the regularized leadership-following method, we establish the variable exist The update rule for the time step, that is, the strategy that determines how the participants should update the equipment resource allocation:

[0037] In the formula, for time step variable, is a first-order strong regularization function, is the learning rate for online convex optimization, for Time step loss; Using imitation learning methods based on historical interaction data, we generate human-like anchoring strategies similar to human decision-making methods, and establish participants based on the benefits. In Strategy The utility function is the benefit advantage that participants gain after adopting a certain strategy for resource allocation:

[0038] In the formula, For participants In Strategy The utility function of is the original reward function, As a regulatory factor, guide participants Whether the action is aligned with the "anchor strategy" is the divergence (also called “relative entropy”), For participants strategy, For the exclusion of participants Other people’s strategies, For anchoring strategy, , Take the negative entropy regularization function; According to the update rules and utility function, the iterative method of equipment resource allocation strategy is obtained, which is the maximum and minimum entropy regularization:

[0039] In the formula, For participants exist Time step strategy, is the strategy space, For strategy Information entropy of multiplayer games can usually be used To describe the style of the game participants; Simplify the iteration method of equipment resource allocation strategy:

[0040] Because of the function About convex function, and for the general , the convex conjugate (Legendre-Fenchel transformation) method is used to solve and obtain the equilibrium solution:

[0041]

[0042] In the formula, For strategy Corresponding rewards, For the updated strategy, Status Conditions, take action strategy, is the game participant style parameter, Status Take action of income, Status Take action of income, is a state-action pair.

[0043] In this step, we focus on using reinforcement learning methods based on game theory and use bootstrap methods such as self-playing to learn low-availability countermeasures.

[0044] In establishing participants In Strategy When the utility function is constructed, the reward transformation method is adopted to guide the generation of strategies by adding a real term (usually a regularization term) to the original reward function.

[0045] For multiplayer games, since everyone has different styles, usually The value of is different.

[0046] For a two-person zero-sum game, define the equilibrium solution probability in this step as The method of iterating in the above manner is called the MaxMinEnt method. This method considers the exploration of both the anchoring strategy and its own strategy. It can be used not only for the calculation of Nash equilibrium strategy in the iterative solution of the blueprint strategy, but also for strategy search during online reasoning. The pseudo code of its implementation steps is shown in Algorithm 1:

[0047] Step 208, calculate the objective value function of each participant according to the equilibrium solution; calculate the approximate Nash equilibrium response according to the objective value function; obtain the state value function according to the approximate Nash equilibrium response and the expected function value vector of the state; establish the loss function according to the state value function, and iteratively search the equipment resource allocation strategy with the goal of minimizing the loss function until the preset conditions are met, and output the current equipment resource allocation strategy as the optimal solution of the Markov model.

[0048] Specifically: As a joint action learning method, Nash Q Learning (NQL) has been proven to converge to Nash equilibrium in random game problems. Its main learning target action value function , when performing a single-step update The operator is replaced with the Nash equilibrium calculation of this "stage game". Assuming that from the state arrive Execute actions during the process , according to the status Take action The corresponding state action value function Nash equilibrium of the game in the calculation phase ; According to the equilibrium solution, calculate the objective value function of each participant, that is, the target value of the benefits obtained by each participant in the scenario when taking actions around the equipment resource allocation strategy in this state:

[0049] In the formula, is the target state action value function (i.e., target value function), is the learning rate of reinforcement learning, Status Take action The rewards you get when is the discount factor, To take action The probability of Status Take action The corresponding state action value function; Double Oracle (DO) is an iterative Nash equilibrium calculation method. Its calculation process mainly starts from the subset of alternative actions of each game reference person. , in each iteration, the best response of the restricted Nash equilibrium is calculated Add to Alternative Action Set ; According to the objective value function, calculate the response strategy of the approximate Nash equilibrium:

[0050] In the formula, is a response strategy that approximates the Nash equilibrium, An action taken by a participant, is the action vector taken by the participant, The state-action value function corresponding to the action taken by the participant, For participants The combined strategy of all others; Since the action space of the game is very large, the reward calculation of each game reference party needs to traverse all "actions" and the next state , it is difficult to calculate the Q value function, so this application adopts the state value iteration method, using To approximate the state value function, according to the response strategy of the approximate Nash equilibrium and the expected function value vector of the state, the state value function is obtained, that is, the benefits obtained when competing according to the resource allocation strategy in this state:

[0051] In the formula, is the state value function, For status rewards, To take action The probability of Status The value function of Since the action space is particularly large, the Nash equilibrium strategy of the "one-step-ahead" matrix game in each stage of the multi-stage game is still challenging. A strategy proposal network can be designed To approximate the state The distribution of feasible actions, establish the loss function of the corresponding value network and the policy network: (Formula 1) In the formula, is the state loss function, Status Value function, For status rewards, To use hyperparameters Status The value function of is the strategy loss function, For participants Actions, For participants Adopt an action The probability of For participants In Status Take action strategy.

[0052] In this step, since the MaxMinEnt method can provide a solution to the Nash equilibrium strategy, this application designs an iterative multiplayer game strategy method based on the Max Min Entropy Nash Value Oracle (MME-NVO), which mainly relies on the Nash Q learning method and dual oracle-assisted Nash value iteration.

[0053] This step uses a distributed architecture to accelerate strategy training and reasoning. During training, we first rely on human data to imitate the anchoring strategy and value function as a hot start for reinforcement learning, rather than randomly initializing from scratch. During training, we mainly use the MaxMinEnt method to solve the equilibrium strategy of the single-step forward game at each stage. The pseudo code for the implementation steps of the MME-NVO method is shown in Algorithm 2:

[0054] In this embodiment, if Figure 3As shown in the figure, the game strategy solution method of "offline (training) blueprint strategy learning and online (testing) adaptive strategy search" is studied. The input is the pre-deployed and interactive strategy, and the output structure can be used to build the situation chain and action chain. The offline training stage adopts the iterative population game strategy learning method, and the online reasoning stage adopts the search method based on the generative game strategy model to generate the best counter-strategy. The two-stage strategy solution method is adopted to meet the strategy search needs. The multi-participant resource allocation game is modeled as a multi-stage Markov game. Since this type of game usually contains unknown opponent information, the relevant technology of multi-person imperfect information game can be used to assist in game strategy solution.

[0055] The above-mentioned strategy search method for multi-participant resource allocation focuses on offline blueprint strategy learning and training and online adaptation and counter-strategy search, and provides a multi-participant resource allocation game strategy search framework based on maximum and minimum entropy. In the face of the challenge of incompletely knowable (imperfect) information, in order to address the exploration difficulties caused by the non-transitive suppression of strategies, based on the regularization theory in online convex optimization, and focusing on the needs of iterative strategy rapid solution, a Nash equilibrium strategy iterative solution method based on maximum and minimum entropy is proposed; first, based on the anchoring strategy, a Nash equilibrium strategy solution method guided by maximum and minimum entropy regularization is designed; then, based on the blueprint strategy training, based on Nash Q-learning and dual oracles, a maximum and minimum Nash oracle strategy learning framework is designed using Nash value iteration, which effectively handles the uncertainty challenge brought by opponents with imperfect information. In addition, the effectiveness of the proposed method is verified on the "Diplomacy" board game platform, and the practicality of the method is empirically analyzed using the "Brotto Game".

[0056] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0057] In a specific embodiment, since the present application is an iterative algorithm, in order to verify the effectiveness of each module, the performance of the present method is tested using the "Real-World Meta Game", "Diplomatic Board Game" and Brotto Game as benchmarks. The "Real-World Meta Game" is the intermediate normalized result of various strategic games, the Brotto Game is a typical multi-participant resource allocation game model, and the "Diplomatic Board Game" is an enhanced version of the Brotto Game, and is also a new benchmark for current multi-person imperfect information games, which is convenient for testing various baseline algorithms.

[0058] 1) Use MME-PSRO to solve the Broto game and conduct strategy space analysis, that is, test the performance of MME-PSRO in the Broto game.

[0059] The Brotto game is used as the test benchmark and implemented using the open source game model provided by the OpenSpiel platform. The Brotto game is a type of peer action zero-sum resource allocation game. Each player has N troops that can be deployed to K areas. If the player allocates more troops than the opponent in a certain area, then the area will be conquered and rewarded. The player who conquers more areas wins the final game. This model is widely used in political selection, auctions, and resource deployment. Like the "diplomatic board game", it has an action space that explodes with the increase of parameter values; secondly, the Brotto game is also a "simultaneous action" game, so the game participants must use mixed strategies; thirdly, the state and action space of the Brotto game is small enough to accurately calculate various equilibria.

[0060] The baseline algorithms compared with the MME-PSRO algorithm are mainly: ①PSRO: Traditional model based on supervised learning using offline datasets; ②DPP-PSRO: A strategy learning method for policy space response prediction based on diversity measurement.

[0061] The "strategy space response oracle" algorithm is used for comparative analysis to solve the Broto game. The relevant experimental results are as follows: Figures 4 to 6 As shown, the method of the present application converges faster than the other two methods.

[0062] 2) Design a “Diplomacy Board Game” game strategy based on the MME-NOV method, that is, the performance test of MME-NVO in the “Diplomacy Board Game” task.

[0063] The multiplayer No Press Diplomacy board game is used as a test benchmark to analyze the solution of the multiplayer imperfect information game. The "standard version" of the board game supports seven players, representing "Area A, B, C, D, E, F and G". Since it is a synchronous action game, each player usually writes down the instructions synchronously in the real match and announces them at the same time to assist in the decision. Due to the imperfect information involved, a single player usually adopts a random mixed strategy to prevent the opponent from predicting his own strategy. Since the strategy space of the game is large, only "single-step forward search" is used in each stage game. The selected typical task examples are as follows: Figure 7 As shown in the figure. In the first stage of the game, there are 22 units in total, each game reference person has 3-4 units, and each unit has about 10 legal actions, so the total action space is about 10 22 As the game progresses, there are at most 34 units, so the legal action space is about 10 64 ; Since the game has no repetition rules and no stage restrictions, the median size of the game tree node is about 10 for a 20-stage game instance. 867 , the game is very complicated.

[0064] Anchoring strategy: The humanoid anchoring strategy training mainly adopts the imitation learning method, using the "Diplomacy Board Game" data provided by webdiplomacy.net for training. During the data processing, empty actions are filtered out, and only the actions of the first half of the players in the ranking are trained. During the training process, 7 players are randomly arranged to enhance the data.

[0065] Model architecture: The model design is based on the “original map” assumption. The network architecture is as follows Figure 8 As shown, it mainly includes a shared encoding module, and separate "strategy" and "value function" decoding modules. Information is encoded for 81 possible land locations and 81 possible coastal locations, and the number of feature channels is 38. In order to encode the instruction history information corresponding to each location, an additional 202 channels are provided for each location to record the historical instructions at that location; each game participant corresponds to 1 channel, for a total of 7 channels; in order to ensure the equivariant characteristics of information encoding, a global state encoding channel for each party is added, for a total of 7 channels. For the position, participant, and global state tensors, 224 channel information is output through a linear layer respectively and spliced ​​into an 89*224-dimensional tensor. Position deviation information is added in the middle, and then the Transformer architecture is used for information encoding. Finally, the same LSTM decoder is used for decoding, using The layer guides the encoder to output the participants' benefit information.

[0066] The status coding information of each desktop position is shown in Table 1, the status coding information of each player's desktop is shown in Table 2, and the global desktop status coding information is shown in Table 3.

[0067] Table 1 Status coding information for each desktop position

[0068] Table 2 Encoding information of each player's desktop status

[0069] Table 3 Global desktop status encoding information

[0070] MME-NVO algorithm: In the specific implementation process, the distributed architecture supports the rapid learning of the strategy. The detailed algorithm-related hyperparameter settings are shown in Table 4.

[0071] Table 4. Hyperparameter settings of MME-NVO algorithm

[0072] Model performance evaluation: The main evaluation indicator used is the sum of squares (SoS), which represents the geographical situation controlled by the participants. For each player, ,in Indicates the number of supply centers controlled by the game participants.

[0073] According to the performance indicators of the model, the mElo score of each strategy is obtained through two-by-two battles, and the model strategies are ranked. The main calculation steps are as follows:

[0074]

[0075] like Figures 9 to 16 As shown in the figure, as the number of iterations increases, the corresponding SoS reward information indicators change when the algorithm model plays different roles. When playing the roles of area A and area B, the learned strategy rewards fluctuate.

[0076] The game mode of 1 vs. 6 was adopted, and the results are shown in Table 5.

[0077] The baseline algorithms compared with the MME-NVO algorithm are mainly: ①DipNet: A model obtained by supervised learning using an offline dataset; ②BlutPrint: Blueprint strategy, mainly using supervised learning; ③DORA: The dual oracle action exploration method is used for offline training, and regret matching is used for online reasoning. This method achieves good results in the "1 vs. 1" two-person zero-sum game, but performs poorly in the "1 vs. 6" multi-person game. ④SearchBot: Uses offline data supervision to train the blueprint strategy and adopts single-step forward search during online reasoning.

[0078] Table 5 Average SoS score results of MME-NVO algorithm in 1 vs. 6 games

[0079] As can be seen from Table 5, the strategy learned by the MME-NVO algorithm of this application has better performance than DipNet and BluePrint, and the strategy performance has been further improved compared with the DORA and SearchBot methods.

[0080] The present application also provides a strategy search device for multi-participant resource allocation, such as Fig.17 As shown, in one embodiment, it includes: an acquisition module 1702, a balancing module 1704, a solution module 1706 and an output module 1708, wherein: The acquisition module 1702 is used to acquire a multi-participant resource allocation scenario, which includes: a competitive resource allocation scenario in which multiple participants invest in equipment funds, multiple merchants invest in multiple types of products to occupy the market, or multiple campaign teams invest in elections in multiple regions to win public support, and model the multi-participant resource allocation scenario to obtain a Markov model; according to the Markov model, obtain the expected function value vector of the state; The equilibrium module 1704 is used to construct a Nash equilibrium strategy and optimize it using a neural network to obtain an approximate Nash equilibrium as a constraint condition for the expected function value vector of the state; The solution module 1706 is used to establish the update rule of the strategy at the time step by using the regularized leader-follower method; generate the anchor strategy by using the imitation learning method based on the historical interaction data, and establish the utility function of the participants in the strategy; obtain the iteration method of the strategy according to the update rule and the utility function; simplify the iteration method of the strategy, and solve it by using the convex conjugate method to obtain the equilibrium solution; Output module 1708 is used to calculate the target value function of each participant according to the equilibrium solution; calculate the response of the approximate Nash equilibrium according to the target value function; obtain the state value function according to the response of the approximate Nash equilibrium and the expected function value vector of the state; establish the loss function according to the state value function, and iteratively search the strategy with the goal of minimizing the loss function until the iteration is stopped when the preset conditions are met, and output the current strategy as the optimal solution of the Markov model.

[0081] The specific definition of the strategy search device for multi-participant resource allocation can be found in the definition of the strategy search method for multi-participant resource allocation above, which will not be repeated here. Each module in the above device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0082] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.18 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a strategy search method for multi-participant resource allocation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0083] Those skilled in the art will understand that Fig.18 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0084] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0085] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0086] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0087] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0088] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A strategy search method for multi-participant resource allocation, characterized in that: include: Obtain multi-participant resource allocation scenarios, including: competitive resource allocation scenarios of multiple participants around equipment capital investment, multiple merchants investing in multiple types of products to occupy the market, or multiple campaign teams investing in elections in multiple regions to win public support, and model the multi-participant resource allocation scenarios to obtain a Markov model; according to the Markov model, obtain the expected function value vector of the state; Construct a Nash equilibrium strategy and use a neural network to optimize it to obtain an approximate Nash equilibrium as a constraint on the expected function value vector of the state; The regularized leader-follower method is used to establish the update rules of the strategy at the time step; the imitation learning method based on historical interaction data is used to generate the anchor strategy and establish the utility function of the participants in the strategy; according to the update rules and utility function, the iteration method of the strategy is obtained; the iteration method of the strategy is simplified and solved by the convex conjugate method to obtain the equilibrium solution; According to the equilibrium solution, the objective value function of each participant is calculated; according to the objective value function, the response of the approximate Nash equilibrium is calculated; according to the response of the approximate Nash equilibrium and the expected function value vector of the state, the state value function is obtained; according to the state value function, a loss function is established, and with the goal of minimizing the loss function, the strategy is iteratively searched until the preset conditions are met, and the iteration is stopped, and the current strategy is output as the optimal solution of the Markov model.

2. The strategy search method for multi-participant resource allocation according to claim 1, characterized in that: The regularized leader-follower method is used to establish the update rules of the strategy at the time step, including: In the formula, for time step variable, is a first-order strong regularization function, is the learning rate for online convex optimization, for Time step loss.

3. The strategy search method for multi-participant resource allocation according to claim 2, characterized in that: An imitation learning method based on historical interaction data is used to generate anchoring strategies and establish the utility function of participants in the strategy, including: In the formula, For participants In Strategy The utility function of is the original reward function, is the regulating factor, is the divergence, For participants strategy, For the exclusion of participants Other people’s strategies, Anchoring strategy.

4. The strategy search method for multi-participant resource allocation according to claim 3 is characterized in that: According to the update rules and utility function, the iterative method of the strategy is obtained, including: In the formula, For participants exist Time step strategy, is the strategy space, For strategy Information entropy.

5. The strategy search method for multi-participant resource allocation according to any one of claims 1 to 4, characterized in that: According to the equilibrium solution, calculate the objective value function of each participant, including: In the formula, is the target value function, is the learning rate of reinforcement learning, Status Take action The rewards you get when is the discount factor, To take action The probability of Status Take action The corresponding state value function.

6. The strategy search method for multi-participant resource allocation according to claim 5, characterized in that: According to the objective value function, the response of the approximate Nash equilibrium is calculated, including: In the formula, is the response approximating the Nash equilibrium, An action taken by a participant, is the action vector taken by the participant, The state-action value function corresponding to the action taken by the participant, For the exclusion of participants The combined strategy of all others.

7. The strategy search method for multi-participant resource allocation according to claim 6, characterized in that: Establish a loss function, including: In the formula, is the state loss function, Status Value function, For status rewards, To use hyperparameters Status The value function of is the strategy loss function, For participants Actions, For participants Adopt an action The probability of For participants In Status Take action strategy.

8. A strategy search device for multi-participant resource allocation, characterized in that: include: The acquisition module is used to acquire multi-participant resource allocation scenarios, which include: competitive resource allocation scenarios of multiple participants focusing on equipment capital investment, multiple merchants investing capital around multiple types of products to occupy the market, or multiple campaign teams investing election funds in multiple regions to win public support, and modeling the multi-participant resource allocation scenarios to obtain a Markov model; according to the Markov model, the expected function value vector of the state is obtained; The equilibrium module is used to construct the Nash equilibrium strategy and optimize it using a neural network to obtain an approximate Nash equilibrium as a constraint condition for the expected function value vector of the state; The solution module is used to establish the update rules of the strategy at the time step by using the regularized leader-follower method; to generate the anchor strategy by using the imitation learning method based on the historical interaction data, and to establish the utility function of the participants in the strategy; to obtain the iteration method of the strategy according to the update rules and the utility function; to simplify the iteration method of the strategy, and to solve it by using the convex conjugate method to obtain the equilibrium solution; The output module is used to calculate the objective value function of each participant according to the equilibrium solution; calculate the approximate Nash equilibrium response according to the objective value function; obtain the state value function according to the approximate Nash equilibrium response and the expected function value vector of the state; establish the loss function according to the state value function, and iteratively search the strategy with the goal of minimizing the loss function until the iteration is stopped when the preset conditions are met, and output the current strategy as the optimal solution of the Markov model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images