Environmental disturbance-oriented pursuit game strategy solving method

Abstract

The invention provides a pursuit game strategy solving method oriented to environmental disturbance. According to the method, initial strategies of pursuers and escapers at different decision points are set according to strategy sets of the pursuers and the escapers, and a game tree is traversed to obtain strategy variable initial values and auxiliary variable initial values in a sequence form; in the iterative optimization process, in each time step, the pursuer and the escaper update the decision variables by using the obtained feedback information of the previous time step; traversing the game tree to respectively obtain feedback information of the pursuer and the escaper at the current time step; the pursuer and the escaper update the auxiliary variable using feedback information obtained at the current time step. A regularization item is added into an updating formula of a decision variable and an auxiliary variable, the regularization technology is utilized to stabilize the negative influence of random noise, the convergence performance of the algorithm is ensured, the convergence of the algorithm can be ensured without calculating an average strategy, the calculation and storage burden is reduced, the algorithm efficiency is improved, and faster convergence is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for solving pursuit-escape game strategies in the presence of environmental perturbations. Background Technology

[0002] The pursuit-escape game problem is a hot research topic in game theory. Imperfect-information extended-form games can accurately simulate the multiple decision interactions between the pursuer and the escapee in the pursuit-escape game problem, as well as the important characteristic of the existence of private information during the interaction process. In the pursuit-escape game problem, although the pursuer and the escapee make decisions simultaneously, by utilizing the concept of information sets in imperfect-information extended-form games, the pursuit-escape game problem can be modeled as a turn-based decision-making problem. The pursuer and the escapee make decisions at different stages based on the game state information they have obtained, in order to maximize their own gains. For the participants in the game, making fast and accurate decisions is the key to improving their intelligence.

[0003] Currently, numerous methods exist for solving the pursuit-escape game problem using extended game theory models, such as Counterfactual Regret Minimization and Online Convex Optimization. When the environment is undisturbed, these methods theoretically guarantee that the average policy they generate will be stable at an approximate Nash equilibrium. However, calculating the average policy of the participants incurs significant computational and storage costs; for example, some algorithms require neural network fitting when calculating the average policy. Furthermore, in practical pursuit-escape games, both the pursuer and the escapee often lack precise feedback information due to noise or uncertainties in the environment, and can only update their policies based on potentially noisy observed feedback. This can not only affect the convergence speed of the algorithm but may even prevent it from converging. Therefore, for pursuit-escape games with perturbed environments, specialized methods are needed to further improve learning efficiency. Summary of the Invention

[0004] In view of this, the present invention provides a strategy solution for a chase-escape game with environmental perturbations. After the iterative calculation is completed, it is not necessary to calculate the policy mean at each time step as the Nash equilibrium solution. Instead, it directly converges to the Nash equilibrium solution through iteration, thereby reducing the computation and storage burden in the policy learning process, ensuring the algorithm converges quickly when there are environmental perturbations, and improving learning efficiency.

[0005] To solve the above-mentioned technical problems, the present invention is implemented as follows.

[0006] A method for solving pursuit-escape game strategies in environments with perturbations includes:

[0007] Step 1: Based on the strategy sets of the pursuer and the escapee, set their initial strategies at different decision points, and traverse the game tree to obtain the initial values ​​of the strategy variables and auxiliary variables in sequence form;

[0008] Step 2: At each time step, the pursuer and the escapee update their decision variables using the feedback information obtained from the previous time step; traverse the game tree to obtain the feedback information of the pursuer and the escapee at the current time step; the pursuer and the escapee update their auxiliary variables using the feedback information obtained at the current time step.

[0009] The feedback information represents the participant's expected payoff assessment of all pure strategies at the current time step, where a pure strategy is defined as having a probability of 1 for choosing a particular strategy in the strategy set and a probability of 0 for choosing any of the other strategies. The higher the participant's expected payoff assessment of the pure strategy at the current time step, the greater the probability of choosing that pure strategy in the next time step. The feedback information is obtained by traversing the entire game tree, specifically by multiplying the updated decision variables by the payoff matrix, and is considered to contain noise.

[0010] A regularization term is added to the update formulas for decision variables and auxiliary variables. The regularization term is strongly convex with respect to the distance generation function in the update formula, which makes the policy obtained in the last iteration of policy learning converge to the Nash equilibrium solution without having to calculate the policy mean at each time step.

[0011] Preferably, the updated decision variable is:

[0012] At time step t+1, both the pursuer and the escapee update their decision variables using the feedback information obtained at time step t:

[0013]

[0014] Where x and Let y and y represent the sequential decision variable and auxiliary variable of the pursuer, respectively. Let these represent the decision variables and auxiliary variables in sequential form for the escapees, respectively; and These represent the sets of strategies for the pursuer and the escapee, respectively.

[0015] and These represent the feedback information of the pursuer and the escapee at time step t under environmental disturbance conditions, respectively. Each component in the vector represents the hunter's expected payoff assessment of its chosen pure strategy at time step t. Each component in the vector represents the escapee's expected payoff assessment for choosing the corresponding pure strategy at time step t. and This represents the environmental disturbance variables for both the pursuer and the escapee at time step t; As a whole, by traversing the game tree, that is, by multiplying the payoff matrix U by the escapee's decision variable y at time step t. t This is an approximation obtained by including the environmental perturbation variables of the pursuer at time step t. Similarly, As a whole, the payoff matrix U is multiplied by the pursuer's decision variable x at time step t. t The calculated value is obtained by multiplying by an approximation and includes the environmental disturbance variable of the escapee at time step t. <,> indicates the inner product; η t β is the learning rate that decays, and β is the strength adjustment coefficient of the regularization term;

[0016] g1(x) and g2(y) represent the regularization functions set for the pursuer and the escapee, respectively, and Φ1(x) and Φ2(y) represent the distance generation functions set for the pursuer and the escapee, respectively. and Let Φ1(x) and Φ2(y) represent the Bregman divergences calculated by Φ1(x) and Φ2(y), respectively; the regularization function g i (·) and distance generation function Φ i (·) satisfies: g i (·) relative to Φ i (·) is μ-strongly convex, where μ is a non-negative number, i∈{1,2}.

[0017] Preferably, the updated auxiliary variable is:

[0018] At time step t+1, both the pursuer and the escapee update auxiliary variables using the feedback information obtained at time step t+1:

[0019]

[0020]

[0021] Where x and Let y and y represent the sequential decision variable and auxiliary variable of the pursuer, respectively. Let these represent the decision variables and auxiliary variables in sequential form for the escapees, respectively; and These represent the sets of strategies for the pursuer and the escapee, respectively.

[0022] and They represent the feedback information obtained by the pursuer and the escapee at time step t under environmental disturbance conditions, respectively. Each component in the vector represents the hunter's expected payoff assessment of its chosen pure strategy at time step t. Each component in the vector represents the escapee's expected payoff assessment for choosing the corresponding pure strategy at time step t. and This represents the environmental disturbance variables for both the pursuer and the escapee at time step t+1; As a whole, multiply the payoff matrix U by the escapee's decision variable y at time step t+1. t+1 This is an approximation, and the calculated value contains the environmental perturbation variables of the pursuer at time step t+1. Similarly, As a whole, the payoff matrix U is multiplied by the pursuer's decision variable x at time step t+1. t+1 This is an approximation, and the calculated value includes the environmental disturbance variables of the escapee at time step t+1. <,> indicates the inner product; η t β is the learning rate that decays, and β is the strength adjustment coefficient of the regularization term;

[0023] g1(x) and g2(y) represent the regularization functions set for the pursuer and the escapee, respectively, and Φ1(x) and Φ2(y) represent the distance generation functions set for the pursuer and the escapee, respectively. and Let Φ1(x) and Φ2(y) represent the Bregman divergences calculated by Φ1(x) and Φ2(y), respectively; the regularization function g i (·) and distance generation function Φ i (·) satisfies: g i (·) relative to Φ i (·) is μ-strongly convex, where μ is a non-negative number, i∈{1,2}.

[0024] Preferably, the learning rate satisfies

[0025] Preferably, g is set i (·)=μΦ i (·)

[0026] Preferably, step 1 includes: selecting a suitable probability distribution for each decision point as the initial point for the strategy learning of the pursuer and the escapee, based on the action space of the pursuer and the escapee at each decision point; traversing the game tree to obtain the strategy variables and auxiliary variables in sequence form for the pursuer and the escapee, and obtaining feedback information in the initialization phase.

[0027] Beneficial effects:

[0028] The strategy learning method for the pursuit-escape game of this invention stabilizes the negative impact of random noise by using regularization techniques, ensuring the convergence performance of the algorithm. It can guarantee the convergence of the algorithm without calculating the average strategy, reducing the computational and storage burden, improving the efficiency of the algorithm, and achieving faster convergence.

[0029] It is precisely because of the effect of stabilizing random noise that the efficiency of strategy solution can still be guaranteed when the pursuer and the escapee can only obtain feedback information that may contain random noise. This breaks through the limitation that the optimistic mirror gradient descent method can only be applied to scenarios with no noise assumption when solving pursuit and escape game problems that can be modeled as extended game problems. This lays the foundation for pushing the pursuit and escape game strategy solution based on the optimistic mirror gradient descent method from theory to practical application. Attached Figure Description

[0030] Figure 1 A diagram illustrating the game of pursuit and escape;

[0031] Figure 2 Game tree for a chase game modeled using extended game theory with imperfect information;

[0032] Figure 3 A flowchart for learning strategies in a pursuit-escape game where environmental disturbances exist.

[0033] Figure 4 Convergence curves for the availability of the proposed method using different regularization operators;

[0034] Figure 5 A complete game tree for the drone mission allocation problem;

[0035] Figure 6 A game tree drawn from the perspective of drone 1 in the drone task allocation problem;

[0036] Figure 7 A game tree drawn from the perspective of drone 2 in the drone task allocation problem. Detailed Implementation

[0037] This invention provides a method for solving pursuit-escape game strategies in environments with disturbances. The pursuit-escape game problem is modeled as an extended game problem. This method employs the optimistic mirror gradient descent method to solve the pursuit-escape game strategy and improves the update process to overcome the limitation that the optimistic mirror gradient descent method can only be applied to scenarios with no noise assumptions. Its core solution is as follows:

[0038] Based on the strategy sets of the pursuer and the escapee, their initial strategies are set at different decision points. The game tree is traversed to obtain the initial values ​​of the strategy variables and auxiliary variables in sequence form. At each time step, the pursuer and the escapee update their decision variables using the feedback information obtained from the previous time step. The game tree is traversed to obtain the feedback information of the pursuer and the escapee at the current time step. The pursuer and the escapee update their auxiliary variables using the feedback information obtained at the current time step.

[0039] It is particularly noteworthy that this invention features a specially designed update formula for decision variables and auxiliary variables, incorporating a regularization term. This regularization term is strongly convex relative to the distance generation function in the update formula. This regularization technique stabilizes the negative impact of random noise, guiding policy learning and ensuring that the policy obtained in the final iteration converges to the Nash equilibrium solution without needing to calculate the policy mean at each time step. This reduces the computational and storage burden during policy learning, ensuring rapid convergence even with environmental disturbances and improving learning efficiency. Furthermore, the efficiency of policy solving is also guaranteed when the pursuer and escapee only receive feedback information that may contain random noise.

[0040] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0041] This embodiment proposes an efficient strategy learning method for pursuit-escape game under environmental perturbation conditions. The pursuit-escape game problem it addresses is as follows: Figure 1 As shown. In this pursuit game, the pursuer controls two patrolmen P1 and P2, whose initial positions are respectively located at... Figure 1 The escapee is represented by nodes A and G. Each time the escapee moves, it selects an action for P1 and P2. Based on the escapee's current position, its reachable locations are shown in Table 1. The escapee initially resides at... Figure 1 Node S in the map, based on the escapee's current position, has reachable locations as shown in Table 2. During each move, if the escapee chooses to stay at their current position and wait one step, then if any subsequent patrol reaches a node previously visited by the escapee, the patrol will not know that the escapee had previously reached that node. If the escapee is not cleared, the pursuers will know that the escapee had previously reached that node. If the escapee reaches... Figure 1 If the escapee reaches any of the rightmost nodes (X, Y, Z), the escapee wins and the game ends. If the escapee and any patroller are on the same node within the allotted time, the pursuer wins and the game ends. If the escapee does not reach the rightmost node within the allotted time, and the pursuer fails to arrest the escapee, the game ends. The escapee's payoff at the end of the game is shown in Table 3.

[0042] Table 1. Locations reachable by pursuers

[0043]

[0044] Table 2. Locations accessible to escapees

[0045]

[0046] Table 3. Earnings of Escapees

[0047]

[0048] The pursuit-escape game problem is modeled using an extended game theory model with imperfect information as follows: Figure 2 The diagram illustrates a turn-based pursuit game problem, where the shaded area represents a set of information about the escapee, whose actions regarding the pursuer are unknown. The process for solving pursuit game strategies in a perturbed environment is as follows: Figure 3 As shown in the diagram, in this example of a chase problem, the escapee has 24 pieces of information, and the pursuer has 321 pieces of information. Therefore, the payoff matrix U is a 321×24 matrix. The payoff matrix U is related to the payoff settings in Table 3. If a sequence seq1 of the pursuer and a sequence seq2 of the escapee can guarantee reaching a leaf node of the game tree, i.e., ending the game, then both the pursuer and the escapee will receive the payoff value corresponding to that leaf node. In this case, the elements in the payoff matrix U corresponding to the pursuer's sequence seq1 and the escapee's sequence seq2 are the escapee's payoff value, and the pursuer's payoff value is the negative of the escapee's payoff. If a sequence seq1 of the pursuer and a sequence seq2 of the escapee cannot guarantee reaching a leaf node of the game tree, i.e., the current game has not ended, then the elements in the payoff matrix U corresponding to the pursuer's sequence seq1 and the escapee's sequence seq2 are 0, i.e., there is no payoff.

[0049] Step 1: Based on the action spaces of the pursuer and the escapee at each decision point, select an appropriate probability distribution for each decision point as the initial point for policy learning for both participants. For example, the initial policies of the pursuer and the escapee at each decision point can be set to a uniform distribution.

[0050] Step 2: At each time step, perform the following computational tasks:

[0051] (1) Calculate the learning rate (step size) η at the current time step t. t The learning rate should be selected to meet the following conditions and transmitted to the participants:

[0052]

[0053] (2) The pursuer and the escapee update their decision variables at each decision point using the feedback information from the previous time step. The update method for the decision variables of the pursuer and the escapee is as follows:

[0054]

[0055] Where x and Let y and y represent the sequential decision variable and auxiliary variable of the pursuer, respectively. Let these represent the decision variables and auxiliary variables in sequential form for the escapees, respectively; and These represent the sets of strategies for the pursuer and the escapee, respectively. and These represent the feedback information of the pursuer and the escapee at time step t under environmental disturbance conditions. The feedback information represents the local directional information generated by the current decision, providing a numerical reference for which direction to pursue / escape in the next step. Specifically, the physical meaning of the feedback information is: It is a vector, where different components correspond to the expected returns obtained by different strategies; each component represents the hunter's assessment of the expected return for choosing the corresponding pure strategy at time step t. Let be a vector, where each component represents the escapee's expected payoff assessment for choosing a corresponding pure strategy at time step t. Here, a pure strategy represents a strategy with a probability of 1 in the strategy set and a probability of 0 in choosing any of the other strategies. The expected payoff assessment provides directional information for the participant, guiding them to adjust their strategy in the next step. The higher the participant's (escapee / hunter's) expected payoff assessment of the pure strategy at the current time step t, the greater the probability of choosing that pure strategy in the next time step.

[0056] and This represents the environmental disturbance variables for both the pursuer and the escapee at time step t; As a whole, by traversing the game tree, that is, by multiplying the payoff matrix U by the escapee's decision variable y at time step t. t This is an approximation obtained by including the environmental perturbation variables of the pursuer at time step t. Similarly, As a whole, the payoff matrix U is multiplied by the pursuer's decision variable x at time step t. t This is an approximation obtained by including the environmental disturbance variables of the escapee at time step t. <,> indicates the inner product; η t β is the learning rate that decays, and β is the strength adjustment coefficient of the regularization term;

[0057] g1(x) and g2(y) represent the regularization functions set for the pursuer and the escapee, respectively, and Φ1(x) and Φ2(y) represent the distance generation functions set for the pursuer and the escapee, respectively. and Let Φ1(x) and Φ2(y) represent the Bregman divergences calculated by Φ1(x) and Φ2(y), respectively; the regularization function g i (·) and distance generation function Φ i (·) satisfies: g i (·) relative to Φ i (·) is μ-strongly convex, where μ is a non-negative number, i∈{1,2}.

[0058] For example, g can be set i (·)=μΦ i (·). The distance generation functions Φ1(x) and Φ2(y) between the pursuer and the escapee can be set as follows:

[0059]

[0060] in and Let I represent the information sets (or decision point sets) of the pursuer and the escapee, respectively, and let x[p1(I)] and y[p2(I)] represent the probabilities of the pursuer and the escapee reaching decision point set I, respectively. and Let p1(I) represent the probability distributions of the pursuer and the escapee in decision point set I, respectively, and let p1(I) represent the parent sequence of the sequences contained in decision point set I. and They can be set to respectively or j∈{1,2}, s is a probability distribution, s i It represents a component of the probability distribution s.

[0061] (3) Traverse the game tree to obtain the noisy feedback information of the pursuer and the escapee at the current time step. and

[0062] (4) The pursuers and the escapees respectively update the auxiliary variables using the feedback information that may contain noise:

[0063]

[0064] in and These represent the feedback information observed by the two participants at time step t+1 under environmental disturbance conditions. and This represents the environmental disturbance variables for both the pursuer and the escapee at time step t+1. The meanings of other parameters are consistent with the setting method and the settings during the decision variable update process.

[0065] After the above calculations are completed, a new round of iterative updates begins until the accuracy requirements are met. Accuracy can be represented by availability.

[0066]

[0067] Where ξ t Indicates the decision variable (x) t ,y t The availability is calculated, U represents the payoff matrix, t is the number of iterations, and the decision variable (x) is... t ,y t ) represents the strategy used in the t-th iteration.

[0068] Using a sequence-based strategy, vectors x, y, The decomposition proceeds from bottom to top to the decision point sets corresponding to the pursuer and the escapee. Taking the pursuer's decision variable x as an example, the decomposition method is as follows: The probability distribution of the pursuer in decision point set I can be obtained by dividing the sequence of the pursuer's decision components in decision point set I by the probability of reaching decision point set I.

[0069] Then, the proposed algorithm is used to solve the problem. The mean of the random noise is set to 0, and the standard deviation is set to σ = 0.1. The optimization algorithm is iterated continuously, with 2000 iterations. The distance generation functions are set to the extended Euclidean norm and the extended entropy, respectively. The iteration results are as follows. Figure 4 As shown, the horizontal axis represents the number of iterations, and the vertical axis represents exploitability. A lower exploitability indicates that the current strategy is closer to a Nash equilibrium solution. Figure 4 It can be seen that even though the participants can only obtain noisy feedback information, the proposed strategy solution method can still quickly converge to an approximate Nash equilibrium solution, which improves the solution efficiency and reduces the consumption of computing and storage resources.

[0070] The invention will be illustrated below using a simple example of a drone task allocation problem.

[0071] A reconnaissance mission needs to be carried out, and two drones are available for selection. The commander randomly assigns two of three different equipment levels (L1, L2, and L3, where L1 is the lowest and L3 is the highest) to the two drones. Each drone only knows its own equipment level. The drones then compete for the right to execute the mission. The drone that wins the mission will receive corresponding rewards after completing it, while the other drone will incur corresponding losses. Assuming this is a zero-sum game, the sum of the rewards for the two drones is zero. The drone with the higher-level equipment has priority in winning the mission.

[0072] Figure 5 The game tree representing the task allocation problem contains all possible decision scenarios, where check, bet, fold, and call represent possible strategies.

[0073] exist Figure 5 In the diagram, triangles represent opportunity nodes. In this task allocation problem, opportunity nodes can be considered as commanders who randomly allocate equipment. These commanders only participate in equipment allocation and do not participate in the game itself. Circles indicate that it's Drone 1's (UAV1) turn to make a decision, squares indicate that it's Drone 2's (UAV2) turn, and diamonds represent leaf nodes, signifying the end of the game. The number below each diamond represents Drone 1's payoff. Because this task allocation problem is a zero-sum game, Drone 2's payoff is the opposite of Player 1's payoff.

[0074] An information set, also known as a decision point: Figure 5 Nodes connected by dashed lines (red / blue) indicate that the drone is currently unaware of its actual node position. Figure 5 Taking the two nodes connected by the red dotted line in the upper left corner as an example, it is currently Drone 1's turn to make a decision: Drone 1 has received L1 equipment, but Drone 1 does not know whether Drone 2 has received L2 or L3 equipment. Therefore, these two nodes are the same for Drone 1. Thus, these two nodes can be merged into one information set (decision point).

[0075] Will Figure 5 Nodes that cannot be distinguished by drone 1 are merged, and a game tree is drawn from drone 1's perspective to obtain... Figure 6 Decision Point indicates that it is Drone 1's turn to make a decision, and Observation Point indicates that it is Drone 2's turn to make a decision.

[0076] Similarly, Figure 5Nodes that cannot be distinguished by UAV 2 are merged, and a game tree is drawn from UAV 2's perspective to obtain... Figure 7 Decision Point indicates that it is Drone 2's turn to make a decision, and Observation Point indicates that it is Drone 1's turn to make a decision.

[0077] The decision variable for UAV 1 is x = [P(j0,start),P(j1,check),P(j1,bet),P(j2,check),P(j2,bet),P(j3,check),P(j3,bet),P(j4,fold),P(j4,call),P(j5,fold),P(j5,call),P(j6,fold),P(j6,call)]. The decision variable x contains the probability of UAV 1 choosing each sequence, where (j0,start), (j1,check), ... represent sequences, and P(j0,start) represents the probability of choosing sequence (j0,start). Other expressions with the same form have similar meanings.

[0078] The payoff matrix is ​​shown in Table 4. The first column of the table represents all sequences of drone 1, and the first row represents all sequences of drone 2. For example, the sequence (j1, bet) of drone 1 means that drone 1 chooses the strategy bet at decision point j1, and the sequence (j4, fold) of drone 1 means that drone 1 chooses the strategy check at decision point j1 and the strategy fold at decision point j4.

[0079] The following uses drone 1 as an example. Figure 6 The game tree drawn from the perspective of drone 1:

[0080] 1. The sum of the probabilities of drone 1 choosing different strategies at a certain decision point is equal to 1. For example, the probability of drone 1 choosing strategy check at decision point j2 + the probability of drone 1 choosing strategy bet at decision point j2 = 1.

[0081] 2. Taking the sequences (j6, fold) and (j6, call) of drone 1 as an example. Because drone 1 is in sequence (j3, check)...

[0082] The sequence that follows is (j6, fold) and (j6, call). Therefore, the sequence (j3, check) is the parent sequence of the sequences (j6, fold) and (j6, call), and the sequences (j6, fold) and (j6, call) are the sequences (j3, check).

[0083] subsequence.

[0084] 3. For UAV 1, the information set (set of decision points)

[0085] 4. Assume x represents the decision variable of UAV 1, and x[I] represents the decision components in x corresponding to all sequences contained in information set (decision point) I, where j0, j1, j2, j3, j4, j5, and j6 are all information sets, also called decision points. Figure 6 Taking information set (decision point) j4 as an example, information set j4 contains the sequences (j4, fold) and (j4, call), x[j4] = [P(j4, fold), P(j4, call)]. Since the parent sequence of the sequence contained in information set j4 is (j1, check), x[p1(j4)] = P(j1, check). Therefore, the probability distribution of UAV 1 in information set j4 is x[j4] / x[p1(j4)] = x[j4] / P(j1, check) =

[0086] [P(j4,fold),P(j4,call)] / P(j1,check). Each element of the auxiliary variable has a similar meaning to the decision variable.

[0087] 5. The probability P(j6,fold) of drone 1 choosing sequence (j6,fold) = the probability of drone 1 choosing strategy check in j3 × the probability of drone 1 choosing strategy fold in j6. The probability P(j6,call) of drone 1 choosing sequence (j6,call) = the probability of drone 1 choosing strategy check in j3 × the probability of drone 1 choosing strategy fold in j6.

[0088] The probability of making a call.

[0089] 6. The probability of drone 1 selecting a certain sequence is equal to the sum of the probabilities of all its subsequences (if any). Taking the sequence (j1, check) as an example, its subsequences are (j4, fold) and (j4, call), meaning that the parent sequence of both (j4, fold) and (j4, call) is the sequence (j1, check).

[0090] P(j1,check)=P(j4,fold)+P(j4,call).

[0091] 7. The probability of choosing a certain strategy at a certain decision point = the probability corresponding to the sequence / the probability of the parent sequence of the sequence. Taking the sequence (j5, fold) as an example, the parent sequence of the sequence (j5, fold) is the sequence (j2, check). Therefore, the probability of UAV 1 choosing fold at decision point j5 = P(j5, fold) / P(j2, check).

[0092] In this task allocation problem, drone 1 has 13 sequences and drone 2 has 13 sequences, therefore the payoff matrix has a dimension of 13×13. Of course, not all game problems have the same number of sequences for both participants, so the payoff matrix is ​​not necessarily a square matrix. This payoff matrix is ​​sparse, with many elements being 0 because a certain sequence from drone 1 or drone 2 cannot be guaranteed to reach a leaf node, and therefore has no payoff. If a sequence from drone 1 or drone 2 can be guaranteed to reach a leaf node, then it will receive the corresponding payoff.

[0093] Table 4. Reward Matrix for Unmanned Aerial Vehicle (UAV) Task Allocation Problem

[0094]

[0095] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.

Claims

1. A method for solving pursuit-escape game strategies in the presence of environmental perturbations, characterized in that, include: Step 1: Based on the strategy sets of the pursuer and the escapee, set their initial strategies at different decision points, and traverse the game tree to obtain the initial values ​​of the strategy variables and auxiliary variables in sequence form; Step 2: At each time step, the pursuer and the escapee update their decision variables using the feedback information obtained from the previous time step; traverse the game tree to obtain the feedback information of the pursuer and the escapee at the current time step; the pursuer and the escapee update their auxiliary variables using the feedback information obtained at the current time step. The feedback information is obtained by traversing the entire game tree, specifically by multiplying the updated decision variables by the payoff matrix, and is considered to contain noise. A regularization term is added to the update formulas for decision variables and auxiliary variables. The regularization term is strongly convex with respect to the distance generation function in the update formula, which makes the policy obtained in the last iteration of policy learning converge to the Nash equilibrium solution without having to calculate the policy mean at each time step.

2. The method as described in claim 1, characterized in that, The updated decision variable is: At time step t+1, both the pursuer and the escapee update their decision variables using the feedback information obtained at time step t: Where x and Let y and y represent the sequential decision variable and auxiliary variable of the pursuer, respectively. Let these represent the decision variables and auxiliary variables in sequential form for the escapees, respectively; and These represent the sets of strategies for the pursuer and the escapee, respectively. and These represent the feedback information of the pursuer and the escapee at time step t under environmental disturbance conditions, respectively. Each component in the vector represents the hunter's expected payoff assessment of its chosen pure strategy at time step t. Each component in the vector represents the escapee's expected payoff assessment for choosing the corresponding pure strategy at time step t. and This represents the environmental disturbance variables for both the pursuer and the escapee at time step t; As a whole, by traversing the game tree, that is, by multiplying the payoff matrix U by the escapee's decision variable y at time step t. t This is an approximation obtained by including the environmental perturbation variables of the pursuer at time step t. Similarly, As a whole, the payoff matrix U is multiplied by the pursuer's decision variable x at time step t. t The calculated value is obtained by multiplying by an approximation and includes the environmental disturbance variable of the escapee at time step t. <,> indicates the inner product; η t β is the learning rate that decays, and β is the strength adjustment coefficient of the regularization term; g1(x) and g2(y) represent the regularization functions set for the pursuer and the escapee, respectively, and Φ1(x) and Φ2(y) represent the distance generation functions set for the pursuer and the escapee, respectively. and Let Φ1(x) and Φ2(y) represent the Bregman divergences calculated by Φ1(x) and Φ2(y), respectively; the regularization function g i (·) and distance generation function Φ i (·) satisfies: g i (·) relative to Φ i (·) is μ-strongly convex, where μ is a non-negative number, i∈{1,2}.

3. The method as described in claim 1, characterized in that, The updated auxiliary variable is: At time step t+1, both the pursuer and the escapee update auxiliary variables using the feedback information obtained at time step t+1: Where x and Let y and y represent the sequential decision variable and auxiliary variable of the pursuer, respectively. Let these represent the decision variables and auxiliary variables in sequential form for the escapees, respectively; and These represent the sets of strategies for the pursuer and the escapee, respectively. They represent the feedback information obtained by the pursuer and the escapee at time step t under environmental disturbance conditions, respectively. Each component in the vector represents the hunter's expected payoff assessment of its chosen pure strategy at time step t. Each component in the vector represents the escapee's expected payoff assessment for choosing the corresponding pure strategy at time step t. and This represents the environmental disturbance variables for both the pursuer and the escapee at time step t+1; As a whole, multiply the payoff matrix U by the escapee's decision variable y at time step t+1. t+1 This is an approximation, and the calculated value contains the environmental perturbation variables of the pursuer at time step t+1. Similarly, As a whole, the payoff matrix U is multiplied by the pursuer's decision variable x at time step t+1. t+1 This is an approximation, and the calculated value includes the environmental disturbance variables of the escapee at time step t+1. <,> indicates the inner product; η t β is the learning rate that decays, and β is the strength adjustment coefficient of the regularization term; g1(x) and g2(y) represent the regularization functions set for the pursuer and the escapee, respectively, and Φ1(x) and Φ2(y) represent the distance generation functions set for the pursuer and the escapee, respectively. and Let Φ1(x) and Φ2(y) represent the Bregman divergences calculated by Φ1(x) and Φ2(y), respectively; the regularization function g i (·) and distance generation function Φ i (·) satisfies: g i (·) relative to Φ i (·) is μ-strongly convex, where μ is a non-negative number, i∈{1,2}.

4. The method as described in claim 2 or 3, characterized in that, The learning rate satisfies 5. The method as described in claim 2 or 3, characterized in that, Set g i (·)=μΦ i (·) 6. The method as described in claim 1, characterized in that, Step 1 includes: selecting a suitable probability distribution for each decision point based on the action space of the pursuer and the escapee at each decision point, as the initial point for the strategy learning of the pursuer and the escapee; traversing the game tree to obtain the strategy variables and auxiliary variables in sequence form for the pursuer and the escapee respectively, and obtaining the feedback information in the initialization phase.

Images