Multi-agent Bayesian strategy reusing method and system based on view angle invariance

Abstract

The invention discloses a multi-agent Bayesian strategy reuse method and system based on view angle invariance, and the method comprises the steps: taking different opponent strategies as different tasks, carrying out the interactive learning of an optimal coping strategy with opponents using different strategies one by one, and adding the optimal coping strategy to a controlled strategy library of a controlled agent as a controlled strategy; then, a controlled agent equipped with a strategy library interacts with an opponent agent, a track is generated, the track is encoded through a view angle invariant context awareness module to obtain context features covering environment dynamic change information, and a supervision model is constructed in combination with task categories corresponding to the context features to serve as a local observation model; and calculating likelihood through a local observation model based on the local trajectory, updating belief according to a Bayesian rule, and selecting an optimal coping strategy from a controlled strategy library by the controlled agent according to the belief. According to the method, opponent strategies can be quickly and accurately identified and the optimal coping strategy can be reused in multi-agent collaborative confrontation.

Description

Technical Field

[0001] The present invention relates to the field of control technology of multi-agent systems, and in particular to a multi-agent Bayesian strategy reuse method and system based on perspective invariance. Background Art

[0002] A multi-agent system is a generalization of classic decision-making problems in which multiple agents (usually robots) interact in a shared environment to achieve common or conflicting goals. Multi-agent reinforcement learning is one of the most effective techniques for studying such systems and has been widely used in traffic light control, team games, smart grids and other fields in recent years. Classic multi-agent reinforcement learning methods, such as QMIX, MAPPO, BiCNet and other algorithms, mainly focus on how to learn a single task, assuming that the environment is partially observable or that the enemy strategy is fixed in a team competition environment. Therefore, the main purpose of the above methods is to adopt corresponding techniques (e.g., value decomposition, learning communication, etc.) to promote collaboration within the team. However, the opponent's strategy in the real world may be changing, that is, the environment is non-stationary.

[0003] Considering the above problems, training from scratch will lead to inefficient learning. In contrast, strategy reuse based on the transfer framework is a direct and efficient method that has been widely studied. This method explicitly classifies the opponent's strategy and allows the opponent's strategy to show great differences, which is impossible to achieve with implicit opponent modeling and meta-learning methods. Through this strategy reuse method, existing knowledge and experience can be better utilized, learning efficiency can be improved, and better results can be achieved in complex environments.

[0004] The Bayesian Policy Reuse (BPR) framework has shown significant potential in addressing the above issues. BPR is a general transfer framework that reuses existing source policies when faced with unlabeled (but previously seen) tasks. However, existing BPR algorithms have the following main limitations: 1) The number of controlled agents is limited to one, making it difficult to adapt to scenarios where multiple controlled agents appear at the same time; 2) The controlled agent can obtain complete environmental information online, including global state and opponent trajectory, making it difficult to adapt to partially observable scenarios. 3) The observation model designed by the variant method is heavily dependent on the opponent information, and these methods may fail when the opponent information is not available.

[0005] In the scenario of multi-agent distributed strategy reuse under partial observability, each agent has a separate brain, which is more flexible and easier to expand than the centralized one. However, as mentioned above, it is impossible to directly expand the existing BPR method. In order to solve this challenge, the following conditions need to be met. First, only the information that can be obtained by partial observability can be used as observation signals. Second, when executing online, each agent needs to carry an independent observation model. Finally, when facing the same task, the strategies selected by the agents should be consistent, that is, to reach a cognitive consensus. If the strategies selected by the controlled agents are inconsistent, the final joint strategy will become chaotic, thereby reducing the overall performance. Summary of the invention

[0006] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a multi-agent Bayesian strategy reuse method and system based on perspective-invariant is provided. The present invention aims to quickly and accurately identify opponent strategies and reuse the optimal response strategies in multi-agent collaborative confrontation.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: A multi-agent Bayesian strategy reuse method based on perspective invariance is used for robot control in a multi-agent system consisting of two types of agents: a controlled robot as a controlled agent and an adversarial robot as an opponent agent. The controlled robot selects a controlled strategy from a preset controlled strategy library. The best response strategies include: S101, treat different opponent strategies as different tasks, for the opponent strategy library The opponent's strategy in , interact with the opponent agent to learn the optimal response strategy and use it as a controlled strategy Add to the library of controlled policies for controlled agents In; According to the controlled strategy library , the opponent strategy library of the opponent agent The trajectories are generated interactively, and then the trajectories are encoded through the perspective-invariant situational awareness module to obtain contextual features covering the dynamic changes of the environment. The supervision model is constructed as a local observation model based on the task category corresponding to the contextual features. ; S102, based on the local trajectory through the local observation model Calculate the likelihood and update the belief according to Bayes' rule. The controlled agent selects from the controlled policy library according to the belief. Choose the best response strategy.

[0008] Optionally, in step S101, interactively learning the optimal response strategy with the opponent agent means that the controlled agent uses the MAPPO algorithm to interactively learn the optimal response strategy with the opponent agent, and the opponent strategy library of the opponent agent The function expression is: , In the above formula, ~ are the opponent agents The opponent's strategy; The resulting controlled strategy library The function expression is: , In the above formula, ~ The controlled agents are A controlled strategy.

[0009] Optionally, when the trajectory is encoded by the view-invariant scenario-aware module in step S101 to obtain context features covering the dynamic change information of the environment, the encoder used includes a global context encoder and a local context encoder, wherein the function expression of the global context encoder is: , , In the above formula, is the global context feature at time t, is the global context encoder, are the network parameters of the global context encoder, Use strategy for time t The global trajectory generated by the interaction with the environment, is the initial global state of the controlled agent, is the initial action of the controlled agent, is the initial action of the opponent agent, For controlled agents The global state at the moment, For the controlled agent The action of the moment, For the opponent agent The action at the moment; the function expression of the local context encoder is: , , In the above formula, For controlled agents exist The local context features at the moment, is the local context encoder, are the network parameters of the local context encoder, for Always use strategy Local trajectories resulting from interaction with the environment, For controlled agents The initial local observations, For controlled agents Initial action, For controlled agents exist The local observation at time, For controlled agents exist Actions, are the shared local context encoder network parameters for all controlled agents.

[0010] Optionally, the loss function expression used by the global context encoder and the local context encoder when training at any perspective is: , In the above formula, is the loss function, is the sample size, for and Contrasting loss between two views, for and Contrasting loss between two views, is the feature output by the global context encoder, For controlled agents The features output by the local context encoder are For controlled agents The features output by the local context encoder of , and the contrast loss function expression of the two perspectives is: , In the above formula, For is the contrast loss of the aiming point, For is the contrast loss of the aiming point. In order to ensure that the currently controlled agent can use the corresponding observation model to infer the task category when facing any task, for each response strategy , a context encoder can be constructed for it separately, and there are: , , In the above formula, represents the expected value, For the strategy The global context feature vector under For controlled agents In Strategy The local context feature vector under is the number of tasks, To store data units, For the strategy The local context feature vector under For the strategy The global context feature vector under is the similarity score function, and , , in the superscript is transposed.

[0011] Optionally, in step S101, a supervision model is constructed as a local observation model in combination with the task category corresponding to the context feature. When , the task category corresponding to the context feature is obtained by learning the mapping from the local context to the corresponding task category using the K nearest neighbor algorithm.

[0012] Optionally, step S102 includes: S201, based on experience queue Record each controlled agent The local track is input into the local context encoder corresponding to the selected strategy To calculate the corresponding local context features , the length of the experience queue Q is To store each controlled agent The local trajectory of ,in For controlled agents The initial local observations, For controlled agents The initial action, For controlled agents exist The local observation at time, For controlled agents The action at time t; S202, each controlled intelligent body The online interactive local trajectory is input into the corresponding local observation model In order to predict the task category ,in ~ are the predicted labels corresponding to the local trajectories at time 0 to t respectively; S203, calculate each controlled intelligent agent according to the following formula The likelihood: , In the above formula, For when the task is to use When The probability of To predict the label Medium Task The number of occurrences, is the number of tasks, ; S204, according to the controlled intelligent agent shown in the following formula The Bayesian rule updates the belief: , In the above formula, is the controlled agent at the current moment Use strategy against your opponent The belief is the controlled agent at the previous moment Use strategy against your opponent The belief is the strategy of the opponent's strategy library, For controlled agents Using strategies against opponents Generate experience queue The probability of is the controlled agent at the previous moment Use strategy against your opponent belief; S205: The controlled agent selects a controlled strategy library according to the belief and the preset selection strategy. Choose the best response strategy.

[0013] Optionally, in step S205, the controlled agent selects from the controlled strategy library according to the belief and the preset selection strategy. When selecting the optimal response strategy, the function expression of the preset selection strategy is: , In the above formula, For the best response strategy, is the upper limit of points, is the lower limit of the integral, For controlled agents Target tasks The belief For the strategy The utility value is The probability of is the integration variable, and: , In the above formula, For controlled agents Use strategy against your opponent The belief For the strategy The expected utility under .

[0014] In addition, the present invention also provides a multi-agent Bayesian strategy reuse system based on perspective invariance, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the multi-agent Bayesian strategy reuse method based on perspective invariance.

[0015] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the perspective-invariant multi-agent Bayesian strategy reuse method through a processor.

[0016] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the perspective-invariant multi-agent Bayesian strategy reuse method through a processor.

[0017] Compared with the prior art, the present invention has the following advantages: In the collaborative confrontation of multiple agents, a single strategy is often difficult to deal with complex and changeable opponents. The Bayesian strategy reuse framework focuses on how to select the optimal strategy from the existing source strategy library to deal with non-stationary opponents who suddenly switch strategies. However, most previous studies limit the number of controlled agents to one and assume that global information is observable. In order to be closer to the actual situation and weaken the above assumptions, the present invention proposes a multi-agent Bayesian strategy reuse method based on perspective invariance. First, in order to meet the setting of local observability of multiple agents, the present invention uses local trajectories as observation signals instead of opponent trajectories or round rewards commonly used in most BPR algorithms. Secondly, since the changes of agents are all reflected in the global trajectory, the present invention designs a perspective-invariant context contrast perception module, which aims to enable locally observable controlled agents to perceive the dynamic changes of global information. Specifically, the module uses contrastive learning to bring different perspectives of the same task closer and push the perspectives of different tasks farther to extract context features. This module not only helps the controlled agents reach a cognitive consensus on the dynamic changes of the environment, but also establishes clear boundaries for tasks in the feature space. Furthermore, in order to establish the relationship with the task category, the present invention uses supervised learning to map the learned context features to the corresponding task labels. Then, the learned supervised model is used as the local observation model. In two confrontation scenarios, it is verified that the method of the present invention can quickly and accurately identify the opponent's strategy and reuse the optimal response strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of the basic principle of the method of the embodiment of the present invention.

[0019] Figure 2 This is the pursuit environment in the embodiment of the present invention.

[0020] Figure 3 This is a hierarchical foraging environment in an embodiment of the present invention.

[0021] Figure 4 The average round reward is reused for the pursuit environment strategy in the embodiment of the present invention.

[0022] Figure 5 The cumulative reward is reused for the hunting environment strategy in the embodiment of the present invention.

[0023] Figure 6 This is the strategy accuracy for identifying opponents in the pursuit environment in the embodiment of the present invention.

[0024] Figure 7 The average round reward is reused for the hierarchical foraging environment strategy in the embodiment of the present invention.

[0025] Figure 8 The hierarchical foraging environment strategy in the embodiment of the present invention reuses the cumulative reward.

[0026] Fig. 9 This is the accuracy of identifying opponent strategies in a hierarchical foraging environment in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In the following, a 3v3 team competition example is taken as an example. It is assumed that the opponent agent has three strategies and the corresponding controlled agent also has three response strategies. The multi-agent Bayesian policy reuse method based on view-invariant contrastawareness (BPR-VCA method for short) of the present invention is further explained in detail.

[0028] like Figure 1 As shown in the figure, the perspective-invariant multi-agent Bayesian strategy reuse method of this embodiment is used for robot control in a multi-agent system composed of two types of agents, a controlled robot as a controlled agent and an adversarial robot as an opponent agent, including the controlled robot selecting a controlled strategy library from a preset controlled strategy library. The best response strategies include: S101, treat different opponent strategies as different tasks, for the opponent strategy library The opponent's strategy in , interact with the opponent agent to learn the optimal response strategy and use it as a controlled strategy Add to the library of controlled policies for controlled agents In; According to the controlled strategy library , the opponent strategy library of the opponent agent The trajectories are generated interactively, and then the trajectories are encoded through the perspective-invariant situational awareness module to obtain contextual features covering the dynamic changes of the environment. The supervision model is constructed as a local observation model based on the task category corresponding to the contextual features. ; S102, based on the local trajectory through the local observation model Calculate the likelihood and update the belief according to Bayes' rule. The controlled agent selects from the controlled policy library according to the belief. Choose the best response strategy.

[0029] Step S101 is an offline stage. Figure 1 As shown in (a) in the offline phase, the coping strategy and opponent strategy Interactively generate trajectories, and then use these trajectories to generate local observation models through the VCA module ; Step S102 is the online stage, such as Figure 1 As shown in (b); in the online stage, based on the local trajectory Calculate the likelihood and update the belief according to Bayes’ rule. Further, the controlled agent updates the belief from Choose the best strategy.

[0030] In step S101 of this embodiment, the controlled agent interacts with the opponent agent to learn the optimal response strategy by using the MAPPO algorithm (known method) to interact with the opponent agent to learn the optimal response strategy, and the opponent agent's opponent strategy library The function expression is: , In the above formula, ~ are the opponent agents The opponent's strategy; The resulting controlled strategy library The function expression is: , In the above formula, ~ The controlled agents are This embodiment mainly focuses on the problem of strategy reuse, so both the controlled agent and the opponent agent are equipped with a strategy library. Before officially performing the task, the first task of the controlled agent is to learn the optimal response strategy. , the controlled agent uses the MAPPO algorithm to interact with the opponent agent and learn the optimal response strategy Then, the learned coping strategies are added to the controlled strategy library Repeat this process until the controlled strategy library is filled. Given The opponent strategy library is represented as . Accordingly, the controlled agent strategy library is represented as The controlled agent uses the MAPPO algorithm to interact with the opponent agent to learn the optimal response strategy, including: for each opponent strategy , using the MAPPO algorithm to learn the response strategy and add it to Then, for For each controlled strategy m among the opponent strategies, for each opponent strategy other than opponent strategy m , using a controlled strategy Countering opponent strategies ,Will and the corresponding task tags Add to training dataset From the training data set Get the sample training data set from the dataset and use the KNN algorithm to train the local observation model .

[0031] This embodiment, in the perspective-invariant multi-agent Bayesian strategy reuse method, for the first time extends the BPR framework to partially observable multi-agent team competition scenarios with local trajectories as observation signals. This embodiment proposes a perspective-invariant situational awareness module that can perceive the dynamic changes of the environment and promote cognitive consensus among agents. In the BPR method, observation model modeling is the most core because it is a key component for updating beliefs online. It determines the degree of refinement of strategy detection frequency and the effectiveness of strategy reuse. At present, the observation model constructed by the BPR algorithm mainly considers signals such as round rewards, transition states, and opponent trajectories. However, the above-mentioned observation signals have several defects. First, round rewards can only be obtained at the end of a round. In addition, task errors under sparse rewards may lead to zero benefits, resulting in indistinguishable performance models. Then, the remaining two signals may be completely ineffective in partially observable scenarios. In order to overcome the above challenges, this embodiment uses local trajectories as observation signals, which are more general. However, relying solely on pure local information may not be able to fully detect changes in opponent strategies. This is because after the controlled agent conducts multi-step exploration, changes in local information may still be weak and difficult to detect. Since the changes of the agent are all reflected in the global information, this embodiment expects that the local information can approximate the global information. At the same time, the observation model needs to have a clear boundary when facing different tasks to better identify the opponent's strategy. In order to meet the above expectations, this embodiment designs a perspective-invariant contrast situation perception module (such as Figure 1 As shown). This module has two main purposes: 1) to realize the perception of dynamic changes in the environment and promote cognitive consensus among controlled intelligence; 2) to form clear boundaries between different task trajectories in the feature space. In particular, this embodiment brings the feature representations of different perspectives under the same task closer in the feature space, while pushing the feature representations under different tasks further away. That is to say, the global perspective and the local perspective under the same task are both used as positive samples, while the global perspective and the local perspective under different tasks are both used as negative samples. Specifically, in step S101 of this embodiment, when the trajectory is encoded by the perspective-invariant scenario perception module to obtain contextual features covering information on dynamic changes in the environment, the encoder used includes a global context encoder and a local context encoder, wherein the function expression of the global context encoder is: , , In the above formula, is the global context feature at time t, is the global context encoder, are the network parameters of the global context encoder, Use strategy for time t The global trajectory generated by the interaction with the environment, is the initial global state of the controlled agent, is the initial action of the controlled agent, is the initial action of the opponent agent, For controlled agents The global state at the moment, For the controlled agent The action of the moment, For the opponent agent The action at the moment; the function expression of the local context encoder is: , , In the above formula, For controlled agents exist The local context features at the moment, is the local context encoder, are the network parameters of the local context encoder, for Always use strategy Local trajectories resulting from interaction with the environment, For controlled agents The initial local observations, For controlled agents Initial action, For controlled agents exist The local observation at time, For controlled agents exist Actions, are the shared local context encoder network parameters for all controlled agents.

[0032] This embodiment first uses the global context encoder and the local context encoder to embed the historical interaction information into a compact but information-rich feature space. The global encoder is defined as ,in Indicated in Always use strategy The global trajectory generated by the interaction with the environment, is the global state, is the joint action of the controlled agents, is the joint action of the opponent agent, is the global context feature vector, is the network parameter of the global context encoder. At the same time, define the agent The local encoder ,in Indicated in Moments, use strategies Local trajectories resulting from interaction with the environment, It is a local observation. is the local context feature vector, is the network parameter of the local context encoder, which is shared. In order to ensure that the currently controlled agent can use the corresponding observation model to infer the task category when facing any task. Therefore, for each response strategy , a context encoder can be constructed for it separately. Given the trajectory data generated by the interaction between the agent strategy and any opponent strategy, if the global trajectory is fixed, then from the agent Enumerating positive and negative examples from the local perspective of , the loss can be written as: , in, is a similarity score function. It can be related to mutual information and provides a tractable lower bound on the mutual information objective: , in, is the number of tasks, is the global context feature Local context features Therefore, minimizing the loss It means maximizing the mutual information between the global and local views of the same task. Symmetrically, the contrast loss of the two views is: , By extending the two perspectives to any number of perspectives and considering the relationship between any two perspective pairs, the optimization objective can be written as: .

[0033] Therefore, the loss function expression used by the global context encoder and the local context encoder in this embodiment when training at any perspective is: , In the above formula, is the loss function, is the sample size, for and Contrasting loss between two views, for and Contrasting loss between two views, is the feature output by the global context encoder, For controlled agents The features output by the local context encoder are For controlled agents The features output by the local context encoder of , and the contrast loss function expression of the two perspectives is: , In the above formula, For is the contrast loss of the aiming point, For is the contrast loss of the aiming point. In order to ensure that the currently controlled agent can use the corresponding observation model to infer the task category when facing any task, for each response strategy , a context encoder can be constructed for it separately, and there are: , , In the above formula, represents the expected value, For the strategy The global context feature vector under For controlled agents In Strategy The local context feature vector under is the number of tasks, To store data units, For the strategy The local context feature vector under For the strategy The global context feature vector under is the similarity score function, and , , in the superscript is the transpose. From the perspective of each task, For each task m in the tasks, the training process of the global context encoder and the local context encoder includes: (1) initializing the global context encoder corresponding to task m and local context encoder Network parameters; (2) for the maximum number of samples Each of the samples and Every other task in the : By strategy Generate Trajectory and , , ,Will and Add to dedicated policy Replay buffer (3) From the replay buffer Get Trajectory, calculate the loss function by formula And optimize the global context encoder and local context encoder The network parameters of task m are obtained by and local context encoder .

[0034] The standard BPR uses round rewards as observation signals, which contain very limited information and need to wait until the end of the round to be obtained. Secondly, the performance models for different opponents may be the same. For example, under sparse rewards, only successful tasks can obtain benefits, and any mistakes may result in zero benefits. In the previous step, this embodiment discussed how to encode local trajectories through the VCA module (view-invariant situational awareness module) to obtain contextual features that cover information about dynamic changes in the environment. However, in order to accurately identify the opponent's strategy during online execution, it is necessary to establish a direct relationship between contextual features and specific task categories. Therefore, a natural idea is to use these contextual features and their corresponding task categories as input-output for supervised learning. This embodiment uses the classic K-nearest neighbor (KNN) algorithm to learn the mapping from local context to the corresponding task category. Then, the supervised model is regarded as a local observation model. Specifically, in step S101 of this embodiment, a supervision model is constructed as a local observation model in combination with the task category corresponding to the context feature. When , the task category corresponding to the context feature is obtained by learning the mapping from the local context to the corresponding task category using the K nearest neighbor algorithm.

[0035] In the online stage, the degree of confidence of the controlled agent in its belief is crucial to the accuracy of its strategy selection. Figure 1 As shown in (b) of , this embodiment updates the belief based on the local observation model and reuses the source strategy from the local trajectory. Specifically, this embodiment maintains the experience queue Q, and the length of each queue is . It is used to store each controlled agent The online interaction local trajectory can be expressed as Then, it is input to each currently controlled agent The local encoder corresponding to the selected strategy To calculate the corresponding local context vector Next, the local context vector is input into the corresponding local observation model to predict the task category , is a predicted label. Specifically, step S102 in this embodiment includes: S201, based on experience queue Record each controlled agent The local track is input into the local context encoder corresponding to the selected strategy To calculate the corresponding local context features , the length of the experience queue Q is To store each controlled agent The local trajectory of ,in For controlled agents The initial local observations, For controlled agents The initial action, For controlled agents exist The local observation at time, For controlled agents The action at time t; S202, each controlled intelligent body The online interactive local trajectory is input into the corresponding local observation model In order to predict the task category ,in ~ are the predicted labels corresponding to the local trajectories at time 0 to t respectively; S203, calculate each controlled intelligent agent according to the following formula The likelihood: , In the above formula, For when the task is to use When The probability of To predict the label Medium Task The number of occurrences, is the number of tasks, ; S204, according to the controlled intelligent agent shown in the following formula The Bayesian rule updates the belief: , In the above formula, is the controlled agent at the current moment Use strategy against your opponent The belief is the controlled agent at the previous moment Use strategy against your opponent The belief is the strategy of the opponent's strategy library, For controlled agents Using strategies against opponents Generate experience queue The probability of is the controlled agent at the previous moment Use strategy against your opponent belief; S205: The controlled agent selects a controlled strategy library according to the belief and the preset selection strategy. Choose the best response strategy.

[0036] In step S205 of this embodiment, the controlled agent selects a selected strategy from the controlled strategy library according to the belief and the preset selection strategy. When selecting the optimal response strategy, the function expression of the preset selection strategy is: , In the above formula, For the best response strategy, is the upper limit of points, is the lower limit of the integral, For controlled agents Target tasks The belief For the strategy The utility value is The probability of is the integration variable, and: , In the above formula, For controlled agents Use strategy against your opponent The belief For the strategy The expected utility under .

[0037] The following three specific application environments illustrate the effectiveness of the method (BPR-VCA) proposed in this embodiment. The comparison algorithms include the well-known methods BPR+, Bayes-ToMoP, and Deep BPR+. BPR+ extends BPR to adversarial settings and only relies on the performance model to update the belief at the end of each round. BPR-ToMoP assumes that the opponent also uses Bayesian reasoning strategies to exhibit more complex behaviors and considers the opponent's predictions in belief updates. Deep BPR+ explicitly models the opponent's strategy to correct the belief model during online execution. In the experiment, Deep BPR+ was extended to multiple controlled agents and used global information to update the belief model. and local information The modeled adversaries are referred to as Deep BPR+(G) and Deep BPR+(L) in this embodiment.

[0038] The experimental design of this embodiment is as follows: the initial state of the intelligent agent in the pursuit environment is as follows: Figure 2 As shown in the figure, the agents in the multi-agent system in this environment include controlled robots and dynamic confrontation robots. The controlled robots include three hunting robots A1-A3, and the dynamic confrontation robots include three preys B1-B3. The system is surrounded by black walls, and the agents cannot cross the walls. The goal of the hunting robots is to capture the prey, while the goal of the prey is to avoid capture. Before each round starts, the controlled robots can select from the controlled strategy library. A controlled policy is chosen based on the belief. At each time step , the controlled robot can move from the action space Select and perform an action in the game. If the prey successfully evades the hunting robot, the prey group will receive a reward of 1. If one prey is captured, the prey group will receive a reward of -1; if two prey are captured, they will receive a reward of -2; if three prey are captured, they will receive a reward of -5. After the time step, a game ends. In addition, the environment is partially observable, and the prey can only observe other agents within a certain range. In the experiment, this embodiment controls the prey group and regards the three hunting robots as a unified opponent. And the opponent strategy library Four strategies were designed: 1) three hunting robots track prey in a clockwise trajectory; 2) three hunting robots track prey in a counterclockwise trajectory; 3) two hunting robots track prey in a clockwise trajectory, while one hunting robot tracks prey in a counterclockwise trajectory; 4) two hunting robots track prey in a counterclockwise trajectory, while one hunting robot tracks prey in a clockwise trajectory; 5) three hunting robots track prey in the horizontal direction; 6) three hunting robots track prey in the vertical direction.

[0039] The initial state of the agent in the hierarchical foraging environment is as follows: Figure 3As shown in the figure, the agents in the multi-agent system in this environment include controlled robots and adversarial robots, where the controlled robots include two red team robots (A1 and A2) and the adversarial robots include two blue team robots (B1 and B2). Apples represent different tasks performed by robots, including two red apples (R1 and R2) and one yellow apple (Y1). Each grid in the figure can only accommodate one agent, except for the position of apples. The goal of the robots of the red team and the blue team is to eat as many apples as possible within the specified time (equivalent to reaching the position and performing the specified task). In this environment, the rewards for reaching the position and performing different types of specified tasks are different, and the number of controlled robots required for different types of tasks may also be different. The required rules can be set according to actual needs. For example, eating a yellow apple can get a reward of 1, while eating a red apple can get a reward of 2. However, the red apple requires that the team members eat it at the same time to get the reward. Once the apples are eaten or the maximum time step T=100 is reached, the round ends and the position of the agent is reset. In the experiment, the red team robot group (A1 and A2) is controlled, and the two blue team robots (B1 and B2) are regarded as an opponent as a whole, and the opponent strategy library is Three strategies were designed: (1) B1 eats the yellow apple first, then B1 and B2 eat the red apple R1 and red apple R2 in turn; (2) B2 eats the yellow apple first, then B1 and B2 eat the red apple R1 and red apple R2 in turn; (3) B1 and B2 eat the red apple R2 first, then B1 eats the yellow apple, and finally eats the red apple R1 together. Both agents move along the horizontal axis first and then along the vertical axis.

[0040] In the online phase, the pursuit experiment environment was divided into three phases and ran for a total of 100 rounds, with 100 experiments repeated. In the first 30 rounds, the adversary inferred the strategy type of the controlled agent based on the standard BPR framework at the beginning of each round. In rounds 30-60, the adversary initially adopted strategy (3) and switched to strategy (5) after 2 steps every 2 rounds. In rounds 60-100, the adversary used: The sequential switching strategy. Figures 4 and 5 show the round reward and cumulative reward respectively, and Figure 6 shows the recognition accuracy when facing different opponent strategies at different stages. The different color legends in the figure represent different algorithms, and the shaded area and gray line represent the standard deviation. The alternating background colors in Figures 4 and 5 are used to distinguish different stages.

[0041] In the first 30 rounds, it can be seen from the figure that the method of this embodiment (BPR-VCA) can achieve a 100% recognition rate when the opponent also uses BPR. At the same time, from the perspective of round rewards and cumulative rewards, the method of this embodiment (BPR-VCA) is also the best. In rounds 30-60, the opponent switches strategies every 5 rounds. In this stage, BPR-ToMoP, BPR+, Deep BPR+(G) and Deep BPR+(L) perform consistently in terms of recognition accuracy and round rewards. However, due to the accumulation of the previous stage, Deep BPR+(L) has the lowest cumulative reward. The method of this embodiment (BPR-VCA) performs well in reward evaluation and recognition accuracy, indicating that it can cope with opponents who switch strategies within rounds to a certain extent. In rounds 60-100, the opponent selects strategies in a certain order, which covers the opponent's strategy library. BPR-ToMoP, BPR+, and Deep BPR+(G) performed consistently in terms of recognition accuracy, with Deep BPR+(L) performing the worst. Although the method in this embodiment (BPR-VCA) had some errors, it still achieved a recognition rate of 92.5%. From the reward curve in the figure, it can be seen that the method in this embodiment (BPR-VCA) can track the opponent's strategy at an early moment in a round.

[0042] The comparison of strategy recognition accuracy obtained in this embodiment is shown in Table 1.

[0043] Table 1 Comparison of strategy recognition accuracy (mean ± variance)

[0044] As shown in Table 1, from the results of the entire experiment, the recognition rate of the method in this embodiment (BPR-VCA) can reach 97%, while the recognition rates of other baseline algorithms are only between 50% and 60%. At the same time, the method in this embodiment (BPR-VCA) also obtains the most rewards. This shows that the method in this embodiment (BPR-VCA) can accurately identify the opponent's strategy before a round ends and switch its own strategy in time, so it performs best in terms of rewards and recognition accuracy. At the same time, it is noted that BPR+ and Deep BPR+ (G) have consistent performance throughout the experiment. This is because when the rewards in the evaluation environment are not sparse and the performance model is distinguishable, BPR+ can meet the requirements by relying solely on the performance model.

[0045] Similar to the previous example, this example also analyzes the experimental results of the online reuse phase in the hierarchical foraging environment. It is also divided into three phases and runs a total of 100 rounds, and the experiment is repeated 100 times. The legend is the same as PP. In the first 30 rounds, the opponent infers the strategy type of the controlled agent based on the standard BPR framework at the beginning of each round. In the 30th to 60th rounds, the opponent randomly selects a strategy from the strategy library every 5 rounds. In the 60th to 100th rounds, the opponent uses: Sequential switching strategy.

[0046] In the first 30 rounds, Figures 7 and 9 clearly show that only the method of this embodiment (BPR-VCA) achieves the best round reward and its detection accuracy reaches 100% when the opponent also has reasoning ability. Although Bayes-ToMoP is designed for BPR opponents, it needs to determine that its opponent uses BPR after its average reward exceeds a predefined threshold. The performance of Deep BPR+(G) is the second best, BPR+ is the worst, and Deep BPR+(L) has difficulty in accurately modeling the opponent's strategy using local information. In rounds 30-60, the variance of several algorithms increases because the opponent switches randomly. The performance of the method of this embodiment (BPR-VCA) also decreases, but it is still the best with a recognition accuracy of 91.2%. It is followed by the Deep BPR+(G) method with a recognition accuracy of 87.4%; followed by the BPR+ method with a recognition accuracy of 84%. The remaining two algorithms are between 60% and 80%. It can be observed from Figure 9 that the frequency of the opponent switching strategies has decreased in this stage, and the method of this embodiment (BPR-VCA) is slightly higher than Deep BPR+(G). However, based on the reward advantage accumulated in the previous stage, it can be clearly observed from the data in Figure 8 that the cumulative rewards obtained by the method of this embodiment (BPR-VCA) far exceed those of the baseline method. In rounds 60-100, this embodiment sets the opponent to switch between strategies (2) and (3) at intervals of strategy (1). According to the performance model constructed in the offline Deep BPR+(G) stage, the performance model may be the same when the controlled agent uses strategy (1) and the opponent uses strategy (2) or (3). At this stage, if Figure 2As shown in Figure 1, the performance of BPR+ and Bayes-ToMoP, which strongly depend on the performance model, has dropped significantly, and their recognition accuracy is less than 60%. However, although Deep BPR+(G) is designed to solve this problem, its recognition accuracy is limited because it can only update its beliefs at the end of the round. As shown in Table 1, from the overall experimental results, the method of this embodiment (BPR-VCA) performs well in identifying the opponent's strategy, with a recognition accuracy of up to 96.8%. It is followed by the Deep BPR+(G) method, with a recognition accuracy of 67.6%. The recognition rates of the remaining three algorithms are all less than 60%. At the same time, it can be seen from Figures 7 and 8 that the reward obtained by the method of this embodiment (BPR-VCA) is also the most generous, which shows that the method of this embodiment (BPR-VCA) can identify the opponent's strategy faster. From the above analysis, it can be seen that the algorithm of this embodiment (BPR-VCA) can identify opponents with high-frequency switching strategies in the middle of each round of interaction by relying only on local trajectories, and the benefits obtained are also the highest.

[0047] In summary, in the collaborative confrontation of multiple agents, a single strategy is often difficult to deal with complex and changeable opponents. The Bayesian strategy reuse framework focuses on how to select the optimal strategy from the existing source strategy library to deal with non-stationary opponents who suddenly switch strategies. However, most previous studies limit the number of controlled agents to one and assume that global information is observable. In order to be closer to the actual situation and weaken the above assumptions, the method of this embodiment proposes a multi-agent Bayesian strategy reuse method based on perspective invariance. First, in order to meet the setting of local observability of multiple agents, this embodiment uses local trajectories as observation signals instead of opponent trajectories or round rewards commonly used in most BPR algorithms. Secondly, since the changes of agents are all reflected in the global trajectory, this embodiment designs a perspective-invariant context contrast perception module, which aims to encourage locally observable controlled agents to perceive the dynamic changes of global information. Specifically, this module uses contrast learning to bring different perspectives of the same task closer and push away perspectives under different tasks to extract contextual features. This module not only helps the controlled agents reach a cognitive consensus on the dynamic changes of the environment, but also establishes clear boundaries for tasks in the feature space. Furthermore, in order to establish a relationship with the task category, this embodiment uses supervised learning to map the learned context features to the corresponding task labels. Then, the learned supervised model is used as a local observation model. Finally, this embodiment verifies in two confrontation scenarios that the perspective-invariant multi-agent Bayesian strategy reuse method of this embodiment can quickly and accurately identify the opponent's strategy and reuse the optimal response strategy.

[0048] In addition, this embodiment also provides a multi-agent Bayesian strategy reuse system based on perspective invariance, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the multi-agent Bayesian strategy reuse method based on perspective invariance.

[0049] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the perspective-invariant multi-agent Bayesian strategy reuse method through a processor.

[0050] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the perspective-invariant multi-agent Bayesian strategy reuse method through a processor.

[0051] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0052] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A multi-agent Bayesian strategy reuse method based on perspective invariance, characterized in that: It is used for robot control in a multi-agent system consisting of two types of agents, a controlled robot as a controlled agent and an adversarial robot as an opponent agent, wherein the controlled robot selects a preset controlled strategy library. The best response strategies include: S101, treat different opponent strategies as different tasks, for the opponent strategy library The opponent's strategy in , interact with the opponent agent to learn the optimal response strategy and use it as a controlled strategy Add to the library of controlled policies for controlled agents In; According to the controlled strategy library , the opponent strategy library of the opponent agent The trajectories are generated interactively, and then the trajectories are encoded through the perspective-invariant situational awareness module to obtain contextual features covering the dynamic changes of the environment. The supervision model is constructed as a local observation model based on the task category corresponding to the contextual features. ; S102, based on the local trajectory through the local observation model Calculate the likelihood and update the belief according to Bayes' rule. The controlled agent selects from the controlled policy library according to the belief. Choose the best response strategy.

2. The perspective-invariant multi-agent Bayesian strategy reuse method according to claim 1 is characterized in that: In step S101, the controlled agent interacts with the opponent agent to learn the optimal response strategy by using the MAPPO algorithm to interact with the opponent agent to learn the optimal response strategy, and the opponent agent's opponent strategy library The function expression is: ， In the above formula, ~ are the opponent agents The opponent's strategy; The resulting controlled strategy library The function expression is: ， In the above formula, ~ are the controlled agents A controlled strategy.

3. The perspective-invariant multi-agent Bayesian strategy reuse method according to claim 1 is characterized in that: When the trajectory is encoded by the view-invariant scenario-aware module in step S101 to obtain context features covering the dynamic change information of the environment, the encoder used includes a global context encoder and a local context encoder, wherein the function expression of the global context encoder is: ， ， In the above formula, is the global context feature at time t, is the global context encoder, are the network parameters of the global context encoder, Use strategy for time t The global trajectory generated by the interaction with the environment, is the initial global state of the controlled agent, is the initial action of the controlled agent, is the initial action of the opponent agent, For controlled agents The global state at the moment, For the controlled agent The action of the moment, For the opponent agent The action at the moment; the function expression of the local context encoder is: ， ， In the above formula, For controlled agents exist The local context features at the moment, is the local context encoder, are the network parameters of the local context encoder, for Always use strategy Local trajectories resulting from interaction with the environment, For controlled agents The initial local observations, For controlled agents Initial action, For controlled agents exist The local observation at time, For controlled agents exist Actions, are the shared local context encoder network parameters for all controlled agents.

4. The perspective-invariant multi-agent Bayesian strategy reuse method according to claim 3 is characterized in that: The loss function expression used by the global context encoder and the local context encoder when training at any perspective is: ， In the above formula, is the loss function, is the sample size, for and Contrasting loss between two views, for and Contrasting loss between two views, is the feature output by the global context encoder, For controlled agents The features output by the local context encoder are For controlled agents The features output by the local context encoder of , and the contrast loss function expression of the two perspectives is: ， In the above formula, For is the contrast loss of the aiming point, For is the contrast loss of the aiming point. In order to ensure that the currently controlled agent can use the corresponding observation model to infer the task category when facing any task, for each response strategy , a context encoder can be constructed for it separately, and there are: ， ， In the above formula, represents the expected value, For the strategy The global context feature vector under For controlled agents In Strategy The local context feature vector under is the number of tasks, To store data units, For the strategy The local context feature vector under For the strategy The global context feature vector under is the similarity score function, and , , in the superscript is transposed.

5. The perspective-invariant multi-agent Bayesian strategy reuse method according to claim 1 is characterized in that: In step S101, a supervision model is constructed as a local observation model by combining the task category corresponding to the context feature. When , the task category corresponding to the context feature is obtained by learning the mapping from the local context to the corresponding task category using the K nearest neighbor algorithm.

6. The perspective-invariant multi-agent Bayesian strategy reuse method according to claim 3 is characterized in that: Step S102 includes: S201, based on experience queue Record each controlled agent The local track is input into the local context encoder corresponding to the selected strategy To calculate the corresponding local context features , the length of the experience queue Q is To store each controlled agent The local trajectory of ,in For controlled agents The initial local observations, For controlled agents The initial action, For controlled agents exist The local observation at time, For controlled agents The action at time t; S202, each controlled intelligent body The online interactive local trajectory is input into the corresponding local observation model In order to predict the task category ,in ~ are the predicted labels corresponding to the local trajectories at time 0 to t respectively; S203, calculate each controlled intelligent agent according to the following formula The likelihood: ， In the above formula, For when the task is to use When The probability of To predict the label Medium Task The number of occurrences, is the number of tasks, ; S204, according to the controlled intelligent agent shown in the following formula The Bayesian rule updates the belief: ， In the above formula, is the controlled agent at the current moment Use strategy against your opponent The belief is the controlled agent at the previous moment Use strategy against your opponent The belief is the strategy of the opponent's strategy library, For controlled agents Using strategies against opponents Generate experience queue The probability of is the controlled agent at the previous moment Use strategy against your opponent belief; S205: The controlled agent selects a controlled strategy library according to the belief and the preset selection strategy. Choose the best response strategy.

7. The perspective-invariant multi-agent Bayesian strategy reuse method according to claim 6 is characterized in that: In step S205, the controlled agent selects from the controlled strategy library according to the belief and the preset selection strategy. When selecting the optimal response strategy, the function expression of the preset selection strategy is: ， In the above formula, For the best response strategy, is the upper limit of the score, is the lower limit of the integral, For controlled agents Target tasks The belief For the strategy The utility value is The probability of is the integration variable, and: ， In the above formula, For controlled agents Use strategy against your opponent The belief For the strategy The expected utility under .

8. A multi-agent Bayesian strategy reuse system based on perspective invariance, comprising interconnected microprocessors and memories, characterized in that: The microprocessor is programmed or configured to execute the perspective-invariant multi-agent Bayesian strategy reuse method described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the perspective-invariant multi-agent Bayesian strategy reuse method described in any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the perspective-invariant multi-agent Bayesian strategy reuse method described in any one of claims 1 to 7 through a processor.

Images