Multi-agent exploration method and device fusing priori knowledge

Abstract

The invention relates to the technical field of reinforcement learning, and provides a multi-agent exploration method and device fusing priori knowledge. The method comprises the following steps: acquiring priori knowledge through environment learning; and the intelligent agent interacts with the environment based on the overall strategy and the prior knowledge to obtain a data set so as to realize strategy training of the intelligent agent. The priori knowledge is obtained through environment learning, the priori knowledge is utilized to restrain environment interaction, namely, the action space of the intelligent agent in a special state is limited, sampling of the intelligent agent to the environment is reduced, the action space is reduced, the problems of slow convergence and the like of multi-intelligent-agent reinforcement learning are solved, the exploration efficiency is improved, and the method is suitable for large-scale popularization and application. And collaborative exploration among multiple agents and optimization of the whole system are realized.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a multi-agent exploration method and device integrating prior knowledge. Background Art

[0002] As the third paradigm of machine learning, reinforcement learning (RL) overcomes the constraint of supervised learning that requires a large number of labeled data sets, and has achieved good performance in solving the problems of difficult and inaccurate modeling. Since Q-learning (full name: Q-Learning), reinforcement learning has rapidly developed from single-agent algorithms to multi-agent algorithms. To date, multi-agent reinforcement learning has been successfully applied in logistics, Go, robotics, and games.

[0003] However, in a multi-agent environment, as the number of agents increases, the exponential growth of the state-action space leads to the Curse of Dimensionality. This makes exploration more difficult, especially in an environment with sparse rewards, where policy convergence requires an unimaginable number of interactions. Slow convergence and low exploration efficiency hinder the widespread application of multi-agent reinforcement learning in the real world.

[0004] In view of this, how to overcome the defects of the existing technology, reduce the sampling of the environment by the intelligent agent and accelerate the learning process is a problem to be solved in this technical field. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a multi-agent exploration method integrating prior knowledge to alleviate problems such as slow convergence of multi-agent reinforcement learning and improve exploration efficiency.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a multi-agent exploration method integrating prior knowledge, comprising:

[0008] Acquire prior knowledge through environmental learning;

[0009] The agent interacts with the environment based on the overall strategy and prior knowledge to obtain data sets to achieve strategy training of the agent.

[0010] Preferably, the acquiring prior knowledge through environmental learning specifically includes:

[0011] Build the original virtual environment;

[0012] A learning environment is obtained by adding noise to an original virtual environment; wherein the noise includes one or more of action noise, state noise, parameter noise and time noise;

[0013] The prior knowledge is obtained from the learning environment; wherein the prior knowledge is a knowledge base related to the original virtual environment obtained by integrating expert knowledge learned in the learning environment and knowledge obtained by inverse reinforcement learning.

[0014] Preferably, the agent interacts with the environment to obtain a data set based on an overall strategy and prior knowledge, specifically including:

[0015] In each round, the environment state is reset to the initial state and the initial joint observation of all agents is obtained. In each time step of a round, the agent interacts with the environment based on the overall strategy and prior knowledge to obtain the transition tuple of the corresponding time step, and each transition tuple is stored to obtain a data set;

[0016] Each round includes at least one time step, and the initial joint observation of the agents in one round is used as the joint observation of the first time step to perform interactions within the first time step.

[0017] Preferably, in each time step of a round, the agent interacts with the environment based on the overall strategy and prior knowledge to obtain the transfer tuple of the corresponding time step, specifically including:

[0018] At time step t, the current state is used as the starting state s t , each agent has a starting state s t According to the prior knowledge and overall strategy, the action to be performed at the current time step is selected, and the actions selected by all agents are combined into a joint action.

[0019] Perform joint action a t , the state after the joint action is executed is taken as the successor state s t+1 , and the agent receives the environmental reward r t and the joint observation o at the next time step t+1 ;

[0020] Set the starting state s of the current time step to t , the joint observation o at the current time step t , joint action a t , successor state s t +1 , the joint observation o at the next time step t+1 and environmental rewards t The transition tuple (s) that constitutes the current time step t ,o t ,at ,s t+1 ,o t +1 ,r t ).

[0021] Preferably, each agent is based on the initial state s t According to prior knowledge and overall strategy, the actions to be performed at the current time step are selected, including:

[0022] If the corresponding action can be obtained from prior knowledge, then this action is used as the action to be performed at the current time step;

[0023] Otherwise, the action to be performed at the current time step is determined based on the overall policy.

[0024] Preferably, after performing the joint action, each agent observes the environment to obtain the local observation of the agent at the next time step, and the local observations of all agents at the next time step constitute the joint observation of the next time step. t+1 .

[0025] Preferably, in each round, initialize the target strategy π of each agent i ; then the overall strategy of all agents is

[0026] Preferably, after completing the strategy training of the agent, a sequence consisting of samples of the maximum cumulative discounted reward in the process of transitioning from the initial state to the target state is obtained from the data set, which is the optimal path of the agent.

[0027] In a second aspect, the present invention further provides a multi-agent exploration device based on prior knowledge, which is used to implement the multi-agent exploration method based on prior knowledge described in the first aspect, and the device comprises:

[0028] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the multi-agent exploration method based on prior knowledge described in the first aspect.

[0029] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0030] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method of the first aspect.

[0031] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method according to the first aspect.

[0032] The present invention acquires prior knowledge through environmental learning and uses the prior knowledge to constrain environmental interaction, that is, to limit the action space of the agent under special conditions, thereby reducing the agent's sampling of the environment and achieving a reduction in the action space, thereby alleviating problems such as slow convergence of multi-agent reinforcement learning, improving exploration efficiency, and achieving collaborative exploration among multiple agents and optimization of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 It is a flowchart of a first multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of a multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0036] Figure 3 is a schematic diagram of another multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0037] Figure 4 is a flow chart of a second multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0038] Figure 5 is a flow chart of a third multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0039] Figure 6 is a flowchart of a fourth multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0040] Figure 7 is a flowchart of a fifth multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0041] Figure 8 is a flowchart of a sixth multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0042] Fig. 9 is a flowchart of a seventh multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0043] Fig.10 is a schematic diagram of another multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0044] Fig.11 is a flow chart of an eighth multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0045] Fig.12 is a flowchart of a ninth multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0046] Fig.13 is a flowchart of a tenth multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention;

[0047] Fig.14 It is a schematic diagram of the architecture of a multi-agent exploration device integrating prior knowledge provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0049] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0050] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0051] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific characteristic contents) will be involved, and the corresponding expressions include the following three combinations: only A, only B, and a combination of A and B.

[0052] As used herein, "about," "substantially," or "approximately" includes the stated value and an average value that is within an acceptable range of deviation from the particular value as determined by one of ordinary skill in the art taking into account the measurements in question and the errors associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0053] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0054] Embodiment 1:

[0055] In order to solve the dimensionality curse problem during multi-agent exploration, a variety of exploration methods have been proposed. For example, Successor Representation (full name in English: Successor representation) and Count-Based Exploration via Embedded State Space for Deep Reinforcement Learning (WCMC) both use count-based exploration strategies to encourage agents to visit states that are rarely visited; SAC (full name: Soft Actor-Critic) and KME (full name: k-Means Maximum Entropy Exploration) promote the full exploration of the environment by agents based on the maximum entropy exploration method.

[0056] However, slow convergence is a common problem. This is not only due to the complexity and diversity of the environment, but also closely related to the interdependence and lack of information exchange between agents. These factors together lead to the fact that in practical applications, MARL algorithms often require more training time and samples to achieve satisfactory performance. In order to solve this problem, embodiment 1 of the present invention provides a multi-agent exploration method that integrates prior knowledge, such as Figure 1 As shown, including:

[0057] In step 201, prior knowledge is acquired through environmental learning; specifically: a virtual environment is constructed by a person skilled in the art according to requirements, and prior knowledge is acquired through learning in the virtual environment.

[0058] In step 202, the agent interacts with the environment based on the overall strategy and prior knowledge to obtain a data set to implement the strategy training of the agent. In actual implementation, if necessary, the interaction between the agent and the environment can be completed through reinforcement learning. The specific reinforcement learning method selected is obtained by those skilled in the art based on empirical analysis. Figure 2 Online reinforcement learning (On-line RL), in which the agent and the environment are constantly interacting, and the dataset is continuously expanded as the interaction proceeds; or Figure 3 On-policy RL, in which the sample data comes from the current policy and each sample data is used only once. In the process of interaction between the agent and the environment, the overall strategy and prior knowledge are used to limit the actions selected by the agent, that is, the overall strategy and prior knowledge can be understood as action selection rules under specific states. In other optional embodiments, the task can be decomposed based on prior knowledge or the reward function can be designed based on prior knowledge.

[0059] This embodiment acquires prior knowledge through environmental learning and uses prior knowledge to constrain environmental interactions, that is, to limit the action space of the agent under special conditions, thereby reducing the agent's sampling of the environment and achieving a reduction in the action space, thereby alleviating problems such as slow convergence of multi-agent reinforcement learning, improving exploration efficiency, and achieving collaborative exploration among multiple agents and optimization of the overall system.

[0060] In a specific application scenario, the prior knowledge is acquired through environmental learning, such as Figure 4 As shown, specifically including:

[0061] In step 301, an original virtual environment is constructed.

[0062] In step 302, a learning environment is obtained by adding noise to the original virtual environment; wherein the noise includes one or more of action noise, state noise, parameter noise and time noise.

[0063] In step 303, the prior knowledge is obtained from the learning environment; wherein the prior knowledge is a knowledge base related to the original environment obtained by integrating the expert knowledge learned in the learning environment and the knowledge obtained by inverse reinforcement learning. That is, a person skilled in the art constructs a virtual environment according to the requirements, and obtains a noisy version of the learning environment by adding action noise, state noise, parameter noise, and time noise to the original environment; then, the expert knowledge learned in the virtual environment and the knowledge obtained by inverse reinforcement learning are integrated to obtain a knowledge base related to the original environment.

[0064] In an optional embodiment, the agent interacts with the environment to obtain a data set based on the overall strategy and prior knowledge, such as Figure 5 As shown, specifically including:

[0065] In step 401, in each round, the environment state is reset to the initial state, and the initial joint observations of all agents are obtained; that is, the local observations of all agents in the initial state constitute the initial joint observations of the agents.

[0066] In step 402, in each time step of a round, the agent interacts with the environment based on the overall strategy and prior knowledge to obtain the transition tuple of the corresponding time step, and stores each transition tuple to obtain a data set.

[0067] Each round includes at least one time step, and the initial joint observation of the agents in one round is used as the joint observation of the first time step to perform interactions within the first time step.

[0068] In each time step of a round, the agent interacts with the environment based on the overall strategy and prior knowledge to obtain the transfer tuple of the corresponding time step, such as Figure 6 As shown, specifically including:

[0069] In step 501, at time step t, the current state is used as the starting state s t , each agent has a starting state s t According to the prior knowledge and overall strategy, the action to be performed at the current time step is selected, and the actions selected by all agents are combined into a joint action.

[0070] In step 502, a joint action a is performed. t , the state after the joint action is executed is taken as the successor state s t+1 , and the agent receives the environmental reward r t and the joint observation o at the next time step t+1 .

[0071] In step 503, the initial state s of the current time step ist , the joint observation o at the current time step t , joint action a t , successor state s t+1 , the joint observation o at the next time step t+1 and environmental rewards t The transition tuple (s) that constitutes the current time step t ,o t ,a t ,s t+1 ,o t+1 ,r t ).

[0072] Moreover, after performing the joint action, each agent observes the environment to obtain the local observation of the agent at the next time step, and the local observations of all agents at the next time step constitute the joint observation of the next time step. t+1 In each round, initialize the target strategy π of each agent i ; then the overall strategy of all agents is That is, each agent has a target strategy. For multiple agents, the entire algorithm is regarded as the joint strategy of all agents, that is, the overall strategy.

[0073] In specific application scenarios, the priority of prior knowledge is greater than the priority of the overall strategy, that is, each agent is based on the initial state s t According to the prior knowledge and the overall strategy, the action to be performed at the current time step is selected, such as Figure 7 As shown, specifically including:

[0074] In step 601, if the corresponding action can be obtained from prior knowledge, the action is used as the action to be performed in the current time step.

[0075] In step 602, otherwise, the action to be performed at the current time step is determined according to the overall strategy.

[0076] After completing the strategy training of the intelligent agent, the sequence of samples with the maximum cumulative discounted reward in the process of transferring from the initial state to the target state is obtained from the data set, which is the optimal path of the intelligent agent.

[0077] This embodiment constructs a virtual environment based on adding noise and acquires relevant knowledge through environmental learning and fusion knowledge base, realizes the collection of prior knowledge, and uses prior knowledge to constrain action selection, thereby reducing the action space, alleviating the slow convergence of multi-agent reinforcement learning and other problems, and improving exploration efficiency.

[0078] Embodiment 2:

[0079] The present invention is based on the method described in Example 1, combined with specific application scenarios, and uses technical descriptions in related scenarios to illustrate the implementation process of the present invention in characteristic scenarios.

[0080] like Figure 8 As shown, the specific steps of a multi-agent exploration method integrating prior knowledge provided by an embodiment of the present invention are as follows:

[0081] In step 701, prior knowledge is acquired through environmental learning. In specific implementation, a virtual environment needs to be constructed. A noisy version of the learning environment is obtained by adding action noise, state noise, parameter noise, and time noise to the original environment (i.e., the original virtual environment); then, the expert knowledge learned in the virtual environment (i.e., the learning environment) and the knowledge acquired through inverse reinforcement learning are combined to obtain a knowledge base related to the original environment.

[0082] In step 702, based on prior knowledge and overall strategy, transition tuples are acquired through the interaction between the agent and the environment, and all acquired transition tuples are used as experience pool data sets.

[0083] In order to train multiple agents, it is first necessary to obtain sample data for training. In this embodiment, the agent selects actions to interact with the environment based on prior knowledge and overall strategy, and obtains the subsequent state of the environment state and the environment reward after the agent performs different actions to form a transition tuple. After storing the transition tuples of the agent at different time steps, a continuously updated data set can be obtained, and each transition tuple is used as a sample in the data set.

[0084] In step 703, the strategy training is completed by updating the data set. In the specific implementation, the data set is updated through the continuous interaction between the agent and the environment, so as to obtain the best convergence strategy for the agent to access the terminal state from the initial state - to obtain the maximum cumulative discounted reward.

[0085] This embodiment trains multiple agents as a whole by using the method described in steps 701 to 703, and uses the prior knowledge obtained from environmental learning to limit the action space, thereby reducing the interaction between the agent and the environment, accelerating the learning process of the agent, and alleviating the problem of dimensionality curse.

[0086] In actual implementation, the interaction between the agent and the environment is completed through reinforcement learning as needed and necessary. The specific reinforcement learning method used is: Figure 2 Online reinforcement learning (On-line RL), in which the agent and the environment are constantly interacting, and as the interaction proceeds, the dataset is continuously expanded; Figure 3 This is On-policy RL. In this method, the sample data comes from the current policy and each sample data is used only once.

[0087] However, before training, it is necessary to obtain prior knowledge to limit the action space. The noisy version of the learning environment is obtained by adding action noise, state noise, parameter noise, and time noise to the original environment; then, the expert knowledge learned in the virtual environment and the knowledge obtained by inverse reinforcement learning are combined to obtain a knowledge base related to the original environment.

[0088] When training, first initialize the target strategy π of each agent i ; Then, the target strategy of all agents is

[0089] After completing the target strategy initialization, it is also necessary to complete the interaction between the agent and the environment based on the overall strategy, and use the transfer tuples obtained from the interaction to expand / update the dataset D.

[0090] The data set is updated through the continuous interaction between the agent and the environment, that is, each transfer tuple is obtained through the interaction between the overall strategy and the environment, such as Fig. 9 As shown, specifically including:

[0091] In step 801, at the beginning of each round, the environment state is reset to the initial state and the joint observations of all agents are obtained.

[0092] When exploring, the agent and the environment need to interact for multiple rounds, each round is represented by i, i∈[1...E]. When entering each round i, the environment state needs to be reset to s first. 1 , and observe the first state o when the environment is initialized 1 .

[0093] At the same time, each agent observes the initial state to obtain the local observation of the agent, and the local observations of all agents are combined into the initial joint observation. The local observation obtained by the yth agent is expressed as The first joint observation of n agents is expressed as

[0094] In step 802, at each time step, all agents select a joint action according to the initial global state based on the overall strategy and prior knowledge.

[0095] In the specific implementation, each round includes at least one time step, each time step is denoted as t, t∈(1,2…H), and the agent performs an action in each time step.

[0096] At each time step t, t∈(1,2…H), according to the global state s t, each agent uses the overall strategy (that is, each agent uses its own corresponding target strategy) to select its own actions to form a joint action. The action selected by the yth agent according to the overall strategy is expressed as The joint action consisting of the actions selected by n agents is expressed as

[0097] In order to reflect the priority of prior knowledge, one method that can be selected during programming is to obtain the action a selected by the strategy. t At this time, using prior knowledge in the environment state s t When We can let the agent perform joint actions If the prior knowledge does not restrict the actions to be taken in the current state, the agents select their own actions and then perform a joint action a t .

[0098] In step 803, after performing the joint action at each time step, the starting state is transformed into the successor state, and the agent obtains the environment reward and the next joint observation.

[0099] At each time t, before executing the joint action, the global state of the environment is s at the current time step t , the joint observation o at the current time step t After executing the joint action, the global state of the environment changes to the successor state s t+1 , all agents receive a total environmental reward r t , the joint observation of all agents is transformed into the second joint observation o t+1 .

[0100] In step 804, the starting state, joint observation, joint action, successor state, next joint observation and environmental reward of each time step are combined into a transfer tuple of the current time step, which is stored to obtain an experience replay data set.

[0101] After obtaining the global state and joint observation before and after the joint action is executed, as well as the corresponding environmental rewards, the corresponding parameters can be combined into the transfer tuple of the time step. The transfer tuple can be expressed as (s t ,o t ,a t ,s t+1 ,o t+1 ,r t ).

[0102] In this embodiment, after completing the interactions of all time steps in all rounds using the method described in steps 801 to 804, the final updated data set D can be obtained. Based on D, the path / sequence of the agent's transition from the initial state to the final state can be determined.

[0103] In the above process, the method of multi-agent training includes building a virtual environment, using virtual environment learning and knowledge base to obtain prior knowledge, realizing the interaction between the agent and the environment based on the overall strategy and prior knowledge, and completing the strategy training. In a specific scenario, the complete training process of multi-agent is as follows: Fig.10 As shown in Figure 1, the training process of the agent includes: acquiring prior knowledge, interacting with the environment under the guidance of prior knowledge to obtain samples. When the agent interacts with the environment, at each time step t, the agent performs an action a guided by prior knowledge. t , so that the environment state changes from the initial state s t Change to the successor state s t+1 , and obtain the corresponding reward value r t In the next time step, the successor state s in the previous time step is t+1 As the starting state s of this time step t After each time step, the target policy π is trained using the updated dataset D until the multi-agent training is completed.

[0104] This embodiment can reduce the sampling of the environment by the agent, realize the accelerated convergence of multiple agents and the optimization of the whole system. Specifically: the virtual environment is constructed by adding noise and the relevant knowledge is acquired through environmental learning and fusion knowledge base, realizing the collection of prior knowledge. And the action selection is constrained by prior knowledge, realizing the reduction of action space, alleviating the problems of slow convergence of multi-agent reinforcement learning, and improving the exploration efficiency.

[0105] Embodiment 3:

[0106] Considering that there may be multi-target exploration scenarios in actual use, in order to cope with this scenario, this embodiment also provides a preferred implementation method based on Embodiment 1 and Embodiment 2. Embodiment 1 of the present invention provides a multi-agent exploration method integrating prior knowledge, such as Fig.11 As shown, including:

[0107] In step 901, agents are reallocated to each target according to the loss function at every preset number of rounds; wherein multiple agents assigned to one target form an agent group; the preset number is obtained by those skilled in the art based on empirical analysis. In order to realize agent collaborative exploration in a multi-target reinforcement learning environment, it is necessary to group agents and targets, allocate agents (make some agents as professional agents of the target); and in order to rationalize the allocation, a loss function is constructed and minimized; when a target reaches the final state, the agent allocated to the target is released so that it can serve other targets.

[0108] In addition, in order to rationalize the use of resources and prevent a certain target from occupying a certain agent for a long time, it is necessary to regroup the agent and the target, and release the agent occupied by the target so that it can be used for exploration by other agents. At the same time, in order not to make the grouping process occupy too much computing resources, it is planned to regroup once every M rounds.

[0109] In step 902, the target strategy of each agent is determined according to the target of each agent group, and each agent is trained for multiple rounds using the target strategy and prior knowledge, that is, the agent in Example 1 interacts with the environment to obtain a data set based on the overall strategy and prior knowledge to achieve the strategy training of the agent, wherein the target strategy of all agents constitutes the overall strategy in Example 1. The target strategy is used to train the agent to access the target and collect data to obtain the maximum environmental reward.

[0110] In a specific implementation, the target strategy of each agent is determined according to the target of each agent group, which can be: setting a corresponding strategy for each target in advance, and taking the strategy of the target of the agent group to which the agent belongs as the target strategy of the agent.

[0111] This embodiment dynamically groups agents and targets (i.e., allocates agents) based on the constructed loss function, solves the difficult exploration problem in multi-target reinforcement learning, and realizes the collaborative allocation among agents. It also considers the optimal allocation of resources, so as to regroup agents and targets in a timely manner, and realizes efficient training of multiple agents.

[0112] In an optional implementation, the loss function is Where N is the number of targets participating in the allocation, k v is the number of agents required to participate in the vth goal of the assignment, dist vu is the distance between the vth target and the corresponding uth agent involved in the allocation; When dist vu The smaller it is, the closer the agent u is to the target v; x is the horizontal coordinate of the u-th agent, u y is the ordinate of the u-th agent, v x is the horizontal coordinate of the vth target, v y is the ordinate of the vth target. The smaller the value of the loss function, the better the allocation method.

[0113] Considering that in actual use, the number of agents may not completely match the number required by the target, in order to cope with various scenarios, this embodiment also provides a preferred implementation method, that is, according to the loss function, the agents are reallocated to each target, such as Fig.12 As shown, specifically including:

[0114] In step 1001, when the total number of agents When the total number of agents can meet the needs of all unfinished targets, all unfinished targets are used as targets for allocation, and the targets for allocation are allocated agents to minimize the value of the loss function.

[0115] In step 1002, when the total number of agents When the total number of agents cannot meet the needs of all unfinished targets, w targets are selected from all unfinished targets as targets to be allocated, and agents to minimize the value of the loss function; wherein w can be predetermined by those skilled in the art, or a target selection strategy can be predetermined by those skilled in the art, and w targets are selected according to the target selection strategy.

[0116] Among them, K i is the target of the i-th participant in the allocation, K v is the number of agents required for the vth unfinished target, and M is the number of unfinished targets. The unfinished target means that the target has not reached the required position, that is, the agent has not reached the required position of the target.

[0117] There is another situation: P <K v , that is, the total number of agents is less than the number of agents required for the vth unfinished goal. At this time, the vth unfinished goal does not participate in the allocation of agents; when P <min(K v ), that is, when the total number of agents is less than the number of agents required for each goal, no allocation of agents is performed and the exploration task cannot be completed.

[0118] Among them, the above method selects w targets from all unfinished targets as targets to be allocated, and allocates agents to minimize the value of the loss function, such as Fig.13 As shown, specifically including:

[0119] In step 1101, multiple combinations are taken from unfinished goals; wherein the number of goals included in each combination is the maximum number corresponding to the required number of agents less than or equal to P; wherein P is the total number of agents; the number of goals included in each combination is the maximum number corresponding to the required number of agents less than or equal to P, that is, the total number of agents required for all goals in the combination is just less than or equal to P, and when any other goal is added to the combination, the number of agents required in the combination exceeds P.

[0120] In step 1102, taking each combination as the target for participation in allocation, calculate the values of the loss function under different agent allocation methods; the agent allocation method refers to which agents are allocated to which targets.

[0121] In step 1103, use the combination corresponding to the minimum loss function value as the target for participation in allocation, and use the agent allocation method corresponding to the minimum loss function value to allocate agents; step 1103 can be understood as selecting, among all possible combinations and all agent allocation methods under all combinations, the one that minimizes the value of the loss function.

[0122] In step 1104, alternatively, take the combination with the largest number of targets as the target for participation in allocation, and use the agent allocation method corresponding to the minimum loss function value of the targets participating in allocation to allocate agents. Step 1104 can be understood as first determining the combination that can explore the most targets, and then determining the optimal agent allocation method under this combination.

[0123] In a specific application scenario, that is, when the number of agents required for each unfinished target is the same, where w is the number of targets participating in allocation, k is the number of agents required for each unfinished target, P%k represents the remainder of P divided by k, represents rounding down.

[0124] Taking another case as an example, that is, when the number of agents required for all targets is the same and is k, then reallocating agents for each target according to the loss function specifically includes:

[0125] If P < k, the exploration task will not be able to be completed.

[0126] If P ≥ k×N, the agents required for the target are sufficient. At this time, the loss function for selecting k×N agents for allocation is Minimize the loss function to allocate k agents for each agent v to complete the grouping of agents and targets; when a certain target (assumed to be numbered N) reaches the final state, release the agents allocated to this target, and minimize Allocate k×(N - 1) agents for the remaining N - 1 targets.

[0127] If k < P < k×N, it is not possible to allocate sufficient agents for all targets. At this time, only targets can be allocated, select k×w agents for allocation; the loss function is Minimize the loss function to assign k agents to each agent v to complete the grouping of agents and targets; after every M rounds, release the agents assigned to all targets and regroup. If a target reaches the required position, assign k×w′ agents to w′ targets among the remaining N-1 targets; minimize Assign k×(N-1) agents to the remaining N-1 goals.

[0128] It should be noted that the methods in Example 1 and Example 2 are applicable to this embodiment and will not be described in detail here.

[0129] Embodiment 4:

[0130] like Fig.14 , is a schematic diagram of the architecture of a multi-agent exploration device integrating prior knowledge according to an embodiment of the present invention. The multi-agent exploration device integrating prior knowledge according to this embodiment includes one or more processors 21 and a memory 22. Fig.14 A processor 21 is taken as an example.

[0131] The processor 21 and the memory 22 may be connected via a bus or other means. Fig.14 The example of connecting through bus is taken in the following.

[0132] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the multi-agent exploration method integrating prior knowledge in Example 1. The processor 21 executes the multi-agent exploration method integrating prior knowledge by running the non-volatile software programs and instructions stored in the memory 22.

[0133] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0134] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the multi-agent exploration method integrating prior knowledge in the above-mentioned embodiment 1 is executed.

[0135] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0136] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-agent exploration method integrating prior knowledge, characterized in that: include: Acquire prior knowledge through environmental learning; The agent interacts with the environment based on the overall strategy and prior knowledge to obtain data sets to achieve strategy training of the agent.

2. The multi-agent exploration method integrating prior knowledge according to claim 1 is characterized in that: The acquisition of prior knowledge through environmental learning specifically includes: Build the original virtual environment; A learning environment is obtained by adding noise to an original virtual environment; wherein the noise includes one or more of action noise, state noise, parameter noise and time noise; The prior knowledge is obtained from the learning environment; wherein the prior knowledge is a knowledge base related to the original virtual environment obtained by integrating expert knowledge learned in the learning environment and knowledge obtained by inverse reinforcement learning.

3. The multi-agent exploration method integrating prior knowledge according to claim 1 is characterized in that: The agent interacts with the environment based on the overall strategy and prior knowledge to obtain the data set, specifically including: In each round, the environment state is reset to the initial state and the initial joint observation of all agents is obtained. In each time step of a round, the agent interacts with the environment based on the overall strategy and prior knowledge to obtain the transition tuple of the corresponding time step, and each transition tuple is stored to obtain a data set; Each round includes at least one time step, and the initial joint observation of the agents in one round is used as the joint observation of the first time step to perform interactions within the first time step.

4. The multi-agent exploration method integrating prior knowledge according to claim 3 is characterized in that: In each time step of a round, the agent interacts with the environment based on the overall strategy and prior knowledge to obtain the transition tuple of the corresponding time step, which specifically includes: At time step t, the current state is used as the starting state s t , each agent has a starting state s t According to the prior knowledge and overall strategy, the action to be performed at the current time step is selected, and the actions selected by all agents are combined into a joint action. Perform joint action a t , the state after the joint action is executed is taken as the successor state s t+1 , and the agent receives the environmental reward r t and the joint observation o at the next time step t+1 ; Set the starting state s of the current time step to t , the joint observation o at the current time step t , joint action a t , successor state s t+1 , the joint observation o at the next time step t+1 and environmental rewards t The transition tuple (s) that constitutes the current time step t ,o t ,a t ,s t+1 ,o t+1 ,r t ).

5. The multi-agent exploration method integrating prior knowledge according to claim 4 is characterized in that: Each agent is based on the initial state s t According to prior knowledge and overall strategy, the actions to be performed at the current time step are selected, including: If the corresponding action can be obtained from prior knowledge, then this action is used as the action to be performed at the current time step; Otherwise, the action to be performed at the current time step is determined based on the overall policy.

6. The multi-agent exploration method integrating prior knowledge according to claim 4 is characterized in that: After performing the joint action, each agent observes the environment to obtain the local observation of the agent at the next time step. The local observations of all agents at the next time step constitute the joint observation of the next time step. t+1 .

7. The multi-agent exploration method integrating prior knowledge according to claim 3 is characterized in that: In each round, initialize the target strategy π of each agent i ; then the overall strategy of all agents is 8. The multi-agent exploration method integrating prior knowledge according to any one of claims 1 to 7, characterized in that: After completing the strategy training of the intelligent agent, the sequence of samples with the maximum cumulative discounted reward in the process of transferring from the initial state to the target state is obtained from the data set, which is the optimal path of the intelligent agent.

9. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the multi-agent exploration method integrating prior knowledge as described in any one of claims 1-8.

10. A multi-agent exploration device integrating prior knowledge, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the multi-agent exploration method integrating prior knowledge as described in any one of claims 1-8.

Images