Multi-agent target cooperative search method and system

Abstract

The application discloses a multi-agent target cooperative search method and system, relates to the fields of swarm intelligence and multi-agent target search technology, and comprises the following steps: acquiring environment detection information perceived by any agent in a search simulation environment; determining a deterministic policy action based on an expected return gradient setting; acquiring updated environment detection information after the agent executes the deterministic policy action; determining internal reward data based on action emotion change data and external environment reward data; the internal reward data and the external environment reward data constitute search overall reward data; the search overall reward data, current state data, updated state data and the deterministic policy action constitute experience quadruples; each agent randomly extracts experience quadruples from an experience pool for training to obtain and execute optimal policy actions, so that target cooperative search is realized. The application solves the problem of sparse rewards and improves the efficiency of multi-agent search in a complex unknown environment.

Description

Technical Field

[0001] This invention relates to the field of swarm intelligence and multi-agent target search technology, and in particular to a multi-agent target cooperative search method and system based on an improved deep deterministic policy gradient. Background Technology

[0002] In recent years, with the convergence of control, communication, and computer technologies, multi-agent cooperative control has attracted widespread attention from scholars in many countries. Compared to single-agent systems, multi-agent systems can accomplish more complex tasks and have advantages such as high efficiency, high fault tolerance, and inherent parallelism. Target search using multiple agents is one of the main challenges in the multi-agent field. Because multiple agents are assigned to search for different targets, they may interfere with each other, reducing the overall task efficiency.

[0003] Existing multi-agent target cooperative search techniques are mostly based on intelligent biomimetic algorithms, such as ant colony optimization, neural network algorithms, genetic algorithms, and particle swarm optimization. However, these intelligent biomimetic algorithms typically require multiple iterative planning steps to obtain the optimal solution. When faced with search problems in complex dynamic environments, they are prone to getting trapped in local optima, making it difficult to find feasible solutions. Deep reinforcement learning has become a popular topic in multi-agent target cooperative search in recent years. It combines the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning, providing a solution to the perception and decision-making problems of complex multi-agent systems.

[0004] Multi-agent deep reinforcement learning algorithms build upon the optimal decision-making capabilities provided by reinforcement learning, combining them with the powerful high-dimensional data representation capabilities of deep learning to fit value functions or policies. This allows for training based on interactive samples to obtain the optimal value function or policy. It uses neural networks as function approximators to generalize and approximate the value function, overcoming the shortcomings of traditional reinforcement learning and multi-agent reinforcement learning—especially the curse of dimensionality. This enables multiple agents to: 1) observe their states (or decision factors); 2) exchange information with neighboring agents (e.g., immediate rewards, Q-values, value functions, and optimal policies); 3) interact with their operational environment; and 4) learn knowledge independently and choose appropriate actions in an unsupervised manner, thereby enhancing system performance. In particular, deep deterministic policy gradient algorithms (DPRQs) can effectively handle tasks with high-dimensional or continuous action spaces, such as multi-agent search and path planning. By optimizing parameters, the policy is directly updated iteratively to maximize the cumulative expected reward. Compared to other methods, DPRQs are simpler and have better convergence. However, in environments with sparse rewards and random noise, deep reinforcement learning algorithms struggle to obtain state-action samples containing effective reward information through random exploration, resulting in inefficient training processes or even failure to learn effective policies. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-agent target cooperative search method and system, which solves the reward sparsity problem by introducing sentiment data and improves the efficiency of multi-agent search in complex and unknown environments.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A multi-agent cooperative target search method includes:

[0008] A search simulation environment is constructed; multiple agents, obstacles, and search targets are randomly set within the search simulation environment; each agent includes an Actor network and a Critic network; the Actor network is used to select the action to be executed by the agent, and the Critic network is used to evaluate the expected reward of the agent;

[0009] For any agent in the search simulation environment, acquire the environmental detection information perceived by the agent; the environmental detection information includes the current state data of all agents within the agent's detection range.

[0010] Based on the expected return gradient, deterministic policy actions are set according to the environmental detection information, the Actor network, and the Critic network;

[0011] The updated environment detection information is obtained after the agent executes the deterministic policy action; the updated environment detection information includes action and emotion change data, external environment reward data, and updated state data;

[0012] The intrinsic reward data is determined based on the action and emotion change data and the external environment reward data; the intrinsic reward data and the external environment reward data constitute the overall search reward data; the overall search reward data, the current state data, the updated state data, and the deterministic policy action constitute an experience quadruple; the experience quadruples corresponding to multiple agents constitute an experience pool;

[0013] Each agent randomly selects an experience quadruple from the experience pool and uses the selected experience quadruple to train the Actor network and the Critic network to obtain the optimal policy action for each agent; each agent executes the corresponding optimal policy action to achieve cooperative target search.

[0014] Optionally, the calculation formula for the action-emotion change data is as follows:

[0015]

[0016] in, θ represents the data on the agent's changes in action and emotion. i ,η i ,λ i Let represent the weight vectors of the internal changes of the first, second, and third emotions in the i-th agent, respectively. These represent the number of times the i-th agent reaches the first emotional state, the second emotional state, and the third emotional state within one time step, respectively. This indicates the pre-defined intrinsic emotional reward corresponding to the first emotion. This indicates the pre-set intrinsic emotional reward corresponding to the second emotion. This indicates the pre-set intrinsic emotional reward corresponding to the third emotion.

[0017] Optionally, intrinsic reward data is determined based on the action-emotion change data and the external environmental reward data, specifically including:

[0018] According to the formula

[0019]

[0020] Calculate the emotional steady-state value; where, Data representing changes in the agent's actions and emotions. H represents the initial emotional value preset within the agent before time t. t This represents the steady-state value of emotion within the intelligent body at time t;

[0021] According to the formula

[0022]

[0023] Calculate the sentiment function value; where E represents the sentiment function value, and H... t-1 This represents the steady-state value of emotion within the intelligent body at time t-1;

[0024] According to the formula

[0025]

[0026] Calculate the sentiment coefficient; where C represents the sentiment coefficient, k represents the constant coefficient, and e represents the constant e; according to the formula

[0027]

[0028] Calculate the intrinsic reward data; where T represents the maximum time step. This represents the intrinsic reward data of the i-th agent. This represents data on external environmental rewards.

[0029] Optionally, the process of determining the external environmental reward data specifically includes:

[0030] If the updated environmental detection information includes the search target corresponding to the agent, then the external environment reward data is assigned a first preset value;

[0031] If the updated environmental detection information includes any obstacle, and the distance between the obstacle and the agent is less than a first preset distance value, then a second preset value is assigned to the external environment reward data.

[0032] If the updated environment detection information includes other intelligent agents, and the distance between the other intelligent agents and the intelligent agent is less than the second preset distance value, then the external environment reward data is assigned a value according to the preset collision penalty formula;

[0033] If the updated state data of the agent in the updated environment detection information represents the agent's movement, then a third preset value is assigned to the external environment reward data.

[0034] To achieve the above objectives, the present invention also provides the following technical solution:

[0035] A multi-agent target cooperative search system, comprising:

[0036] A simulation environment construction module is used to construct a search simulation environment; multiple agents, multiple obstacles, and multiple search targets are randomly set in the search simulation environment; each agent includes an Actor network and a Critic network; the Actor network is used to select the action to be executed by the agent, and the Critic network is used to evaluate the expected reward of the agent;

[0037] The detection information acquisition module is used to acquire environmental detection information perceived by any intelligent agent in the search simulation environment; the environmental detection information includes the current state data of all intelligent agents within the detection range of the intelligent agent;

[0038] The strategy action determination module is used to set deterministic strategy actions based on the expected return gradient, the environmental detection information, the Actor network, and the Critic network.

[0039] The detection information update module is used to acquire updated environment detection information after the agent executes the deterministic policy action; the updated environment detection information includes action and emotion change data, external environment reward data, and updated state data;

[0040] An experience quadruple construction module is used to determine intrinsic reward data based on the action-emotion change data and the external environment reward data; the intrinsic reward data and the external environment reward data constitute the overall search reward data; the overall search reward data, the current state data, the updated state data, and the deterministic policy action constitute an experience quadruple; the experience quadruples corresponding to multiple agents constitute an experience pool;

[0041] The multi-agent search module is used to randomly select experience quadruples from the experience pool by each agent, and use the selected experience quadruples to train the Actor network and the Critic network to obtain the optimal policy action for each agent; each agent executes the corresponding optimal policy action to achieve target cooperative search.

[0042] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0043] This invention discloses a multi-agent target cooperative search method and system. First, a simulation environment is constructed, and any agent within the simulation environment performs environmental perception to obtain environmental detection information. Based on the expected reward gradient, deterministic policy actions are set according to the environmental detection information and the Actor and Critic networks within the agent, thereby applying deep deterministic policy gradients to the multi-agent cooperative search problem. This combines the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning, enabling the multi-agent system to possess higher autonomous learning capabilities. It effectively improves the search speed and accuracy in complex dynamic environments, thus adapting to different scene changes and solving complex dynamic environments that traditional algorithms and intelligent biomimetic algorithms cannot address.

[0044] Then, the agent's emotional change data, external environment reward data, and updated state data after executing the aforementioned deterministic policy action are obtained. Based on the emotional change data and external environment reward data, intrinsic reward data is determined. Intrinsic reward data and external environment reward data constitute the overall search reward data. The overall search reward data, current state data, updated state data, and deterministic policy action constitute an experience quadruple. Multiple agents' corresponding experience quadruples constitute an experience pool. In other words, this invention introduces intrinsic emotional motivation, which can be mapped to intrinsic reward signals in the deep deterministic policy gradient algorithm. The intrinsic and external rewards of the agent are used together as the overall reward for the agent's search process, forming a highly heuristic exploration strategy. This effectively solves the reward sparsity problem, improves the efficiency of multi-agent search in complex and unknown environments, and solves the problem of learning search strategies under reward sparsity conditions in traditional DDPG. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the multi-agent target cooperative search method of the present invention;

[0047] Figure 2 This is a multi-agent algorithm framework diagram of a specific example of the present invention;

[0048] Figure 3 This is a schematic diagram of a multi-agent collaborative target search scenario according to the present invention;

[0049] Figure 4 This is a flowchart illustrating the multi-agent target cooperative search process, as shown in a specific example of the present invention.

[0050] Figure 5 This is a schematic diagram of the structure of the multi-agent target cooperative search system of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] The purpose of this invention is to provide a multi-agent goal cooperative search method and system. For the multi-agent goal cooperative search problem, an emotion intrinsic motivation module is designed and introduced into the deep deterministic policy gradient algorithm to solve the problem that multi-agents cannot learn effective search strategies in sparse reward environments.

[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] To address the exploration difficulties arising from sparse rewards and random noise interference in large-scale state-action spaces, researchers have proposed deep reinforcement learning exploration methods based on goals, uncertainty metrics, and intrinsic motivation. Compared to goal-based and uncertainty-metric-based methods, deep reinforcement learning methods based on intrinsic motivation, starting from the mechanism in behavioral and psychological studies of how intrinsic motivation drives higher organisms to autonomously explore unknown environments, formalize various heuristic concepts derived from intrinsic motivation, such as "novelty," into intrinsic reward signals to drive agents to autonomously and efficiently explore the environment. This embodies a more abstract and anthropomorphic approach. Specifically, intrinsic motivation stems from the pleasure gained by higher organisms in pursuing increased autonomy, ability, or control, and is the driving force for exploring unknown environments without external stimuli. In deep reinforcement learning, intrinsic motivation can be mapped to intrinsic reward signals, which, combined with deep reinforcement learning methods based on value functions or policy gradients, form highly heuristic exploration strategies to improve the efficiency of agents exploring complex and unknown environments.

[0055] Therefore, to address the problem of insufficient exploration motivation and inability to learn effective search strategies during multi-agent collaborative search caused by reward sparsity, an intrinsic deep deterministic policy gradient algorithm was designed for multi-agent systems. Psychological research shows that emotions have the function of driving behavioral adaptation; both positive and negative emotions generate behavioral motivation, and changes in emotions trigger changes in learning motivation. Therefore, by incorporating an intrinsic emotional motivation module, which can generate current emotions and intrinsic rewards based on environmental stimuli and cognitive states, the reward sparsity problem can be effectively solved.

[0056] Example 1

[0057] like Figure 1 As shown, this embodiment provides a multi-agent target cooperative search method, including:

[0058] Step 100: Construct a search simulation environment; multiple agents, multiple obstacles, and multiple search targets are randomly set in the search simulation environment; each agent includes an Actor network and a Critic network; the Actor network is used to select the action to be executed by the agent, and the Critic network is used to evaluate the expected benefit of the agent.

[0059] Step 200: For any agent in the search simulation environment, acquire the environmental detection information perceived by the agent; the environmental detection information includes the current state data of all agents within the agent's detection range. Here, "all agents" includes the agent itself and other agents within the detection range.

[0060] In a specific instance, such as Figure 2As shown, S represents the state space set of the agent, S = {s} k |k=1,2,...,m},s k Let $k$ represent the k-th state of the agent, and $m$ represent the number of states of the agent. O represents the observation set, O = {O1, O2, ..., O2}. N Let N represent the Nth agent. At each time step, the agent observes the environmental state through perception to obtain environmental detection information; this information also includes the positions of obstacles and the search target within the agent's detection range. A represents the action space, A = {A1, A2, ..., A...}. N}. A N Let represent the action of the Nth agent. Then, the action of the Nth agent at time t+1 is represented as follows:

[0061]

[0062] Where, θ t+1 Let represent the angle of motion of the agent at time t+1, μ represent the rate of change of the angle of motion of the agent, v(t+1) represent the velocity of the agent at time t+1, and a represent the acceleration. The actions of the agent are mainly related to velocity and direction.

[0063] Furthermore, to facilitate subsequent calculations, it is necessary to pre-set the rules for assigning rewards to the agent's external environment. This means the agent receives a reward for moving to the target location and incurs a penalty for colliding with obstacles. A dynamic penalty function is set for collisions between agents to minimize the occurrence of unsafe states. By setting environmental rewards, the agent learns and adopts a search strategy that maximizes cumulative rewards, helping it find the target more quickly.

[0064] Specifically, the process of determining external environmental reward data includes:

[0065] 1) If the environmental detection information includes the search target corresponding to the agent, that is, the agent has found the target, then the external environmental reward data is assigned a first preset value; specifically, the external environmental reward data...

[0066] 2) If the environmental detection information includes any obstacle, and the distance between the obstacle and the agent is less than a first preset distance value, i.e., the agent collides with the obstacle, then a second preset value is assigned to the external environmental reward data; specifically,

[0067] 3) If the environmental detection information includes other intelligent agents, and the distance between the other intelligent agents and the intelligent agent is less than the second preset distance value, that is, the current intelligent agent collides with other intelligent agents, then the external environmental reward data is assigned a value according to the preset collision penalty formula.

[0068] The preset collision penalty formula is as follows:

[0069] λ is the collision penalty factor, and i and j represent the i-th and j-th agents, respectively.

[0070] 4) If the updated state data of the agent in the environmental detection information represents the agent's movement, then a third preset value is assigned to the external environmental reward data; specifically, the agent's movement cost,

[0071] Step 300: Based on the expected reward gradient, and according to the environmental detection information, the Actor network, and the Critic network, set a deterministic policy action. Each agent corresponds to an Actor network μ(o i ;θ i ) and a Critic network Q(s,a;ω) i The Actor network is deterministic, for input o. i The output action a i =μ(o i ;θ i The input to the Critic network is the global state and the actions of all agents, and the output is a real number. It represents the degree of merit of performing action a based on state s. The Critic network evaluates all actions, and the Actor policy network makes improvements.

[0072] Step 300 specifically includes:

[0073] 1) Based on the environmental detection information and the Actor network, randomly select the action to be executed.

[0074] 2) Based on the action to be executed and the Critic network, the expected return value is calculated using the expected return gradient formula.

[0075] 3) Determine whether the expected return value meets the preset optimal condition; if the expected return value meets the preset optimal condition, mark the action to be executed as a deterministic strategy action; if the expected return value does not meet the preset optimal condition, adjust the network parameters of the Actor network according to the expected return value, and then return to the step of randomly selecting the action to be executed based on the environmental detection information.

[0076] The role of the Actor network is to improve the parameters θ through training. i This increases the average value of the Critic network. The gradient formula for the expected return of agent i is as follows:

[0077]

[0078] in, Let θ represent the gradient of the expected reward for the i-th agent. i Indicates the action strategy parameters, where E represents the mathematical expectation and M represents the preset memory module. Represents the action gradient term. Q represents the gradient term of the action policy; i (o,a1,…,a n (a1, ..., a) represents the ensemble evaluation value output by the Critic network for the i-th agent. n ) represents all actions that the i-th agent can choose; o = (o 1 ,o 2 ,...,o n ), o n This represents the current state data of the nth agent; a i Indicates o i The actions to be executed by the agent are output when input into the Actor network of the agent; n represents the number of all actions that the i-th agent can choose. Let represent the deterministic policy of the i-th agent, where i = 1, 2, ..., N, and N represents the total number of agents. Additionally, the preset memory module is an experience replay array used to store collected experiences; each experience is a quadruple (s...). t ,a t ,R t ,s t+1 ).

[0079] Step 400: Obtain updated environment detection information after the agent executes the deterministic policy action; the updated environment detection information includes action emotion change data, external environment reward data, and updated state data.

[0080] Specifically, emotions have the function of driving behavioral adaptation; both positive and negative emotions generate behavioral motivation, and changes in emotions trigger changes in learning motivation. To address the state changes during the agent's goal search process, three emotions—happiness (first emotion), anger (second emotion), and fear (third emotion)—are introduced, and an intrinsic reward function based on emotional motivation is set up to solve the reward sparsity problem.

[0081] As shown in Table 1, the intrinsic emotional reward is set first.

[0082]

[0083] Based on Table 1, the calculation formula for the action-emotion change data is as follows:

[0084]

[0085] in, θ represents the data on the agent's changes in action and emotion. i ,η i ,λ i Let represent the weight vectors indicating the internal changes of the first emotion (happiness), the second emotion (anger), and the third emotion (fear) in the i-th agent, respectively. These represent the number of times the i-th agent reaches the first emotional state, the second emotional state, and the third emotional state within one time step, respectively. This indicates the pre-defined intrinsic emotional reward corresponding to the first emotion. This indicates the pre-set intrinsic emotional reward corresponding to the second emotion. This indicates the pre-set intrinsic emotional reward corresponding to the third emotion.

[0086] θ i =[θ1,θ2,...,θ M ]

[0087] η i =[η1,η2,...,η O ].

[0088] λ i =[λ1,λ2,...,λ P ]

[0089] Where M, O, and P represent happy states, respectively. Anger Fear The quantity. θ M ,η O ,λ P The weight parameter, with a value of [0, 1], determines the degree to which external environmental information affects internal emotional changes.

[0090] Step 500: Determine intrinsic reward data based on the action-emotion change data and the external environment reward data; the intrinsic reward data and the external environment reward data constitute the overall search reward data; the overall search reward data, the current state data, the updated state data, and the deterministic policy action constitute an experience quadruple; the experience quadruples corresponding to multiple agents constitute an experience pool.

[0091] Step 500 specifically includes:

[0092] To address the three emotions involved in the agent's search process, a steady-state emotion variable function H based on emotion changes is designed. t According to the formula

[0093]

[0094] Calculate the emotional steady-state value; where, Data representing changes in the agent's actions and emotions. H represents the initial emotional value preset within the agent before time t. t This represents the steady-state value of emotion within the intelligent body at time t.

[0095] The difference in the value of the agent's emotion steady-state variable function between two adjacent learning steps will lead to changes in the agent's emotion. Therefore, considering only one emotion dimension, the emotion function E is defined as follows:

[0096]

[0097] Where E represents the sentiment function value, H t-1 This represents the steady-state value of emotion within the intelligent body at time t-1; according to the formula...

[0098]

[0099] Calculate the sentiment coefficient; where C represents the sentiment coefficient, k represents the constant coefficient, and e represents the constant e.

[0100] According to the formula

[0101]

[0102] The computational agent uses intrinsic reward data based on emotion; where T represents the maximum time step. This represents the intrinsic reward data of the i-th agent. This represents data on external environmental rewards.

[0103] The process of determining external environmental reward data specifically includes:

[0104] 1) If the updated environment detection information includes the search target corresponding to the agent, then the external environment reward data is assigned a first preset value.

[0105] 2) If the updated environmental detection information includes any obstacle, and the distance between the obstacle and the agent is less than a first preset distance value, then a second preset value is assigned to the external environment reward data.

[0106] 3) If the updated environment detection information includes other intelligent agents, and the distance between the other intelligent agents and the intelligent agent is less than the second preset distance value, then the external environment reward data is assigned a value according to the preset collision penalty formula.

[0107] 4) If the updated state data of the agent in the updated environment detection information represents the agent's movement, then a third preset value is assigned to the external environment reward data.

[0108] Step 600: Each agent randomly selects an experience quadruple from the experience pool, and uses the selected experience quadruple to train the Actor network and the Critic network to obtain the optimal policy action for each agent; each agent executes the corresponding optimal policy action to achieve cooperative target search.

[0109] The Critic network serves to train the system by combining observations and actions. The evaluation of deterministic action policies is optimized by updating the i-th value network with TD error, allowing the value network to better fit the value function Q(s,a).

[0110] To accelerate the learning process of the agents, the input to the Critic network includes the observed states and actions taken by other agents. The parameters of the Critic network are updated by minimizing the loss. Then, the parameters of the action network are calculated and updated using gradient descent. The update formula for the Critic network is:

[0111]

[0112] Where, L(θ) i R represents the expected reward value output by the Critic network, and R represents the overall search reward data. i Let Q' represent the overall search reward data for the i-th agent. i (o',a1',…,a n ') represents the input of the i-th agent in the empirical quadruple as (o', a1', ..., a n When '), the criterion network outputs the concentrated evaluation value; o' represents the updated state data in the empirical quadruple, a n ' represents the action to be executed by the agent when 'o' is input into the Actor network of the agent, and μ' represents the deterministic policy after being updated by the Actor network.

[0113] The TD error update formula is as follows:

[0114] δ(t)=R+γQ′(s,a)-Q(s,a)

[0115] Both the Critic and Actor target networks use a soft update method for parameter updates, as follows:

[0116]

[0117] In the improved deep deterministic policy gradient algorithm of this invention, each agent obtains the action to be executed based on its own policy and interacts with the environment to gain experience, which is stored in its own memory module. After all agents have interacted with the environment, each agent randomly draws experience from the experience pool to train its own neural network, thereby outputting the optimal action at each time step.

[0118] Multi-agent target cooperative search scenarios such as Figure 3 As shown: Each agent is treated as a particle, and a multi-agent particle simulation environment is established to complete the multi-agent target cooperative search task. The multi-agent target cooperative search task requires agents to avoid collisions and find the target. Based on this, the multi-agent target cooperative search method of this invention further includes:

[0119] 1) During the execution of the corresponding optimal policy action by each agent, for any agent, calculate the distance between the agent and other agents, the distance between the agent and any obstacle, and the distance between the agent and the corresponding search target.

[0120] 2) Based on a first preset metric, the distance between the agent and other agents is evaluated to obtain a first evaluation result; based on a second preset metric, the distance between the agent and any obstacle is evaluated to obtain a second evaluation result; based on a third preset metric, the distance between the agent and the corresponding search target is evaluated to obtain a third evaluation result. The specific evaluation formula is as follows:

[0121]

[0122] Among them, D i D n D o D t R represents the center coordinates of the current agent, the nth agent, the obstacle, and the target, respectively. i R n R o R t These represent the radii of the current agent, the nth agent, the obstacle, and the target, respectively.

[0123] 3) If the first evaluation result, the second evaluation result, and the third evaluation result all indicate that the preset metrics are met, that is, the distances between agents, between agents and obstacles, and between agents and targets are all greater than the preset metrics, then a search completion signal is generated; if any of the first evaluation result, the second evaluation result, and the third evaluation result indicates that the preset metrics are not met, then return to the step of obtaining the updated environmental detection information after the agent executes the deterministic policy action.

[0124] In a specific practical application, such as Figure 4 As shown, the multi-agent target collaborative search process of the present invention is as follows: 1) Construct a simulation environment, randomly arrange multiple agents, obstacles, and targets, and perform initialization environmental modeling; 2) Determine whether the serial number i of the agent satisfies the range of 1 < i ≤ N; 3) If it is satisfied, update the Actor network of each agent by calculating the deterministic policy of each agent. The specific steps include: Agent i is based on the currently perceived environmental information, including the positions, states s, and action information of all agents within the detectable range, environmental rewards, etc., and then randomly selects the action a to execute; Update the expected return of agent i according to the expected return gradient formula, and determine whether the currently selected action is the optimal policy of the Critic network. If so, set it as the deterministic policy; if not, repeat the above steps in 3) until a deterministic policy is obtained. 4) Obtain the emotional changes, emotional intrinsic motivation rewards, and new states of the agents based on the actions. 5) Determine whether the new state s' is the final state; 6) If it is the final state, end and update the next agent network; if it is not the final state, obtain the intrinsic reward based on emotional motivation according to the emotional changes of the agents, and use the intrinsic reward and extrinsic reward of the agents together as the overall reward R for the agent search process. Store the quadruple (s, a, R, s') as an experience in the memory module; sample from the memory module, then update the new expected return gradient, and update the Critic network parameters according to the minimum loss function to obtain the overall optimal policy. 7) After obtaining the overall optimal policy, each agent independently executes the search task according to the overall optimal policy. During the search process of each agent, calculate the quality metrics between each agent and the target at all times. When each agent meets the preset metric requirements, it represents that the multi-agent target collaborative search task is completed.

[0125] The multi-agent target collaborative search method of the present invention can be applied to:

[0126] 1) Frequency and Voltage Coordinated Control of Multi-Microgrid Electricity-Gas Systems. For multi-microgrid systems, different case models are established as simulation environments. Due to the strong interaction between microgrids and the natural gas network, coordinated control of multiple microgrids can be achieved based on the "centralized training, decentralized execution" cooperative regulation concept of the MADDPG algorithm. The MADDPG controller can significantly suppress frequency deviations caused by wind power and load disturbances, as well as gas pressure fluctuations caused by natural gas pipeline load fluctuations. Furthermore, the MADDPG controller can effectively coordinate the overall stability among the sub-microgrids of the multi-microgrid system.

[0127] 2) In the field of autonomous driving, this invention addresses lane-following systems in autonomous vehicles. Leveraging the powerful nonlinear fitting and generalization capabilities of deep deterministic policy gradient algorithms, the invention treats the driving vehicle as an agent, other vehicles in the lane or road directions as obstacles, and the vehicle's destination as the search target. It also determines the reward function for lane-following performance. Then, the multi-agent target cooperative search method of this invention is used to perform cooperative target search, achieving good following performance under various road conditions.

[0128] Example 2

[0129] like Figure 5 As shown, in order to implement the technical solution in Embodiment 1, this embodiment provides a multi-agent target cooperative search system, including:

[0130] The simulation environment construction module 101 is used to construct a search simulation environment; multiple agents, multiple obstacles, and multiple search targets are randomly set in the search simulation environment; each agent includes an Actor network and a Critic network; the Actor network is used to select the action to be executed by the agent, and the Critic network is used to evaluate the expected benefit of the agent.

[0131] The detection information acquisition module 201 is used to acquire environmental detection information perceived by any intelligent agent in the search simulation environment; the environmental detection information includes the current state data of all intelligent agents within the detection range of the intelligent agent.

[0132] The strategy action determination module 301 is used to set deterministic strategy actions based on the expected return gradient, the environmental detection information, the Actor network, and the Critic network.

[0133] The detection information update module 401 is used to acquire updated environment detection information after the agent executes the deterministic policy action; the updated environment detection information includes action and emotion change data, external environment reward data, and updated state data.

[0134] The experience quadruple construction module 501 is used to determine intrinsic reward data based on the action-emotion change data and the external environment reward data; the intrinsic reward data and the external environment reward data constitute the overall search reward data; the overall search reward data, the current state data, the updated state data, and the deterministic policy action constitute an experience quadruple; the experience quadruples corresponding to multiple agents constitute an experience pool.

[0135] The multi-agent search module 601 is used to randomly select experience quadruples from the experience pool by each agent, and use the selected experience quadruples to train the Actor network and the Critic network to obtain the optimal policy action of each agent; each agent executes the corresponding optimal policy action to achieve target cooperative search.

[0136] Compared with the prior art, the present invention also has the following advantages:

[0137] The basic idea of ​​applying the deep deterministic policy gradient algorithm of this invention to a multi-agent environment is centralized training and decentralized execution. That is, during the training process, the critic network of each agent collects the state and action information of all agents, but during the training phase, only the actor network of each agent makes decisions based on local information (i.e., the agent's own actions and state).

[0138] Since intrinsic emotions have the function of driving behavioral adaptation, both positive and negative emotions can generate behavioral motivation, and changes in emotions can trigger changes in learning motivation. Therefore, introducing intrinsic emotional motivation allows for the generation of current emotions and intrinsic rewards based on environmental stimuli and cognitive states. By combining the agent's intrinsic rewards and environmental rewards as the overall reward for the agent's search process, the problem of reward sparsity can be effectively solved, and collisions can be avoided to the greatest extent possible.

[0139] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0140] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-agent target cooperative search method, characterized in that, The multi-agent target cooperative search method includes: A search simulation environment is constructed; multiple agents, obstacles, and search targets are randomly set within the search simulation environment; each agent includes an Actor network and a Critic network; the Actor network is used to select the action to be executed by the agent, and the Critic network is used to evaluate the expected reward of the agent; For any agent in the search simulation environment, acquire the environmental detection information perceived by the agent; the environmental detection information includes the current state data of all agents within the agent's detection range. Based on the expected return gradient, deterministic policy actions are set according to the environmental detection information, the Actor network, and the Critic network; The updated environment detection information is obtained after the agent executes the deterministic policy action; the updated environment detection information includes action and emotion change data, external environment reward data, and updated state data; The intrinsic reward data is determined based on the action and emotion change data and the external environment reward data; the intrinsic reward data and the external environment reward data constitute the overall search reward data; the overall search reward data, the current state data, the updated state data, and the deterministic policy action constitute an experience quadruple; the experience quadruples corresponding to multiple agents constitute an experience pool; Each agent randomly selects an experience quadruple from the experience pool and uses the selected experience quadruple to train the Actor network and the Critic network to obtain the optimal policy action for each agent; each agent executes the corresponding optimal policy action to achieve cooperative target search.

2. The multi-agent target cooperative search method according to claim 1, characterized in that, The calculation formula for the action-emotion change data is as follows: in, Data representing changes in the agent's actions and emotions. Let represent the weight vectors of the internal changes of the first, second, and third emotions in the i-th agent, respectively. These represent the number of times the i-th agent reaches the first emotional state, the second emotional state, and the third emotional state within one time step, respectively. This indicates the pre-defined intrinsic emotional reward corresponding to the first emotion. This indicates the pre-set intrinsic emotional reward corresponding to the second emotion. This indicates the pre-set intrinsic emotional reward corresponding to the third emotion.

3. The multi-agent target cooperative search method according to claim 1, characterized in that, The intrinsic reward data is determined based on the aforementioned action-emotion change data and the aforementioned external environmental reward data, specifically including: According to the formula Calculate the emotional steady-state value; where, Data representing changes in the agent's actions and emotions. H represents the initial emotional value preset within the agent before time t. t This represents the steady-state value of emotion within the intelligent body at time t; According to the formula Calculate the sentiment function value; where E represents the sentiment function value, H t-1 This represents the steady-state value of emotion within the intelligent body at time t-1; According to the formula Calculate the sentiment coefficient; where, Indicates the sentiment coefficient. k Denotes the constant coefficient, and e represents the constant e; According to the formula Calculate the intrinsic reward data; where T represents the maximum time step. This represents the intrinsic reward data of the i-th agent. This represents data on external environmental rewards.

4. The multi-agent target cooperative search method according to claim 1, characterized in that, The step of setting deterministic policy actions based on the expected return gradient, according to the environmental detection information, the Actor network, and the Critic network, specifically includes: Based on the environmental detection information and the Actor network, an action to be executed is randomly selected. Based on the action to be executed and the Critic network, the expected return value is calculated using the expected return gradient formula. Determine whether the expected return value meets the preset optimal conditions; If the expected return value meets the preset optimal condition, then the action to be executed is marked as a deterministic strategy action; If the expected return value does not meet the preset optimal condition, the network parameters of the Actor network are adjusted according to the expected return value, and then the process returns to the step of randomly selecting an action to be executed based on the environmental detection information.

5. The multi-agent target cooperative search method according to claim 4, characterized in that, The formula for the expected return gradient is as follows: in, Indicates the first i The expected reward gradient of an agent Indicates the action strategy parameters, Indicates the first i Current state data of each agent ,action It conforms to the mathematical expectation of distribution M; Indicates a preset memory module. Represents the action gradient term. Represents the gradient term of the action policy; This represents the centralized evaluation value output by the Critic network for the i-th agent. This represents all actions that the i-th agent can choose; , This represents the current state data of the nth agent; a i Indicates will The action to be executed by the agent when input into the Actor network of the agent; n This represents the number of all actions that the i-th agent can choose. Let i represent the deterministic policy of the i-th agent, i=1,2,…,N, where N represents the total number of agents.

6. The multi-agent target cooperative search method according to claim 1, characterized in that, During the training of the Actor network and the Critic network using the extracted empirical quadruples, the update formula for the Critic network is: ； in, This represents the expected return value output by the Critic network. Indicates the current state data o and the action. Search overall reward data R, updated state data in experience quaternions It conforms to the mathematical expectation of distribution M; Indicates a preset memory module. R represents the action policy parameters. i This represents the overall search reward data for the i-th agent. The input of the i-th agent in the empirical quadruple is... At that time, the centralized evaluation value output by the Critic network; Indicates will The action to be executed by the first agent output when input into the Actor network of the agent. Indicates will The action to be executed by the nth agent output when input into the Actor network of the agent. Indicates will The action to be executed by the i-th agent output when input into the Actor network of the agent; This represents the centralized evaluation value output by the Critic network for the i-th agent. This represents all actions that the i-th agent can choose. n This represents the number of all agents within the detection range of the stated agent. This represents the deterministic policy after being updated by the Actor network. For the first i The current state data of each agent, i=1,2,…,N, where N represents the total number of agents.

7. The multi-agent target cooperative search method according to claim 1, characterized in that, The multi-agent target cooperative search method also includes: During the execution of the corresponding optimal policy action by each agent, for any agent, the distance between the agent and other agents, the distance between the agent and any obstacle, and the distance between the agent and the corresponding search target are calculated; Based on a first preset index, the distance between the intelligent agent and other intelligent agents is evaluated to obtain a first evaluation result; Based on a second preset index, the distance between the intelligent agent and any obstacle is evaluated to obtain a second evaluation result; Based on a third preset index, the distance between the intelligent agent and the corresponding search target is evaluated to obtain a third evaluation result; If the first evaluation result, the second evaluation result, and the third evaluation result all indicate that the preset indicators have been met, then a search completion signal is generated. If any of the first evaluation result, the second evaluation result, or the third evaluation result fails to meet the preset index, the process returns to the step of obtaining the updated environment detection information after the agent executes the deterministic policy action.

8. The multi-agent target cooperative search method according to claim 1, characterized in that, The process of determining the external environmental reward data specifically includes: If the updated environmental detection information includes the search target corresponding to the agent, then the external environment reward data is assigned a first preset value; If the updated environmental detection information includes any obstacle, and the distance between the obstacle and the agent is less than a first preset distance value, then a second preset value is assigned to the external environment reward data. If the updated environment detection information includes other intelligent agents, and the distance between the other intelligent agents and the intelligent agent is less than the second preset distance value, then the external environment reward data is assigned a value according to the preset collision penalty formula; If the updated state data of the agent in the updated environment detection information represents the agent's movement, then a third preset value is assigned to the external environment reward data.

9. A multi-agent target cooperative search system, characterized in that, The multi-agent target cooperative search system includes: A simulation environment construction module is used to construct a search simulation environment; multiple agents, multiple obstacles, and multiple search targets are randomly set in the search simulation environment; each agent includes an Actor network and a Critic network; the Actor network is used to select the action to be executed by the agent, and the Critic network is used to evaluate the expected reward of the agent; The detection information acquisition module is used to acquire environmental detection information perceived by any intelligent agent in the search simulation environment; the environmental detection information includes the current state data of all intelligent agents within the detection range of the intelligent agent; The strategy action determination module is used to set deterministic strategy actions based on the expected return gradient, the environmental detection information, the Actor network, and the Critic network. The detection information update module is used to acquire updated environment detection information after the agent executes the deterministic policy action; the updated environment detection information includes action and emotion change data, external environment reward data, and updated state data; An experience quadruple construction module is used to determine intrinsic reward data based on the action-emotion change data and the external environment reward data; the intrinsic reward data and the external environment reward data constitute the overall search reward data; the overall search reward data, the current state data, the updated state data, and the deterministic policy action constitute an experience quadruple; the experience quadruples corresponding to multiple agents constitute an experience pool; The multi-agent search module is used to randomly select experience quadruples from the experience pool by each agent, and use the selected experience quadruples to train the Actor network and the Critic network to obtain the optimal policy action for each agent; each agent executes the corresponding optimal policy action to achieve target cooperative search.

Images