Training method for multi-robot collaborative search

Abstract

The present invention discloses a training method for multi-robot collaborative search, comprising: S1: establishing a multi-robot collaborative search task and a kinematic model of the multi-robots; S2: designing an overall reward for the multi-robot collaborative search, including an environmental reward and a motivational reward; S3: training a multi-robot collaborative search model based on the overall reward, thereby obtaining a personality-based emotional system from the perspective of reward supplementation, and combining it with a multi-agent deep deterministic policy gradient algorithm, thereby effectively improving the search efficiency and speed of the multi-robots in a complex dynamic environment.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning, and in particular to a training method for multi-robot collaborative search. Background Art

[0002] In recent years, deep reinforcement learning (DRL) theory has provided a new approach to solving the problem of multi-robot collaborative search. Deep reinforcement learning, with its powerful perception and decision-making capabilities, integrates the perception, learning, and decision-making capabilities of intelligent agents into a single framework, achieving end-to-end perception and decision-making from raw input to decision-making action. This provides a solution to the complex and uncertain environments encountered by multi-robot systems.

[0003] Although deep reinforcement learning adopts an end-to-end strategy, which is more advantageous than traditional methods and has achieved great success in the field of multi-robots, existing technologies are unable to handle environments with characteristics such as sparse rewards and random noise. As a result, search robots only receive sparse rewards and do not receive continuous and meaningful feedback, resulting in poor training results and even inability to learn effective strategies, leading to search difficulties.

[0004] Therefore, there is an urgent need to provide an improved multi-robot collaborative search training method that can optimize the reward function in the traditional method, improve the robot's self-learning ability, and thus improve the search efficiency and stability of the multi-robot system. Summary of the Invention

[0005] The present invention aims to provide a multi-robot collaborative search training method, which can solve the above technical problems.

[0006] According to one aspect of the present invention, a training method for multi-robot collaborative search is provided, comprising: S1: establishing a multi-robot collaborative search task and a kinematic model of the multi-robot; S2: designing an overall reward for the multi-robot collaborative search, including an environmental reward and a motivational reward; S3: training a multi-robot collaborative search model based on the overall reward.

[0007] Preferably, the total reward R(t) of the search robot i at time t is:

[0008]

[0009] Where N is the total number of search robots, d i is the distance between the search robot and the search target, r i e (t) is the environmental reward, r i em (t) is the motivation reward, xi (t), y i (t), x tar (t), y tar (t) are the position coordinates of the search robot i and the search target at time t respectively.

[0010] Preferably, the search robot i triggers one or more environmental rewards at time t, including: a first environmental reward r i 1 (t) = 10, used to reward the search robot i for searching the target; the second environment rewards r i 2 (t) = -2, used to reward the search robot i for colliding with an obstacle; the third environment reward Used to reward the collision between the search robots, where λ is the collision penalty factor; the fourth environment reward r i 4 (t) = -10 × (max (|x i (t)|,|y i (t)|)-0.9), which is used to reward the search robot i for moving to the boundary.

[0011] Preferably, the formula for the motivation reward is:

[0012]

[0013] Among them, E i (t) is the emotional intensity of the search robot i, which can be specifically expressed as:

[0014] in, Personality mapping for the search robot i, For the emotional decay of the search robot i, is the external emotional stimulus received by the search robot i, g(x) is the emotional intensity E i The value of (t) is kept in the range of [-1,1].

[0015] Preferably, the personality map for:

[0016]

[0017] Among them, v is the basic emotional state of the robot (v=1,2,...m), is the coordinate corresponding to the search robot i in the PAD emotion space when it is in any basic emotion state v, represents the coordinates of the personality of the search robot i in the PAD emotional space, disi,v (t) represents the degree of closeness between the personality of the search robot i and any of the basic emotional states, D i (t) represents the overall sentiment tendency of the search robot i.

[0018] Preferably, the emotion decays for:

[0019]

[0020] Among them, E i (t-1) represents the emotion intensity of the search robot i at the last moment, φ(k i ) represents the emotion decay function, k i It represents the rate of emotional decay determined by personality, and T represents the decay period.

[0021] Preferably, the external emotional stimulus for:

[0022]

[0023] Among them, S i (t-1) is the stimulus intensity that the search robot i received at the last moment, r i e (t-1) is the environmental reward of the search robot i at the previous moment.

[0024] Preferably, the multi-robot collaborative search model includes: a strategy network Output the current action a according to the current state s(t) i (t); Evaluation network According to the current state s(t) and the current action a i (t) Output action value Q i (t); an experience pool for storing an experience replay array M, which includes the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots, and the next moment state s(t+1) based on the current action a(t) of all search robots, wherein the current state s(t) and the next moment state s(t+1) adopt the current joint observation state o(t) and the next moment joint observation state o(t+1) of all search robots.

[0025] Preferably, according to the overall reward, training the multi-robot collaborative search model includes: S31: initializing the policy network, the evaluation network and the current state s(t); S32: according to the policy network and the current state s(t), obtaining the current action a of each search robot i i(t); S33: Execute action a of each search robot i i (t), get the next moment state s of each search robot i i (t+1) and the current total reward R of each search robot i i (t); S34: store the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots and the next moment state s(t+1) as an experience replay array M in the experience pool; S35: sample the experience replay array M, update the policy network and the evaluation network; S36: repeat steps S32-S35 until the training is completed.

[0026] Preferably, sampling the experience playback array M and updating the strategy network and the evaluation network include: S351: according to the expected benefit of the search robot i The gradient of t updates the policy network:

[0027]

[0028] in, is the policy network parameter, o(t) is the current joint observation state, o(t)=(o1(t),o2(t),...,o n (t)), Q i (t) is the centralized evaluation value of the search robot i at the current moment; S352: Update the strategy network according to the temporal difference error:

[0029]

[0030] in, To evaluate the network parameters, is the loss value of the evaluation network, y(t) is the target value of the evaluation network, R(t) is the total reward of the search robot i at time t, γ is the discount factor, Q i (t) is the centralized evaluation value of the search robot i at the next moment, is the deterministic strategy updated by the search robot i at the next moment.

[0031] The present invention discloses a training method for multi-robot collaborative search, comprising: S1: establishing a multi-robot collaborative search task and a kinematic model of the multi-robots; S2: designing an overall reward for the multi-robot collaborative search, including an environmental reward and a motivational reward; S3: training the multi-robot collaborative search model based on the overall reward, thereby obtaining a personality-based emotional system from the perspective of reward supplementation, and combining the system with a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, thereby effectively improving the search efficiency and speed of the multi-robots in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0033] Figure 1 is a flow chart of a training method for multi-robot collaborative search according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of a multi-robot collaborative search task scenario according to an embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of a robot kinematic model according to an embodiment of the present invention; and

[0036] Figure 4 2 is a block diagram of a MADDPG algorithm based on emotional intrinsic motivation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] Figures 1-4 The figure shows a flowchart of a multi-robot collaborative search training method according to an embodiment of the present invention, a schematic diagram of a multi-robot collaborative search task scenario, a schematic diagram of a robot kinematic model, and a block diagram of a MADDPG algorithm based on emotional intrinsic motivation. This embodiment of the present invention provides a multi-robot collaborative search training method, comprising: S1: establishing a multi-robot collaborative search task and a kinematic model of the multi-robots; S2: designing an overall reward for the multi-robot collaborative search, including an environmental reward and a motivational reward; and S3: training the multi-robot collaborative search model based on the overall reward.

[0039] In related technologies, multi-robot target search methods mostly use probabilistic graph methods and swarm optimization algorithms, but existing methods usually require complex mathematical calculations to solve multi-robot collaborative search problems and are prone to falling into local optimality, which is not suitable for complex search scenarios.

[0040] This embodiment of the present invention adds motivational rewards based on the emotional system to the traditional MADDPG algorithm to improve the efficiency and speed of multi-robot searches in complex dynamic environments. Specifically:

[0041] like Figure 2 As shown in the figure, on a bounded, two-dimensional, continuous plane with obstacles, there are m dynamic targets and n search robots, where n > m. All search robots and targets can appear at any coordinate and move in any direction. Search robots select a target to pursue through task assignment. Search robots with the same target form a target search team. A successful search is considered successful when the distance between the robots and the target is less than the search radius r.

[0042] Emotional robot i is defined as a five-tuple P i = <Pos i ,E i ,Per i ,R i ,f i >, where: Pos i Represents the coordinates (x i ,y i );E i Represents the basic emotional state of the i-th robot. i R represents the personality of the i-th robot. i Indicates the reward value obtained by the i-th robot. i represents the emotional cooperation willingness of the i-th robot.

[0043] The emotion selection of the emotional robot is based on the famous six-emotion classification method proposed by Ekman, which divides emotions into fear, anger, sadness, happiness, disgust and surprise. In modern psychology research, happiness, sadness, fear and anger are often regarded as basic emotions. The embodiment of the present invention is aimed at multi-robot collaborative search scenarios, using E i Represents the basic emotion of robot i, E i =[e1,e2,...,e m ], here E i represents the emotional state of robot i, e m Indicates the emotional intensity of basic emotions (such as happiness, fear, sadness), where e m ∈[0,1], the larger the value, the stronger the emotion.

[0044] like Figure 3 As shown, in the multi-robot collaborative search task, the embodiment of the present invention regards each robot as a particle and Figure 2 The robot kinematic model is established in the simulation environment.

[0045] The kinematic model of each emotional robot is the same, including position (coordinates), speed, and direction of movement. The positions of the robot at time t and time t+1 are (x N (t),y N (t)) and (x N (t+1),y N (t+1)), the robot's kinematic model is as follows:

[0046]

[0047] Where v is the robot's movement speed, which is set to a fixed value in the embodiment of the present invention, and Δt is the time step. N Indicates the angle between the robot's current direction of motion and the positive x-axis of the world coordinate system, indicating the robot's forward direction.

[0048] The goal of a multi-robot collaborative search task is to avoid collisions and find the target. Therefore, the distance between robots, the distance between robots and obstacles, and the distance between robots and the target are used as evaluation criteria to measure the multi-robot search strategy. Therefore, the evaluation criteria for the quality of a multi-robot collaborative search strategy are designed as follows:

[0049]

[0050] Among them, D i 、D n 、D o and D t The center coordinates of the i-th robot, the center coordinates of the n-th robot, the center coordinates of the obstacle, and the center coordinates of the target. i 、Ra n 、Ra o and Ra t The radius of the i-th robot, the radius of the n-th robot, the radius of the obstacle, and the radius of the target. Through quality evaluation indicators, we can judge the status of the robot at the current location and the quality of the current search strategy.

[0051] like Figure 4As shown in Figure 1, this embodiment of the present invention proposes an artificial emotion-based MADDPG algorithm, called MADDPG-E. In MADDPG-E, an emotion system is added to the MADDPG algorithm. Based on environmental stimuli and cognitive state, it generates a motivational reward for each action. This intrinsic motivational reward, along with the robot's environmental rewards, serves as the overall reward for the robot's search process.

[0052] This not only effectively solves the sparse reward problem in related technologies but also better avoids collisions. Furthermore, by incorporating the willingness of robots to collaborate into multi-robot collaborative search tasks based on their individual personalities, the efficiency of collaborative search can be significantly improved.

[0053] The algorithm mainly consists of five elements:<S,O,A,R,M> , the specific meaning of each element is as follows:

[0054] (1) S: S = {s k |k=1,2,...,m} is the robot state space set, s k Represents the kth state of the robot, m represents the number of states of the robot, and all robots share a state space set.

[0055] (2) O: O = {O1, O2, ..., O N} is the multi-robot observation set, and N is the number of robots. At each time step, each robot observes the state of the environment through perception. The robot can obtain the positions of obstacles within the detectable range, its own position, the positions of other robots, and the position of the target.

[0056] (3) A: A={A1,A2,...,A N} is the action space set of multiple robots. N The action of robot N is mainly related to speed and direction. The action taken by robot N at time t+1 is as follows:

[0057]

[0058] Here, α(t+1) is the movement angle of the robot N at time t+1, β is the rate of change of the robot's movement angle, v(t+1) is the movement speed of the robot N at time t+1, and v′ represents the acceleration.

[0059] (4) R: is the reward function of the multi-robot system.

[0060] (5) M: represents the memory module, which uses an experience playback array to store the collected experience. Each of them is a four-tuple {s(t), a(t), s(t+1), r e(t)} means:

[0061]

[0062] Among them, N (t) represents the observed state of robot N at time t, a N (t) represents the action selected by robot N at time t.

[0063] According to an embodiment of the present invention, the multi-robot collaborative search model includes: a strategy network Output the current action a according to the current state s(t) i (t); Evaluation network According to the current state s(t) and the current action a i (t) Output action value Q i (t); an experience pool for storing an experience replay array M, which includes the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots, and the next moment state s(t+1) based on the current action a(t) of all search robots, wherein the current state s(t) and the next moment state s(t+1) adopt the current joint observation state o(t) and the next moment joint observation state o(t+1) of all search robots.

[0064] According to an embodiment of the present invention, training a multi-robot collaborative search model according to the overall reward includes: S31: initializing the policy network, the evaluation network, and the current state s(t);

[0065] S32: According to the strategy network and the current state s(t), the current action a of each search robot i is obtained. i (t); S33: Execute action a of each search robot i i (t), get the next moment state s of each search robot i i (t+1) and the current total reward R of each search robot i i (t); S34: store the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots and the next moment state s(t+1) as an experience replay array M in the experience pool; S35: sample the experience replay array M, update the policy network and the evaluation network; S36: repeat steps S32-S35 until the training is completed.

[0066] According to an embodiment of the present invention, sampling the experience playback array M, updating the strategy network and the evaluation network includes: S351: according to the expected benefit of the search robot i The gradient of t updates the policy network:

[0067]

[0068] in, is the policy network parameter, o(t) is the current joint observation state, o(t)=(o1(t),o2(t),...,o n (t)), Q i (t) is the centralized evaluation value of the search robot i at the current moment; S352: Update the strategy network according to the temporal difference error:

[0069]

[0070] in, To evaluate the network parameters, is the loss value of the evaluation network, y(t) is the target value of the evaluation network, R(t) is the total reward of the search robot i at time t, γ is the discount factor, Q i (t) is the centralized evaluation value of the search robot i at the next moment, is the deterministic strategy updated by the search robot i at the next moment.

[0071] In the MADDPG-E algorithm, for each of the N robots in a multi-robot system, there is a policy (Actor) network. and a critic network Actor networks are deterministic and for certain inputs (S i ,O i ), output action a i The input of the Critic network is the global state and the actions of all robots. The output is a real number that represents the degree of goodness of executing action a based on state s. The Critic network is used to evaluate all the actions of the robot and guide the Actor policy network to make improvements. The role of the Actor network is to improve the parameters through training. Increase the average value of the critic network. The critic network is trained using the robot's combined observations and actions. Optimizing the centralized evaluation of deterministic action strategies and updating the critic network with temporal difference errors allows the critic network to better fit the value function Q(s,a).

[0072] The multi-robot collaborative search process is converted into a Markov decision process. In each time step t, robot i receives an overall reward and takes an action according to its current state. And the dynamic model p is transformed according to the environment. i (s i(t+1)|s i (t),a i (t)) changes the current state to the next state. The goal of each search robot is to maximize its final expected reward. According to an embodiment of the present invention, the total reward R(t) of search robot i at time t is:

[0073]

[0074] Where N is the total number of search robots, d i is the distance between the search robot and the search target, r i e (t) is the environmental reward, r i em (t) is the motivation reward, x i (t), y i (t), x tar (t), y tar (t) are the position coordinates of the search robot i and the search target at time t respectively.

[0075] According to an embodiment of the present invention, the search robot i triggers one or more environmental rewards at time t, including: a first environmental reward r i 1 (t) = 10, used to reward the search robot i for searching the target; the second environment rewards r i 2 (t) = -2, used to reward the search robot i for colliding with an obstacle; the third environment reward Used to reward the collision between the search robots, where λ is the collision penalty factor; the fourth environment reward r i 4 (t) = -10 × (max (|x i (t)|,|y i (t)|)-0.9), which is used to reward the search robot i for moving to the boundary.

[0076] The multi-robot collaborative search method of the embodiment of the present invention first requires artificial emotion modeling of the emotion system. The emotion robot generates basic emotions based on internal states such as personality and emotion attenuation, as well as external stimuli. It then provides emotional intrinsic rewards based on these basic emotions to adjust deep reinforcement learning. This allows emotions to directly and effectively participate in task allocation decisions, optimizing algorithm performance. According to the embodiment of the present invention, the formula for the motivation reward is:

[0077]

[0078] Among them, E i(t) is the emotional intensity of the search robot i, which can be specifically expressed as:

[0079] in, Personality mapping for the search robot i, For the emotional decay of the search robot i, is the external emotional stimulus received by the search robot i, g(x) is the emotional intensity E i The value of (t) is kept in the range of [-1,1].

[0080] According to an embodiment of the present invention,

[0081] In related technologies, rewards are often sparse during a robot's search process. This means that the robot often needs to go through a series of attempts before completing a task before receiving a reward. However, intrinsic emotions can enable the robot to generate current emotions and motivational rewards based on environmental stimuli and cognitive states, thereby improving the robot's search efficiency.

[0082] Personality is an inherent characteristic of a robot, which determines the overall behavioral and psychological tendencies of individual robots and reflects the heterogeneity and diversity of robots in a group. For example, compared with individuals with cautious personalities, individuals with bold personalities are more likely to perform dangerous tasks. At present, the OCEAN personality model is the most common personality representation method. Therefore, the embodiment of the present invention describes the personality of a robot through five aspects: openness, conscientiousness, extroversion, agreeableness, and neuroticism: per = [Op, Co, Ex, Ag, Ne], where Op represents openness, Co represents conscientiousness, Ex represents extroversion, Ag represents agreeableness, and Ne represents neuroticism. The values ​​of all attributes are between [-1, 1]. The personality space is mapped to the PAD emotional space, and the following conversion relationship is established for this purpose:

[0083]

[0084] According to an embodiment of the present invention, the personality mapping for:

[0085]

[0086] Among them, v is the basic emotional state of the robot (v=1,2,...m), is the coordinate corresponding to the search robot i in the PAD emotion space when it is in any basic emotion state v, represents the coordinates of the personality of the search robot i in the PAD emotional space, dis i,v (t) represents the degree of closeness between the personality of the search robot i and any of the basic emotional states, D i (t) represents the overall sentiment tendency of the search robot i.

[0087] The emotional decay mechanism is based on the "Third Law of Emotional Intensity" in psychology. Emotional states of any intensity will decay infinitely and approach zero over time. If the influence of external values ​​is ignored, the emotional state intensity decay curve and exponent y=e -x When studying emotion decay in the embodiment of the present invention, the current emotional state of the robot decays based on the emotional state at the previous moment.

[0088] According to an embodiment of the present invention, the emotion decay for:

[0089]

[0090] Among them, E i (t-1) represents the emotion intensity of the search robot i at the last moment, φ(k i ) represents the emotion decay function, k i It represents the rate of emotional decay determined by personality, and T represents the decay period.

[0091] External stimuli are the primary cause of emotional change, and task rewards are a crucial determinant of these stimuli. Tasks with higher rewards generate stronger positive stimuli. Therefore, this embodiment of the present invention defines stimuli S based on the environmental rewards received by robots during different tasks. The types of stimuli correspond to basic emotion types.

[0092] According to an embodiment of the present invention, the external emotional stimulus for:

[0093]

[0094] Among them, S i (t-1) is the stimulus intensity that the search robot i received at the last moment, r i e (t-1) is the environmental reward of the search robot i at the previous moment.

[0095] Obviously, those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0096] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A multi-robot collaborative search training method, characterized in that: include: S1: Establishing a multi-robot collaborative search task and a kinematic model of the multi-robots; S2: Design the overall reward for the multi-robot collaborative search, including environmental rewards and motivation rewards; S3: training a multi-robot collaborative search model based on the overall reward; Among them, the total reward R(t) of the search robot i at time t is: Where N is the total number of search robots, d i is the distance between the search robot and the search target, r i e (t) is the environmental reward, r i em (t) is the motivation reward, x i (t), y i (t), x tar (t), y tar (t) are the position coordinates of the search robot i and the search target at time t, The formula for the motivation reward is: Among them, E i (t) is the emotion intensity of the search robot i, which can be specifically expressed as: in, is the personality mapping of the search robot i, is the emotional decay of the search robot i, is the external emotional stimulus received by the search robot i, g(x) is the emotional intensity E i The value of (t) is kept in the range of [-1,1].

2. The multi-robot collaborative search training method according to claim 1, characterized in that: The search robot i triggers one or more environmental rewards at time t, including: First Block Rewards Used to reward the search robot i for searching the target; Second Block Rewards Used to reward the search robot i for colliding with an obstacle; Third Environment Rewards Used to reward collisions between the search robots, where λ is a collision penalty factor; Fourth Environment Rewards i 4 (t) = -10 × (max (|x i (t)|,|y i (t)|)-0.9), used to reward the search robot i for moving to the boundary.

3. The multi-robot collaborative search training method according to claim 2, characterized in that: The personality map for: Among them, v is the basic emotional state of the robot, is the coordinate corresponding to the search robot i in the PAD emotion space when it is in any of the basic emotion states v, represents the coordinates corresponding to the personality of the search robot i in the PAD emotion space, dis i,v (t) represents the degree of closeness between the personality of the search robot i and any of the basic emotional states, D i (t) represents the overall sentiment tendency of the search robot i.

4. The multi-robot collaborative search training method according to claim 2, characterized in that: The emotional decay for: Among them, E i (t-1) represents the emotion intensity of the search robot i at the last moment, φ(k i ) represents the emotion decay function, k i It represents the rate of emotional decay determined by personality, and T represents the decay period.

5. The multi-robot collaborative search training method according to claim 2, characterized in that: The external emotional stimulus for: S i (t-1)=r i e (t-1)×E i (t-1) Among them, S i (t-1) is the stimulus intensity received by the search robot i at the last moment, r i e (t-1) is the environmental reward of the search robot i at the previous moment.

6. The multi-robot collaborative search training method according to claim 1, characterized in that: The multi-robot collaborative search model includes: Policy Network Output the current action a according to the current state s(t) i (t); Evaluation Network According to the current state s(t) and the current action a i (t) Output action value Q i (t); The experience pool is used to store the experience replay array M, which includes the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots, and the next moment state s(t+1) based on the current action a(t) of all search robots, wherein the current state s(t) and the next moment state s(t+1) adopt the current joint observation state o(t) and the next moment joint observation state o(t+1) of all search robots.

7. The multi-robot collaborative search training method according to claim 6, characterized in that: Based on the overall reward, training the multi-robot collaborative search model includes: S31: Initializing the policy network, the evaluation network and the current state s(t); S32: According to the strategy network and the current state s(t), the current action a of each search robot i is obtained. i (t); S33: Execute action a of each search robot i i (t), get the next moment state s of each search robot i i (t+1) and the current total reward R of each search robot i i (t); S34: The current state s(t), the current actions a(t) of all the search robots, the current total rewards R(t) of all the search robots, and the next state s(t+1) are stored in the experience pool as an experience replay array M; S35: sampling the experience replay array M, and updating the policy network and the evaluation network; S36: Repeat steps S32-S35 until the training is completed.

8. The multi-robot collaborative search training method according to claim 7, characterized in that: Sampling the experience replay array M and updating the policy network and the evaluation network include: S351: According to the expected profit of the search robot i The gradient of the policy network is updated: in, is the policy network parameter, o(t) is the current joint observation state, o(t)=(o1(t),o2(t),...,o n (t)), Q i (t) is the centralized evaluation value of the search robot i at the current moment; S352: Update the strategy network according to the temporal difference error: in, To evaluate the network parameters, is the loss value of the evaluation network, y(t) is the target value of the evaluation network, R(t) is the total reward of the search robot i at time t, γ is the discount factor, Q i (t) is the centralized evaluation value of the search robot i at the next moment, is the deterministic strategy updated by the search robot i at the next moment.

Images