Multi-robot collaborative search training method

Abstract

The invention discloses a multi-robot collaborative search training method. The method comprises the following steps: S1, establishing a multi-robot collaborative search task and a multi-robot kinematics model; s2, designing an overall reward for multi-robot collaborative search, wherein the overall reward comprises an environment reward and an incentive reward; and S3, according to the overall reward, training a multi-robot collaborative search model so as to start from the perspective of reward supplementation to obtain a personality-based emotion system, and combining the personality-based emotion system with a multi-agent depth deterministic strategy gradient algorithm, thereby effectively improving the search efficiency and speed of multiple robots in a complex dynamic environment.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning, and in particular to a training method for multi-robot collaborative search. Background Art

[0002] In recent years, deep reinforcement learning (DRL) theory has provided a new solution to the problem of multi-robot collaborative search. Deep reinforcement learning has powerful perception and decision-making capabilities, integrating the perception, learning, and decision-making capabilities of intelligent agents into the same framework, achieving "end-to-end" perception and decision-making from raw input to decision-making actions, and providing solutions to complex and uncertain environmental problems in multi-robot systems.

[0003] Although deep reinforcement learning adopts an end-to-end strategy, which is more advantageous than traditional methods and has achieved great success in the field of multi-robots, existing technologies are unable to handle environments with characteristics such as sparse rewards and random noise, resulting in search robots only receiving sparse rewards and not receiving continuous and meaningful feedback, causing the robot's training effect to deteriorate or even fail to learn effective strategies, leading to search difficulties.

[0004] Therefore, there is an urgent need to provide an improved multi-robot collaborative search training method that can optimize the reward function in the traditional method, improve the robot's self-learning ability, and thereby improve the search efficiency and stability of the multi-robot system. Summary of the invention

[0005] The present invention aims to provide a multi-robot collaborative search training method, which can solve the above-mentioned technical problems.

[0006] According to one aspect of the present invention, a training method for multi-robot collaborative search is provided, comprising: S1: establishing a multi-robot collaborative search task and a kinematic model of the multi-robots; S2: designing an overall reward for the multi-robot collaborative search, including an environmental reward and a motivational reward; S3: training a multi-robot collaborative search model based on the overall reward.

[0007] Preferably, the total reward R(t) of the search robot i at time t is:

[0008]

[0009] Where N is the total number of search robots, d i is the distance between the search robot and the search target, r i e (t) is the environmental reward, r i em (t) is the motivation reward, xi (t), y i (t), x tar (t), y tar (t) are the position coordinate values ​​of the search robot i and the search target at time t respectively.

[0010] Preferably, the search robot i triggers one or more environmental rewards at time t, including: a first environmental reward r i 1 (t) = 10, used to reward the search robot i for searching the target; the second environment rewards r i 2 (t) = -2, used to reward the search robot i for colliding with an obstacle; the third environment reward It is used to reward the collision between the search robots, where λ is the collision penalty factor; the fourth environment reward r i 4 (t) = -10 × (max (|x i (t)|,|y i (t)|)-0.9), which is used to reward the search robot i for moving to the boundary.

[0011] Preferably, the formula for the motivation reward is:

[0012]

[0013] Among them, E i (t) is the emotion intensity of the search robot i, which can be specifically expressed as:

[0014] in, Personality mapping for the search robot i, For the emotional decay of the search robot i, is the external emotional stimulus received by the search robot i, and g(x) is the value of the emotional intensity E i The value of (t) is kept in the range of [-1,1].

[0015] Preferably, the personality map for:

[0016]

[0017] Among them, v is the basic emotional state of the robot (v = 1, 2, ... m), is the coordinate corresponding to the search robot i in the PAD emotion space when it is in any basic emotion state v, represents the coordinates of the personality of the search robot i in the PAD emotional space, dis i,v(t) represents the closeness between the personality of the search robot i and any of the basic emotional states, D i (t) represents the overall sentiment tendency of the search robot i.

[0018] Preferably, the emotion decay for:

[0019]

[0020] Among them, E i (t-1) represents the emotion intensity of the search robot i at the last moment, φ(k i ) represents the sentiment decay function, k i It represents the decay rate of emotion determined by personality, and T represents the decay period.

[0021] Preferably, the external emotional stimulus for:

[0022]

[0023] Among them, S i (t-1) is the stimulus intensity that the search robot i received at the last moment, r i e (t-1) is the environmental reward of the search robot i at the previous moment.

[0024] Preferably, the multi-robot collaborative search model includes: a strategy network Output the current action a according to the current state s(t) i (t); Evaluation network According to the current state s(t) and the current action a i (t) Output action value Q i (t); an experience pool for storing an experience replay array M, which includes the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots, and the next moment state s(t+1) based on the current action a(t) of all search robots, wherein the current state s(t) and the next moment state s(t+1) adopt the current joint observation state o(t) and the next moment joint observation state o(t+1) of all search robots.

[0025] Preferably, according to the overall reward, training the multi-robot collaborative search model includes: S31: initializing the strategy network, the evaluation network and the current state s(t); S32: according to the strategy network and the current state s(t), obtaining the current action a of each search robot i i (t); S33: Execute action a of each search robot ii (t), get the next moment state s of each search robot i i (t+1) and the current total reward R of each search robot i i (t); S34: store the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots and the next moment state s(t+1) as an experience replay array M in the experience pool; S35: sample the experience replay array M, update the policy network and the evaluation network; S36: repeat steps S32-S35 until the training is completed.

[0026] Preferably, sampling the experience playback array M and updating the strategy network and the evaluation network include: S351: according to the expected benefit of the search robot i The gradient of updates the policy network:

[0027]

[0028] in, is the policy network parameter, o(t) is the current joint observation state, o(t)=(o1(t),o2(t),...,o n (t)), Q i (t) is the centralized evaluation value of the search robot i at the current moment; S352: Update the strategy network according to the temporal difference error:

[0029]

[0030] in, To evaluate the network parameters, is the loss value of the evaluation network, y(t) is the target value of the evaluation network, R(t) is the total reward of the search robot i at time t, γ is the discount factor, Q i (t) is the centralized evaluation value of the search robot i at the next moment, It is the deterministic strategy updated by the search robot i at the next moment.

[0031] The present invention discloses a training method for multi-robot collaborative search, comprising: S1: establishing a multi-robot collaborative search task and a kinematic model of the multi-robots; S2: designing an overall reward for the multi-robot collaborative search, including an environmental reward and a motivation reward; S3: training a multi-robot collaborative search model according to the overall reward, thereby obtaining a personality-based emotional system from the perspective of reward supplementation, and combining the system with a multi-agent deep deterministic policy gradient algorithm (Multi-Agent Deep Deterministic Policy Gradient, MADDPG), thereby effectively improving the search efficiency and speed of the multi-robots in a complex dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0033] Figure 1 is a flow chart of a training method for multi-robot collaborative search according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of a multi-robot collaborative search task scenario according to an embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of a robot kinematics model according to an embodiment of the present invention; and

[0036] Figure 4 4 is a block diagram of a MADDPG algorithm based on emotional intrinsic motivation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0038] Figure 1-Figure 4 A flowchart of a training method for multi-robot collaborative search according to an embodiment of the present invention, a schematic diagram of a multi-robot collaborative search task scenario, a schematic diagram of a robot kinematic model, and a block diagram of a MADDPG algorithm based on emotional intrinsic motivation are shown respectively. An embodiment of the present invention provides a training method for multi-robot collaborative search, comprising: S1: establishing a multi-robot collaborative search task and a kinematic model of the multi-robot; S2: designing an overall reward for the multi-robot collaborative search, including an environmental reward and a motivation reward; S3: training a multi-robot collaborative search model based on the overall reward.

[0039] In the related technologies, multi-robot target search methods mostly use probabilistic graph methods and swarm optimization algorithms, but existing methods usually require complex mathematical calculations to solve multi-robot collaborative search problems and are prone to fall into local optimality, which is not suitable for complex search scenarios.

[0040] The embodiment of the present invention adds motivational rewards combined with the emotion system to the traditional MADDPG algorithm to improve the search efficiency and speed of multiple robots in complex dynamic environments. Specifically:

[0041] like Figure 2 As shown in the figure, there are m dynamic targets and n search robots on a bounded two-dimensional continuous plane with obstacles, n>m, all search robots and targets can appear at any coordinates and can move in any direction. The search robot selects a target for pursuit and search through task allocation, and the search robots with the same target form a target search team. When the distance between the robots and the target is less than the search radius r, it means the search is successful.

[0042] Emotional robot i is defined as a five-tuple P i = <Pos i ,E i ,Per i ,R i ,f i >, where: Pos i represents the coordinates (x i ,y i );E i Represents the basic emotional state of the ith robot. i R represents the personality of the i-th robot. i Represents the reward value obtained by the i-th robot. i Represents the emotional cooperation willingness of the ith robot.

[0043] The emotion selection of the emotion robot is based on the famous 6-emotion classification method proposed by Ekman, which divides emotions into fear, anger, sadness, happiness, disgust and surprise. In modern psychology research, happiness, sadness, fear and anger are often regarded as basic emotions. i represents the basic emotion of robot i, E i =[e1,e2,...,e m ], here E i represents the emotional state of robot i, e m Indicates the intensity of basic emotions (such as happiness, fear, sadness), where e m ∈[0,1], the larger the value, the stronger the emotion.

[0044] like Figure 3 As shown, in the multi-robot collaborative search task, the embodiment of the present invention regards each robot as a particle and Figure 2 The robot kinematic model is established in the simulation environment.

[0045] The kinematic model of each emotional robot is the same, including position (coordinates), speed, and direction of movement. The positions of the robot at time t and time t+1 are (x N (t),y N (t)) and (x N (t+1),y N (t+1)), the kinematic model of the robot is as follows:

[0046]

[0047] Where v is the robot's movement speed, which is set to a fixed value in the embodiment of the present invention, and Δt is the time step. N It indicates the angle between the robot's current direction of movement and the positive x-axis of the world coordinate system, and the robot's forward direction.

[0048] The purpose of multi-robot collaborative search is to avoid collision and search for the target. Therefore, the distance between robots, the distance between robots and obstacles, and the distance between robots and targets need to be used as evaluation criteria for measuring multi-robot search strategies. Therefore, the evaluation criteria for the pros and cons of multi-robot collaborative search strategies are designed as follows:

[0049]

[0050] Among them, D i , D n , D o and D t The center coordinates of the i-th robot, the center coordinates of the n-th robot, the center coordinates of the obstacle, and the center coordinates of the target. i , Ra n , Ra o and Ra t The radius of the i-th robot, the radius of the n-th robot, the radius of the obstacle, and the radius of the target. The quality evaluation indicators can be used to judge the status of the robot at the current location and the quality of the current search strategy.

[0051] like Figure 4As shown, the embodiment of the present invention proposes a MADDPG algorithm based on artificial emotion, called MADDPG-E algorithm. In MADDPG-E, the emotion system is added to the MADDPG algorithm. It can generate motivation rewards every time according to environmental stimuli and cognitive states, and use this intrinsic motivation reward and the robot's environmental reward as the overall reward of the robot's search process.

[0052] This can not only effectively solve the problem of sparse rewards in related technologies, but also better avoid collisions. At the same time, based on the different personalities of each robot, adding the willingness of cooperation between robots to the multi-robot collaborative search task can effectively improve the efficiency of cooperative search.

[0053] The algorithm mainly consists of five elements:<S,O,A,R,M> , the specific meaning of each element is as follows:

[0054] (1) S: S = {s k |k=1,2,...,m} is the robot state space set, s k represents the kth state of the robot, m represents the number of states of the robot, and all robots share a state space set.

[0055] (2) O: O = {O1, O2, ..., O N} is the observation set of multiple robots, and N represents the number of robots. In each time step, each robot observes the state of the environment through perception. The robot can obtain the position of obstacles within the detectable range, its own position, the positions of other robots, and the position of the target.

[0056] (3) A: A={A1,A2,...,A N} is the action space set of multiple robots. N represents the action of robot N, which is mainly related to speed and direction. The action taken by robot N at time t+1 is as follows:

[0057]

[0058] Here, α(t+1) is the movement angle of the robot N at time t+1, β is the rate of change of the robot's movement angle, v(t+1) is the movement speed of the robot N at time t+1, and v′ represents acceleration.

[0059] (4) R: is the reward function of the multi-robot system.

[0060] (5) M: represents the memory module, which uses an experience playback array to store the collected experience. Each of them is a four-tuple {s(t), a(t), s(t+1), r e(t)} means:

[0061]

[0062] Among them, N (t) represents the observed state of robot N at time t, a N (t) represents the action selected by robot N at time t.

[0063] According to an embodiment of the present invention, the multi-robot collaborative search model includes: a strategy network Output the current action a according to the current state s(t) i (t); Evaluation network According to the current state s(t) and the current action a i (t) Output action value Q i (t); an experience pool for storing an experience replay array M, which includes the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots, and the next moment state s(t+1) based on the current action a(t) of all search robots, wherein the current state s(t) and the next moment state s(t+1) adopt the current joint observation state o(t) and the next moment joint observation state o(t+1) of all search robots.

[0064] According to an embodiment of the present invention, based on the overall reward, training a multi-robot collaborative search model includes: S31: initializing the policy network, the evaluation network and the current state s(t); S32: According to the strategy network and the current state s(t), the current action a of each search robot i is obtained. i (t); S33: Execute action a of each search robot i i (t), get the next moment state s of each search robot i i (t+1) and the current total reward R of each search robot i i (t); S34: store the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots and the next moment state s(t+1) as an experience replay array M in the experience pool; S35: sample the experience replay array M, update the policy network and the evaluation network; S36: repeat steps S32-S35 until the training is completed.

[0065] According to an embodiment of the present invention, sampling the experience playback array M, updating the strategy network and the evaluation network includes: S351: according to the expected benefit of the search robot i The gradient of updates the policy network:

[0066]

[0067] in, is the policy network parameter, o(t) is the current joint observation state, o(t)=(o1(t),o2(t),...,o n (t)), Q i (t) is the centralized evaluation value of the search robot i at the current moment; S352: Update the strategy network according to the temporal difference error:

[0068]

[0069] in, To evaluate the network parameters, is the loss value of the evaluation network, y(t) is the target value of the evaluation network, R(t) is the total reward of the search robot i at time t, γ is the discount factor, Q i (t) is the centralized evaluation value of the search robot i at the next moment, It is the deterministic strategy updated by the search robot i at the next moment.

[0070] In the MADDPG-E algorithm, for each of the N robots in a multi-robot system, there is a policy (Actor) network and a critic network Actor networks are deterministic and are i ,O i ), output action a i The input of the Critic network is the global state and the actions of all robots, and the output is a real number, which represents the degree of goodness of executing action a based on state s. The Critic network is used to evaluate all the actions of the robot and guide the Actor strategy network to make improvements. The role of the Actor network is to improve the parameters through training Increase the average value of the Critic network. The role of the Critic network is to use the joint observations and actions of the robot for centralized training. The centralized evaluation of the deterministic action strategy is optimized, and the Critic network is updated with the temporal difference error, which allows the Critic network to better fit the value function Q(s,a).

[0071] The multi-robot collaborative search process is converted into a Markov decision process. In each time step t, robot i receives an overall reward and takes an action based on its current state. The dynamics model p is transformed according to the environment. i (s i (t+1)|si (t),a i (t)) changes the current state to the next state. The goal of each search robot is to maximize its final expected reward. According to an embodiment of the present invention, the total reward R(t) of the search robot i at time t is:

[0072]

[0073] Where N is the total number of search robots, d i is the distance between the search robot and the search target, r i e (t) is the environmental reward, r i em (t) is the motivation reward, x i (t), y i (t), x tar (t), y tar (t) are the position coordinate values ​​of the search robot i and the search target at time t respectively.

[0074] According to an embodiment of the present invention, the search robot i triggers one or more environmental rewards at time t, including: a first environmental reward r i 1 (t) = 10, used to reward the search robot i for searching the target; the second environment rewards r i 2 (t) = -2, used to reward the search robot i for colliding with an obstacle; the third environment reward It is used to reward the collision between the search robots, where λ is the collision penalty factor; the fourth environment reward r i 4 (t) = -10 × (max (|x i (t)|,|y i (t)|)-0.9), which is used to reward the search robot i for moving to the boundary.

[0075] The multi-robot collaborative search method of the embodiment of the present invention firstly performs artificial emotion modeling on the emotion system. The emotion robot generates basic emotions according to internal states such as personality and emotion attenuation and external stimuli, and adjusts deep reinforcement learning based on the emotional intrinsic rewards provided by the basic emotions. In this way, emotions can directly and effectively participate in task allocation decisions, thereby optimizing algorithm performance. According to the embodiment of the present invention, the formula for the motivation reward is:

[0076]

[0077] Among them, E i (t) is the emotion intensity of the search robot i, which can be specifically expressed as:

[0078] in, Personality mapping for the search robot i, For the emotional decay of the search robot i, is the external emotional stimulus received by the search robot i, g(x) is the emotional intensity E i The value of (t) is kept in the range of [-1,1].

[0079] According to an embodiment of the present invention,

[0080] In the related art, during the robot search process, rewards are often sparse, that is, when the robot is performing a task, it often needs to go through a series of attempts until the task is completed before it can get a reward, and there is no reward in the process. However, internal emotions can make the robot generate current emotions and motivation rewards according to environmental stimuli and cognitive states, thereby improving the robot's search efficiency.

[0081] Personality is an intrinsic characteristic of a robot, which determines the overall behavioral and psychological tendencies of individual robots and reflects the heterogeneity and diversity of robots in a group. For example, compared with individuals with cautious personalities, individuals with bold personalities are more likely to perform dangerous tasks. At present, the OCEAN personality model is the most common personality representation method. Therefore, the embodiment of the present invention describes the personality of the robot through five aspects: openness, conscientiousness, extroversion, agreeableness, and neuroticism: per = [Op, Co, Ex, Ag, Ne], where Op represents openness, Co represents conscientiousness, Ex represents extroversion, Ag represents agreeableness, and Ne represents neuroticism. The values ​​of all attributes are between [-1, 1]. The personality space is mapped to the PAD emotional space, and the following conversion relationship is established for this purpose:

[0082]

[0083] According to an embodiment of the present invention, the personality mapping for:

[0001]

[0002] Among them, v is the basic emotional state of the robot (v = 1, 2, ... m), is the coordinate corresponding to the search robot i in the PAD emotion space when it is in any basic emotion state v, represents the coordinates of the personality of the search robot i in the PAD emotional space, dis i,v (t) represents the closeness between the personality of the search robot i and any of the basic emotional states, D i (t) represents the overall sentiment tendency of the search robot i.

[0003] The emotion decay mechanism is proposed based on the "third law of emotion intensity" in psychology. Any emotional state of any intensity will decay infinitely and tend to zero over time. If the influence of external values ​​is ignored, the emotion state intensity decay curve and exponent y=e -x When studying the emotion decay in the embodiment of the present invention, the current emotion state of the robot decays on the basis of the emotion state at the previous moment.

[0004] According to an embodiment of the present invention, the emotion decay for:

[0005]

[0006] Among them, E i (t-1) represents the emotion intensity of the search robot i at the last moment, φ(k i ) represents the sentiment decay function, k i It represents the decay rate of emotion determined by personality, and T represents the decay period.

[0007] External stimulation is the main reason for emotional changes, and the reward of the task is the decisive factor of stimulation. The higher the reward of the task, the stronger the positive stimulation. Therefore, the embodiment of the present invention defines stimulation S based on the environmental rewards received by the robot of different tasks. The types of stimulation correspond to the basic types of emotions.

[0008] According to an embodiment of the present invention, the external emotional stimulus for:

[0009]

[0010] Among them, S i (t-1) is the stimulus intensity that the search robot i received at the last moment, r i e (t-1) is the environmental reward of the search robot i at the previous moment.

[0011] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0012] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-robot collaborative search training method, characterized in that: include: S1: Establishing a multi-robot collaborative search task and a kinematic model of the multi-robots; S2: Design the overall reward for the multi-robot collaborative search, including environmental rewards and motivation rewards; S3: Based on the overall reward, train a multi-robot collaborative search model.

2. The multi-robot collaborative search training method according to claim 1, characterized in that: The total reward R(t) of the search robot i at time t is: Where N is the total number of search robots, d i is the distance between the search robot and the search target, r i e (t) is the environmental reward, r i em (t) is the motivation reward, x i (t), y i (t), x tar (t), y tar (t) are the position coordinate values ​​of the search robot i and the search target at time t respectively.

3. The multi-robot collaborative search training method according to claim 2, characterized in that: The search robot i triggers one or more environmental rewards at time t, including: First Environment Rewards i 1 (t) = 10, used to reward the search robot i for searching the target; Second Environment Rewards i 2 (t) = -2, used to reward the search robot i for colliding with an obstacle; Third Environment Rewards Used to reward collisions between the search robots, where λ is a collision penalty factor; Fourth Environment Rewards i 4 (t) = -10 × (max (|x i (t)|,|y i (t)|)-0.9), used to reward the search robot i for moving to the boundary.

4. The multi-robot collaborative search training method according to claim 2, characterized in that: The formula for the motivation reward is: Among them, E i (t) is the emotion intensity of the search robot i, which can be specifically expressed as: in, is the personality mapping of the search robot i, is the emotion decay of the search robot i, is the external emotional stimulus received by the search robot i, g(x) is the emotional intensity E i The value of (t) is kept in the range of [-1,1].

5. The multi-robot collaborative search training method according to claim 4, characterized in that: The personality map for: Among them, v is the basic emotional state of the robot (v = 1, 2, ... m), is the coordinate corresponding to the search robot i in the PAD emotion space when it is in any of the basic emotion states v, represents the coordinates corresponding to the personality of the search robot i in the PAD emotional space, dis i,v (t) represents the closeness between the personality of the search robot i and any of the basic emotional states, D i (t) represents the overall sentiment tendency of the search robot i.

6. The multi-robot collaborative search training method according to claim 4, characterized in that: The emotional decay for: Among them, E i (t-1) represents the emotion intensity of the search robot i at the last moment, φ(k i ) represents the sentiment decay function, k i It represents the decay rate of emotion determined by personality, and T represents the decay period.

7. The multi-robot collaborative search training method according to claim 4, characterized in that: The external emotional stimulus for: Among them, S i (t-1) is the stimulus intensity received by the search robot i at the last moment, is the environmental reward of the search robot i at the previous moment.

8. The multi-robot collaborative search training method according to claim 1, characterized in that: The multi-robot collaborative search model includes: Policy Network Output the current action a according to the current state s(t) i (t); Evaluation Network According to the current state s(t) and the current action a i (t) Output action value Q i (t); An experience pool is used to store an experience replay array M, wherein the experience replay array includes the current state s(t), the current action a(t) of all search robots, the current total reward R(t) of all search robots, and the next moment state s(t+1) based on the current action a(t) of all search robots, wherein the current state s(t) and the next moment state s(t+1) adopt the current joint observation state o(t) and the next moment joint observation state o(t+1) of all search robots.

9. The multi-robot collaborative search training method according to claim 8, characterized in that: According to the overall reward, training the multi-robot collaborative search model includes: S31: Initializing the strategy network, the evaluation network and the current state s(t); S32: According to the strategy network and the current state s(t), obtain the current action a of each search robot i i (t); S33: Execute action a of each search robot i i (t), get the next moment state s of each search robot i i (t+1) and the current total reward R of each search robot i i (t); S34: storing the current state s(t), the current actions a(t) of all the search robots, the current total rewards R(t) of all the search robots, and the next moment state s(t+1) as an experience playback array M in the experience pool; S35: sampling the experience playback array M, and updating the policy network and the evaluation network; S36: Repeat steps S32-S35 until the training is completed.

10. The multi-robot collaborative search training method according to claim 9, characterized in that: Sampling the experience playback array M and updating the policy network and the evaluation network include: S351: According to the expected profit of the search robot i The gradient of the policy network is updated: in, is the policy network parameter, o(t) is the current joint observation state, o(t)=(o1(t),o2(t),...,o n (t)), Q i (t) is the centralized evaluation value of the search robot i at the current moment; S352: Update the strategy network according to the temporal difference error: in, To evaluate the network parameters, is the loss value of the evaluation network, y(t) is the target value of the evaluation network, R(t) is the total reward of the search robot i at time t, γ is the discount factor, Q i (t) is the centralized evaluation value of the search robot i at the next moment, It is the deterministic strategy updated by the search robot i at the next moment.

Images