Mobile robot multi-target hunting method and system based on density field and deep reinforcement learning

Abstract

The invention discloses a mobile robot multi-target hunting method and system based on a density field and deep reinforcement learning, and the method employs the density field and deep reinforcement learning technology. Comprising seven steps of perception information acquisition, target density calculation, target distribution decision, hunting state judgment, observation state generation, action decision and execution and hunting process iteration, a density field algorithm is combined with a relative position to calculate a score, so that reasonable real-time distribution of targets and robot grouping are realized, and hunting force balance is ensured; and the robots in the same group efficiently surround the target based on a deep reinforcement learning training strategy. According to the method, the system performance is remarkably improved in the aspects that the mobile robot rapidly gets close to the target and rapidly forms the surrounding formation, and an efficient and intelligent solution is provided for a multi-target surrounding scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of mobile robots, and mainly relates to a multi-target encirclement method and system for mobile robots based on density field and deep reinforcement learning. Background Art

[0002] Multi-target encirclement of mobile robots has the characteristics of strong reliability, scalable structure, and diverse task execution, and is currently being increasingly widely applied in military, industrial production, warehouse logistics, science, education and entertainment, etc. Existing multi-target encirclement methods for mobile robots are mainly divided into two types: traditional rule-based methods and learning-based methods. The multi-target encirclement scenario includes two tasks: target allocation and encirclement. As a rule-based spatial description method, the density field has been applied to the multi-target encirclement scenario of mobile robots, and it can effectively complete the target allocation task. However, in multi-robot collaborative encirclement, there are obvious deficiencies in improving the encirclement efficiency.

[0003] In the actual collaborative encirclement process, the lack of intelligent strategies for action coordination and dynamic adjustment between different robots makes it difficult to meet the requirements of efficient encirclement. With the rapid development of deep learning technology, multi-target encirclement methods based on deep reinforcement learning have received increasing attention. This method enables the intelligent agent to automatically learn and optimize its own decision-making strategy through interaction with the environment, showing significant advantages in dynamic and complex tasks. However, currently, most of these methods focus on solving the problem of a single target encirclement by robot collaboration. In the multi-target encirclement scenario, due to the simultaneous training of target allocation and encirclement, it often leads to difficulties in designing the reward function and slow convergence in training. Summary of the Invention

[0004] The present invention precisely aims at the problem that the existing traditional rule-based multi-target encirclement method for mobile robots has low encirclement efficiency due to insufficient dynamic adjustment ability, and the problem that the learning-based method faces high training complexity and difficult convergence for multiple encirclement targets. A multi-target encirclement method and system for mobile robots based on density field and deep reinforcement learning are proposed. The density field is combined with deep reinforcement learning technology, including seven steps: perception information acquisition, target density calculation, target allocation decision, encirclement state judgment, observation state generation, action decision and execution, and encirclement process iteration. The density field algorithm is used to calculate scores in combination with relative positions, so as to realize the reasonable and real-time allocation of targets and robot grouping, ensuring the balance of encirclement forces; the robots within the same group perform efficient encirclement of the targets based on the strategies trained by deep reinforcement learning. The method of the present invention significantly improves the system performance in terms of the mobile robot quickly approaching the target and quickly forming an encirclement formation, providing an efficient and intelligent solution for the multi-target encirclement scenario.

[0005] To achieve the above object, the technical solution adopted by the present invention is: a multi-target pursuit method for a mobile robot based on a density field and deep reinforcement learning, comprising the following steps:

[0006] S1. Perception information acquisition: Obtain the relative position information of the mobile robot i at the current moment with the teammate robots in the environment and all the targets to be pursued and the relative velocity information

[0007] S2. Target density calculation: Calculate the density values of all the targets to be pursued in the environment according to the relative position information obtained in step S1;

[0008] S3. Target allocation decision: Calculate the selection preference score of each mobile robot for each target according to the relative position of the mobile robot and all the targets to be pursued obtained in step S1 and the density values of the targets to be pursued calculated in step S2, and obtain the pursuit target P i and the set N of cooperative robots within the group i ;

[0009] S4. Pursuit state judgment: Judge whether the mobile robot and the cooperative robots within the group meet the preset pursuit target. If so, end the pursuit task; otherwise, continue to execute step S5; Whether the preset pursuit target is met specifically means: for the target P jointly pursued by the robot and the set N of cooperative robots within its group i whether the preset angle condition and distance condition are met; i

[0010] S5. Observation state generation: Extract the observation state from the pursuit target P obtained in step S3 i and the set N of cooperative robots within the group i , the relative position and relative velocity information obtained in step S1 The observation state is a vector with the number of elements being 4*(1 + |N i |);

[0011] S6. Action decision and execution: Input the observation state extracted in step S5 into the pre-trained policy network to obtain the action under the current observation state After parsing the action instruction, control the mobile robot i to execute the action It is represented by the following formula:

[0012]

[0013] where net i ( ) is a multi-layer perceptron network;​

[0014] S7. Iteration of the round-up process: After the mobile robot finishes executing an action it returns to step S1 and continues to execute the next time-step process until the round-up task in step S4 ends.

[0015] As an improvement of the present invention, the calculation method of the density value of the target k to be rounded up in step S2 is specifically: Specifically:

[0016]

[0017] wherein, represents the relative position vector between the target k to be rounded up and the mobile robot i; N represents the number of all mobile robots in the environment; x k represents the position vector of the target k to be rounded up in space; || || 2 is the vector two-norm operation; the W() function is the following piecewise function:

[0018]

[0019] wherein, R is a preset distance threshold.

[0020] As an improvement of the present invention, the calculation method of the selection propensity score in step S3 is specifically: Specifically:

[0021]

[0022] wherein, β 1 and β 2 are constant values that control the importance of the density value and the relative distance

[0023] As another improvement of the present invention, in step S3, the target allocation algorithm adopts one of the Hungarian algorithm, the auction algorithm, the genetic algorithm, or the swarm algorithm, and the rules of the target allocation algorithm are:

[0024] (1) Randomly sort all mobile robot numbers;

[0025] (2) According to the sorting in rule (1), the mobile robots in the sequence sequentially allocate the target with the highest selection propensity score they have. If the target k with the highest score has already been allocated to 3 mobile robots, then select the target corresponding to the second-highest score, and so on until a target is allocated;

[0026] (3) Proceed in this way until all mobile robots are allocated round-up targets.

[0027] ​As another improvement of the present invention, in the step S4, the robot and its collaborative robot set N i For the target jointly surrounded by them, whether the preset angle condition and distance condition are satisfied is specifically:

[0028] And

[0029]

[0030] Among them, And Are respectively the distance between the mobile robots r 1 , r 2 And the included angle relative to the surrounded target; dis safe Is the safe distance that needs to be maintained between the mobile robots; Is the distance between the mobile robot r 1 And the surrounded target.

[0031] As yet another improvement of the present invention, in the step S5, the observation state of the agent Is specifically:

[0032]

[0033] Among them, And Are the relative position and relative speed of the mobile robot i and the target P i , And Are the relative position and relative speed of the mobile robot i and the collaborative robot j within the same group.

[0034] As a further improvement of the present invention, during the iterative process of the surrounding process in the step S7, at time t, the motion logic of the mobile robot i is:

[0035]

[0036] Among them, Is the position coordinate of the mobile robot i at time t, Is the speed of the mobile robot i at time t, The acceleration of the mobile robot i at time t, and Δt is the time unit.

[0037] To achieve the above object, the technical solution adopted by the present invention is also: a multi-target surrounding system for mobile robots based on density field and deep reinforcement learning, including a sensing module, a density field calculation module, a target allocation module, a surrounding judgment module, a policy network module, and a motion control module,

[0038] The perception module: It establishes data connections with the density field calculation module, the target allocation module, and the policy network module respectively, and includes a position and velocity perception unit and a data preprocessing unit. Among them, the position and velocity perception unit is used to collect the relative positions and relative velocity information of the robot with teammate robots and all targets to be surrounded in the environment, and the data preprocessing unit is used to process and optimize the collected data;

[0039] The density field calculation module: It includes a space division unit and a density calculation unit, and is used to calculate the density values of all targets to be surrounded in the environment;

[0040] The target allocation module: It establishes data connections with the density field calculation module, the surrounding judgment module, and the policy network module respectively, and includes a score calculation unit and an allocation algorithm execution unit. Among them, the score calculation unit is used to calculate the tendency score of the robot to select the target to be surrounded, and the allocation algorithm execution unit executes the target allocation algorithm according to the output score of the score calculation unit to determine the surrounding target of each robot and the set of cooperative robots within the group;

[0041] The surrounding judgment module: It establishes data connections with the policy network module and the motion control module respectively, and includes a condition setting unit and a judgment execution unit. Among them, the condition setting unit is used to preset the surrounding target conditions; the judgment execution unit judges whether the surrounding target is satisfied according to the conditions of the condition setting unit;

[0042] The policy network module: It establishes a data connection with the motion control module, and includes an observation state generation unit, a policy network training unit, and an action decision unit. Among them, the observation state generation unit is used to extract the observation state as the input of the policy network training unit; the action decision unit inputs the observation state into the pre-trained policy network to obtain the action of the robot under the current observation state;

[0043] The motion control module: It includes an instruction parsing unit and a driving execution unit, and is used to parse the action instruction and control the robot to execute the action.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] (1) Achieve uniform and rapid encirclement: The present invention innovatively combines the density field and deep reinforcement learning, significantly improving the encirclement effect. In the target allocation stage, the target density value is calculated through the density field algorithm (step S2), and the selection tendency score is calculated in combination with the relative position (step S3). The Hungarian algorithm and other target allocation algorithms are used to achieve reasonable target allocation and robot grouping. This process fully considers the distribution of targets and the positional relationship between robots and targets, enabling the balanced distribution of encirclement forces, avoiding the situation where some targets are over-concerned or ignored, and achieving uniform encirclement. In the encirclement execution stage, the robots within the same group can make intelligent decisions based on the policy trained by deep reinforcement learning (step S6), quickly adjusting their movement directions and speeds according to the real-time observation state.

[0046] (2) Flexibly expand the number of robots and the number of encirclement targets: In the training stage, deep reinforcement learning is only trained for the scenario where some robots encircle one target. By this means, the training process is simplified, the training complexity is reduced, and the policy network can learn the encirclement strategy more efficiently. In the execution stage, based on the policy network model obtained from the previous training, it can be flexibly extended to complex scenarios where more robots encircle multiple targets. In actual encirclement tasks, multiple robots are reasonably allocated to different targets through the target allocation module (step S3), and the robots within each group perform encirclement according to the trained policy. This method improves the versatility and scalability of the system and can adapt to multi-target encirclement tasks of different scales.

[0047] (3) Reduce the information processing burden and improve the system response speed: During the execution process, the grouped robots only need to obtain the environmental information within the group and do not require global information. In the observation state generation step (step S5), the observation state of the intelligent agent only includes the relative position and relative speed information related to the encirclement target and the collaborative robots within the group, greatly reducing the amount of information that the robots need to process. This local information acquisition method reduces the information processing burden of the robots, avoids the computational delay caused by processing a large amount of global information, and enables the robots to respond more quickly to local environmental changes. In the face of a complex and changing encirclement environment, the grouped robots can timely adjust their actions to ensure the smooth progress of the encirclement task, improving the real-time performance and stability of the system. Brief Description of the Drawings

[0048] Figure 1 is the step flow chart of the multi-target encirclement method for mobile robots based on density field and deep reinforcement learning of the present invention;

[0049] Figure 2 is the structural schematic diagram of the multi-target encirclement system for mobile robots based on density field and deep reinforcement learning of the present invention;

[0050] Figure 3 Schematic diagram of the encirclement effect of Scenario 1 in Embodiment 3 of the present invention;

[0051] Figure 4 Schematic diagram of the encirclement effect of Scenario 2 in Embodiment 3 of the present invention;

[0052] Figure 5 Comparison diagram of the distance between the mobile robot and the encirclement target of two encirclement models in Scenario 1 in Embodiment 3 of the present invention;

[0053] Figure 6 Comparison diagram of the encirclement angle of two encirclement models in Scenario 1 in Embodiment 3 of the present invention;

[0054] Figure 7 Comparison diagram of the distance between the mobile robot and the encirclement target of two encirclement models in Scenario 2 in Embodiment 3 of the present invention;

[0055] Figure 8 Comparison diagram of the encirclement angle of two encirclement models in Scenario 2 in Embodiment 3 of the present invention;

[0056] Figure 9 Structure diagram of the policy network of the MADDPG algorithm in Embodiment 3 of the present invention;

[0057] Figure 10 Structure diagram of the value network of the MADDPG algorithm in Embodiment 3 of the present invention. Detailed implementation manners

[0058] The present invention will be further clarified below in conjunction with the accompanying drawings and specific implementation manners. It should be understood that the following specific implementation manners are only used to illustrate the present invention and not to limit the scope of the present invention.

[0059] Embodiment 1

[0060] Before applying deep reinforcement learning combined with density field to multi-target encirclement of mobile robots, it is necessary to train multi-robot encirclement of a single target based on the MADDPG algorithm. The trained policy network is shared by each mobile robot performing the encirclement task. The following gives the observation information of agent i for multi-robot encirclement of a single target based on the MADDPG algorithm Input state of the value network Reward i :

[0061] When training the encirclement model using the MADDPG algorithm, the observation state of the agent is represented by the following formula:

[0062]

[0063] where, and For the relative position and relative velocity of mobile robot i and target P i and and For the relative position and relative velocity of mobile robot i and cooperative robot j within the same group, therefore is a vector with the number of elements being 4*(1 + |N i |);

[0064] The input state state of the value network t Evaluates the policy value by integrating all agent information and is expressed by the following formula:

[0065]

[0066] The total reward obtained by mobile robot i at each time step is Reward i , and this reward includes: the target approaching reward r t , the angle between robots reward r s , and the collision avoidance reward t c , which are respectively expressed by the following formulas:

[0067] Reward i = r t + r s + r c

[0068]

[0069] Similarly, the reward of each mobile robot {Reward 1 , Reward 2 … Reward N} can be calculated.

[0070] Based on the above description and the idea of the MADDPG algorithm, the policy network net of the mobile robot is trained. After sharing the policy network net among all mobile robots performing the encirclement task, the mobile multi-robot multi-target encirclement method based on the density field and deep reinforcement learning as shown in Figure 1 is executed. Figure 1 It includes the following steps:

[0071] Step S1: Perception information acquisition;

[0072] Obtain the relative position information and relative velocity information of mobile robot i and teammate robots in the environment and all targets to be encircled at the current moment The relative position information includes the relative positions of teammate robots and the relative positions of all targets to be encircled The relative speed information includes the relative speed of the teammate robots and the relative speeds of all targets to be surrounded where N and M are the numbers of all mobile robots and targets to be surrounded in the environment respectively, i, j = 1, 2, …, n, (i ≠ j); k = 1, 2, …, M

[0073] Step S2: Target density calculation;

[0074] Based on the relative position information obtained in step S1 calculate the density values of all targets to be surrounded in the environment; for each target k to be surrounded, its relative position vector with the mobile robot i in the environment is and its position vector in space is x k , and the density value of target k can be calculated using the following formula

[0075]

[0076] where || || 2 is the vector two-norm operation; the W( ) function is the following piecewise function

[0077]

[0078] where R is a preset distance threshold

[0079] Step S3: Target assignment decision;

[0080] Combining the relative positions X of the mobile robot i with all targets to be surrounded obtained in step S1 t and the set of density values of all targets to be surrounded obtained in step S2 calculate the score for the robot i to select the target k to be surrounded as obtain the scores of the robot i for all targets to be surrounded Determine the summary of the scores of the robots in the environment for all targets to be surrounded The robot i communicates with the other robots and obtains the surrounded target P i at this moment and the set N of cooperative robots within the group i , where the target assignment algorithm adopts one of the Hungarian algorithm, auction algorithm, genetic algorithm, and swarm algorithm

[0081] The specific calculation formula of

[0082]

[0083] Among them, β 1 and β 2 are constant values that control the importance of the density value and the relative distance .

[0084] Step S4: Judging the encirclement state;

[0085] Judge whether the mobile robot i and the cooperative robots in the group meet the preset encirclement goal. If so, end the encirclement task; otherwise, execute Step S5. The preset method of the encirclement goal is as follows: The robot i and its cooperative robot set N i For the target P i they jointly encircle, the following angle condition and distance condition need to be satisfied:

[0086] and

[0087]

[0088] Among them, and are the distance and the included angle relative to the encirclement target between the mobile robots r 1 , r 2 respectively; dis safe is the safe distance that needs to be maintained between mobile robots; is the distance between the mobile robot r 1 and the encirclement target. When the encirclement is successful, needs to approach the preset value dis trap .

[0089] Step S5: Generating the observation state;

[0090] Combining the encirclement target P i obtained in Step S3 and the cooperative robot set N i in the group, extract the observation state from the relative position and relative velocity information obtained in Step S1

[0091] Train the encirclement model using the MADDPG algorithm. The observation state of the agent is expressed by the following formula:

[0092]

[0093] Among them, and are the relative position and relative velocity of the mobile robot i and the target P i , and is the relative position and relative velocity of the mobile robot i and the collaborative robot j within the same group. Therefore, is a vector with the number of elements being 4*(1 + |N i |).

[0094] Step S6: Action decision and execution;

[0095] Input the observation state into the pre-trained policy network to obtain the action under the current observation state Parse the action instruction and control the mobile robot i to execute the action according to the parsed instruction

[0096]

[0097] where net i ( ) is a multi-layer perceptron network.

[0098] Step S7: Iteration of the encirclement process;

[0099] After the mobile robot finishes executing the action return to Step S1 and continue to execute the next time-step process. And at time t, the motion logic of the mobile robot i is as follows:

[0100]

[0101] where, is the position coordinate of the mobile robot i at time t, is the velocity of the mobile robot i at time t, the acceleration of the mobile robot i at time t, and Δt is the time unit.

[0102] Embodiment 2

[0103] A multi-target encirclement system for mobile robots based on density field and deep reinforcement learning, as Figure 2 shown, includes a sensing module, a density field calculation module, a target allocation module, an encirclement judgment module, a policy network module, and a motion control module. The sensing module respectively establishes data connections with the density field calculation module, the target allocation module, and the policy network module. The target allocation module respectively establishes data connections with the density field calculation module, the encirclement judgment module, and the policy network module. The encirclement judgment module respectively establishes data connections with the policy network module and the motion control module. The policy network module establishes a data connection with the motion control module.

[0104] The perception module includes a position and velocity perception unit and a data preprocessing unit. The density field calculation module includes a space division unit and a density calculation unit. The target allocation module includes a score calculation unit and an allocation algorithm execution unit. The encirclement judgment module includes a condition setting unit and a judgment execution unit. The policy network module includes an observation state generation unit, a policy network training unit, and an action decision-making unit. The motion control module includes an instruction parsing unit and a driving execution unit.

[0105] The position and velocity perception unit in the perception module is used to collect the relative position and relative velocity information of the mobile robot with the teammate robots and all targets to be encircled in the environment. This process can obtain data through sensors such as lidar and cameras. Then, the data preprocessing unit preprocesses the collected data. The preprocessing mainly includes operations such as denoising and filtering to remove the noise data generated by sensor errors or environmental interference, and optimizes the data through algorithms such as Kalman filtering to improve the accuracy and stability of the data. The preprocessed position information of the perception module is input into the density field calculation module. The space division unit divides the encirclement environment into multiple grids, providing a basic framework for density value calculation. The density calculation unit calculates the density values of all targets to be encircled in the environment according to the density field algorithm formula. In the target allocation module, the score calculation unit combines the relative position of the robot and the target and the target density value to calculate the score of the robot for selecting the target to be encircled. The allocation algorithm execution unit executes the target allocation algorithm based on the score to determine the encirclement target of each robot and the set of collaborative robots within the group. The condition setting unit of the encirclement judgment module presets the encirclement target conditions. The encirclement target conditions specifically include the angle condition and distance condition that the robot and its set of collaborative robots need to meet for the common encirclement target. The judgment execution unit judges whether the mobile robot and the collaborative robots meet the encirclement target conditions according to these conditions. If they meet, the encirclement task ends; if not, it enters the next step. The observation state generation unit of the policy network module combines the assigned target and the set of collaborative robots to extract the observation state from the perception information. This observation state is used as the input of the policy network to describe the current encirclement environment information of the robot. The action decision-making unit inputs the observation state into the pre-trained policy network to obtain the action of the robot under the current observation state. The instruction parsing unit of the motion control module parses the action instruction and converts it into a control signal that the mobile robot can understand and execute. The driving execution unit controls the mobile robot to execute the action according to the parsed instruction. After the robot executes the action, the encirclement process is iterated until the mobile robot and the collaborative robots meet the encirclement target conditions. Among them, the policy network training unit uses the MADDPG algorithm to train the encirclement model. During the training process, according to the observation state, action, and reward feedback of the agent, the parameters of the policy network are continuously adjusted to optimize the decision-making strategy. Through a large number of trainings, the policy network can output more reasonable action instructions according to different observation states, improving the encirclement efficiency.

[0106] Example 3

[0107] In this example, the task of 9 robots surrounding 3 stationary targets is simulated under two scenarios, and a multi-target surrounding model combining density field and artificial potential field in the prior art is used as a comparative example; Scenario 1 is as shown in Figure 3 . In a 40m x 40m map, the initial positions of the mobile robots are evenly distributed on a circle with a radius of 4 centered at the coordinate (0, 0). Scenario 2 is as shown in Figure 4 . In a 40m x 40m map, the initial positions of the mobile robots are evenly distributed on a circle with a radius of 10 centered at the coordinate (10, 10). The surrounding target coordinates in Scenario 1 and Scenario 2 are the same, which are (5, 6), (10, 16), and (15, 6) respectively. In this example, the motion limit of all mobile robots is acceleration |a| ≤ 5.0m / s 2 , speed |v| ≤ 5m / s. The density field parameters involved are as follows: The preset distance R = 20m required to calculate the density value, and the control weights β 1 = 60, β 2 = 0.2. The maximum execution time of the mobile robot for the task is 400 time steps, and the time step Δt = 0.1s.

[0108] Before the mobile robot executes the multi-target surrounding task, the policy network is trained using the MADDPG algorithm. For the scenario of 9 robots surrounding 3 targets, 3 robots are required to surround 1 target. Therefore, in the training of the policy network, the training environment is designed as follows: In a 40m x 40m map, 3 mobile robots are randomly distributed at any position on the map, and the surrounding target is stationary at the map center (20, 20); The motion limit of the mobile robot and the task time step are the same as those in the example; The safety distance dis safe between mobile robots = 2m, and the surrounding distance dis trap between the mobile robot and the target = 3m; The environmental rewards r t , r s , r c proportionality coefficients λ t = 1, λ s = 1, λ c = 2. The MADDPG training parameters are as follows: The total number of training episodes = 70000, the maximum number of steps per episode = 400, the capacity of the experience buffer = 10 6 , the training batch size = 1024, the network update frequency steps update = 400, the learning rate lr = 0.002, and the soft update coefficient tau = 0.01. The reward calculation method for mobile robot i is as follows:

[0109]

[0110] Similarly, calculate the rewards of all mobile robots participating in the training {Reward 1 , Reward 2 , Reward 3}. During the training process, all robots share network parameters. The structures of the policy network and the value network are shown in Appendix Figure 9 and Appendix Figure 10 . Both the policy network and the value network are multi-layer perceptron structures with 4 hidden layers (the number of neurons and activation functions involved in the network are shown in the figure). During the training of mobile robot i, the policy network and the value network respectively receive the robot's observation and the state and then respectively output the action and the value (important components for assisting in training the policy network in the MADDPG algorithm). With the above parameters and settings, the parameter of the policy network model of the mobile robot obtained by training is net.

[0111] The purpose of the present invention is to improve the efficiency of multi-target capture of mobile robots based on the density field by introducing a multi-agent reinforcement learning algorithm. The capture efficiency is defined as the time value when all mobile robots meet the preset capture success conditions. The mobile robot group with less time used has higher efficiency.

[0112] Based on the above objective, at the initial moment: t = 0s, on a 40m x 40m map, the mobile robots N = {r 1 , r 2 ,... r 9} are stationary and evenly distributed on a circle with the coordinate (0, 0) as the center and a radius of 4 (taking Scenario 1 as an example). The speed of the mobile robot acceleration The policy network net 1 = net 2 =... net 9 = net. Compared with Scenario 1, only the initial distribution position coordinates of the mobile robots are different in Scenario 2.

[0113] A multi-target capture method for mobile robots based on the density field and deep reinforcement learning is as follows:

[0114] Step S1: For any mobile robot i, use the perception module to obtain the relative position and relative speed information of the teammate robots and all targets to be captured in the map at time t. The relative position information includes the relative position of the teammate robots and the relative positions of all targets to be captured The relative speed information includes the relative speed of the teammate robots Relative position to all targets to be captured Among them, i≠j.

[0115] Step S2: For all mobile robots, the relative position information of the same target in step S1 is Input the density field calculation module, which calculates the density values ​​of all the targets to be captured in the environment. In scenarios 1 and 2, the specific calculation method of the density field calculation module is as follows: For each target k to be captured, its relative position vector with the mobile robot i in the environment Its position vector in the map is x k , the density value of target k can be calculated using the following formula

[0116]

[0117] In scenarios 1 and 2, the target is stationary, so at any time W(||x 1 || 2 )=0.492,W(||x 2 || 2 )=0.001,W(||x 3 || 2 )=0.024.

[0118] Step S3: For all mobile robots, combine the and the target density value calculated in step S2 Calculate the selection tendency score of each mobile robot for each target. For example, for mobile robot i, The selection tendency scores for the three targets are calculated according to the following formula:

[0119]

[0120] All mobile robots communicate and summarize their respective score sets Based on the target allocation algorithm, the capture target P at time t is obtained i and the set of collaborative robots in the group N i (For mobile robot i). The target allocation algorithm is as follows: ① Randomly sort all mobile robot numbers; ② According to the sorting in ①, the mobile robots in the sequence are assigned their selection tendency scores in turn. The highest target, if the target k with the highest score has been assigned to 3 mobile robots, then select the target corresponding to the second highest score, and so on until the target is assigned; ③ In this way, all mobile robots are assigned to capture targets.

[0121] Step S4: For any mobile robot i, the encirclement judgment module determines whether the mobile robot i and the collaborative robots in the group meet the preset encirclement goal. If so, the encirclement task ends; otherwise, step five is executed. Among them, in Scenario 1 and Scenario 2, the preset method for the encirclement goal is as follows: For robot i and its collaborative robot set N i For the target P they jointly encircle i , the following angle conditions and distance conditions need to be satisfied:

[0122] And

[0123]

[0124] Steps S5 and S6: For any mobile robot i, combining the relative position and relative velocity information obtained in step S1, the assigned target P i and the information of the collaborative robot set N i can be used to calculate the observed state at time t

[0125]

[0126] Input into the policy network module to calculate the action at time t:

[0127]

[0128] Step S7: For any mobile robot i, after executing the action , return to step S1 and continue to execute the next time step process. And at time t, the coordinates of the mobile robot i

[0129]

[0130] The methods and systems proposed in Embodiment 1 and Embodiment 2 can achieve target allocation and collaborative encirclement in the above two scenarios. The comparison results of the distances between the mobile robots and the encirclement targets in Scenario 1 for the two encirclement models are as Figure 5 shown, and the comparison results of the encirclement angles are as Figure 6 shown. The comparison results of the distances between the mobile robots and the encirclement targets in Scenario 2 for the two encirclement models are as Figure 7 shown, and the comparison results of the encirclement angles are as Figure 8As shown in the figure, by synthesizing the comparison results in the two scenarios, it can be seen that in the multi-target capture task of mobile robots, the present invention combines the density field and deep reinforcement learning, and shows significant advantages in the control of the average distance (the average value of the distances between all mobile robots and their capture targets) and the average angle (the average value of the included angles formed between all mobile robots and their capture targets), specifically manifested as prominent numerical differences.

[0131] In terms of distance control, in Scenario 1, Figure 5 it shows that at time step 150, the average distance between the mobile robots corresponding to the method of the present invention and the target has approached the expected distance, while the comparison model only reaches the expected distance for the first time at time step 300. In Scenario 2, Figure 7 it shows that at the initial stage of capture, that is, at time step 50, the average distance between the mobile robots of the method of the present invention and the capture target is about 3.5 m, while that of the comparison model is about 5.5 m; by time step 170, this average distance has approached the expected distance, and the comparison model is about 4 m.

[0132] In terms of angle control, the present invention also performs excellently. Figure 6 And Figure 8 the comparison data fully demonstrate its superiority. In Scenario 1, Figure 6 it shows that the capture included angle of the method of the present invention gradually stabilizes at about 2.09 rad after time step 100, which is extremely close to the expected average capture included angle while the average capture included angle of the comparison model reaches the expected value at about time step 260. In Scenario 2, Figure 8 it shows that after time step 20, the capture included angle of the mobile robots under the method of the present invention can stabilize at the expected value and has extremely small fluctuations, meeting the preset included angle conditions; while the comparison model does not reach the expected capture included angle until about 180 steps.

[0133] In summary, the multi-target capture model combining the density field and deep reinforcement learning proposed by the present invention shows obvious numerical advantages in the two key capture indicators of distance and angle, greatly improving the efficiency and quality of multi-target capture.

[0134] It should be noted that the above content only illustrates the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. For those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements all fall within the protection scope of the claims of the present invention.

Claims

1. A mobile robot multi-target capture method based on density field and deep reinforcement learning, characterized by: The steps include: S1. Acquisition of perception information: Acquisition of the relative position and relative speed information of the mobile robot, teammate robots in the environment, and all targets to be captured at the current moment; S2, target density calculation: according to the relative position information obtained in step S1, the density values ​​of all the targets to be captured in the environment are calculated; S3, target allocation decision: According to the relative positions of the mobile robot and all the targets to be captured obtained in step S1 and the density of the targets to be captured calculated in step S2, the selection tendency score of each mobile robot for each target is calculated, and the capture target P at that moment is obtained based on the target allocation algorithm. i and the set of collaborative robots in the group N i ; S4, capture state judgment: judge whether the mobile robot and the collaborative robots in the group meet the preset capture target. If so, end the capture task; otherwise, continue to execute step S5; whether the preset capture target is met is specifically: the robot and its collaborative robot set N in the group i For their common target P i , whether the preset angle condition and distance condition are met; S5, observation state generation: according to the capture target P obtained in step S3 i and the set of collaborative robots in the group N i , extract the observation state from the relative position and relative velocity information obtained in step S1 The observed state The number of elements is 4*(1+|N i |) vector; S6, action decision and execution: The observation state extracted in step S5 Input into the pre-trained policy network to get the current observation state Next action After parsing the action command, control the mobile robot i to perform the action S7, round-up process iteration: the mobile robot completes the action After that, return to step S1 and continue to execute the next time step process until the round-up task in step S4 is completed.

2. The mobile robot multi-target capture method based on density field and deep reinforcement learning as claimed in claim 1, characterized in that: The density value of the target k to be captured in step S2 The specific calculation method is: in, represents the relative position vector of the target k to be captured and the mobile robot i; N represents the number of all mobile robots in the environment; x k represents the position vector of the target k to be captured in space; ||||2 is the vector two-norm operation; W() function is the following piecewise function: Wherein, R is the preset distance threshold.

3. The mobile robot multi-target capture method based on density field and deep reinforcement learning as claimed in claim 2, characterized in that: In step S3, the propensity score is selected The specific calculation method is: Among them, β1 and β2 are the control density values and relative distance The importance of constant value.

4. The mobile robot multi-target capture method based on density field and deep reinforcement learning as claimed in claim 3, characterized in that: In step S3, the target allocation algorithm adopts one of the Hungarian algorithm, auction algorithm, genetic algorithm or swarm algorithm, and the rule of the target allocation algorithm is: (1) Randomly sort all mobile robot numbers; (2) According to the order in rule (1), the mobile robots in the sequence are assigned their selection tendency scores in turn. The highest target, if the target k with the highest score has been assigned a preset number of mobile robots, then the target corresponding to the second highest score is selected, and so on until the target is assigned; (3) This process continues until all mobile robots are assigned a capture target.

5. The mobile robot multi-target capture method based on density field and deep reinforcement learning as claimed in claim 4, characterized in that: In step S4, the robot and its collaborative robot set N i For the target they jointly capture, whether the preset angle conditions and distance conditions are met is as follows: and in, and are the distance between the mobile robots r1 and r2 and the angle relative to the capture target; dis safe The safe distance that needs to be maintained between mobile robots; is the distance between the mobile robot r1 and the capture target.

6. The mobile robot multi-target capture method based on density field and deep reinforcement learning as claimed in claim 1, characterized in that: In step S5, the observed state of the agent Specifically: in, and is the mobile robot i and the target P i The relative position and relative velocity of and is the relative position and relative speed between the mobile robot i and the collaborative robot j in the same group.

7. The mobile robot multi-target capture method based on density field and deep reinforcement learning as claimed in claim 1, characterized in that: During the iteration of the round-up process in step S7, at time t, the motion logic of the mobile robot i is: in, is the position coordinate of mobile robot i at time t, is the speed of mobile robot i at time t, The acceleration of mobile robot i at time t, Δt is the time unit.

8. A mobile robot multi-target capture system based on density field and deep reinforcement learning, characterized by: It includes perception module, density field calculation module, target allocation module, capture judgment module, strategy network module and motion control module. The perception module: establishes data connections with the density field calculation module, the target allocation module and the strategy network module respectively, and includes a position and speed perception unit and a data preprocessing unit; wherein the position and speed perception unit is used to collect the relative position and relative speed information of the robot and the teammate robots and all the targets to be captured in the environment, and the data preprocessing unit is used to process and optimize the collected data; The density field calculation module includes a space division unit and a density calculation unit, which is used to calculate the density values ​​of all targets to be captured in the environment; The target allocation module: establishes data connections with the density field calculation module, the capture judgment module and the strategy network module respectively, and includes a score calculation unit and an allocation algorithm execution unit; wherein the score calculation unit is used to calculate the tendency score of the robot to select the target to be captured, and the allocation algorithm execution unit executes the target allocation algorithm according to the output score of the score calculation unit to determine the capture target of each robot and the set of collaborative robots in the group; The capture judgment module: establishes data connections with the strategy network module and the motion control module respectively, and includes a condition setting unit and a judgment execution unit; wherein the condition setting unit is used to preset the capture target condition; the judgment execution unit judges whether the capture target is met according to the condition of the condition setting unit; The strategy network module: establishes a data connection with the motion control module, and includes an observation state generation unit, a strategy network training unit and an action decision unit; wherein the observation state generation unit is used to extract the observation state as the input of the strategy network training unit; the action decision unit inputs the observation state into the pre-trained strategy network to obtain the action of the robot under the current observation state; The motion control module includes an instruction parsing unit and a drive execution unit, which are used to parse action instructions and control the robot to execute actions.

Images