A multi-robot cooperative exploration method and device, electronic equipment and medium

Abstract

The application discloses a kind of multi-robot cooperative exploration method, device, electronic equipment and medium, including obtaining the exploration information of multiple robots in current environment;Based on behavior network, the behavior corresponding to each robot is predicted according to the exploration information of robot;Multiple robots are divided into regions, and the target point set corresponding to each robot is obtained;Based on the behavior corresponding to each robot, the moving target point is selected from the target point set according to evaluation network;Path planning is carried out according to the moving target point, and the moving path corresponding to robot is obtained;According to the moving path, robot is controlled to move to moving target point.This application accurately segments the detection environment of each robot to reduce the possibility of repeated exploration, and can maximize the exploration range of robot, so that each robot has its own network, so as to make decisions in a distributed execution manner, explore as much unknown environment as possible in a shorter time, and improve the exploration efficiency.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the field of intelligent control technology, and particularly relates to a multi-robot cooperative exploration method and device, an electronic device and a medium. BACKGROUND

[0002] With the rapid development of automation and computer technologies, the research on intelligent robots exploring unknown environments becomes more and more important. Some recent industrial and scientific scenarios, such as Mars landing, cave rescue, power grid planning, etc., further stimulate the demand for autonomous exploration. Compared with single-robot exploration, multi-robot cooperative exploration improves the efficiency and robustness of task execution through information sharing, and can also complete some tasks that single robots cannot complete. Therefore, multi-robot cooperative exploration has attracted widespread attention and research from researchers.

[0003] At present, the existing multi-robot cooperative exploration method often adopts a centralized control mode, among which the contract net model is most commonly used. However, the contract net model uses the bid evaluation by the tenderer, which is easy to cause system bottleneck. Moreover, the coordination strategy calculation of the multi-robot system cooperative work is complex, and the hardware configuration requirement of the central processing unit is high, and the software structure is complex, which increases the cost of the multi-robot system. In addition, the few existing strategies for realizing the distribution of multi-robots in the environment are local and decentralized, resulting in low exploration efficiency. SUMMARY

[0004] The present application aims to provide a multi-robot cooperative exploration method, device, electronic device and medium, which fully mobilizes each robot to explore cooperatively while improving the exploration efficiency.

[0005] In a first aspect, the present application provides a multi-robot cooperative exploration method, comprising: obtaining exploration information of a plurality of robots in a current environment; predicting behaviors corresponding to each robot based on a behavior network according to the exploration information of the robots; dividing the plurality of robots into regions to obtain a target point set corresponding to each robot; selecting a moving target point from the target point set according to an evaluation network based on the behaviors corresponding to each robot; performing path planning according to the moving target point to obtain a moving path corresponding to the robot; and controlling the robot to move to the moving target point according to the moving path to complete a cooperative exploration task of all robots in the current environment.

[0006] In the present application, the exploration information of multiple robots in the current environment is first acquired; then, based on a behavior network, the behaviors corresponding to the robots are predicted according to the exploration information of the robots; then, the multiple robots are regionally divided to obtain a target point set corresponding to each robot; then, based on the behaviors corresponding to each robot, a moving target point is selected from the target point set according to an evaluation network; then, path planning is performed according to the moving target point to obtain a moving path corresponding to the robot; then, the robot is controlled to move to the moving target point according to the moving path, and the collaborative exploration task of all robots in the current environment is completed. The present application precisely segments the detected environment of each robot to reduce the possibility of repeated exploration, and can maximize the exploration range of the robot. The present application enables each robot to have its own network, thereby making decisions in a distributed execution manner, enabling as much unknown environment as possible to be explored in a short time, and relaxing the requirement for the structure of the environment.

[0007] In an implementation form of the first aspect, the exploration information of the multiple robots in the current environment is acquired, including: acquiring the current position and the last moving position of any one of the multiple robots, and the current positions of other robots in the multiple robots; acquiring the relative position between the current position and the last moving position of any one of the robots; and acquiring the relative positions between the current position of any one of the robots and the current positions of other robots.

[0008] In the present application, the exploration information of the multiple robots can include the position information of the multiple robots, wherein the position information includes the current position and the last moving position of any one of the multiple robots, the relative position between the current position and the last moving position of any one of the robots, and the relative positions between the current position of any one of the robots and the current positions of other robots.

[0009] In an implementation form of the first aspect, based on the behavior network, the behaviors corresponding to the robots are predicted according to the exploration information of the robots, including: inputting the exploration information of any one of the robots into the behavior network; extracting the behavior features corresponding to the exploration information through a plurality of first fully connected layers and a first gated recurrent unit layer in the behavior network; and outputting the behavior corresponding to any one of the robots through a first activation function output layer in the behavior network.

[0010] In the present application, the behavior network includes a first input layer, a plurality of first fully connected layers, and a first activation function output layer, the exploration information of any one of the robots is input into the behavior network as the first input layer of the behavior network; then the behavior features corresponding to the exploration information are extracted through a plurality of first fully connected layers and a first gated recurrent unit layer in the behavior network; and then the behavior corresponding to any one of the robots is output through a first activation function output layer in the behavior network.

[0011] In an implementation form of the first aspect, the area partitioning of the plurality of robots to obtain the target point set corresponding to each robot comprises: connecting all adjacent robots in the plurality of robots into a triangle, and drawing a perpendicular bisector of each side of the triangle; enclosing a polygon by a plurality of the perpendicular bisectors around each robot to obtain a polygon area map corresponding to the plurality of robots; and obtaining the target point set corresponding to each robot according to the polygon area map.

[0012] In the present application, when the area partitioning of the plurality of robots is performed, all adjacent robots in the plurality of robots are first connected into a triangle, and a perpendicular bisector of each side of the triangle is drawn; then a polygon is enclosed by a plurality of the perpendicular bisectors around each robot to obtain a polygon area map corresponding to the plurality of robots; and the target point set corresponding to each robot is obtained according to the polygon area map. The polygon area map includes a plurality of polygons, each polygon contains only one robot, the points in the polygon area are closest to the corresponding robot, and the points on the edges of the polygon are equidistant to the two adjacent robots.

[0013] In an implementation form of the first aspect, the moving target point is selected from the target point set according to the behavior of each robot based on the evaluation network, comprising: inputting the behavior of any one robot, the target point set and the exploration information into the evaluation network; extracting the target point feature of any one robot through a plurality of second fully connected layers and a second gated recurrent unit layer in the evaluation network; and outputting the moving target point of any one robot through a first activation function output layer in the behavior network.

[0014] In the present application, the evaluation network includes a second input layer, a plurality of second fully connected layers and a second gated recurrent unit layer. The behavior of any one robot, the target point set and the exploration information can be input into the evaluation network. Then the target point feature of any one robot is extracted through a plurality of second fully connected layers and a second gated recurrent unit layer in the evaluation network. Then the moving target point of any one robot is output through a first activation function output layer in the behavior network.

[0015] In an implementation form of the first aspect, the path planning is performed according to the moving target point to obtain the moving path of the robot, comprising: taking the current position of any one robot as a planning initial point, and taking the moving target point of any one robot as a planning target point, and completing the path planning of any one robot by using a heuristic search algorithm.

[0016] In the application, the feasible trajectory from the current position of the robot to the moving target point can be determined by the heuristic search algorithm, the application can independently design a path for each robot by using the heuristic search algorithm to facilitate the obstacle avoidance walking of each robot, thereby improving the efficiency of the multi-robot cooperative exploration.

[0017] In an implementation form of the first aspect, the method further includes: obtaining the reward value of any one robot and the exploration information after any one robot moves to the moving target point when the robot moves to the moving target point according to the moving path; and calculating the target value function value according to the reward value of any one robot and the exploration information after any one robot moves to the moving target point.

[0018] The loss value is calculated according to the target value function value, and the loss value is used to update the parameters of the evaluation network.

[0019] In the application, the reward value of any one robot and the exploration information after any one robot moves to the moving target point can be obtained when the robot moves to the moving target point according to the moving path, and then the target value function value is calculated according to the reward value of any one robot and the exploration information after any one robot moves to the moving target point, and then the loss value is calculated according to the target value function value, and the loss value is used to update the parameters of the evaluation network. The network precision is improved, and the efficiency of the multi-robot cooperative exploration is also improved.

[0020] In a second aspect, the application provides a multi-robot cooperative exploration device, including: an information acquisition module, configured to acquire exploration information of a plurality of robots in a current environment; a behavior prediction module, configured to predict behaviors corresponding to the robots based on a behavior network according to the exploration information of the robots; a region division module, configured to divide regions of the plurality of robots to obtain a target point set corresponding to each robot; a target point selection module, configured to select a moving target point from the target point set according to an evaluation network based on the behaviors corresponding to the robots; a path planning module, configured to plan a moving path corresponding to the robot according to the moving target point; and a target point execution module, configured to control the robot to move to the moving target point according to the moving path, and complete a cooperative exploration task of all the robots in the current environment.

[0021] In a third aspect, the application provides an electronic device, including: a memory, the memory storing a plurality of instructions; and a processor, the processor loading the instructions from the memory to execute the steps in any one of the multi-robot cooperative exploration methods provided in the embodiments of the application.

[0022] In a fourth aspect, the application provides a computer-readable storage medium, which stores a computer program, and the program is executed by a multi-robot cooperative exploration device to implement the steps in any one of the multi-robot cooperative exploration methods provided in the embodiments of the application.

[0023] The application first acquires exploration information of multiple robots in a current environment; then predicts behaviors corresponding to the robots based on a behavior network according to the exploration information of the robots; then performs regional division on the multiple robots to obtain a target point set corresponding to each robot; then selects a moving target point from the target point set according to an evaluation network based on the behaviors corresponding to the robots; then performs path planning according to the moving target point to obtain a moving path corresponding to the robot; then controls the robot to move to the moving target point according to the moving path, and completes a cooperative exploration task of all robots in the current environment. The application precisely segments the detected environment of each robot to reduce the possibility of repeated exploration, and can maximize the exploration range of the robot.

[0024] The application enables each robot to have its own network, while stimulating the robot to explore in an unknown environment, reducing the possibility of redundant exploration, making decisions in a distributed execution manner, exploring as much unknown environment as possible in a shorter time, and relaxing the requirements for the structure of the environment. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The application is an application scenario diagram of a multi-robot cooperative exploration method of an embodiment of the application.

[0026] Figure 2 The application is a flowchart of a multi-robot cooperative exploration method of an embodiment of the application.

[0027] Figure 3 The application is an update flowchart of a network of an embodiment of the application.

[0028] Figure 4 The application is a simulation training and test environment diagram of an embodiment of the application.

[0029] Figure 5 The application is a training curve comparison diagram of a multi-robot cooperative exploration method of an embodiment of the application and a baseline MADDPG.

[0030] Figure 6 The application is a structural diagram of a multi-robot cooperative exploration device of an embodiment of the application.

[0031] Figure 7 The application is a structural diagram of an electronic device of an embodiment of the application. DETAILED DESCRIPTION

[0032] Following, the implementation of the present application is described through specific examples, and other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the present specification. The present application can also be implemented or applied through other different specific implementations, and various modifications or changes can be made to the details in the present specification based on different views and applications without departing from the spirit of the present application. It should be noted that the following examples and features in the examples can be combined with each other without conflict.

[0033] It should be noted that the diagrams provided in the following examples only illustrate the basic concept of the present application in a schematic manner, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components when actually implemented. The actual implementation of each component may be a random change in type, number and proportion, and the layout pattern of the components may also be more complex.

[0034] The cooperative control technology refers to that a plurality of robots successfully complete a series of tasks without the participation of human control, and realize the autonomous control of the entire multi-robot system. The following examples of the present application provide a multi-robot cooperative exploration method, device, electronic equipment and medium, wherein the multi-robot cooperative exploration device can be integrated in an electronic equipment, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, or a personal computer (PC), etc. The server can be a single server, or a server cluster composed of multiple servers.

[0035] In some examples, the multi-robot cooperative exploration device can also be integrated in multiple electronic equipment, for example, the multi-robot cooperative exploration device can be integrated in multiple servers, and the multiple servers can realize the multi-robot cooperative exploration method of the present application.

[0036] In some examples, the server can also be realized in the form of a terminal.

[0037] For example, referring to Figure 1 The electronic equipment can include a plurality of robots 10, a storage terminal 11, a server 12, etc. The plurality of robots 10 includes a plurality of intelligent robots, the storage terminal 11 is used to store the exploration information of the plurality of robots, and the plurality of robots 10, the storage terminal 11 and the server 12 are connected with each other for communication, which will not be described here.

[0038] The server 12 can include a processor, a memory, and the like. The server 12 can obtain exploration information of a plurality of robots in a current environment; predict behaviors corresponding to the robots based on a behavior network according to the exploration information of the robots; divide the plurality of robots into regions to obtain a target point set corresponding to each robot; select a moving target point from the target point set according to an evaluation network based on the behaviors corresponding to the robots; perform path planning according to the moving target point to obtain a moving path corresponding to the robot; control the robot to move to the moving target point according to the moving path, and complete a cooperative exploration task of all robots in the current environment, and the like.

[0039] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application.

[0040] As shown in Figure 2 Taking the server 12 as an execution subject, the embodiment provides a multi-robot cooperative exploration method, including steps S210 to S260, as follows:

[0041] S210, the server 12 obtains exploration information of a plurality of robots in a current environment.

[0042] In an embodiment, obtaining the exploration information of the plurality of robots in the current environment includes: obtaining a current position and a last moving position of any one of the plurality of robots, and current positions of other robots of the plurality of robots; obtaining a relative position between the current position and the last moving position of any one of the plurality of robots; and obtaining relative positions between the current position of any one of the plurality of robots and current positions of other robots of the plurality of robots.

[0043] In the embodiment, the exploration information of the plurality of robots can include position information of the plurality of robots. The position information refers to information related to the position of the robot, for example, a geographical position of the robot, or a relative position between the robot and other communication nodes, and the like. For example, the position information can include a current position and a last moving position of any one of the plurality of robots, a relative position between the current position and the last moving position of any one of the plurality of robots, and relative positions between the current position of any one of the plurality of robots and current positions of other robots of the plurality of robots.

[0044] In addition, the exploration information of the plurality of robots in the embodiment can also include environment information and state information of the plurality of robots. The environment information refers to information related to the environment around the robot, for example, a terrain, a topography, an obstacle, and the like around the robot; and the state information refers to information related to the state of the robot itself, for example, attributes, power, orientation, control information, and the like of the robot itself.

[0045] S220, the server 12 predicts the behavior corresponding to each robot according to the exploration information of the robot based on the behavior network.

[0046] In an embodiment, the behavior corresponding to each robot is predicted according to the exploration information of the robot based on the behavior network, including: inputting the exploration information of any one robot into the behavior network; extracting the behavior features corresponding to the exploration information through a plurality of first fully connected layers and a first gated recurrent unit layer in the behavior network; and outputting the behavior corresponding to any one robot through a first activation function output layer in the behavior network.

[0047] In this embodiment, each robot includes its own behavior network and evaluation network. The behavior network is a policy network of the robot, which is used to output decision behavior; the evaluation network is used to evaluate and adjust the parameters of the behavior network, thereby updating the parameters of the behavior network.

[0048] The behavior network can include a first input layer, a plurality of first fully connected layers, and a first activation function output layer. The exploration information of any one robot is input into the behavior network as the first input layer of the behavior network; the behavior features corresponding to the exploration information are extracted through a plurality of first fully connected layers and a first gated recurrent unit layer in the behavior network; and the behavior corresponding to any one robot is output through a first activation function output layer in the behavior network. In this embodiment, in order to introduce memory characteristics, the behavior network can be an RNN (Recurrent Neural Network) or an LSTM (Long short-term memory) network.

[0049] S230, the server 12 divides the area for a plurality of robots to obtain a target point set corresponding to each robot.

[0050] In an embodiment, the area is divided for a plurality of robots to obtain a target point set corresponding to each robot, including: connecting all adjacent robots in the plurality of robots into a triangle, and drawing a perpendicular bisector of each side of the triangle; enclosing a polygon by a plurality of perpendicular bisectors around each robot to obtain a polygon area graph corresponding to the plurality of robots; and obtaining a target point set corresponding to each robot according to the polygon area graph.

[0051] In the embodiment, when the multiple robots are divided into regions, all adjacent robots in the multiple robots are connected into a triangle, and a vertical bisector of each side of the triangle is drawn; then a plurality of vertical bisectors around each robot are enclosed into a polygon to obtain a polygon region map corresponding to the multiple robots; and then a target point set corresponding to each robot is obtained according to the polygon region map. The polygon region map includes a plurality of polygons, each polygon contains only one robot, the points in the polygon region are closest to the corresponding robot, and the points on the edges of the polygon are equidistant to the two adjacent robots.

[0052] In S240, the server 12 selects a moving target point from the target point set according to the evaluation network based on the behavior corresponding to each robot.

[0053] In an embodiment, the selecting of the moving target point from the target point set according to the evaluation network based on the behavior corresponding to each robot includes: inputting the behavior corresponding to any one robot, the target point set and the exploration information into the evaluation network; extracting the target point feature corresponding to any one robot through a plurality of second full connection layers and a second gated recurrent unit layer in the evaluation network; and outputting the moving target point corresponding to any one robot through a first activation function output layer in the behavior network.

[0054] In the embodiment, the evaluation network includes a second input layer, a plurality of second full connection layers and a second gated recurrent unit layer, the application can input the behavior corresponding to any one robot, the target point set and the exploration information into the evaluation network; then extract the target point feature corresponding to any one robot through the plurality of second full connection layers and the second gated recurrent unit layer in the evaluation network; and then output the moving target point corresponding to any one robot through the first activation function output layer in the behavior network.

[0055] In the embodiment, in order to introduce the memory feature, the evaluation network can be an RNN (Recurrent Neural Network) or an LSTM (Long short-term memory) network.

[0056] In S250, the server 12 plans a path according to the moving target point to obtain a moving path corresponding to the robot.

[0057] In an embodiment, the planning of the path according to the moving target point to obtain the moving path corresponding to the robot includes: taking the current position of any one robot as a planning initial point and the moving target point of any one robot as a planning target point, and using a heuristic search algorithm to complete the path planning of any one robot.

[0058] In the embodiment, the feasible trajectory from the current position of the robot to the moving target point can be determined by the heuristic search algorithm, the application can use the heuristic search algorithm to independently design a path for each robot to facilitate the obstacle avoidance walking of each robot, thereby improving the efficiency of the multi-robot cooperative exploration. The heuristic search algorithm can be A-Star algorithm.

[0059] Specifically, the search area of the robot is simplified as a set of quantifiable nodes, starting from the current position of the robot as the starting point, first traversing the adjacent nodes around the starting point, then traversing the adjacent nodes of the already traversed nodes, and gradually spreading outward until the end point is found, i.e., the corresponding moving target point of the robot is found.

[0060] S260, the server 12 controls the robot to move to the moving target point according to the moving path, and completes the cooperative exploration task of all robots in the current environment.

[0061] In the embodiment, when the robot is controlled to move to the moving target point according to the moving path, the reward value of any one robot and the exploration information after any one robot moves to the moving target point can be obtained; then the target value function value is calculated according to the reward value of any one robot and the exploration information after any one robot moves to the moving target point; and the loss value is calculated according to the target value function value, and the loss value is used to update the parameters of the evaluation network. The network precision is improved, and the efficiency of the multi-robot cooperative exploration is also improved.

[0062] In an embodiment, it also includes: when the robot is controlled to move to the moving target point according to the moving path, the reward value of any one robot and the exploration information after any one robot moves to the moving target point are obtained; the target value function value is calculated according to the reward value of any one robot and the exploration information after any one robot moves to the moving target point;

[0063] The loss value is calculated according to the target value function value, and the loss value is used to update the parameters of the evaluation network.

[0064] In this embodiment, the exploration information of multiple robots in the current environment is first obtained; then, based on the behavior network, the behaviors corresponding to each robot are predicted according to the exploration information of the robots; then, the multiple robots are divided into regions to obtain a target point set corresponding to each robot; then, based on the behaviors corresponding to each robot, a moving target point is selected from the target point set according to the evaluation network; then, path planning is performed according to the moving target point to obtain a moving path corresponding to the robot; then, the robot is controlled to move to the moving target point according to the moving path, and the collaborative exploration task of all robots in the current environment is completed. The detection environment of each robot is precisely segmented in this application to reduce the possibility of repeated exploration, and the exploration range of the robot is maximized. Each robot has its own network in this application, so that decisions are made in a distributed execution manner, and as much unknown environment as possible can be explored in a shorter time, while the requirements for the environment structure are relaxed.

[0065] The intelligent robots in the distributed structure are completely autonomous and are in an equal position with each other, without primary and secondary division. The intelligent robots interact information with each other through communication means and make decisions independently using local information. The distributed structure improves the stability and flexibility of the system and alleviates the bottleneck problem existing in the control aspect. Considering the flexibility, robustness and robustness of the entire multi-robot collaborative system, the distributed method is adopted in this embodiment to realize the collaborative control among the multiple robots, and the optimization of the collaborative control index is realized.

[0066] In this embodiment, the detection environment of each robot is precisely segmented by the polygon partitioning and deep reinforcement learning method to reduce the possibility of repeated exploration and maximize the exploration range of each robot. In this embodiment, the next ideal position point of each robot, i.e., the moving target point, can also be obtained through centralized training, and a path is independently designed for each robot using a classic heuristic search algorithm to facilitate the obstacle avoidance walking of each robot. Finally, all robots cooperatively complete the exploration task, such as cooperatively completing the construction of the map of the current environment.

[0067] This embodiment also relates to the update of the behavior network and the evaluation network, as shown in Figure 3 The specific steps include the following steps: S1, initializing the behavior network, the evaluation network and the target network corresponding to each robot, and the experience replay pool R.

[0068] In this embodiment, each robot contains its own behavior network and evaluation network. The behavior network is the policy network of the robot, which is used to output the decision behavior; the evaluation network is used to evaluate and adjust the parameters of the behavior network, so as to update the parameters of the behavior network. The experience replay pool R is used to store the experience values obtained by each robot interacting with the environment in each state. The behavior network and the evaluation network of this embodiment each have a respective target network, i.e., the target network corresponding to each robot, which provides periodic updates and iterations of model parameters for the corresponding behavior network or evaluation network.

[0069] S2, the environment is reset, and the Gaussian random process of behavior detection is initialized.

[0070] S3, the initial state s and the observation information o of each robot are received.

[0071] Specifically, the input of the neural network is composed of a concatenated vector of rangefinder data (48-dimensional vector), its relative position with respect to the previous vector (two-dimensional vector), and the relative position with respect to other mobile robots (4-dimensional vector). The input layer is first connected with three fully connected (FC) layers, each containing 256 nodes, and then a gated recurrent unit (GRU) layer containing 256 nodes. The behavior network outputs an action through a sigmoid function, i.e., the corresponding behavior of the robot.

[0072] In this embodiment, all actions of the robot and its observation results are used as inputs of the evaluation network, which are then processed by three fully connected layers and a gated recurrent unit layer. The evaluation network outputs the Q value through a linear activation function, i.e., taking the state S and the action A as inputs, while outputting the Q' value, which refers to the possible reward of executing the action a in the state s.

[0073] S4, the behavior a in the current state is selected according to the policy network π, and the set of selectable target points is calculated through Voronoi partitioning.

[0074] In this embodiment, the set of selectable target points is calculated through Voronoi partitioning, and the set of target points is where Q is the movable area of the robot, p i ,p j are the positions of robots i and j, respectively.

[0075] S5, map the behavior to the target point, and perform path planning through the A * algorithm.

[0076] S6, obtain the reward value r, the next observation state o', and the next state s', and store them in the experience replay pool R.

[0077] In this embodiment, the reward value r can include a map construction reward, a task completion reward, and a penalty part, etc. After the robot interacts with the environment, the server 12 can return a reward signal, i.e., a reward value. The observation state is the information observed by the robot itself, such as position information, etc.; and the state is the state information of the robot itself, etc.

[0078] S7, repeating steps S4 to S6 until all robot corresponding behavior networks are traversed.

[0079] S8, sampling some samples from the experience replay pool R.

[0080] S9, calculating the target value function, and updating the evaluation network by minimizing the loss value.

[0081] In this embodiment, when calculating the target value function, the following is calculated for the zth robot: y = r z + γmin c Q z,c .

[0082] Wherein, y is the target value, Q z,c is the estimated value, r z is the reward value of the zth robot, γ is the discount factor, γ is in the range of 0 to 1, and c = 1, 2 respectively represent the behavior network of the robot and its target network, and the smaller value is selected.

[0083] In this embodiment, the evaluation network is updated by minimizing the loss value, and the loss value is the difference between the target value and the estimated value. Wherein, B is the total number of samples.

[0084] S10, repeating steps S8 to S9 until all evaluation networks are traversed;

[0085] S11, if the iteration number meets a certain interval step, the behavior network is updated using the policy gradient method, and the target network of each robot is updated, otherwise step S13 is executed;

[0086] In this embodiment, the behavior network is updated using the policy gradient method, and the target network of each robot is updated. This embodiment adopts a delayed policy update method, i.e., the update frequency of the behavior network is slower than that of the evaluation network. The behavior network aims to maximize the value of the evaluation network and update its weight by using the sampled policy gradient.

[0087] S12, repeating steps S4 to S11 until T times have been cycled, and T is greater than 0.

[0088] S13, repeating steps S2 to S12 until a certain number of iterations is met.

[0089] The embodiment first builds various 3D simulation environments containing static obstacles based on the ROS platform using Gazebo. Each mobile robot is equipped with a 360-degree laser radar for detecting optional target points and mapping in an unknown environment. It has been proved that this platform can reduce the differences between the simulation environment and the real world as much as possible. Figure 4 The simulation training and test environment.

[0090] The input of the neural network is composed of a concatenated vector of rangefinder data (a 48-dimensional vector), its relative position with respect to the previous vector (a two-dimensional vector), and the relative position with other mobile robots (a four-dimensional vector). The input layer is first connected with three fully connected layers, each containing 256 nodes, and then a gated recurrent unit (GRU) layer containing 256 nodes. The actor network finally outputs an action through a sigmoid function. All actions of the robot and its observations are used as input to evaluate the network, which is then processed by three FC layers and a GRU layer, and outputs Q values through a linear activation function. The multi-robot collaborative exploration method proposed in the application is compared with the baseline MADDPG (Multi-agent Deep Deterministic Policy Gradient). The two methods have the same observation space, action space and reward space, but MADDPG may have more exploration in the early stage, while the exploration degree of the algorithm proposed in the application will gradually increase and surpass over time.

[0091] The training curves of the method proposed in the application and the baseline MADDPG are shown in Figure 5 It can be seen that the return values of the two methods are constantly increasing and gradually converging, and the method proposed in the application learns faster, more stably, and converges to a higher value than the baseline MADDPG.

[0092] The protection scope of the multi-robot collaborative exploration method of the embodiment of the application is not limited to the order of steps listed in the embodiment. Any scheme realized by adding, replacing or modifying steps of the prior art according to the principle of the application is included in the protection scope of the application.

[0093] The embodiment of the application also provides a multi-robot collaborative exploration device, which can realize the multi-robot collaborative exploration method of the application. However, the implementation device of the multi-robot collaborative exploration method of the application includes but is not limited to the structure of the multi-robot collaborative exploration device listed in the embodiment. Any modification and replacement of the structure of the prior art according to the principle of the application is included in the protection scope of the application.

[0094] As Figure 6As shown, the application provides a multi-robot cooperative exploration device, comprising: an information acquisition module 310, configured to acquire exploration information of a plurality of robots in a current environment; a behavior prediction module 320, configured to predict behaviors corresponding to each robot based on a behavior network according to the exploration information of the robots; a region division module 330, configured to divide the plurality of robots into regions to obtain a target point set corresponding to each robot; a target point selection module 340, configured to select a moving target point from the target point set according to an evaluation network based on the behaviors corresponding to each robot; a path planning module 350, configured to plan a path according to the moving target point to obtain a moving path corresponding to the robot; and a target point execution module 360, configured to control the robot to move to the moving target point according to the moving path to complete a cooperative exploration task of all robots in the current environment.

[0095] In this embodiment, the exploration information of a plurality of robots in a current environment is first acquired; then behaviors corresponding to each robot are predicted based on a behavior network according to the exploration information of the robots; the plurality of robots are then divided into regions to obtain a target point set corresponding to each robot; a moving target point is then selected from the target point set according to an evaluation network based on the behaviors corresponding to each robot; a path is then planned according to the moving target point to obtain a moving path corresponding to the robot; and the robot is then controlled to move to the moving target point according to the moving path to complete a cooperative exploration task of all robots in the current environment. The application precisely segments the detected environment of each robot to reduce the possibility of repeated exploration and maximize the exploration range of the robot. The application enables each robot to have its own network to make decisions in a distributed execution manner, enabling the robot to explore as much unknown environment as possible in a short time while relaxing the requirements for the structure of the environment.

[0096] In an embodiment, the information acquisition module 310 comprises a position information acquisition module configured to: acquire the current position and the last moving position of any one of the plurality of robots, and the current positions of other robots; acquire the relative position between the current position and the last moving position of any one of the robots; and acquire the relative positions between the current position of any one of the robots and the current positions of other robots.

[0097] In this embodiment, the exploration information of the plurality of robots can comprise position information of the plurality of robots, wherein the position information comprises the current position and the last moving position of any one of the plurality of robots, the relative position between the current position and the last moving position of any one of the robots, and the relative positions between the current position of any one of the robots and the current positions of other robots.

[0098] In an embodiment, the behavior prediction module 320 comprises a behavior prediction submodule configured to: input the detection information of any one robot into a behavior network; extract the behavior features corresponding to the detection information through a plurality of first fully connected layers and a first gated recurrent unit layer in the behavior network; and output the behavior corresponding to any one robot through a first activation function output layer in the behavior network.

[0099] In this embodiment, the behavior network comprises a first input layer, a plurality of first fully connected layers, and a first activation function output layer. The detection information of any one robot is input into the behavior network as the first input layer of the behavior network. The behavior features corresponding to the detection information are extracted through a plurality of first fully connected layers and a first gated recurrent unit layer in the behavior network. The behavior corresponding to any one robot is output through a first activation function output layer in the behavior network.

[0100] In an embodiment, the region division module 330 comprises a region division submodule configured to: connect all adjacent robots in the plurality of robots into triangles, and draw perpendicular bisectors of each side of the triangles; enclose a plurality of polygons around a plurality of vertical bisectors around each robot to obtain a polygon region map corresponding to the plurality of robots; and obtain a target point set corresponding to each robot according to the polygon region map.

[0101] In this embodiment, when the plurality of robots are regionally divided, all adjacent robots in the plurality of robots are first connected into triangles, and perpendicular bisectors of each side of the triangles are drawn. A plurality of polygons are then enclosed around a plurality of vertical bisectors around each robot to obtain a polygon region map corresponding to the plurality of robots. A target point set corresponding to each robot is then obtained according to the polygon region map. The polygon region map comprises a plurality of polygons, each polygon containing only one robot. The points in the polygon region are closest to the corresponding robot, and the points on the edges of the polygon are equidistant from the two robots on the edges.

[0102] In an embodiment, the target point selection module 340 comprises a target point selection submodule configured to: input the behavior corresponding to any one robot, the target point set, and the exploration information into an evaluation network; extract the target point features corresponding to any one robot through a plurality of second fully connected layers and a second gated recurrent unit layer in the evaluation network; and output the movement target point corresponding to any one robot through a first activation function output layer in the behavior network.

[0103] In this embodiment, the evaluation network comprises a second input layer, a plurality of second fully connected layers, and a second gated recurrent unit layer, the application can input the behavior, target point set and exploration information corresponding to any robot into the evaluation network; then the target point features corresponding to any robot are extracted through the plurality of second fully connected layers and the second gated recurrent unit layer in the evaluation network; and then the moving target point corresponding to any robot is output through the first activation function output layer in the behavior network.

[0104] In an embodiment, the path planning module 350 comprises a path planning submodule, which is configured to take the current position of any robot as a planning initial point, take the moving target point of any robot as a planning target point, and complete the path planning of any robot by using a heuristic search algorithm.

[0105] In this embodiment, the feasible trajectory from the current position of the robot to the moving target point can be determined by using the heuristic search algorithm, the application can independently design a path for each robot by using the heuristic search algorithm to facilitate the obstacle avoidance walking of each robot, thereby improving the efficiency of multi-robot collaborative exploration.

[0106] In an embodiment, the multi-robot collaborative exploration device further comprises a network updating module, which is configured to: after the robot is controlled to move to the moving target point according to the moving path, obtain the reward value of any robot and the exploration information after any robot moves to the moving target point; calculate the target value function value according to the reward value of any robot and the exploration information after any robot moves to the moving target point.

[0107] The loss value is calculated according to the target value function value, and the loss value is used to update the parameters of the evaluation network.

[0108] In this embodiment, after the robot is controlled to move to the moving target point according to the moving path, the reward value of any robot and the exploration information after any robot moves to the moving target point can be obtained; then the target value function value is calculated according to the reward value of any robot and the exploration information after any robot moves to the moving target point; and then the loss value is calculated according to the target value function value, and the loss value is used to update the parameters of the evaluation network. The network precision is improved, and the efficiency of multi-robot collaborative exploration is also improved.

[0109] In specific implementation, each of the above modules can be implemented as an independent entity, or can be combined as the same or several entities, and the specific implementation of each of the above modules can be referred to the method embodiments above, which will not be described here.

[0110] From the above, the multi-robot cooperative exploration device provided in the application can accurately segment the detection environment of each robot to reduce the possibility of repeated exploration, and can maximize the exploration range of the robot. The application enables each robot to have its own network, thereby making decisions in a distributed execution manner, enabling as much unknown environment as possible to be explored in a short time, while relaxing the requirements for the structure of the environment.

[0111] In several embodiments provided in the application, it should be understood that the disclosed system, device or method can be implemented in other manners. For example, the above-described device embodiments are merely illustrative. For example, the division of the modules / units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different modules / units can be indirect couplings or communication connections through some interfaces, devices or modules / units, and can be electrical, mechanical or in other forms.

[0112] The modules / units described as separate components can or can not be physically separate, and the components shown as modules / units can or can not be physical modules, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules / units can be selected according to actual needs to achieve the purpose of the embodiments of the application. For example, the functional modules / units in each embodiment of the application can be integrated into a processing module, or each module / unit can be physically separate, or two or more modules / units can be integrated into one module / unit.

[0113] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0114] The embodiments of the application also provide an electronic device, which can be a terminal, a server, etc. The terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, etc. The server can be a single server or a server cluster composed of multiple servers, etc.

[0115] In some embodiments, the multi-robot cooperative exploration apparatus provided in the present application can also be integrated in multiple electronic devices, for example, the multi-robot cooperative exploration apparatus can be integrated in multiple servers, and the multiple servers can be used to implement the multi-robot cooperative exploration method of the present application. Alternatively, the multi-robot cooperative exploration apparatus can be integrated in multiple servers, and the multiple servers can be used to implement the multi-robot cooperative exploration method of the present application.

[0116] In the present embodiment, the electronic device of the present embodiment is taken as an example to be described in detail, for example, as shown in FIG. 1, which shows a structural schematic diagram of a server related to the embodiments of the present application, in particular: Figure 7

[0117] The server can include a processor 410 with one or more processing cores, a memory 420 with one or more computer readable storage media, a power supply 430, an input module 440, a communication module 450, and the like. Those skilled in the art can understand that the server structure shown in FIG. 1 does not constitute a limitation on the server, and can include more or fewer components than shown in the figure, or combine certain components, or different component arrangements. Among them: Figure 7

[0118] The processor 410 is the control center of the server, which connects various parts of the entire server through various interfaces and lines, executes various functions of the server and processes data by running or executing software programs and / or modules stored in the memory 420 and calling data stored in the memory 420, thereby overall monitoring the server. In some embodiments, the processor 410 can include one or more processing cores; in some embodiments, the processor 410 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 410.

[0119] The memory 420 can be used to store software programs and modules, and the processor 410 executes various functions and data processing by running the software programs and modules stored in the memory 420. The memory 420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the server, etc. In addition, the memory 420 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 420 can also include a memory controller to provide access for the processor 410 to the memory 420.​​

[0120] The server also includes a power supply 430 to supply power to the various components, which in some embodiments can be logically connected to the processor 410 through a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption management, etc. The power supply 430 can also include one or more DC or AC power sources, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, etc.

[0121] The server can also include an input module 440, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0122] The server can also include a communication module 450, which in some embodiments can include a wireless module through which the server can perform short-range wireless transmission, thereby providing the user with wireless broadband Internet access. For example, the communication module 450 can be used to help the user send and receive emails, browse web pages, and access streaming media, etc.

[0123] Although not shown, the server can also include a display unit, etc., which will not be described here. In particular, in the present embodiment, the processor 410 in the server will load one or more executable files corresponding to the processes of the application program into the memory 420 according to the following instructions, and run the application program stored in the memory 420 by the processor 410, thereby implementing various functions of the multi-robot collaborative exploration device.

[0124] The server of the present embodiment can first obtain exploration information of a plurality of robots in a current environment; then predict behaviors corresponding to the robots based on a behavior network according to the exploration information of the robots; then divide the plurality of robots into regions to obtain a set of target points corresponding to each robot; then select a moving target point from the set of target points according to an evaluation network based on the behaviors corresponding to each robot; then perform path planning according to the moving target point to obtain a moving path corresponding to the robot; then control the robot to move to the moving target point according to the moving path, thereby completing the collaborative exploration task of all robots in the current environment. The present application precisely segments the detected environment of each robot to reduce the possibility of repeated exploration, and maximizes the exploration range of the robot. The present application enables each robot to have its own network, thereby making decisions in a distributed execution manner, enabling the robot to explore as much unknown environment as possible in a shorter time, while relaxing the requirements for the structure of the environment, etc.

[0125] In some embodiments, the present application also provides a computer readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps of the method of the above-mentioned embodiments can be instructed by a program to complete by a processor, and the program can be stored in a computer readable storage medium. The storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid state disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0126] The embodiments of the present application can also provide a computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions according to the embodiments of the present application are generated. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer or data center to another website, computer or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode.

[0127] The computer program product is executed by a computer, and the computer executes the method of the above-mentioned method embodiments. The computer program product can be a software installation package, and when the above-mentioned method is needed, the computer program product can be downloaded and executed on the computer.

[0128] The description of the corresponding processes or structures of each of the above-mentioned figures has its own emphasis, and the parts not described in detail in a certain process or structure can be referred to the related description of other processes or structures.

[0129] The above-mentioned embodiments are only illustrative of the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical idea of the present application should be covered by the claims of the present application.

Claims

1. A multi-robot collaborative exploration method, characterized in that, The method includes: Obtain exploration information from multiple robots in the current environment; Based on the behavior network, the behavior of each robot is predicted according to the exploration information of the robots; The multiple robots are divided into regions to obtain a set of target points corresponding to each robot; Based on the behaviors of each robot, a moving target point is selected from the set of target points according to the evaluation network; Based on the target point, a path is planned to obtain the robot's corresponding movement path; The robot is controlled to move to the target point according to the movement path, and the above steps are repeated until the collaborative exploration task of all robots in the current environment is completed; The step of dividing the multiple robots into regions to obtain a set of target points corresponding to each robot includes: Connect all adjacent robots among the plurality of robots to form a triangle, and draw the perpendicular bisectors of each side of the triangle; A polygon is formed by several perpendicular bisectors around each robot, resulting in a polygonal region map corresponding to the multiple robots. The target point set corresponding to each robot is obtained based on the polygonal region map; The step of selecting a moving target point from the target point set based on the behavior of each robot and according to the evaluation network includes: The behavior, target point set, and exploration information corresponding to any one of the robots are input into the evaluation network; The target point features corresponding to any one robot are extracted by several second fully connected layers and a second gated recurrent unit layer in the evaluation network. The first activation function output layer in the actor network outputs the moving target point corresponding to any robot. The step of planning the path based on the target point to obtain the movement path of the robot includes: Using the current position of any robot as the initial planning point and the movement target point of any robot as the planning target point, a heuristic search algorithm is used to complete the path planning of any robot. The behavior-based network-based prediction of each robot's behavior, based on the robot's exploration information, includes: The exploration information of any one of the robots is input into the behavior network; The behavioral features corresponding to the exploration information are extracted through several first fully connected layers and one first gated recurrent unit layer in the behavioral network. The behavior corresponding to any robot is output through the output layer of the first activation function in the behavior network.

2. The multi-robot cooperative exploration method according to claim 1, characterized in that, The acquisition of exploration information from the multiple robots in the current environment includes: Obtain the current position and previous movement position of any one of the multiple robots, as well as the current position of the other robots among the multiple robots; Obtain the relative position between the current position and the previous moving position of any robot; Obtain the relative position between the current position of any one robot and the current positions of the other robots.

3. The multi-robot cooperative exploration method according to claim 1, characterized in that, The method further includes: After the robot is controlled to move to the target point according to the movement path, the reward value of any one robot and the exploration information of any one robot after moving to the target point are obtained. The target value function value is calculated based on the reward value of any one of the robots and the exploration information after any one of the robots moves to the target point. The loss value is calculated based on the target function value, and the loss value is used to update the parameters of the evaluation network.

4. A multi-robot collaborative exploration device, applicable to the multi-robot collaborative exploration method as described in any one of claims 1 to 3, characterized in that, The device includes: The information acquisition module is used to acquire exploration information from multiple robots in the current environment; The behavior prediction module is used to predict the behavior of each robot based on the robot's exploration information, using a behavior network. The region division module is used to divide the multiple robots into regions to obtain a set of target points corresponding to each robot. The target point selection module is used to select a moving target point from the target point set based on the behavior of each robot and according to the evaluation network. The path planning module is used to plan a path based on the target point to obtain the movement path of the robot. The target point execution module is used to control the robot to move to the target point according to the movement path, so as to complete the collaborative exploration task of all robots in the current environment.

5. An electronic device, characterized in that, The electronic device includes: The memory stores multiple instructions; A processor that loads instructions from the memory to perform the steps in the multi-robot cooperative exploration method as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a multi-robot collaborative exploration device, the program implements the steps of the multi-robot collaborative exploration method according to any one of claims 1 to 3.

Images