Multi-robot cooperative region exploration intelligent decision-making method and device

Abstract

The embodiment of the invention provides a multi-robot collaborative regional exploration intelligent decision-making method and device. The method comprises the following steps: regarding exploration of a to-be-explored area as establishment of a two-dimensional cost map of the to-be-explored area, and setting a variable weight cost function for a multi-robot environment exploration decision process; the method comprises the following steps of: modeling a multi-robot environment exploration decision process into a multi-agent part observable Markov decision process model by taking each robot as an agent; a distributed execution MADDPG framework is trained in a centralized mode through multi-agent reinforcement learning, the agents are made to continuously interact with the environment, transition is regarded as experience to be stored in an experience pool, and during the period, various parameters in a value network and a strategy network of the agents are trained based on the MADDPG framework; and after parameter solidification, decision making is carried out on each weight item in a variable weight cost function of the intelligent agent based on a strategy network, so that an optimal leading edge point is selected as a target point of a next step in an environment exploration process, and exploration is carried out based on the optimal leading edge point.

Description

Technical Field

[0001] The present invention relates to the technical field of robot area exploration, and in particular to an intelligent decision-making method and device for multi-robot collaborative area exploration. Background Art

[0002] At present, there has been extensive research on the autonomous exploration of robots in unknown environments at home and abroad. According to the number of robots, it can be divided into two categories: single robot and multi-robot; according to the research method, it can be divided into two categories: traditional methods and deep learning-based methods. In environmental exploration tasks, the robot's prior knowledge of the environment is extremely limited, so in order to complete the task, it must implement an efficient and robust exploration strategy. In the research in this field, the researchers are most concerned about how to find the best strategy to optimize the robot's exploration performance, minimize the total exploration time, and shorten the path.

[0003] It should be noted that there are currently the following defects in robot area exploration:

[0004] (1) A large number of existing technologies are aimed at single-robot environment exploration, rather than multi-robot collaborative environment exploration. If the single-robot environment exploration method is directly applied to multi-robot scenarios, there will be no communication, interaction, or collaboration between robots, which will result in a large waste of energy, computing power, and other resources.

[0005] (2) Existing work on multi-robot collaborative environment exploration often requires strong communication capabilities between robots and between robots and the environment. In fact, the environments to be explored are often environments with weak communication conditions (such as mines, battlefields, alien planets, etc.), which cannot require robots to have strong communication capabilities. Therefore, it is best for robots to make decisions in a distributed manner. Summary of the invention

[0006] In a first aspect, an embodiment of the present invention provides a multi-robot collaborative area exploration intelligent decision-making method, the method comprising:

[0007] The exploration of the area to be explored is regarded as building a two-dimensional cost map of the area to be explored, and a variable weight cost function is set for the multi-robot environment exploration decision process. The robot decides the next target point in the environment exploration process by dynamically calculating the weight items.

[0008] Taking each robot as an intelligent agent, the multi-robot environment exploration decision-making process is modeled as a multi-agent partially observable Markov decision process model;

[0009] Using the distributed execution of the Multi-agent Deep Deterministic Policy Gradient (MADDPG) architecture with centralized training of multi-agent reinforcement learning, the agents are allowed to continuously interact with the environment and store the changes as experience in the experience pool. During this period, the parameters of the value network and policy network of the agents are trained based on the MADDPG architecture.

[0010] After the parameters are solidified, the policy network is used to make decisions on the weight items in the variable weight cost function of the intelligent agent, so as to select the optimal frontier point as the next target point in the environmental exploration process, and explore based on this.

[0011] In some implementations of the first aspect, the state space of the multi-agent partially observable Markov decision process model includes three levels, namely, a global map, agent coordinates, and frontier point coordinates.

[0012] In some implementable embodiments of the first aspect, the observation space of the multi-agent partially observable Markov decision process model is composed of the local local map of the agent, the agent's own coordinates at the current moment, and the frontier point coordinates observed by the agent itself.

[0013] In some implementations of the first aspect, the action space of the multi-agent partially observable Markov decision process model is composed of weight terms of distance terms and information gain terms in a variable weight cost function.

[0014] In some implementations of the first aspect, the value network and the policy network are RNN structures, and a ReLU function is used as an activation function.

[0015] In a second aspect, an embodiment of the present invention provides a multi-robot collaborative area exploration intelligent decision-making device, the device comprising:

[0016] A setting module is used to regard the exploration of the area to be explored as building a two-dimensional cost map of the area to be explored, and to set a variable weight cost function for the multi-robot environment exploration decision process. The robot dynamically calculates the weight terms in the function to decide the next target point in the environment exploration process;

[0017] A modeling module is used to model the multi-robot environment exploration decision process as a multi-agent partially observable Markov decision process model, taking each robot as an intelligent agent;

[0018] The training module is used to use the MADDPG architecture of distributed execution of multi-agent reinforcement learning to centrally train the agents, allowing them to continuously interact with the environment and store the changes as experience in the experience pool. During this period, the parameters of the value network and policy network of the agents are trained based on the MADDPG architecture.

[0019] The decision-making module is used to make decisions on the weight items in the variable weight cost function of the intelligent agent based on the policy network after the parameters are solidified, so as to select the optimal frontier point as the next target point in the environmental exploration process and conduct exploration based on it.

[0020] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described above.

[0021] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable a computer to execute the method described above.

[0022] According to the embodiments of the present invention, at least the following technical effects are achieved:

[0023] Based on the collaborative frontier point decision problem for multiple robots, a multi-robot collaborative regional exploration intelligent decision-making method based on the MADDPG architecture in multi-agent reinforcement learning and a variable weight cost function is proposed. This method uses a centralized training-distributed execution architecture to enable the agents to make collaborative decisions in a distributed manner. Experiments show that compared with the traditional cost function method and the independent DDPG method, the method proposed in the present invention has better results in evaluation indicators such as the total path length and coverage repetition rate of multi-machine collaborative coverage. For example, compared with the independent DDPG method, the total path length in the coverage process is reduced by 10.37%, and the coverage repetition rate is reduced by 31.50%. Compared with the prior art, the present invention has the advantages of high exploration efficiency and short path length.

[0024] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0026] Figure 1 A flowchart of a multi-robot collaborative area exploration intelligent decision-making method provided by an embodiment of the present invention;

[0027] Figure 2 A flowchart of a robot exploring an unknown environment provided by an embodiment of the present invention;

[0028] Figure 3 A schematic diagram of a frontier point provided by an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of information gain of a frontier point provided by an embodiment of the present invention;

[0030] Figure 5 A schematic diagram of the MADDPG architecture provided by an embodiment of the present invention;

[0031] Figure 6 A structural diagram of an executor network-critic network with an RNN structure in a MADDPG architecture provided in an embodiment of the present invention;

[0032] Figure 7 A schematic diagram of a reward curve during a training process provided by an embodiment of the present invention;

[0033] Figure 8 A schematic diagram of the trajectories of the exploration process of three robots provided in an embodiment of the present invention;

[0034] Fig. 9 A schematic diagram of the trajectories of the exploration process of two robots provided in an embodiment of the present invention;

[0035] Fig.10 A structural diagram of a multi-robot collaborative area exploration intelligent decision-making device provided by an embodiment of the present invention;

[0036] Fig.11 The figure is a structural diagram of an exemplary electronic device capable of implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0039] In order to solve the technical problems arising from the background technology, the embodiments of the present invention provide a method, device, equipment and storage medium for intelligent decision-making of multi-robot collaborative regional exploration. Specifically, the exploration of the area to be explored is regarded as the establishment of a two-dimensional cost map of the area to be explored, and a variable weight cost function is set for the multi-robot environmental exploration decision-making process; each robot is used as an intelligent agent, and the multi-robot environmental exploration decision-making process is modeled as a multi-agent partially observable Markov decision process model; the MADDPG architecture of distributed execution of multi-agent reinforcement learning is used to centrally train the intelligent agent to continuously interact with the environment, and the changes are regarded as experience and stored in the experience pool. During this period, the various parameters in the value network and strategy network of the intelligent agent are trained based on the MADDPG architecture; after the parameters are solidified, the various weight items in the variable weight cost function of the intelligent agent are decided based on the strategy network, so as to select the optimal frontier point as the target point for the next step in the environmental exploration process, and explore based on this.

[0040] In this way, multiple robots can be used to collaboratively explore unknown environments, and the MADDPG architecture of multi-agent reinforcement learning centralized training-distributed execution can be used to achieve intelligent decision-making on the weights of different parts of the cost function during the multi-robot exploration process through deep learning neural network training, thereby completing the selection of frontier points and improving exploration efficiency.

[0041] In the following, in conjunction with the accompanying drawings, a multi-robot collaborative area exploration intelligent decision-making method, device, equipment and storage medium provided by an embodiment of the present invention are described in detail through specific embodiments.

[0042] Figure 1 A flowchart of a multi-robot collaborative area exploration intelligent decision-making method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the multi-robot collaborative area exploration intelligent decision-making method 100 may include the following steps:

[0043] S110, the exploration of the area to be explored is regarded as the establishment of a two-dimensional cost map of the area to be explored, and a variable weight cost function is set for the multi-robot environment exploration decision process. The robot decides the next target point in the environment exploration process by dynamically calculating the weight items therein.

[0044] S120, taking each robot as an intelligent agent, the multi-robot environment exploration decision process is modeled as a multi-agent partially observable Markov decision process model.

[0045] S130 uses the MADDPG architecture of distributed execution of multi-agent reinforcement learning centralized training to allow the agent to continuously interact with the environment and store the changes as experience in the experience pool. During this period, the various parameters in the value network and strategy network of the agent are trained based on the MADDPG architecture.

[0046] S140, after the parameters are solidified, decisions are made on the weight items in the variable weight cost function of the agent based on the policy network, so as to select the optimal frontier point as the next target point in the environmental exploration process, and explore based on this.

[0047] For further understanding, the above steps are described below in conjunction with specific embodiments:

[0048] Next, we will establish a mathematical model for the problem of multi-robot collaborative regional exploration. Assume that the unknown area to be explored is a single-connected area, and the goal of the exploration is to establish a two-dimensional cost map M of the area. The robots that perform the task constitute a set Each robot in [t 0 ,t max ] time period to perform the exploration task. Figure 2 As shown, at the initial time t = t 0 , the robot does not have any prior map information about the environment. In order to perform exploration tasks, the robot first needs to use its onboard sensors, such as LiDAR, sonar, cameras, etc., to perceive the surrounding environment, and its mapping radius is ρ. The data collected by these sensors is used to detect obstacles, identify terrain features, and build a preliminary environmental map. As the robot moves, it will continuously collect new sensor data and update the map in real time to reflect the latest state of the environment. The coordinates of the robot u in the two-dimensional map at time t are Speed The sub-map created by the robot at this moment is The robot will use the SLAM algorithm to stitch all sub-maps stored in the local machine into a local map.

[0049] During the exploration process, each robot needs to decide the next target point, that is, the target position, based on the current observation information of the environment. This decision-making process is usually based on one or more exploration strategies, such as maximizing information gain, minimizing path length, balancing exploration and utilization, etc. These strategies are designed to guide the robot to efficiently cover unknown areas while minimizing repeated exploration and missed areas. Once the target location is determined, the robot will plan a path to that location. Path planning needs to take into account the current environment map, the location of obstacles, and the robot's movement capabilities. During the movement, the robot may encounter new obstacles or moving objects (such as other robots exploring the environment). At this time, the path needs to be dynamically adjusted to avoid collisions and continue to move towards the target location. After the robot reaches the target location, it will use sensors to observe again and update the map based on the new observation information. Then, the robot will select a new target location again based on the updated map information and continue the exploration process. The above process will be repeated until the robot covers the entire unknown area or reaches the preset exploration goal.

[0050] Assume that at time t, the robot u is in position The local map is The set of observed frontier points (i.e., the boundary points between the mapped area and the unmapped area during the robot's exploration process) (like Figure 3 As shown). In the exploration based on the cost function, for the target point at this moment The selection strategy is denoted as Defined as:

[0051]

[0052] in, According to the observed frontier point set The generated candidate target point set. c(x,p) is the cost of the robot position x to point p, which is generally defined as:

[0053]

[0054] The first part of is the normalized local map The shortest collision-free distance from x to p. This distance can be calculated and determined by algorithms such as Dijstra algorithm and A* algorithm. Part II Refers to the normalized local map The potential information gain near point p in the figure refers to the unknown area within the circle with radius ρ and center on point p, as Figure 4 As shown in p 1 、p 2 and p3 Take three candidate frontier points as an example. The orange area corresponds to the unknown area, and its area is the information gain. Then calculate the ratio of this area to the circle area and normalize it to The above two parts are w 1 and w 2 The weighted sum constitutes the total cost. However, the method of determining the weights of the two parts is crucial and will directly affect the performance of the robot's exploration decision method. For example, the optimal weights are different in different maps and different exploration stages of the same map.

[0055] Here we propose to use multi-agent reinforcement learning to solve the problem of frontier point selection cost function weight decision-making to optimize the multi-robot collaborative regional exploration strategy. In view of the temporal decision-making problem in multi-robot collaborative regional exploration, each robot is regarded as an agent, and a multi-agent partially observable Markov decision process model is established.

[0056] The state space of the environment It includes three levels: global map M i , robot coordinates The environment map is grouped into 10×10 regions, each of which has three attributes: the proportion of the free area, obstacles, and unknown area to the area ω free ,ω obstacle With ω unknown The frontier point coordinates are clustered by the K-means algorithm, and the present invention takes K=12. According to the above, the total length of the state space is 10×10×3+2×U+2×12. When the total number of robots is set to U=3, the length of the state space is 330.

[0057] Each agent has a local observation of the environment. Similar to the constituent elements of the state space, the local local map of the agent Current time coordinates The total length of the observation space for agent u is 10×10×3+2+2×12=326. The observations of all agents at the same step can be concatenated into joint observations, expressed as the joint observations of this step.

[0058] The action space of the agent is the weight term w of the distance term and information gain term in the cost function 1 With w 2 . Due to the existence of w 1 +w 2 = 1 limit, here any one can be taken as the intelligent agent action, the present invention takes As the agent action, then w 2 By 2 =1―w 1 Calculated. Afterwards, the agent uses this weight to map The cost of each candidate point is calculated, and the frontier point with the smallest cost is selected as the target point, and A is used * The algorithm calculates a collision-free path to the target point and tracks the path to the target point. The present invention assumes that the decision-making moments of all agents are synchronized, that is, when implementing the algorithm, it is assumed that the robots will move to the target point at different speeds and arrive at the same time. The actions of all agents in the same step can be spliced ​​into a vector, which is represented as the joint action of the step.

[0059] During the agent's exploration process, it will receive dense rewards, which can be expressed as follows:

[0060]

[0061] Different from the calculation of candidate points, the information gain and distance cost here use the global map M i Computation, not local local maps The purpose of this treatment is to reduce the rewards when the agent exploration areas overlap, thereby reducing the repeated coverage of the area and the total length of the robot path, thereby improving the coordination of multi-agent exploration.

[0062] Here we use the MADDPG architecture in multi-agent reinforcement learning to train the parameters of the value network (also known as the critic network) and the policy network (also known as the executor network) of the agent. The architecture is as follows: Figure 5 The present invention introduces the RNN structure when designing the executor network and the critic network, and uses the ReLU function as the activation function, as shown in Figure 6 shown.

[0063] After the parameters are solidified, the policy network is used to make decisions on the weight items in the variable weight cost function of the intelligent agent, so as to select the optimal frontier point as the next target point in the environmental exploration process, and explore based on this.

[0064] It is worth noting that this paper selected a large-scale area for training and testing the multi-robot collaborative regional exploration intelligent decision-making method based on MADDPG, and compared it with other methods in the same area. Three robots were set up in the area to start mapping from fixed starting points (60,60), (150,200), and (60,140).

[0065] The present invention conducts comparative experiments on the proposed method and the following four methods:

[0066] 1. Greedy strategy: Each robot will select the candidate point of the cluster center closest to it as the target point.

[0067] 2. Equal weight cost function method: Each robot has a fixed weight w 1 =0.5, w 2 =0.5 sets the cost function.

[0068] 3. Gain priority cost function method: Each robot has a fixed weight w 1 =0.3, w 2 =0.7 sets the cost function.

[0069] 4. Independent DDPG method: Each robot uses an independent DDPG method to make decisions, without a centralized training-distributed execution process.

[0070] The present invention uses two performance evaluation indicators: the robot's motion path length and the repetition rate χ of the robot's exploration area. Among them, for the total motion path length, the present invention sets up four indicators d 90％ ,d 98％ , and They are used to measure the total path length and the corresponding average length when the robot has explored 90% and 98% of the area.

[0071] like Figure 7 As shown in the figure, the reward curve of the method proposed in the present invention during the training process is shown, including the individual rewards and total rewards of the three agents. It can be seen from the figure that during the training process, the rewards and individual rewards of the three agents and each agent show a gradual upward trend and converge after 8,500 rounds.

[0072] Figure 8 , Fig. 9 The trajectories of the exploration process of three and two robots in the simulation environment with an area of ​​20m×20m and 10m×10m respectively are shown. From the trajectories of the robots, the following rules can be seen:

[0073] First, the robots have learned to explore the area in a division of labor, and the overlap in coverage between robots is small.

[0074] Second, the workload of different robots is relatively balanced, and the load balancing is better than the centralized training method of multiple agents proposed in the present invention.

[0075] Third, the robot rarely misses corner areas during the exploration process and is able to comprehensively evaluate and choose between nearby and distant targets, reducing the total length of the path during exploration.

[0076] As shown in Table 1, the performance of different methods in this environment on different indicators is demonstrated.

[0077] Table 1

[0078]

[0079] As can be seen from Table 1, the method proposed in this invention has the best performance in three indicators compared with other methods. The robot can complete 90% and 98% coverage of the area in a shorter path, which shows the efficiency and synergy of the method proposed in this invention, which comes from the variable weight decision of the cost function and the central training characteristics of the method. The following is an analysis of other methods:

[0080] 1. The greedy strategy method makes the robot always go to the target point closest to itself, causing the robot to move back and forth in a small range, resulting in a waste of paths;

[0081] 2. The equal-weight cost function method takes into account the balance between path cost and information gain, allowing the robot to cover 90% and 98% of the area faster, but because its cost weight is set to a fixed value, its performance is not as good as the method proposed in this invention;

[0082] 3. The gain priority cost function method sets a larger weight on information gain, so that the robot can cover 90% of the area quickly. However, since a large number of corners will be missed during the exploration process, the robot must go to these corners again for exploration, resulting in a longer path length required to cover 98% of the area.

[0083] 4. The independent DDPG method regards other robots as part of the environment, and does not explicitly consider the collaborative strategy between robots. The decision-making process of the robots is independent of each other, resulting in the coverage overlap rate and total path length between robots being higher than the method proposed in this invention. However, since the independent DDPG method can also make variable weight decisions on the cost function, its total path length indicator is lower than that of the cost function method with two fixed weights.

[0084] In summary, the method proposed in this paper has the highest agent collaboration and exploration efficiency. 90％ In terms of indicators, it is reduced by 10.37% compared with the gain priority cost function method, which has the second best effect; 98％ and χ indicators are reduced by 6.32% and 31.50% respectively compared with the independent DDPG method.

[0085] It can be seen that according to the embodiments of the present invention, at least the following technical effects are achieved:

[0086] 1. The present invention can realize distributed decision-making in the process of robot exploration. That is, during the training process, the robot is centrally trained in the platform and the network parameters are continuously updated iteratively. During the execution process, the robot can use the best model to realize distributed decision-making through its own observation.

[0087] 2. The method based on deep reinforcement learning only needs to forward propagate the action decision network during the execution process, and does not require iterative optimization calculations. Therefore, the complexity of the method in the execution phase is relatively low.

[0088] 3. Compared with the greedy strategy method, the equal-weight cost function method, the gain-priority cost function method and the independent DDPG method, the MADDPG-based exploration method proposed in this invention has the highest agent collaboration and exploration efficiency.

[0089] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0090] The above is an introduction to a method embodiment. The following is a further explanation of the solution of the present invention through an apparatus embodiment.

[0091] Fig.10 A structural diagram of a multi-robot collaborative area exploration intelligent decision-making device provided by an embodiment of the present invention, such as Fig.10 As shown, the multi-robot collaborative area exploration intelligent decision-making device 1000 may include:

[0092] The setting module 1010 is used to regard the exploration of the area to be explored as establishing a two-dimensional cost map of the area to be explored, and to set a variable weight cost function for the multi-robot environment exploration decision process. The robot decides the next target point in the environment exploration process by dynamically calculating the weight items therein.

[0093] The modeling module 1020 is used to model the multi-robot environment exploration decision process as a multi-agent partially observable Markov decision process model, taking each robot as an intelligent agent.

[0094] The training module 1030 is used to utilize the MADDPG architecture of distributed execution of centralized training of multi-agent reinforcement learning, so that the agent continuously interacts with the environment and stores the changes as experience in the experience pool. During this period, various parameters in the value network and strategy network of the agent are trained based on the MADDPG architecture.

[0095] The decision module 1040 is used to make decisions on the weight items in the variable weight cost function of the intelligent agent based on the strategy network after the parameters are solidified, so as to select the optimal frontier point as the next target point in the environmental exploration process and conduct exploration based on it.

[0096] Understandably, Fig.10 Each module / unit in the multi-robot collaborative area exploration intelligent decision-making device 1000 has the function of realizing Figure 1 The functions of each step in the multi-robot collaborative area exploration intelligent decision-making method 100 shown in the figure can achieve its corresponding technical effects, so for the sake of brevity, they will not be repeated here.

[0097] Fig.11 1 is a block diagram of an exemplary electronic device capable of implementing an embodiment of the present invention. Electronic device 1100 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 1100 may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown in the present invention, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required in the present invention.

[0098] like Fig.11 As shown, the electronic device 1100 may include a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the electronic device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0099] Multiple components in the electronic device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the electronic device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0100] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer program product, including a computer program, which is tangibly contained in a computer-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the method 100 in any other appropriate manner (eg, by means of firmware).

[0101] The various embodiments described above in the present invention can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0103] In the context of the present invention, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] It should be noted that the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute method 100 and achieve the corresponding technical effect achieved by executing the method in an embodiment of the present invention. For the sake of concise description, they will not be repeated here.

[0105] In addition, the present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method 100 is implemented.

[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and the present invention is not limited here.

[0107] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-robot collaborative area exploration intelligent decision-making method, characterized in that: The method comprises: The exploration of the area to be explored is regarded as building a two-dimensional cost map of the area to be explored, and a variable weight cost function is set for the multi-robot environment exploration decision process. The robot decides the next target point in the environment exploration process by dynamically calculating the weight items. Taking each robot as an intelligent agent, the multi-robot environment exploration decision-making process is modeled as a multi-agent partially observable Markov decision process model; Using the MADDPG architecture of distributed execution of multi-agent reinforcement learning centralized training, the agent is allowed to continuously interact with the environment and store the changes as experience in the experience pool. During this period, the parameters of the value network and policy network of the agent are trained based on the MADDPG architecture. After the parameters are solidified, the policy network is used to make decisions on the weight items in the variable weight cost function of the intelligent agent, so as to select the optimal frontier point as the next target point in the environmental exploration process, and explore based on this.

2. The method according to claim 1, characterized in that The state space of the multi-agent partially observable Markov decision process model includes three levels, namely, a global map, agent coordinates and frontier point coordinates.

3. The method according to claim 1, characterized in that: The observation space of the multi-agent partially observable Markov decision process model is composed of the local local map of the agent, the agent's own coordinates at the current moment and the frontier point coordinates observed by the agent itself.

4. The method according to claim 1, characterized in that: The action space of the multi-agent partially observable Markov decision process model is composed of weight terms of distance terms and information gain terms in a variable weight cost function.

5. The method according to claim 1, characterized in that The value network and the policy network are RNN structures, and use the ReLU function as the activation function.

6. A multi-robot collaborative area exploration intelligent decision-making device, characterized in that: The device comprises: A setting module is used to regard the exploration of the area to be explored as building a two-dimensional cost map of the area to be explored, and to set a variable weight cost function for the multi-robot environment exploration decision process. The robot dynamically calculates the weight terms in the function to decide the next target point in the environment exploration process; A modeling module is used to model the multi-robot environment exploration decision process as a multi-agent partially observable Markov decision process model, taking each robot as an intelligent agent; The training module is used to use the MADDPG architecture of distributed execution of multi-agent reinforcement learning to centrally train the agents, allowing them to continuously interact with the environment and store the changes as experience in the experience pool. During this period, the parameters of the value network and policy network of the agents are trained based on the MADDPG architecture. The decision-making module is used to make decisions on the weight items in the variable weight cost function of the intelligent agent based on the policy network after the parameters are solidified, so as to select the optimal frontier point as the next target point in the environmental exploration process and conduct exploration based on it.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.

Images