Global path coverage method based on distributed game multi-agent reinforcement learning

Abstract

The invention discloses a global path coverage method based on distributed game multi-agent reinforcement learning, which comprises the following steps that: 1, a distributed robot executes an independent reinforcement learning plan to access global paths of all sensor nodes according to a current state, and moves along with the paths after completing the global coverage of the paths, so that the global path coverage is completed; the robot senses the sensors in the communication range and other robots, and performs information interaction; 2, the robot reaches a communication range of the sensor nodes and communicates with the sensor nodes, multi-agent reinforcement learning input is updated according to node states, independent learning is executed again, and coverage paths of the remaining sensor nodes are planned; 3, in a distributed game stage, the robots perceive each other, interact information of the accessed sensor nodes and carry out a multi-agent game so as to distribute the remaining sensor nodes; and guiding multiple robots to cooperatively cover all access points. The method is characterized in that 1) a centralized multi-agent game which needs to share information in real time in traditional multi-agent reinforcement learning is avoided, a small amount of information is utilized to share traffic, and global cooperative movement is planned in a local area in a staged manner, and 2) through dynamically updated multi-agent reinforcement learning input, the adaptability of a robot to a working environment is improved.

Description

Technical Field

[0001] The invention relates to a global path coverage method based on distributed game multi-agent reinforcement learning, belonging to the technical field of robot collaborative control. Background Art

[0002] Multi-agent reinforcement learning is an effective technical means to drive multiple unmanned devices to collaborate with each other and assign coverage tasks, and is widely studied in the collaborative coverage tasks of unmanned systems. However, existing multi-agent reinforcement learning technologies are usually based on reliable and real-time information sharing between agents. It is difficult to ensure reliable information interaction in high-interference industrial environments and weak-communication underwater environments, which reduces the real-time nature of collaborative tasks and thus affects the autonomy and safety of robot collaboration. Summary of the invention

[0003] In order to make up for the problems existing in the background technology and improve the shortcomings of the existing solutions. The present invention proposes a multi-robot global path coverage method based on distributed game multi-agent reinforcement learning. The method uses multi-agent reinforcement learning to plan multi-target coverage paths and guide robots to traverse the task area. The method is mainly divided into two stages. The first is an independent learning stage, in which the robot's path coverage is completed through reinforcement learning; then, there is a distributed game stage, in which robots conduct agent game training through information exchange, and finally guide unmanned equipment to perform global coverage tasks to achieve full access to nodes.

[0004] The technical solution of the present invention is as follows: A global path coverage method based on distributed game multi-agent reinforcement learning includes the following steps: Step 1-1. The distributed robot performs independent reinforcement learning planning to access the global path of all sensor nodes according to the current state; after completing the global coverage path, the robot moves along the path, the robot perceives the sensors and other robots within the communication range, and exchanges information; Step 1-2. The robot reaches the communication range of the sensor node, communicates with the sensor node, updates the multi-agent reinforcement learning input according to the node status, re-executes independent learning, and plans the coverage path of the remaining sensor nodes; Step 1-3. In the distributed game stage, robots perceive each other, exchange information about visited sensor nodes, and conduct multi-agent games to allocate remaining sensor nodes, guiding multi-robot collaboration to cover all access points.

[0005] Preferably, in step 1-1, multi-agent reinforcement learning is used to plan a multi-target coverage path to guide the robot to traverse the task area. The robot plans the navigation direction by observing the environment, completes information access and interaction, and each robot performs reinforcement learning to plan a global coverage path; Preferably, in steps 1-3, when the same sensor node receives access requests from multiple robots, a game is conducted between the robots. The robot that wins the game accesses the sensor node, while the robot that loses the access right uses reinforcement learning to reselect the sensor node to be accessed in this stage from the remaining generation access nodes. If the first node that the robot plans to visit next has been visited by other robots, an access conflict occurs between the robots. At this time, the robot uses reinforcement learning to reselect the next node to be visited from the remaining nodes. During the game, the robots will exchange information, that is, exchange their respective historical access records, information on the next node to be visited, and interaction information with other robots. Ultimately, each robot completes the coverage task by conducting distributed game collaboration.

[0006] Beneficial effects of the present invention: The characteristics of this method are: 1) it avoids the centralized multi-agent game that requires real-time information sharing in traditional multi-agent reinforcement learning, and uses a small amount of information sharing traffic to plan global collaborative movement in stages in local areas; 2) it improves the robot's adaptability to the working environment through dynamically updated multi-agent reinforcement learning input. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is an example of global path coverage based on distributed game multi-agent reinforcement learning. DETAILED DESCRIPTION

[0008] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application are described below. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0009] A global path coverage method based on distributed game multi-agent reinforcement learning includes the following steps: Step 1-1. The distributed robot performs independent reinforcement learning planning to access the global path of all sensor nodes according to the current state (such as Figure 1 (a)); After completing the global coverage path, the robot moves along the path, the robot perceives the sensors and other robots within the communication range, and exchanges information; Step 1-2. The robot reaches the communication range of the sensor node, communicates with the sensor node, updates the multi-agent reinforcement learning input according to the node status, re-executes independent learning, and plans the coverage path of the remaining sensor nodes (such as Figure 1 (b)); Step 1-3. In the distributed game stage, robots perceive each other, exchange information about visited sensor nodes, and conduct multi-agent games to allocate remaining sensor nodes, guiding multi-robot collaboration to cover all access points.

[0010] Preferably, in the above step 1-1, multi-agent reinforcement learning is used to plan a multi-target coverage path to guide the robot to traverse the task area. The robot plans the navigation direction by observing the environment, completes information access and interaction, and each robot performs reinforcement learning to plan a global coverage path.

[0011] Preferably, in step 1-2, after completing the global path coverage, the robot moves along the path, reaches the communication range of the sensor node, starts information exchange, and updates the information of the area that has not been visited. Then, the robot uses reinforcement learning technology again to plan the subsequent coverage path, such as Figure 1 As shown, after completing the global path coverage, robot 1 moves along the path, reaches the communication range of the sensor node (C), and starts communicating ( Figure 1 (a) (b)). When robot 1 successfully communicates with the sensor node, that is, the sensor (C) is not currently accessed, path coverage begins, and C records the information of the sensor nodes that robot 1 has visited. Later, when robot 2 reaches the communication range of the sensor node (C), C has completed path coverage with robot 1, so it rejects robot 2's path coverage, records the information of the sensor nodes that robot 1 and robot 3 have visited (green dots), and replies to robot 3. At this time, all robots within the communication range of C (including robot 1) receive information containing the covered path and update their coverage targets. Each robot updates the multi-agent reinforcement learning input according to the covered sensor nodes, re-executes independent learning, and plans the coverage path of the remaining sensor nodes.

[0012] Preferably, in the above steps 1-3, when the same sensor node receives access requests from multiple robots, a game is conducted between the robots. The robot that wins the game accesses the sensor node, while the robot that loses the access right uses reinforcement learning to reselect the sensor node to be visited in this stage from the remaining generation access nodes. If the first node that the robot plans to visit next has been visited by other robots, an access conflict occurs between the robots. At this time, the robot uses reinforcement learning to reselect the next node to be visited from the remaining nodes. During the game, the robots will exchange information, that is, exchange their respective historical access records, information on the next node to be visited, and interaction information with other robots. Finally, each robot completes the path coverage task by conducting distributed game collaboration.

[0013] The game training in steps 1-3 adopts a mixed mode of self-game and priority fictitious self-game. In the initial stage of the game, there are no other opponent samples in the opponent pool, so the preset rounds of self-game are executed to train the network. After the rounds of self-game are over, the opponent pool is shared among the robots. Each robot extracts an opponent with a higher reward value than the current sample from the opponent pool, executes priority fictitious self-game, trains the network, and finally each robot learns the local function value and takes corresponding actions. The information interaction between robots occurs in the sharing process of the opponent pool; This example refers to an observation application scenario based on robot collaboration. For example, the base station provides terrain information to the robot, assigns movement tasks, and receives observation data collected by the robot from sensor nodes; the robot traverses various areas, accesses and collects observation data from sensor nodes, and transmits the data to the base station.

[0014] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A global path coverage method based on distributed game multi-agent reinforcement learning, characterized in that: The following steps are involved: Step 1-1. The distributed robot performs independent reinforcement learning planning to access the global path of all sensor nodes according to the current state. After completing the global coverage path, the robot moves along the path, perceives the sensors and other robots within the communication range, and exchanges information; Step 1-2. The robot reaches the communication range of the sensor node, communicates with the sensor node, updates the multi-agent reinforcement learning input according to the node status, re-executes independent learning, and plans the coverage path of the remaining sensor nodes; Step 1-3. In the distributed game stage, robots perceive each other, exchange information about visited sensor nodes, and conduct multi-agent games to allocate remaining sensor nodes; guiding multiple robots to collaborate to cover all access points.

2. The global path coverage method based on distributed game multi-agent reinforcement learning according to claim 1 is characterized in that: In step 1-1, multi-agent reinforcement learning is used to plan a multi-target coverage path to guide the robot to traverse the coverage area. The robot plans the navigation direction by observing the environment, completes information access and interaction, and each robot performs reinforcement learning to plan the global coverage path.

3. The global path coverage method based on distributed game multi-agent reinforcement learning according to claim 1 is characterized in that: In the steps 1-3, when the same sensor node receives access requests from multiple robots, a game is conducted between the robots. The winning robot accesses the sensor node, while the robot that loses the access right uses reinforcement learning to reselect the sensor node to be accessed in this stage from the remaining generation access nodes. If the first node that the robot plans to visit next has been visited by other robots, an access conflict occurs between the robots. At this time, the robot uses reinforcement learning to reselect the next node to be visited from the remaining nodes. During the game, the robots will exchange information, that is, exchange their respective historical access records, information on the next node to be visited, and interaction information with other robots. Finally, each robot completes the path coverage task by conducting distributed game collaboration.

Images