Man-machine cooperative work system and method, robot and human end

Abstract

The embodiment of the invention provides a man-machine cooperative work system, a man-machine cooperative work method, a robot and a human end, and relates to the technical field of robotics.The man-machine cooperative work method includes the steps that a communication relay link is formed by selecting part of a plurality of robots for map exploration and used for achieving human aided tasks, and part of the robots conduct distributed multi-machine cooperative exploration; and the other part of robots form a stable communication relay link to execute the human aided task, and each robot in the communication relay link can return to the exploration team to continue the map exploration task after completing the human aided task. Compared with a scheme only capable of realizing multi-machine collaborative exploration, the embodiment of the invention has the advantages that two working modes of multi-machine collaborative exploration and human-machine collaborative execution of the human auxiliary task can coexist, and the two working modes can be switched back and forth. Thus, cooperative exploration can be carried out in a communication limited environment, human aided tasks can also be carried out, and user requirements can be better met.

Description

Technical Field

[0001] The present invention relates to the technical field of robots, and in particular to a human-robot collaborative working system, method, robot, and human terminal. Background Art

[0002] In the field of robots, communication limitations between robots are often overlooked in related technologies, and it is assumed that the communication between robots is fully connected. In fact, in typical scenarios of underground exploration, reconnaissance, search and rescue, etc., such as in environments like caves, ruins, and mountains and forests, due to the huge and complex terrain and numerous obstacles, the communication between robots is often restricted, which poses challenges to the implementation of tasks in the actual application process. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide a human-robot collaborative working system, method, robot, and human terminal, so as to enable collaborative exploration and human-assisted tasks to be carried out not only in a communication-restricted environment, and be able to better meet the needs of users. The specific technical solutions are as follows:

[0004] In a first aspect, a human-robot collaborative working system is provided, including: a human terminal and multiple robots;

[0005] The human terminal is configured to receive a human-assisted task request including a task location; obtain link information based on the task location, the current location of the human terminal, and a first communication plan, and send the link information to a first messenger robot, where the link information includes the topological information of the communication relay link and a second communication plan; the first communication plan is the communication plan of each robot for map exploration in the current ring topology structure, including the first communication time and the first communication location for communication between two robots;

[0006] The first messenger robot is configured to send the link information to a first successor robot according to the current communication order, and reach a second communication location at a scheduled second communication time;

[0007] The first successor robot and the robots located after the first successor robot are configured to transmit the link information, and respectively determine whether they are robots on the communication relay link according to the link information. If so, they reach the second communication location at the scheduled second communication time; if not, they continue to perform map exploration; some of the multiple robots for map exploration in the current ring topology structure serve as nodes of the communication relay link;

[0008] The end robot at the end of the communication relay link is configured to execute the human-assisted task;

[0009] The robots in the communication relay link other than the end robot are used to receive the data sent by the previous node and transmit the received data to the subsequent node.

[0010] Optionally, the first messenger robot is specifically configured to receive the link information sent by the human terminal, where the first messenger robot is the robot that returns to the human terminal during the human-machine communication interval; based on the link information, locally update the communication plan and topology information stored by itself, itself and the first other robot, where the first other robot is the robot other than the first messenger robot among the multiple robots performing map exploration in the current ring topology; move to the first communication location for communicating with the first successor robot in the current ring topology, and send the link information to the first successor robot, and reach the second communication location at the agreed second communication time according to the communication plan of the first messenger robot in the second communication plan; the first successor robot is the robot located after the first messenger robot in the current ring topology according to the current communication order;

[0011] The first successor robot is specifically configured to locally update the communication plan and topology information stored by itself, itself and the second other robot, where the second other robot is the robot other than the first successor robot among the multiple robots performing map exploration in the current ring topology, and send the link information to the robot located after the first successor robot in the current ring topology according to the current communication order. The robot located after the first successor robot performs the same operation as the first successor robot until all robots in the current ring topology have obtained the topology information of the communication relay link and the second communication plan.

[0012] Optionally, the human terminal is further configured to confirm the task information of the human-assisted task and send the task information to the robot communicating with the human terminal on the communication relay link, where the task information includes the task type;

[0013] If the robot communicating with the human terminal is the end robot, the robot communicating with the human terminal receives the task information and performs the human-assisted task based on the task information; if the robot communicating with the human terminal is not the end robot, the robot communicating with the human terminal continues to transmit the task information in the direction from the human terminal to the end robot until the task information is sent to the end robot, and the end robot receives the task information and performs the human-assisted task based on the task information.

[0014] Optionally, after the end - effector robot executes the human - assisted task and obtains a task result, if the entity communicating with the end - effector robot is the human - side, the end - effector robot sends the task result to the human - side; if the entity communicating with the end - effector robot is not the human - side, the end - effector robot sends the task result to the robot it communicates with, and the robot communicating with the human - side continues to transmit the task result along the direction from the end - effector robot to the human - side until the task result is sent to the human - side.

[0015] Optionally, the human - side is further configured to receive a new position selected by an operator; based on the task location, the new position, and the communication plans of multiple robots performing map exploration in the current ring topology, re - perform task planning and update the communication plans of each robot to obtain a communication relay link and the updated communication plans of each robot.

[0016] Optionally, the human - side is further configured to determine an estimated completion time of the current human - assisted task; if the estimated completion time is less than a predefined threshold, determine whether there are still unprocessed human - assisted tasks; if so, plan for link transfer, and if not, plan for link disbandment.

[0017] Optionally, the human - side is specifically configured to determine the operator position corresponding to the unprocessed human - assisted task and a new task location; based on the operator position and the new task location, determine the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, use the relay nodes in the current communication relay link as the relay nodes of the new communication relay link; if the number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task, select a first target robot from the robots used for map exploration, and the sum of the number of the first target robots and the number of relay nodes in the current communication relay link reaches the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task, select a second target robot from the relay nodes in the current communication relay link as the relay nodes of the new communication relay link.

[0018] Optionally, the human - side is specifically configured to obtain a third communication plan from a second messenger robot, where the third communication plan is the new communication plan of multiple robots currently performing map exploration; based on the third communication plan, form a new ring topology with multiple robots currently performing map exploration and the robots in the current communication relay link, and update the communication plans of the robots in the current communication relay link to obtain a fourth communication plan, and send the topology information of the new ring topology and the fourth communication plan to the second messenger robot and the robots in the current communication relay link;

[0019] The second messenger robot is configured to locally update the topological information of the new ring topology and the communication plan after receiving the topological information of the new ring topology and the fourth communication plan sent by the human terminal; according to the communication plan of the second messenger robot in the fourth communication plan, reach the fourth communication location at the fourth communication time, and send the topological information of the new ring topology and the fourth communication plan to the second successor robot of the second messenger robot in the new ring topology;

[0020] The second successor robot is configured to receive the topological information of the new ring topology and the fourth communication plan, and send the topological information of the new ring topology and the fourth communication plan to the subsequent robot of the second successor robot;

[0021] The robots in the current communication relay link are configured to locally update the topological information of the new ring topology and the fourth communication plan, and reach the agreed communication location according to the fourth communication plan;

[0022] Each robot in the new ring topology is configured to perform map exploration.

[0023] In a second aspect, a human-machine collaborative working method is provided, which is applied to the human-machine collaborative working system according to any one of the first aspect. The human-machine collaborative working system includes: a human terminal and multiple robots. The method includes:

[0024] Through the human terminal, receive a human-assisted task request including a task location; according to the task location, the current location of the human terminal, and the first communication plan, obtain link information and send it to the first messenger robot. The link information includes the topological information of the communication relay link and the second communication plan; the first communication plan is the communication plan of each robot performing map exploration in the current ring topology, including the first communication time and the first communication location for communication between two robots;

[0025] Through the first messenger robot, send the link information to the first successor robot in the current communication order, and reach the second communication location at the agreed second communication time;

[0026] Through the first successor robot and the robots located after the first successor robot, transfer the link information, and respectively judge whether it is a robot on the communication relay link according to the link information. If so, reach the second communication location at the agreed second communication time; if not, continue to perform map exploration; some of the multiple robots performing map exploration in the current ring topology are used as nodes of the communication relay link;

[0027] Execute the human-assisted task through the end robot at the end of the communication relay link;

[0028] Through the robots other than the end robot in the communication relay link, receive the data sent by the previous node and transmit the received data to the subsequent node.

[0029] In a third aspect, a robot is provided, including a plurality of robots included in the human-robot collaborative work system according to any one of the first aspects.

[0030] In a fourth aspect, a human terminal is provided, including a human terminal included in the human-robot collaborative work system according to any one of the first aspects.

[0031] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the method steps of the method described in the second aspect are implemented.

[0032] An embodiment of the present invention further provides a computer program product containing instructions, which when running on a computer, causes the computer to execute the human-robot collaborative work method described above.

[0033] Advantages of the embodiments of the present invention:

[0034] In the human-robot collaborative work system provided by the embodiments of the present invention, by selecting some of the multiple robots performing map exploration in the current ring topology to form a communication relay link, first, based on the current ring topology, the topology information of the communication relay link and the second communication plan are sent to each robot in the current ring topology. Then, the robots selected as nodes on the communication relay link reach the second communication location at the agreed second communication time, forming a communication relay link from the human terminal to the end robot in a communication-restricted environment. Through this communication relay link, the interaction between the human terminal and the end robot in the communication-restricted environment can be realized to complete the human-assisted task. Moreover, the robots communicate at the agreed communication time and at the agreed communication location, realizing communication in the communication-restricted environment. In this way, it is realized that not only collaborative exploration can be carried out in the communication-restricted environment, but also human-assisted tasks can be carried out, which can better meet the user's needs.

[0035] Of course, it is not necessary for any product or method implementing the present invention to achieve all the above-mentioned advantages simultaneously. Description of the Drawings

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can also be obtained based on these drawings.

[0037] Figure 1 This is a schematic diagram of a human-machine collaborative work system provided by an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the communication topology in an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of switching from the exploration mode to the relay mode in an embodiment of the present invention;

[0040] Figure 4A This is a schematic diagram of a link transfer in an embodiment of the present invention;

[0041] Figure 4B This is another schematic diagram of a link transfer in an embodiment of the present invention;

[0042] Figure 4C This is yet another schematic diagram of a link transfer in an embodiment of the present invention;

[0043] Figure 5A This is a schematic diagram of re-establishing a ring topology in an embodiment of the present invention;

[0044] Figure 5B This is another schematic diagram of re-establishing a ring topology in an embodiment of the present invention;

[0045] Figure 5C This is yet another schematic diagram of re-establishing a ring topology in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of switching from the relay mode to the exploration mode in an embodiment of the present invention;

[0047] Figure 7 This is a schematic diagram of the interaction between the human terminal and the robot in an embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of the robot in the relay mode in an embodiment of the present invention;

[0049] Figure 9 This is a schematic diagram of the link processing flow in an embodiment of the present invention;

[0050] Figure 10 This is an overall schematic diagram when the robot only works in the "exploration" mode;

[0051] Figure 11 This is the overall schematic diagram when a robot works in the "relay" mode in the embodiments of the present invention. Specific embodiments

[0052] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art based on the present invention belong to the scope of protection of the present invention.

[0053] Refer to Figure 1 , the embodiments of the present invention provide a human-machine collaborative work system, including: a human terminal and multiple robots;

[0054] The human terminal is used to receive a human-assisted task request including a task location; according to the task location, the current location of the human terminal, and the first communication plan, obtain link information and send it to the first messenger robot. The link information includes the topological information of the communication relay link and the second communication plan; the first communication plan is the communication plan of each robot performing map exploration in the current ring topology structure, including the first communication time and the first communication location for communication between two robots.

[0055] The first messenger robot is used to send the link information to the first successor robot in accordance with the current communication order and reach the second communication location at the agreed second communication time.

[0056] The first successor robot and the robots located after the first successor robot are used to transfer the link information and respectively determine whether they are robots on the communication relay link according to the link information. If so, they reach the second communication location at the agreed second communication time; if not, they continue to perform map exploration; some of the multiple robots performing map exploration in the current ring topology structure are used as nodes of the communication relay link.

[0057] The end robot at the end of the communication relay link is used to execute the human-assisted task.

[0058] The robots other than the end robot in the communication relay link are used to receive the data sent by the previous node and transmit the received data to the subsequent node.

[0059] In the embodiments of the present invention, by selecting some of the multiple robots performing map exploration in the current ring topology to form a communication relay link, first, based on the current ring topology, the topology information of the communication relay link and the second communication plan are sent to each robot in the current ring topology. Then, the robots selected as nodes on the communication relay link reach the second communication location at the agreed second communication time, forming a communication relay link from the human terminal to the end robot in a communication-restricted environment. Through this communication relay link, the interaction between the human terminal and the end robot in the communication-restricted environment can be realized to complete human-assisted tasks. Moreover, the robots communicate at the agreed communication time and location, achieving communication in the communication-restricted environment. In this way, in the communication-restricted environment, not only can collaborative exploration be carried out, but also human-assisted tasks can be performed, better meeting the user's needs.

[0060] Among them, the human-assisted task request can be a request triggered by the operator. The human-assisted task request is used to request the establishment of a communication relay link for a target area on the map to perform human-assisted tasks in the target area, and this target area is the task location.

[0061] The first messenger robot returns to the human terminal and sends the latest information in the current ring topology to the human terminal.

[0062] The human terminal performs task planning and updates the first communication plan according to the task location, the current location of the human terminal, and the first communication plan, obtaining the topology information of the communication relay link and the second communication plan.

[0063] The first messenger robot sends the link information to the first successor robot according to the current communication order, and the specific content may include the following:

[0064] The first messenger robot is specifically used to receive the link information sent by the human terminal. Among them, the first messenger robot is the robot that returns to the human terminal during the human-machine communication interval; based on the link information, locally update the communication plan and topology information stored by itself, itself and the first other robot. The first other robot is the robot other than the first messenger robot among the multiple robots performing map exploration in the current ring topology; move to the first communication location for communicating with the first successor robot in the current ring topology, and send the link information to the first successor robot, and reach the second communication location at the agreed second communication time according to the communication plan of the first messenger robot in the second communication plan; the first successor robot is the robot located after the first messenger robot in the current ring topology according to the current communication order.

[0065] The first successor robot and the robots located after the first successor robot are used to transmit link information and respectively determine whether they are robots on the communication relay link according to the link information.

[0066] Among them, the first successor robot and the robots located after the first successor robot are used to transmit link information. Specifically, after obtaining the link information transmitted by the previous robot, they complete the previously agreed exploration tasks and then transmit the link information when communicating with their respective successor robots.

[0067] The first successor robot is specifically used to locally update the communication plans and topological information of itself and the second other robots stored in itself. The second other robots are the robots other than the first successor robot among the multiple robots performing map exploration in the current circular topological structure. And the first successor robot sends the link information to the robots located after it in the current circular topological structure according to the current communication order. The robots located after the first successor robot perform the same operations as the first successor robot until all the robots in the current circular topological structure have obtained the topological information of the communication relay link and the second communication plan.

[0068] Among them, the robots located after the first successor robot in the current circular topological structure according to the current communication order can also be understood as the other robots except the first messenger robot and the first successor robot.

[0069] The first successor robot and the robots located after the first successor robot, when determining that they are robots on the communication relay link, reach the second communication location at the respective communication plans in the second communication plan at the agreed second communication time, and reach the second communication location at the agreed second communication time.

[0070] The robots except the end robot in the communication relay link receive the data sent by the previous node and transmit the received data to the subsequent node. It can also be understood that the robots except the end robot in the communication relay link are used for information transmission.

[0071] In the field of robotics, autonomous exploration means that a robot relies on its own sensors and on-board processors to sense and move in an unknown environment in real time to obtain environmental information of the unknown map. In tasks such as underground exploration, reconnaissance, and search and rescue, autonomous robots are needed for positioning, exploration, and mapping. Tasks such as underground exploration, reconnaissance, and search and rescue generally need to work in environments such as caves, ruins, and mountains. In such environments as caves, ruins, and mountains, due to the huge and complex terrain and numerous obstacles, there is often no communication signal.

[0072] A commonly used robot exploration method is Frontier Based Exploration, which guides exploration by identifying and utilizing the so-called "frontiers". In this context, a "frontier" refers to the boundary between known and unknown areas, that is, the transition area between known and unknown regions. Using this exploration method, the robot can minimize the length of the exploration path while ensuring effective exploration. Simultaneous Localization And Mapping (SLAM) technology refers to the technology by which a robot incrementally builds a map in an unknown environment. The initial position of the robot is unknown. During movement, it locates itself based on its position and the map, and then incrementally builds the map, ultimately achieving autonomous positioning and navigation of the robot. Due to the large area and complex terrain of such task environments, the efficiency of a single robot is too low. Using the method of multi-robot collaborative exploration can greatly improve efficiency. As the name implies, multi-robot collaborative exploration means that multiple robots exchange information through a communication network, cooperate with each other, and thus divide the work to complete the exploration task of the entire environmental map. However, in extreme environments, such as caves and underground tunnels, the communication between robots is severely restricted, which poses challenges to collaborative exploration.

[0073] In related technologies, there are solutions for multiple robots to explore distributively and independently, but in this solution, robots cannot communicate in real time; there are also solutions for distributed multi-unmanned aerial vehicle exploration, but in this solution, inter-robot communication is not considered, or it is assumed that the communication network is fully connected and ideal.

[0074] It can be seen that in related technologies, the communication limitations between robots are often ignored, and it is assumed that the communication between robots is fully connected. In fact, in typical scenarios of underground exploration, reconnaissance, search and rescue and other tasks, the communication of robots is usually restricted and can only use a temporary network to exchange data within a short unobstructed distance. Therefore, communication constraints should be considered in such multi-robot communication environments.

[0075] Moreover, the interaction between the robot and the operator is not considered in related technologies, ignoring the reasonable needs and overall planning ability of the operator during collaborative work, and ignoring the handling ability of the operator in case of sudden accidents. And the operator's role is very important in the task of human-multi-robot cooperation.

[0076] Related technologies often only focus on maximizing exploration efficiency, without considering the interaction between the robot and the operator, and ignoring the important role of the operator in the process of human-multi-robot collaborative work. In fact, the operator has many reasonable requirements, such as: 1. Timely understanding of the exploration progress and the status of the robot swarm; 2. Based on the transmitted map and photo information, making overall plans, giving priority to certain specific areas, or asking the robot to focus on monitoring a certain area; 3. If an unexpected situation occurs (such as a robot getting stuck in a narrow area and unable to withdraw autonomously, or the operator discovers some hidden suspicious areas unknown to certain robots, etc.), the operator can make reasonable judgments and request to intervene in the robot's working process to perform human-assisted work.

[0077] For example, for the second type of task mentioned above, the operator may need to focus on real-time monitoring of a certain area, which requires the robot to transmit real-time audio and video streams; when necessary, the operator needs to intervene in the robot's working process to perform human-assisted work and remotely operate the robot manually, which requires the operator to be able to send control commands to the remote robot in real time. A necessary condition for fulfilling the above-mentioned requirements is that there is a stable and high-quality communication relay link between the operator and the remote robot, so as to transmit real-time audio and video streams and real-time control signals. Moreover, in complex or dangerous environments such as caves and disaster areas, it is difficult for humans to reach near the remote robot in person. At this time, only multiple robots can form a stable and high-quality communication relay link including multiple robots as "relay" nodes through the "relay" method to meet the above requirements.

[0078] In addition, after a human-assisted task is completed, if there are no other human-assisted tasks, the communication relay link will be disbanded, and the robot will switch from the "relay" mode back to the "exploration" mode; if there are still multiple human-assisted tasks, the communication relay link can directly transfer between multiple tasks without mode switching.

[0079] In environments such as caves, ruins, and mountains and forests, due to the huge and complex terrain and numerous obstacles, there is often no communication signal, and only wireless ad-hoc network technology (Wireless Ad-Hoc Network) can be relied on. A wireless ad-hoc network is a multi-hop mobile peer-to-peer network composed of multiple nodes that uses wireless communication methods and supports dynamic networking. It has no strict control center, and the status of all nodes is equal, that is, it is a peer-to-peer network. Multiple nodes coordinate their behaviors through a distributed algorithm and can quickly and automatically form an independent network after power-on. When a node needs to communicate with a farther node, multi-hop forwarding by intermediate nodes is required. By using wireless ad-hoc network devices, even in a cave environment without a signal, multiple robots can form their own network, and as long as two robots are within a certain communication range, they can communicate with each other.

[0080] In the related art, this new form of human-robot collaborative work that conducts human-assisted tasks through relay links under communication constraints has not been considered. The embodiments of the present invention implement a solution for humans and multiple robots to collaborate with each other in an environment with communication constraints, relying on wireless ad-hoc network technology and centered around the operator. The solution for humans and multiple robots to collaborate with each other is specifically as follows: a solution for humans and multiple robots to conduct distributed multi-robot collaborative exploration and human-assisted collaborative operations. Among them, multi-robot collaboration and human-robot collaboration are involved. Some robots conduct distributed multi-robot collaborative exploration, and some other robots can form stable communication relay links to perform human-assisted tasks.

[0081] In the embodiments of the present invention, the human side is an electronic device held by the operator, which can be a tablet, a laptop computer, etc. Among them, the operator can also be referred to as a human operator.

[0082] The human-assisted task can also be understood as a collaborative task assisted by humans. Or it can also be understood as any task that requires humans to form a link to help. If the length of the formed link is 1, it can be understood that humans directly help the end robot without an intermediate robot. The specific task is determined according to actual needs.

[0083] For example, the human-assisted task can be: turning on the camera to capture audio and video of the surrounding environment and targets and returning the audio and video stream to the human side, or receiving remote control instructions issued by the human side and performing corresponding actions, and so on. Among them, receiving remote control instructions and performing corresponding actions can also be understood as realizing remote control. It should be noted that this is only an example of the human-assisted task. In the embodiments of the present invention, the human-assisted task can specifically be any task that a robot may complete.

[0084] Among them, the human side obtains the data transmitted by the first messenger robot and can observe information such as the fused overall map, the plans of all current robots (the meeting time and location between two robots), and photos of suspicious areas on the human-machine interface such as a display screen. Based on this information, the operator may have specific operation requirements, and corresponding task requests can be triggered according to the operation requirements. For example, the operator requests the robot to form a high-quality communication relay link for a certain area on the map and conduct human-assisted collaborative tasks in a certain area.

[0085] Perform task planning and update the first communication plan according to the task location, the current location of the human side, and the first communication plan to obtain the topological information of the communication relay link and the second communication plan. Specifically, it can include performing task planning according to the task location and the current location of the human side to obtain the topological information of the communication relay link; and updating the first communication plan according to the topological information of the communication relay link to obtain the second communication plan.

[0086] The first communication plan is the communication plan for each robot performing map exploration in the current ring topology, including the first communication time and the first communication location for communication between every two robots. Among them, the ring topology can also be called a ring communication topology. As shown on the left side in Figure 2 the ring communication topology shown, which includes the ring communication topology where robots i, j, k, l, and m are connected in sequence and communicate in the direction indicated by the arrow. Figure 2 The single arrow in the left ring communication topology shown represents the occurrence order of pairwise communication events. Here, the occurrence order can also be understood as the communication order of robots in the ring topology.

[0087] The second communication plan includes the robots located on the communication relay link, the second communication time, and the second communication location for performing human-assisted tasks.

[0088] Moreover, the topological information of the communication relay link can be understood as a chain communication topology. As shown on the right side in Figure 2 the chain communication topology shown, which includes the human end q and the chain communication topology composed of robots p, o, and n. Figure 2 The double arrow in the chain communication topology shown on the right represents that data can be transmitted bidirectionally through the two robots o and p in the middle of the link.

[0089] The first messenger robot is the robot that returns to the human end during the human-robot communication interval.

[0090] Among them, the human-robot communication interval can be determined according to actual needs or experience, etc., such as 300 seconds, 400 seconds, 500 seconds, and so on.

[0091] In one implementable manner, the human-robot communication interval can be preset. During the actual application process, the human-robot communication interval can change, or it can also remain unchanged.

[0092] It should be noted that in the embodiments of the present invention, the messenger robot is dynamically determined to ensure the requirement that "a robot needs to return to the human end to update information every preset time interval (human-robot communication interval)". That is, the messenger robot is not fixed. The messenger robot that returns to the human end in each round may be different from the messenger robot that returned in the previous round, or it may be the same. The embodiments of the present invention do not limit this. During a round of communication, after the messenger robot completes communication with the human end and returns to the exploration team, the "messenger" identity of the messenger robot that returned to the human end in this round is lifted.

[0093] For example, Figure 1 the first messenger robot may be different in different rounds of communication.

[0094] Moreover, in the embodiments of the present invention, the messenger robot is dynamically determined during pairwise communication between robots. For example, during the pairwise communication of each robot performing map exploration in the current ring topology, the first messenger robot for returning to the human terminal is determined.

[0095] After the first messenger robot sends the topology information of the communication relay link and the second communication plan to the first successor robot, the messenger identity of the first messenger robot is released.

[0096] When all robots in the current ring topology have obtained the topology information of the communication relay link and the second communication plan, the messenger robot (here referring to the first messenger robot) arrives at the second communication location at the agreed second communication time according to the communication plan of the first messenger robot in the second communication plan, and when other robots except the messenger robot in the current ring topology determine that they are robots on the communication relay link, they arrive at the second communication location at the agreed second communication time according to their respective communication plans in the second communication plan. At this time, the robots selected as those on the communication relay link in the current ring topology all reach their respective predetermined communication nodes, forming a communication relay link. In this way, the human terminal and the end robot can communicate through this communication relay link to complete the implementation of the human assistance task.

[0097] In the embodiments of the present invention, the communication order can be dynamically changed or fixed.

[0098] Among them, the end robot at the end of the communication relay link is used to execute the human assistance task. Specifically, it can turn on the camera to capture audio and video of the surrounding environment and targets to return an audio and video stream to the human terminal, receive the remote control instructions issued by the human terminal and execute corresponding actions, etc.

[0099] The end robot can be a robot located in the above target area. For example, the human assistance task is to capture the environment of the target area and return the obtained audio and video stream to the human terminal.

[0100] The other robots except the end robot in the communication relay link are used for information transmission, receiving the data sent by the previous node and transmitting the received data to the subsequent node.

[0101] For any robot other than the end - effector robot, the so - called previous node and subsequent node refer to the robots that communicate with this robot in the information transmission direction. If the information transmission direction is from the end - effector robot to the human side, the previous node of this robot is the robot that is close to the end - effector robot and communicates with this robot, and the subsequent node of this robot is the robot that is close to the human side and communicates with this robot. If the information transmission direction is from the human side to the end - effector robot, the previous node of this robot is the robot that is close to the human side and communicates with this robot, and the subsequent node of this robot is the robot that is close to the end - effector robot and communicates with this robot. The node close to the human side may be the human side itself, and the node close to the end - effector robot may be the end - effector robot itself.

[0102] For example, for Figure 2 robot o shown on the right side in [reference], if the information transmission direction is from robot n (end - effector robot) to human side q, the previous node of this robot o is robot n, and the subsequent node of this robot o is robot p; if the information transmission direction is from human side q to end - effector robot n, the previous node of this robot o is robot p, and the subsequent node of this robot o is robot o.

[0103] In the embodiment of the present invention, by selecting some of the multiple robots performing map exploration in the current ring - shaped topology to form a communication relay link, first, based on the current ring - shaped topology, the topology information of the communication relay link and the updated second communication plan are sent to each robot in the current ring - shaped topology. Then, the robots selected as nodes on the communication relay link reach the second communication location at the agreed second communication time, forming a communication relay link from the human side to the end - effector robot in a communication - restricted environment. Through this communication relay link, the interaction between the human side and the end - effector robot in the communication - restricted environment can be realized to complete the human - assisted task. Among them, some robots perform distributed multi - robot collaborative exploration, and some other robots can form a stable communication relay link to execute the human - assisted task.

[0104] As Figure 3 shown on the left side pointed by the arrow in [reference], assume that the communication order of the current exploration team is "-i - j - k - l - m - n - o - i -", and k is the messenger robot and has received the plan of "assigning 3 robots from the exploration team to the relay node". Messenger k originally had two communication events, which were and , then should be deleted, where represents the communication event between robot k and robot l, represents the communication event between robot j and robot k, and a new , which represents a special communication event. Position indicates the position where k is assigned as a relay node, meaning k should go to the location where Position is. Thus, messenger k first communicates with l, transfers the information to l, then deletes the original plan to communicate with j, and directly goes to the location where Position is to serve as a relay node for subsequent communication. For l and m, they can both first receive the message that they are selected from the communication with their predecessor nodes, and leave the exploration team to go to the position of the relay node only after sending the message to their successor robots. To ensure the communication loop, it is also necessary to modify the communication events of the predecessor robot j of the messenger and the successor robot n of m. j originally had and , and now it becomes and . n originally had and , and now new communication events are added at the back, becoming , , , represents the communication event of robot m communicating with robot m, represents the communication event of robot n communicating with robot o, represents the communication event of robot j communicating with robot n. Briefly understood, the circular network topology composed of "-i-j-k-l-m-n-o-i-" in the current exploration team is split into: a circular network topology composed of "-i-j-n-o-" and another chain network topology composed of robots l, k, and m.

[0105] In an optional embodiment, the human side is further configured to confirm the task information of the human-assisted task and send the task information to the robot communicating with the human side on the communication relay link, where the task information includes the task type;

[0106] If the robot communicating with the human side is the end robot, the robot communicating with the human side receives the task information and executes the human-assisted task based on the task information; if the robot communicating with the human side is not the end robot, the robot communicating with the human side continues to transmit the task information in the direction from the human side to the end robot until the task information is sent to the end robot, and the end robot receives the task information and executes the human-assisted task based on the task information.

[0107] Specifically, the human terminal can provide a human-machine interface for the operator, and the human terminal can obtain task information through this human-machine interface. When a human-assisted task starts, the human terminal will ask the operator about the required task type through the human-machine interface. For example: (1) An inspection task, which requires the end robot of the link to turn on the camera to capture audio and video of the surrounding environment and targets; (2) An operation task, which requires the end robot of the link to receive the remote control instruction from the operator and perform corresponding actions.

[0108] In an alternative embodiment, after the end robot executes the human-assisted task and obtains the task result, if the human terminal communicates with the end robot, the end robot will send the task result to the human terminal; if the human terminal does not communicate with the end robot, the end robot will send the task result to the robot that communicates with the end robot, and the robot that communicates with the human terminal will continue to transmit the task result in the direction from the end robot to the human terminal until the task result is sent to the human terminal.

[0109] The task result may specifically include the audio and video stream obtained by capturing the surrounding environment, or may also include the result after performing corresponding actions according to the remote operation issued by the human terminal.

[0110] Generally speaking, that is, according to the communication relay link and the information transmission direction, the interaction between the human terminal and the end robot is realized, where the information transmission direction specifically includes the above-mentioned from the end robot to the human terminal or from the human terminal to the end robot.

[0111] In an alternative embodiment, the human terminal is further configured to receive the new position selected by the operator; re-plan the task and update the communication plans of each robot according to the task location, the new position, and the communication plans of multiple robots performing map exploration in the current ring topology, so as to obtain the communication relay link and the updated communication plans of each robot.

[0112] In the actual application process, it supports the operator to move from the current position to another position.

[0113] For example, the human terminal initially determines that 5 robots including the messenger will be relay robots on the later communication relay link for human-assisted tasks. The human terminal shows the planning result to the operator and asks the operator whether to choose to move.

[0114] The operator believes that the communication relay link of the five robots is too long. As a result, only two robots remain in the exploration team, which will affect the exploration efficiency. Therefore, the operator decides to move to a place closer to the task location to work. The operator clicks on the working position they want to go to on the map (i.e., the new position selected by the operator as mentioned above). The human terminal then re-plans the communication relay link based on the planned task location and the working position selected by the operator. Since the operator is closer to the task location, only three robots are needed for the communication relay link this time. The human terminal will re-plan the communication events for the three robots including the messenger and display the planning results to the operator for viewing.

[0115] In the above Figure 3 After completing the calculation of the communication relay link and the task planning, the planning results are displayed to the operator, and the operator is asked whether to choose to move. The planning results can specifically include information such as the positions of the robots on the communication relay link and the communication times between the nodes (including the human terminal and the robots) on the communication relay link.

[0116] Among them, the reason for asking the operator whether to choose to move is as follows: In the collaborative work between humans and multiple robots, the operator also has their own task requirements. Sometimes, the working position needs to be changed, and at this time, the communication relay link needs to be re-planned. On the other hand, if the operator chooses to move to a position closer to the target point (i.e., the task location), fewer relay robots are required in the human-assisted task, and more robots can stay in the "exploration" mode, which can improve the overall exploration efficiency. Therefore, allowing the operator to move can make the planning process more flexible and obtain more satisfactory results.

[0117] In an alternative embodiment, the human terminal is further configured to determine the estimated completion time of the current human-assisted task; if the estimated completion time is less than a predefined threshold, it is determined whether there are any unprocessed human-assisted tasks; if so, a link transfer plan is made, and if not, a link disbandment plan is made.

[0118] Among them, the predefined threshold can be preset or calculated on the spot based on the efficiency according to the communication situation.

[0119] The planning of link transfer can be carried out through the following process: the human side is specifically used to determine the operator position corresponding to the unprocessed human-assisted task and the new task location; based on the operator position and the new task location, determine the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, then use the relay nodes in the current communication relay link as the relay nodes of the new communication relay link; if the number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task, then select the first target robot from the robots used for map exploration, and the sum of the number of the first target robots and the number of relay nodes in the current communication relay link reaches the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task, then select the second target robot from the relay nodes in the current communication relay link as the relay nodes of the new communication relay link.

[0120] Among them, the relay node can be understood as the robot on the communication relay link.

[0121] As Figure 4A shown, the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, both are 3. This situation can also be understood as the situation where the number of nodes in the new and old links is equal. Then, the relay nodes (robots l, k, m) in the current communication relay link can be directly used as the relay nodes of the new communication relay link.

[0122] As Figure 4B shown, the number of relay nodes in the current communication relay link is 3, while the number of relay nodes required for the new task is 4. The number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task. This situation can also be understood as the situation where the number of nodes in the new link is more. Then, select the first target robot from the robots used for map exploration, such as robot j. It can also be understood as borrowing robot j from the exploration team and jointly forming the relay nodes of the new communication relay link with the relay nodes (robots l, k, m) in the current communication relay link.

[0123] As Figure 4C shown, the number of relay nodes in the current communication relay link is 4, while the number of relay nodes required for the new task is 3. The number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task. This situation can also be understood as the situation where the number of nodes in the new link is less. Then, select the second target robot (robots l, k, m) from the relay nodes in the current communication relay link as the relay nodes of the new communication relay link. It can also be understood as returning robot j among the relay nodes in the current communication relay link to the exploration team.

[0124] Plan for link disassembly, which may specifically include the following processes:

[0125] The human terminal is specifically used to determine the operator's location corresponding to the unprocessed human-assisted tasks and the new task location; based on the operator's location and the new task location, determine the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, then use the relay nodes in the current communication relay link as the relay nodes of the new communication relay link; if the number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task, then select the first target robot from the robots used for map exploration, and the sum of the number of the first target robots and the number of relay nodes in the current communication relay link reaches the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task, then select the second target robot from the relay nodes in the current communication relay link as the relay nodes of the new communication relay link.

[0126] The human terminal is specifically used to obtain the third communication plan from the second messenger robot, and the third communication plan is the new communication plan of multiple robots currently conducting map exploration; based on the third communication plan, form a new ring topology structure with multiple robots currently conducting map exploration and the robots in the current communication relay link, and update the communication plans of the robots in the current communication relay link to obtain the fourth communication plan, and send the topology information of the new ring topology structure and the fourth communication plan to the second messenger robot and the robots in the current communication relay link;

[0127] The second messenger robot is used to locally update the topology information and communication plan of the new ring topology structure after receiving the topology information of the new ring topology structure and the fourth communication plan sent by the human terminal; according to the communication plan of the second messenger robot in the fourth communication plan, arrive at the fourth communication location at the fourth communication time, and send the topology information of the new ring topology structure and the fourth communication plan to the second successor robot of the second messenger robot in the new ring topology structure;

[0128] The second successor robot is used to locally update the topology information and communication plan of the new ring topology structure, and send the topology information of the new ring topology structure and the fourth communication plan to the subsequent robots of the second successor robot;

[0129] The robots in the current communication relay link are used to locally update the topology information and communication plan of the new ring topology structure, and arrive at the agreed communication location according to the fourth communication plan;

[0130] Each robot in the newly formed ring topology structure is used for map exploration.

[0131] Among them, every other human-machine communication interval has a robot returning to the human side. Both the second messenger robot and the above-mentioned first messenger robot are robots that return to the human side within the human-machine communication interval, and they can be the same robot or different robots. For example, the human-machine communication interval is 300 seconds, the first messenger robot is the robot that returns to the human side every 300 seconds, and the second messenger robot is the robot that returns to the human side every 600 seconds.

[0132] The second successor robot is the successor robot of the second messenger robot, that is, the robot that communicates with the second messenger robot according to the new ring topology structure, that is, the robot located after the second messenger robot according to the communication sequence of the fourth communication plan.

[0133] The subsequent robot of the second successor robot, that is, the robot that communicates with the second successor robot according to the new ring topology structure, that is, the robot located after the second successor robot according to the communication sequence of the fourth communication plan.

[0134] After the second messenger robot arrives at the fourth communication location and sends the topology information of the new ring topology structure and the fourth communication plan to the second successor robot, the messenger identity of the second messenger robot is removed.

[0135] The third communication plan is the new communication plan for multiple robots currently conducting map exploration, and the fourth communication plan is the new communication plan obtained by updating the communication plan of the robots in the current communication relay link.

[0136] In this embodiment, the above process executed by the human side, the second messenger robot, and the robots in the current communication relay link can also be understood as a mode switch from "relay" to "exploration", and a process of re-establishing a ring topology structure among the robots.

[0137] During the process of establishing the ring topology structure, the robots that already know the fourth communication plan arrive at the communication location according to the communication time in the fourth communication plan, and the robots that do not yet know the fourth communication plan follow the communication time and communication location in the third communication plan.

[0138] In Example 1, as Figure 5A shown, the current communication environment includes two independent links: the current communication relay link (including Robot 5, Robot 6, and Robot 7) and the ring topology structure composed of multiple robots currently conducting map exploration (including Robot 1, Robot 2, Robot 3, and Robot 4), and the second messenger robot is Robot 1. In this embodiment, it is necessary to insert the robots in the current communication relay link into the ring topology structure to obtain a new ring topology structure, as Figure 5AAs shown on the right, the new ring topology includes Robot 1, Robot 2, Robot 6, Robot 5, Robot 7, Robot 3, and Robot 4 in the communication order. For the convenience of description, in this embodiment, the current communication relay link can be called a direct link, the ring topology composed of multiple robots currently performing map exploration can be called a small ring, and the new ring topology to be formed can be called a large ring.

[0139] In the description of this embodiment, the exploration process between communication events is ignored, and only how the robots re - establish the topology of the large ring is described.

[0140] In this case, it can be understood that the direct link is inserted at the middle position of the ring topology.

[0141] It can be observed that the fourth communication plan and the third communication plan are partially consistent, that is, the arrow A in both represents the original communication events of the small ring and does not need to be changed. Therefore, the communication events represented by arrow A are the same in the third communication plan and the fourth communication plan.

[0142] 1) On the one hand, Robot 1 (the second messenger robot) arrives at the appointed place at the appointed time according to the fourth communication plan and communicates with Robot 2. Robot 2 arrives at the appointed place at the appointed time according to the third communication plan and communicates with Robot 1. In this step, the third communication plan and the fourth communication plan are consistent, so both can successfully meet. After that, Robot 1 waits for Robot 4 to communicate according to the fourth communication plan.

[0143] 2) After Robot 2 communicates with Robot 1, it already knows the fourth communication plan, so it arrives at the appointed place at the appointed time according to the fourth communication plan and waits for Robot 6 to communicate.

[0144] 3) On the other hand, after the tasks of Robots 5 - Robot 6 - Robot 7 on the direct link are completed (Note: They do not necessarily start at the same time as Robot 1 acting as the messenger. They have to wait until the above - mentioned human - assisted tasks are completed. The waiting time here is the estimated completion time), they arrive at the appointed place at the appointed time according to the fourth communication plan. Specifically, Robot 6 waits for Robot 2 to communicate, Robot 5 waits for Robot 6, and Robot 7 waits for Robot 5.

[0145] 4) So according to the description in 2) and 3), Robot 2 and Robot 6 will meet and communicate according to the fourth communication plan. After the end, Robot 6 will continue to wait for Robot 5 according to the fourth communication plan. And 3) mentioned that Robot 5 is waiting for Robot 6, so Robot 6 and Robot 5 will also meet and communicate according to the fourth communication plan. After the end, Robot 5 will continue to wait for Robot 7 according to the fourth communication plan, and 3) mentioned that Robot 7 is waiting for Robot 5, so Robot 5 and Robot 7 will also meet and communicate according to the fourth communication plan.

[0146] 5) After 4) is finished, Robot 7 goes to wait for Robot 3 according to the fourth communication plan. It should be noted that the communication time and location between Robot 7 and Robot 3 in the fourth communication plan are the same as the communication time and location between Robot 3 and Robot 2 in the third communication plan. This is the key point. Robot 3 still arrives at the agreed place at the agreed time to wait for Robot 2 according to the third communication plan, but it is Robot 7 that is waiting. Robot 7 will communicate with Robot 3 in the name of Robot 2, and then tell Robot 2 about the fourth communication plan during the communication process.

[0147] 6) After 5) is finished, robot 3 waits for robot 4 according to the fourth communication plan, and robot 4 waits for robot 3 according to the third communication plan. In this step, the third communication plan and the fourth communication plan are consistent, so both can successfully meet. After the end, robot 4 waits for robot 1 according to the fourth communication plan, and according to the description in 1), robot 1 is waiting for robot 4, so they can communicate successfully. At this point, all robots in the big ring know the fourth communication plan, and the communication topology has successfully changed to a big ring.

[0148] In Example 2, if Figure 5B As shown, the straight link is directly inserted into the beginning of the small loop, so the key lies in the communication between robot 7 and robot 2. This step is similar to 5) in Example 1.

[0149] In Example 3, Figure 5C As shown, the straight link is directly inserted into the end of the small ring, so at the time when the straight link is added, the robots in the original small ring already know the fourth communication plan.

[0150] When it is necessary to dismiss the robot from the communication relay link and return it to the exploration team, the link dismissal process is involved, such as Figure 6 The specific contents are as follows:

[0151] The human terminal obtains the latest communication plan of the robots in the exploration team from the information brought back by the messenger. The human terminal formulates a suitable "return to the team" plan for all the relay robots to be disbanded. Generally speaking, it is to find a suitable communication event in the current plan of the exploration team, and then let the robots to be disbanded reach the communication location before this communication event occurs, and insert these robots into the circular communication topology of the exploration team, so that they can be re-inserted into the communication team. Specifically, it is divided into the following steps:

[0152] i) The human terminal will query the latest plan of the exploration team (including the time and location of a series of communication events) from the information sent back by the messenger, and then calculate the communication time for each robot to be disbanded to move from the current position to the communication locations of each communication event after completing the human-assisted task.

[0153] ii) If all the robots to be disbanded can reach the communication location where the communication event occurs before a certain communication event, record it as a candidate communication event.

[0154] iii) Finally, find the earliest-occurring communication event from the candidate communication events. That is to say, if the currently disbanded robots set off immediately after completing the human-assisted task and head to the location where the communication event occurs, all the robots can reach before the scheduled occurrence time of this communication event. After finding the target communication event, obtain the communication topology order of each robot according to the time sequence of the robots reaching the target communication event, and plan the corresponding communication events.

[0155] For example, in Figure 6 , the target communication event is . The robots that reach the location where the target communication event occurs are l, k, m in sequence. Then, the originally planned for robots i and j should be deleted, add to robot i, and add to robot j. For robot l, its predecessor robot is i and its successor robot is k. Then add and . Similarly, add and to robot k, and add and to robot m. Simply understood, new communication edges are added. After the planning is completed, the human terminal transmits the above "return to the team" plan to the messenger.

[0156] In the embodiments of the present invention, multiple robots are used to perform autonomous exploration tasks and human-assisted collaboration tasks of distributed cyclic multi-rounds between humans and multiple machines in a complex environment with communication restrictions. Each robot has two modes: "exploration" and "relay".

[0157] In the "exploration" mode, the robot will explore the environment in a distributed and cyclic manner for multiple rounds according to the pattern of "-communication-exploration-communication-exploration-" (when information is transmitted in a circular communication topology for one round, it is called "one round"). Each robot is equipped with an ad-hoc networking device and communicates pairwise through a circular communication topology structure to agree on the next plan (communication time and communication location), thus ensuring the timeliness of communication. The schematic diagram of its communication topology is as shown Figure 2 on the left side. The arrows in the figure indicate the occurrence order of pairwise communication events.

[0158] In the "relay" mode, the robot will form a high-quality communication relay link with the human side to perform human-assisted tasks. The schematic diagram of the communication topology of the communication relay link is as shown Figure 2 on the right side. Regarding n as the end robot and q as the human side, the operator interacts with the end robot n through the human side q. At this time, the double arrows no longer represent the occurrence order of communication events, but emphasize that data can be transmitted bidirectionally through the two robots o and p in the middle of the link.

[0159] Therefore, in the overall group of collaborative robots, there are two teams, namely the "exploration" team and the "relay" team. The communication topology of the robots in the "exploration" team is a circular structure, and information is transmitted and updated in the circular team. While the "relay" team is a linear structure, with the human side at the head of the team and the working robot at the end of the team, which performs human-assisted tasks such as transmitting video and audio data to the human side and receiving control instructions from the human side to execute tasks, etc. The middle of the team is the robot acting as a relay, which is responsible for maintaining the communication relay link and data transmission. Each robot can switch between the "exploration" and "relay" modes according to the distributed algorithm and the operator's requirements to meet various complex requirements of human-multi-robot cooperation. In terms of human-robot interaction, every certain period of time, a robot will "return" to the human side and communicate with the operator. At this time, the operator can obtain the latest global environment map, the robot's plan, photos of suspicious areas, etc.

[0160] According to their own needs, the operator may request (1) no special treatment and continue the original task; (2) prioritize exploring certain areas; (3) the operator moves to a certain area by himself; (4) request to form a relay link and perform human-assisted tasks in a certain area.

[0161] For the first type of task, return to the communication location of the current round plan. For the second type of task, return to the communication location of the current round plan and incorporate the areas to be explored preferentially into the subsequent robot planning for priority consideration; for the third type of task, return to the communication location of the current round plan and update the position of the human side.

[0162] In the embodiments of the present invention, the implementation of the fourth task is mainly completed. Based on the existing multi-robot exploration limited to the interaction between multiple robots and the operator, a new mode, namely the "relay" mode, is added according to actual needs to perform human-assisted tasks. It mainly includes the following parts:

[0163] A. When the robot is in the "exploration" mode, it performs the following steps:

[0164] A1. The robot arrives at the agreed location at the specified time to form a circular communication topology structure.

[0165] A2. The robots communicate with each other pairwise. For example, robot i and robot j exchange information including the local environmental map established, the plan of the current robot passed by the previous robot (the communication time and location between the two robots), photos of suspicious areas, etc.

[0166] A3. Plan the exploration path and agree on the next plan for the two robots (i, j). When planning, consider the time difference between the next communication meeting time and the previous human-machine meeting time. If this time difference is greater than the pre-set human-machine communication interval time, discard the current planning result and perform re-planning so that robot j returns to meet the operator before the next communication event with robot i, and perform human-machine interaction, then enter step C1. If this time difference is less than the pre-set human-machine communication interval time, continue to step A4.

[0167] A4. After the planning is completed, the robot completes the communication process and conducts map exploration along the planned path. Use radar to collect map data, use cameras and image recognition algorithms to capture environmental features, and determine whether it is a suspicious area. After completing this round of exploration, return to step A1.

[0168] B. When the robot is in the "relay" mode, it performs the following steps:

[0169] B1. The robot arrives at the agreed location at the specified time and becomes a node in the communication relay link. At this time, the robot waits in place for all the robots on the communication relay link or the operator with the human terminal to take their positions. After the last robot on the communication relay link and the operator are in place, the following human-assisted tasks are started.

[0170] B2. If it is the robot at the end of the link (such as Figure 2As shown on the right, for robot n), the following operations are performed according to human requirements: (1) Use a camera to capture the surrounding environment and the target, and transmit the video and audio back to the previous relay node (such as the previous robot o of robot n); (2) Receive the remote control instructions from the human terminal and perform actions such as moving or grasping (if the end robot is a mobile manipulator). If it is not the robot at the end of the link (such as robot p), it is responsible for information transmission. According to the direction of information transfer, receive information from the previous node (which may be a robot or the human terminal depending on the specific situation) and transmit it to the subsequent node.

[0171] B3. Receive the end instruction transmitted from the human terminal and perform the following actions according to the actual situation: (1) Switch to the next human-assisted task, go to the next location, and return to step B1; (2) End the "relay" mode, enter the mode switching state, and enter step F3.

[0172] C. When the messenger in the "exploration" mode returns to the human terminal, human-machine interaction will occur (which can also be understood as when there is no ongoing human-assisted task, the interaction between the human terminal and the messenger and between the human terminal and the operator at this time), and the following steps are executed:

[0173] C1. The robot returns to the human terminal within the pre-set human-machine communication interval. This robot that returns to the human terminal is called a "messenger", that is, the above-mentioned messenger robot. The messenger sends data such as the environmental map, the plans of all current robots transmitted by the previous robot (the communication time and location between two robots), and photos of suspicious areas to the human terminal. And wait for the operator's interaction request.

[0174] C2. The human terminal obtains the data transmitted by the messenger and observes information such as the fused overall map, the plans of all current robots (the communication time and location between two robots), and photos of suspicious areas on the display screen.

[0175] C3. The human terminal supports the processing of 4 types of interaction requests according to the operator's self-requirements. Among them, the first 3 types of interaction requests are: (1) Do not perform special processing and let the robot continue its own task; (2) Specify the priority exploration area and let the robot give priority to exploring this area in subsequent planning; (3) Specify the area to go to, and the robot calculates the reference movement path of the operator.

[0176] C4. The messenger obtains the request from the human terminal and performs the following corresponding actions according to the content of the first 3 requests described in C3: (1) Return to the communication location of this round of plan and return to step A1; (2) Return to the communication location of this round of plan, incorporate the priority exploration area into the subsequent robot planning for priority consideration, and return to step A1; (3) Return to the communication location of this round of plan, update the location of the human terminal, and return to step A1.

[0177] C5. The fourth type of interaction request received by the human side from the operator is a human-assisted task request: (4) The operator requests the robot to form a high-quality communication relay link for a certain area on the map and perform human-assisted collaborative tasks in a certain area.

[0178] C6. Then, based on the planned task location, the human side conducts task planning and allocation. According to the next communication event of the robot in the current plan and the communication quality model in the environment, a high-quality communication relay link is planned, the relevant communication events are modified, the planning results are displayed to the operator, and the operator is asked whether to choose to move.

[0179] Specifically:

[0180] i) The human side determines the task location and the current position of the human side on the currently known fused map.

[0181] ii) Subsequently, referring to the communication quality model in the current environment (which can be obtained according to empirical formulas or learned through artificial intelligence algorithms), a search algorithm is used to obtain a high-quality communication relay link that connects the human side and the task location through multiple relay robots. The head of the link is the human side, and the end of the link is the end robot. While ensuring that the communication quality between adjacent nodes on the link is greater than the minimum threshold, the number of required robots is minimized as much as possible.

[0182] iii) After obtaining the communication relay link, the human side arranges a sufficient number of robots from the exploration team as nodes on the communication relay link. Suppose the communication relay link requires x robots. Then, starting from the messenger robot, a total of x robots including the messenger are sequentially selected backward according to the order of its communication topology as nodes of the communication relay link.

[0183] iv) For the messenger, it has originally planned 2 communication events in the subsequent exploration tasks. At this time, it is necessary to retain the first communication event in the original plan, delete the second communication event, and add a special communication event for the relay task; for the subsequent x - 1 selected robots, a third special communication event for the relay task needs to be added after the originally planned 2 communication events. To keep the original circular communication topology in a loop, it is also necessary to establish a communication relationship between the original predecessor robot of the messenger and the last selected relay robot before and after. Therefore, the corresponding communication events of these two robots also need to be modified. The position of the special communication event mentioned above is set at the position of the relay node, and the time is calculated according to the time required for the robot to reach the relay node after the last communication. For example, as shown above Figure 3 shown.

[0184] C7. If the operator needs to move to a new position, the operator will select the new position on the human interface, and the human interface will re-plan the communication relay link to obtain the final communication relay link.

[0185] C8. The human interface will publish the finally determined human-assisted task information, along with the new communication plan and topology information, such as the allocation results of the communication relay link and other information related to human-assisted tasks, the communication plan of each robot after planning and updating, and the new communication topology, to the messenger.

[0186] C9. The messenger receives the human-assisted task published by the human interface, enters the mode switching state, and proceeds to step E1.

[0187] D. When the human interface is performing a human-assisted task, the robot communicating with the human interface at this time will have human-machine interactions (interactions between the human interface and the operator, interactions between the human interface and the robots on the communication relay link, etc.). The following steps are executed:

[0188] D1. At the start of the human-assisted task, the human interface will ask the operator about the type of task required through the human-machine interface. For example: (1) Inspection task, requiring the robot at the end of the link to turn on the camera to capture audio and video of the surrounding environment and targets; (2) Operation task, requiring the robot at the end of the link to receive the remote control instructions from the operator and perform corresponding actions.

[0189] D2. After the operator determines the task type, the human interface will transmit the information to the first robot on the communication relay link.

[0190] D3. The human interface receives the real-time information (such as audio and video) transmitted by the first robot and displays it to the operator. At the same time, it receives the remote control instructions from the operator and transmits them to the first robot on the link.

[0191] D4. During the period when the human interface is performing a human-assisted task, if a messenger (from the exploration team) returns to the human interface, the human interface will first execute and complete the human-machine interaction process C1 - C4 of the "exploration" team according to the steps in C. Since the operator is currently in the process of a human-assisted task, there are only two interaction methods, (1) and (2), in step C3, and after executing the interaction content of (1) and (2) in C4, the messenger does not enter A1 but waits for the instruction from the human interface.

[0192] D5. The human interface asks the operator about the current human-assisted task and the estimated completion time. If the operator estimates that the completion time is too long or cannot be estimated yet (greater than a certain threshold, and this threshold is used as the boundary point for whether the link can be disassembled later. The threshold can be determined according to actual needs or experience), then no link processing will be performed in this round, and the messenger returns to step A1. If the estimated time is not too long (0 or less than a certain value), then the human interface will enter the link processing process D6.

[0193] D6. After the human terminal enters the link processing process, it first determines whether there are still undone human-assisted tasks in the current task pool. If there are, it plans the link transfer according to the operator's selection (select an undone human-assisted task) and enters step D7; if not or the operator refuses to continue with the human-assisted task, it plans the link disbandment and enters step D11.

[0194] D7. The human terminal first asks the operator to select the next operator position, then generates a new link based on the position of the new human-assisted task and the operator's position, and then compares the information of the current link and the new link.

[0195] D8. If the number of relays on the current link is equal to the number of relays required on the new link, directly allocate the current relay robots to the new relays, calculate the time-optimal allocation, and complete the planning. After the current human-assisted task is completed, the current link will be transferred to the new link according to the allocation result, and the messenger returns to step A1; if the number of relays on the current link is less than the number of relays required on the new link, sufficient robots still need to be "borrowed" from the exploration team in subsequent steps and enter step D9; if the number of relays on the current link is greater than the number of relays required on the new link, only allocate the number of robots required for the new link, calculate the time-optimal allocation, and these robots will be transferred to the new link according to the allocation result after the current task is completed. The extra robots will be "returned" to the exploration team in subsequent steps and enter step D11.

[0196] D9. The human terminal plans the robots required for the "borrowing" event, communication events (including arrival time and arrival location), and communication topology changes. This planning process is similar to the process of generating a relay link in C6, which is equivalent to generating a new human-assisted task. The human terminal publishes the completed plan to the messenger.

[0197] D10. The messenger receives the "borrowing" requirement from the human terminal and enters the mode switching state, entering step E1.

[0198] D11. The human terminal obtains the latest plan of the robots in the exploration team from the information brought back by the messenger.

[0199] D12. The human side plans a suitable "return to the team" plan for all relay robots to be dismissed. Generally speaking, it is to find a suitable communication event in the current plan of the exploration team, then let the robots to be dismissed reach the communication location before this communication event occurs, and insert these robots into the circular communication topology of the exploration team, so that they can be re-inserted into the communication team. After the planning is completed, the human side transmits the above "return to the team" plan to the messenger and the robots on the communication relay link respectively. Among them, the "return to the team" plan can specifically include the topological information of the above new circular topology and the fourth communication plan.

[0200] Specifically, it is divided into the following steps:

[0201] i) The human side will query the latest plan of the exploration team (including the time and location of a series of communication events) from the information returned by the messenger, and then calculate the time for all robots to be dismissed to move from their current positions to each communication event location after completing the human-assisted tasks.

[0202] ii) If all robots to be dismissed can reach the location of a certain communication event before the occurrence of this communication event, then record it as a candidate communication event.

[0203] iii) Find the earliest-occurring communication event from the candidate communication events. That is to say, if the current robots to be dismissed set off immediately after completing the human-assisted tasks and head to the location of this communication event, then all robots can reach before the scheduled occurrence time of this communication event. After finding the target communication event, according to the time sequence of the robots reaching the target communication event, obtain the communication topology order of each robot and plan the corresponding communication events. Make the robots to return to the team (the robots currently on the communication relay link, and these robots are to return to the circular network topology for map exploration) set up communication events according to their positions in the new circular communication topology.

[0204] For example, in Figure 6 , the target communication event is , and the robots arriving at the location of the target communication event are l, k, m in sequence. Then, the originally planned of robot i and robot j should be deleted, add to robot i, and add to robot j. For robot l, its predecessor robot is i and its successor robot is k, then add and . Similarly, add and to robot k, and add and to robot m.After the planning is completed, the human side will transmit the above "return to the team" plan to the messenger. After the planning is completed, the human side will transmit the "return to the team" plan to the messenger.

[0205] D13. The messenger receives the "return to the team" plan from the human side and proceeds to step F1.

[0206] D6 - D13 can be understood as the link processing process. Among them, D7 - D10 are the planning of link transfer, and D11 - D13 are the planning of link dissolution.

[0207] E. When switching from the "exploration" mode to the "relay" mode, the following steps are executed:

[0208] E1. The messenger receives the request for the human - assisted task issued by the human side, updates its own and other robots' communication plans and topology information, and returns to the communication location in the original exploration team according to the plan.

[0209] E2. The messenger meets with the successor robot, transmits the information of the human - assisted task and the updated communication plan and topology information to the successor robot, then switches to the "relay" mode, and enters B1 according to the time and location planned in C8.

[0210] E3. After the successor robot of the messenger obtains the information of the human - assisted task of the messenger and the updated communication plan and topology information, it updates its own and other robots' communication plans and topology information, and transmits the relevant information to the successor robot. At the same time, it judges whether it is the relay robot planned in C8. If it is the relay robot, it switches to the "relay" mode and enters B1 according to the time and location planned in C8. If it is not the relay robot, it continues with the original task.

[0211] E4. After the information of the human - assisted task and the updated communication plan and topology information are transmitted once in the exploration team, all the robots in the exploration team know this information. At this time, the communication topology has changed, and some robots in the exploration team are assigned to the "relay" team to form a communication relay link.

[0212] F. When switching from the "relay" mode to the "exploration" mode, the following steps are executed:

[0213] F1. After the messenger receives the return - to - team request information issued by the human side, it updates the communication event, changes the stored communication topology information, changes the communication order of the exploration team, forms a new ring - shaped communication topology, and returns to the communication location of the new ring - shaped communication topology according to the new communication plan. The communication location agreed upon by the new communication plan is the same as the communication location of the messenger in the original exploration team. Then it transmits the return - to - team request information to the successor node. The return - to - team request information can also be understood as the above "return to the team" plan.

[0214] The return request is a request triggered when the human operator ends the human-assisted task.

[0215] F2. The successor node reads the return request information, also changes the stored communication topology information, changes the communication order of the exploration team, and forms a new ring-shaped communication topology. And it passes this information to the successor node.

[0216] F3. After receiving the return request information issued by the human operator, the relay robot to be disbanded updates the communication event, changes the stored communication topology information, changes the communication order of the exploration team, and forms a new ring-shaped communication topology. And it returns to the communication location of the new ring-shaped communication topology according to the new communication plan.

[0217] F4. The disbanded relay robot (the robot on the communication relay link) will arrive at the communication location at the agreed communication time, communicate with the previous robot (for a relay robot, the previous robot of this relay robot refers to the robot that is located before the relay robot according to the re-planned communication order) and the subsequent robot (for a relay robot, the subsequent robot of this relay robot refers to the robot that is located after the relay robot according to the re-planned communication order), and then switches to the "exploration" mode. When the last relay robot successfully returns to the team, the mode switching of all robots is completed, and a new exploration team is generated. When the information is transmitted once in the exploration team, all the robots in the exploration team know the new communication topology order.

[0218] Among them, step F3 has no chronological relationship with F1 and F2 and can be executed independently.

[0219] The embodiments of the present invention consider communication limitations and can be applied to more complex multi-robot exploration tasks in large environments. It proposes a brand-new collaborative working mode between humans and multi-robots, which can form a stable communication relay link in complex geographical and communication environments to meet actual needs. And, centered around humans, it considers the importance of the operator's role, takes into account the complex needs of humans, utilizes the advantages of humans in emergency handling and overall planning, strengthens the operator's supervision and control over the multi-robot exploration task, improves the system robustness and task completion rate. Also, it provides the exploration priority order to meet more complex task requirements. In addition, it also points out the feasible area for the operator to move, improving the operator's safety. And since the operator is movable, compared with the fixed communication base station method, the exploration range is significantly expanded.

[0220] Specifically, when the robot is in the "exploration" mode, it may include the following process:

[0221] The robot arrives at the predetermined location at the specified time to form a ring network topology structure;

[0222] Robots communicate with each other pairwise, exchanging information such as environmental maps and positions;

[0223] Plan the exploration path and agree on the time and ground for the next communication meeting between the two, that is, the communication time and communication location for pairwise communication between robots;

[0224] Judge whether the meeting time is greater than the human-machine timestamp;

[0225] If so, re-plan the next communication event, insert a human-side interaction event before the next communication, and then the robot and the human side interact. Specifically, as Figure 7 shown; if not, the robot performs distributed multi-robot map exploration according to the planned path and judges whether the exploration task and the assistance task are completed.

[0226] In the embodiment of the present invention, the interaction process between the human side and the robot is as Figure 7 shown:

[0227] The messenger starts to return to the human side;

[0228] The messenger communicates with the human side and waits for the interaction request from the human side;

[0229] The operator holds the electronic device and obtains the transmitted information;

[0230] The operator issues an interaction request;

[0231] The interaction options may include: (1) Do not perform special processing and let the messenger continue its own task; (2) Specify the subsequent priority exploration area; (3) The operator specifies the area to go to, and the human side calculates the safe feasible area; (4) The operator requests to perform a human-assisted collaboration task in a certain area, that is, the above-mentioned human-assisted task.

[0232] For the first, second, and third types of interaction operations, the messenger returns to perform the exploration task. For the fourth type of interaction option, the following process is executed:

[0233] The operator selects the human-assisted task location, that is, the above-mentioned task location;

[0234] The human side obtains the communication relay link node according to the operator's position and the task location (that is, Figure 7 the communication link node in

[0235] ), and completes the planning and allocation; The operator selects whether to move; if the operator selects to move to a new position, return to the step where the human side obtains the communication relay link node according to the operator's position and the task location and completes the planning and allocation; if the operator does not select to move to a new position, the human side publishes the final result to the messenger, and then the messenger returns to the exploration team and enters the mode switching process from "exploration" to "relay".

[0236] In the embodiment of the present invention, the workflow in the relay mode is as Figure 8 shown, including:

[0237] The operator / human terminal and the relevant robot reach the positions on the communication relay link (which can also be called the relay link);

[0238] The end robot and the human terminal establish a real-time connection through the communication relay link;

[0239] The operator / human terminal issues an inspection task or an operation task to the remote robot;

[0240] The end robot and the operator / human terminal transmit information such as audio and video (video and audio information), control signals, etc. through the communication relay link;

[0241] Judge whether the human terminal initiates link processing;

[0242] If so, perform the link processing process. Specifically, the link processing process is as Figure 9 shown; if not, return to the step of issuing the inspection task or the operation task;

[0243] In addition, the messenger returns to the human terminal; the operator / human terminal and the messenger complete the human-machine interaction of the "exploration" team; the human terminal asks the operator about the estimated completion time of the human-assisted task; if the time is too long or cannot be estimated, the link processing is abandoned in this round and the messenger returns to the exploration team; if the time is short, such as less than a certain threshold, enter the above step of judging whether the human terminal initiates link processing.

[0244] In the embodiment of the present invention, the link processing flow is as Figure 9 shown:

[0245] The link processing process starts;

[0246] Judge whether there are undone human-assisted tasks;

[0247] If not, the link starts to be disbanded, and the messenger returns to the exploration team while information is transmitted;

[0248] If so, judge whether to switch to the next human-assisted task;

[0249] If not switched to the next human-assisted task, enter the step of starting to disband the link; if switched to the next human-assisted task, execute the task location where the operator selects the next human-assisted task;

[0250] After the operator selects the task location of the next human-assisted task, compare the number of new link relay nodes and the number of old link relay nodes;

[0251] If the two are equal, directly perform link transfer planning, and then the messenger returns to the exploration team while transmitting new link information;

[0252] If the number of relay nodes in the new link is less than that in the old link, perform link transfer planning by "returning" robots to the exploration team. Moreover, the robots to be disbanded start to "rejoin the team", as specifically shown in Figure 6 the figure, and the messenger returns to the exploration team and transmits the "rejoin the team" information;

[0253] If the number of relay nodes in the new link is greater than that in the old link, perform link transfer planning by "borrowing" robots from the exploration team. Moreover, new human-assisted task planning is carried out at the human side, as specifically shown in Figure 6 the figure, and the messenger returns to the exploration team and enters the mode switching process from "exploration" to "relay", transmitting the "borrowing" information;

[0254] In this way, the robots on the original link (old link) start to transfer according to the plan, as specifically shown in Figure 4A 、 Figure 4B and Figure 4C the figure.

[0255] The following uses a specific example to elaborate in detail on the human-machine collaborative work system provided by the embodiment of the present invention.

[0256] Suppose 1 operator holds an electronic device (i.e., the above-mentioned human side) and intends to use 7 robots (numbered i, j, k, l, m, n, o) to perform human-machine collaborative tasks in an unknown cave environment. In such an environment, there is no communication signal covering the entire area, and due to terrain limitations, when the robots are far apart, the communication quality will be very poor or even incommunicable. Therefore, only ad-hoc communication can be carried out between the robots and the human side through wireless ad-hoc network devices. In this scenario, the behavior of the robots in the exploration mode and the overall process of human-machine interaction are as shown in Figure 10 the figure.

[0257] At the moment when the task starts, neither the operator nor the robots have any map data. They judge the initial relative positions through initial communication and establish an initial map. At the same time, the human side sets the human-machine communication interval , assuming , which means that every 300 seconds, at least one robot needs to return to the human side as a messenger for communication and respond to the interaction requirements of the human side. Subsequently, the robots start the first round of planning. According to the current map, the robots assign different robots to go to different map "boundaries" for exploration tasks, and according to the circular communication topology (refer to Figure 2 the left side of the figure), agree on the next round of communication events between the robots in pairs, including the communication time and communication location.

[0258] After the first-round planning is completed, each robot starts the exploration task according to the planning result. Each robot explores the boundary of the map assigned to it and updates its own map in real time during the exploration. If a robot meets other robots during the exploration, it will communicate quickly, exchange data such as maps, and fuse the maps locally. After completing the exploration process, the robot arrives at the communication location on time according to the previous agreement and communicates with the next robot in the order of the ring communication topology.

[0259] Take the communication event between robot i and robot j as an example. When they communicate pairwise, robot i only needs to perform one segment of planning, while robot j needs to perform two segments of planning. This is because in the ring communication topology, robot i needs to communicate with robot o and robot j. By the time it communicates with robot j, it has already completed the communication with robot o and planned the time, location of the next communication, and the path planning of the exploration task. Therefore, in this communication, robot i only needs to consider how to meet robot j after that. Robot j needs to first perform the first segment of path planning between time and and then consider the second segment of path planning between time and Since the human-robot communication interval is set previously, it is also necessary to determine whether the next human-robot communication is during the second segment of path planning. If so, the second segment of path replanning is required. Before the communication between robot i and j, robot j needs to go to the human side as a messenger first for communication and interaction; if not, robot j does not need to return to the human side and can directly perform the second segment of path planning. These several times of planning can all be implemented through the idea of the greedy algorithm. Refer to the content in the related technology, and the embodiments of the present invention do not limit this.

[0260] According to the foregoing setting, every 300 seconds, at least one robot needs to return to the human side as a messenger for communication and respond to the interaction requirements of the human side. After the messenger returns to the human side, it sends data such as the fused environmental map to the human side and waits for the interaction request from the human side. At this time, the human side (the electronic device held by the operator) can receive and display the map and other data returned by the robot and perform map fusion locally. Thus, the human side can observe information such as the fused environmental map, environmental photos, and photos of suspicious areas on the display screen.

[0261] The operator makes upper-level decisions based on his own needs and the overall situation. At this time, the operator can issue four types of interaction requests: (1) No special processing, let the robot continue its task. At this time, the robot will directly complete the interaction with the human end and enter the next round of exploration tasks. (2) Specify the priority exploration area, which is the area of ​​interest to the human end. After receiving the instruction, the robot returns to the next round of exploration tasks and gradually transmits the human end's exploration needs for the priority area to other robots in subsequent communication events. (3) Request the operator's feasible movement domain. The operator issues an interaction request for the feasible movement domain to let the robot calculate the current safe area where humans can move. The operator then specifies the destination to go to in the safe area, and the robot will calculate the operator's reference movement path. (4) Request to form a relay link to perform human-assisted collaborative tasks in a certain area.

[0262] For the first three interaction modes, after the messenger completes the interaction with the human end, the robot returns to the previously planned task and enters the execution of the next round of exploration tasks. For the fourth interaction mode, detailed description will be given later.

[0263] At this point, the robot has completed a cycle of "-exploration-communication-(interaction)-", in which the "interaction" stage is not required in each cycle, but is determined by the interaction time set by the human end (determined based on the human-machine communication interval). Each robot conducts collaborative exploration of the environment in a distributed manner through the "-exploration-communication-(interaction)-" cycle until the environment map is explored. Figure 10 As shown in the figure, multiple robots work in a distributed manner according to the planned tasks. Robots i, m, n, and o are performing exploration tasks, robots k and l are communicating, and robot j is returning to the human end and is about to send the local map back to the operator. The operator can update his own map in a timely manner and put forward his own interaction needs by constantly interacting with the returning robots.

[0264] For the fourth interaction mode, the embodiment of the present invention is implemented in the following specific manner:

[0265] The original cave scene is still used for introduction. The overall process is as follows Figure 11 shown.

[0266] When the robot is exploring according to the above process, it often finds some suspicious areas. Such areas require the operator to intervene for real-time audio and video monitoring and also require manual operation of the robot for tasks. However, due to the special terrain, narrow passages or dangers in such areas, it is not suitable for the operator to enter the area for operation, so only remote control can be carried out. Due to the communication limitations in the cave scenario, such a long distance cannot support communication, so only through the relay of multiple robots can a communication relay link be formed between the human end and the remote robot to meet the above requirements.

[0267] The robot records the photos and locations of the suspicious areas and transmits them via communication to other robots. When the messenger robot returns to the human end, the human end can receive the information of the suspicious areas. At this time, the operator can initiate the 4th type of human-robot interaction through the human end: request to form a communication relay link to perform human-assisted collaboration tasks in the suspicious areas.

[0268] The human end will, based on the planned task location and the current position of the human end, and according to the communication quality model in the current environment, plan a high-quality communication relay link. Suppose this communication relay link requires 5 relay nodes. Then starting from the messenger robot, in the communication order of the exploration team, 4 successor robots are found in sequence, and their communication events are re-planned. So that each selected robot can first receive the message that it has been selected and can only leave the exploration team to move to the position of the relay node after transmitting the message to the successor robot. In this way, 5 robots including the messenger will serve as relay robots on the subsequent communication relay link to perform human-assisted tasks. The human end shows the planning result to the operator and asks the operator whether to choose to move.

[0269] For example, the operator believes that the communication relay link of 5 robots is too long, leaving only 2 robots in the exploration team, which will affect the exploration efficiency. So the operator decides to move to a place closer to the task location to work. The operator clicks on the working position on the map that they want to go to, and the human end re-plans the communication relay link based on the planned task location and the working position selected by the operator. Since the operator is closer to the task location, this communication relay link only requires 3 robots. The human end will re-plan the communication events for 3 robots including the messenger and show the planning result to the operator for viewing.

[0270] The operator can see on the interaction interface of the human end: audio-visual information, map information, joystick and other information, such as the above planning results.

[0271] The communication relay link can include robot k (the remote robot, also known as the end robot), robot l, and robot j, and is used to complete human-assisted tasks such as transmitting video and audio or remote control instructions. Robots i, m, n, and o cooperate to conduct map exploration.

[0272] The operator was satisfied with the result of this link planning and thus ended this human-machine interaction. The human side sent the planning result and the new communication plan to the messenger. The messenger received the request for the human-assisted task issued by the human side, updated its own and other robots' communication plans, and then returned to the communication location in the original exploration team according to the plan, communicated with the subsequent robots, transmitted the information of the human-assisted task and the updated communication plan to the subsequent robots, and finally directly went to the node position assigned to it on the communication relay link. Similarly, all the selected subsequent robots first obtained the information of the human-assisted task and the updated communication plan from their previous robots, then transmitted them to their subsequent robots, and finally directly went to the node position assigned to them on the communication relay link. As a result, all the selected robots in the original exploration team would finally reach the corresponding positions on the communication relay link, while the unselected robots only transmitted the information of the human-assisted task and the information of the updated communication plan until the information was transmitted once in the exploration team and all the robots knew this information before stopping the transmission. At this time, the communication topology had changed, and some robots in the exploration team were assigned to the "relay" team to form a link, and the original ring communication topology became the updated ring topology. This process is as Figure 3 shown.

[0273] When all the relay robots and the operator are in place, the communication relay link has been established. The operator can establish communication with the remote robot through the human side to perform human-assisted tasks. The operator intends to send an instruction to the end robot to capture the surrounding environment. This instruction is forwarded step by step along the relay link through the human side to the end robot. After receiving it, the end robot uses the camera to capture the surrounding environment and the target. For example, robot k captures the surrounding environment and forwards the video and audio back to the human side step by step along the relay link. The human side displays the audio and video on the interaction interface. The operator observes the audio and video information and initiates a remote operation request. After receiving this request, the end robot starts to receive remote control instructions. Subsequently, the operator can, according to the interaction interface of the human side, while observing the surrounding environment and the target object in real time, use buttons such as the virtual joystick on the control interface to remotely control the end robot to make it perform actions such as moving or grasping to complete the predetermined task.

[0274] During the execution of the human-assisted task, the exploration team continues to explore the map in parallel. Every predetermined human-machine communication event T, a messenger robot returns to the human side for communication to exchange the latest map information, communication plan, photos of new suspicious areas, etc. According to the above description of the multi-level collaborative exploration process, the operator can interact with the messenger through the human side. The difference is that the operator is performing a human-assisted task, so there is no need to move and there is no need to initiate the human-assisted task again. Therefore, there are only two interaction methods: (1) and (2). After the operator issues an instruction of (1) or (2), the messenger does not immediately return to the exploration team but waits for the next instruction from the human side.

[0275] The human side asks the operator about the current human-assisted task and the estimated completion time. If the estimated completion time by the operator is too long or cannot be estimated yet (greater than a certain threshold, which is used as the boundary point for whether link disconnection can be performed subsequently), then no link processing is carried out in this round and the messenger returns to the exploration team. If the estimated time is not too long (0 or less than a certain value), then the human side will enter the link processing process.

[0276] The human side first determines whether the messenger has discovered a new human-assisted task and asks the operator to make a decision. The operator can choose one of the remaining human-assisted tasks or refuse to continue performing the human-assisted task. If the operator selects a task, then the link transfer process is entered; if the operator refuses to continue the task, then the link disconnection process is entered.

[0277] The link transfer process is as Figure 4A 、 Figure 4B and Figure 4CAs shown below: The human terminal first allows the operator to select the next operator position, then generates a new link based on the position of the new human-assisted task and the operator's position, and subsequently compares the information of the current link and the new link. There are three cases: (1) If the number of relays on the current link is equal to the number of relays required on the new link, the current relay robots are directly assigned to the new relays, and the time-optimal assignment is calculated to complete the planning. After the current human-assisted task is completed, the current link will be transferred to the new link according to the assignment result. This case does not involve the exchange of robots between the relay team and the exploration team, and the messenger will directly return to the exploration team; (2) If the number of relays on the current link is less than the number of relays required on the new link, a sufficient number of robots still need to be "borrowed" from the exploration team in subsequent steps. The human terminal plans the robots, communication events (including arrival time and arrival position), and communication topology changes required to complete the "borrowing" event according to the communication plan of the exploration team given by the messenger. This planning process is similar to the process of generating the relay link described above, which is equivalent to generating a new human-assisted task. The human terminal publishes the planned result to the messenger. The messenger receives the "borrowing" requirement from the human terminal and enters the mode switching state. The subsequent steps are the same as described above. The messenger will transfer the human-assisted task information, new communication events, and communication topology information to the robots in the exploration team. When the information is transmitted once, the communication topology changes, and a sufficient number of individuals are assigned to the robots in the exploration team and "borrowed" to the predetermined positions on the new relay link; (3) If the number of relays on the current link is greater than the number of relays required on the new link, only the number of robots required for the new link is assigned, and the time-optimal assignment is calculated. After the current task is completed, these robots will be transferred to the new link according to the assignment result. The redundant robots will be "dismissed" back to the exploration team in subsequent steps.

[0278] Specifically, the link dismissal process is as described above Figure 6 as shown.

[0279] After the messenger receives the return request information released by the recipient end, it updates the communication event, changes the stored communication topology information, changes the communication order of the exploration team, and forms a new ring-shaped communication topology. Then it returns to the communication location in the original exploration team as planned. Here, it actually means: returning to the communication location of the new ring-shaped communication topology according to the new communication plan, and the communication location agreed upon by the new communication plan is the same as the communication location of the messenger in the original exploration team. Subsequently, it passes the return request information, the new communication event, and the communication topology backward to the successor node. The successor node reads the return information, also updates the communication event, changes the stored communication topology information, changes the communication order of the exploration team, forms a new ring-shaped communication topology, and passes this information to the successor node. The dispatched relay robot will arrive at the communication location at the agreed communication time, communicate with the previous robot and the subsequent robot, and then switch to the "exploration" mode. When the last relay robot successfully returns to the team, the mode switching of all robots is completed, and a new exploration team is generated. When the information is transmitted once in the exploration team, all the robots in the exploration team know the new communication topology order.

[0280] Through the above content, a human-centered system for human-multi-robot collaborative work under communication constraints is completely realized.

[0281] The embodiment of the present invention also provides a human-machine collaborative work method, which is applied to the human-machine collaborative work system in the above embodiment. The human-machine collaborative work system includes: a human end and multiple robots. The human-machine collaborative work method includes:

[0282] Through the human end, receive a human-assisted task request including the task location; according to the task location, the current location of the human end, and the first communication plan, obtain the link information and send it to the first messenger robot. The link information includes the topology information of the communication relay link and the second communication plan; the first communication plan is the communication plan of each robot for map exploration in the current ring topology structure, including the first communication time and the first communication location for communication between two robots.

[0283] Through the first messenger robot, send the link information to the first successor robot according to the current communication order, and arrive at the second communication location at the agreed second communication time.

[0284] Through the first successor robot and the robots located after the first successor robot, transmit the link information, and respectively judge whether they are robots on the communication relay link according to the link information. If so, arrive at the second communication location at the agreed second communication time; if not, continue to perform map exploration; some of the multiple robots for map exploration in the current ring topology structure serve as nodes of the communication relay link.

[0285] Execute human-assisted tasks through the end robot at the end of the communication relay link;

[0286] Robots other than the end robot in the communication relay link receive data sent by the previous node and transmit the received data to the subsequent node.

[0287] An embodiment of the present invention also provides a robot, including a plurality of robots included in the human-machine collaborative work system in the above embodiment.

[0288] An embodiment of the present invention also provides a human terminal, including the human terminal included in the human-machine collaborative work system in the above embodiment.

[0289] In another embodiment provided by the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above human-machine collaborative work methods are implemented.

[0290] In another embodiment provided by the present invention, a computer program product including instructions is also provided. When it runs on a computer, it causes the computer to execute any of the human-machine collaborative work methods in the above embodiments.

[0291] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a semiconductor medium (for example, solid state disk (SSD)).

[0292] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0293] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the embodiments of methods, robots, human terminals, computer storage media, and computer program products, since they are basically similar to the system embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the system embodiments for the relevant content.

[0294] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A human-machine collaborative work system, characterized in that: include: Human side and multiple robots; The human end is used to receive a human-assisted task request including a task location; According to the task location, the current location of the human terminal and the first communication plan, link information is obtained and sent to the first messenger robot, wherein the link information includes topology information of the communication relay link and the second communication plan; the first communication plan is the communication plan of each robot performing map exploration in the current ring topology structure, including a first communication time and a first communication location for communication between two robots; The first messenger robot is used to send the link information to the first successor robot according to the current communication sequence, and arrive at the second communication location at the agreed second communication time; The first successor robot and the robot behind the first successor robot are used to transmit the link information, and determine whether they are robots on the communication relay link according to the link information, and if so, arrive at the second communication location at the agreed second communication time; if not, continue to perform map exploration; some of the multiple robots performing map exploration in the current ring topology structure serve as nodes of the communication relay link; The terminal robot at the end of the communication relay link is used to perform the human-assisted task; The robots other than the terminal robot in the communication relay link are used to receive data sent by the preceding node and transmit the received data to the subsequent node.

2. The system according to claim 1, characterized in that The first messenger robot is specifically used to receive the link information sent by the human end, wherein the first messenger robot is a robot that returns to the human end within the human-machine communication interval; based on the link information, locally updates the communication plan and topology information of itself and the first other robot stored in itself, the first other robot being a robot other than the first messenger robot among multiple robots performing map exploration in the current ring topology structure; moves to a first communication location in the current ring topology structure that communicates with a first successor robot, and sends the link information to the first successor robot, and arrives at a second communication location at an agreed second communication time according to the communication plan of the first messenger robot in the second communication plan; the first successor robot is a robot that is located after the first messenger robot in the current ring topology structure according to the current communication order; The first successor robot is specifically used to locally update the communication plan and topology information stored in itself and the second other robot. The second other robot is a robot other than the first successor robot among multiple robots exploring the map in the current ring topology structure, and sends the link information to the robot that is located after the first successor robot in the current ring topology structure according to the current communication order. The robot that is located after the first successor robot performs the same operation as the first successor robot until all robots in the current ring topology structure have obtained the topology information of the communication relay link and the second communication plan.

3. The system according to claim 1, characterized in that The human end is further used to confirm the task information of the human-assisted task and send the task information to the robot communicating with the human end on the communication relay link, wherein the task information includes a task type; If the robot communicating with the human end is a terminal robot, the robot communicating with the human end receives the task information and performs the human-assisted task based on the task information; if the robot communicating with the human end is not a terminal robot, the robot communicating with the human end continues to transmit the task information in the direction from the human end to the terminal robot until the task information is sent to the terminal robot, and the terminal robot receives the task information and performs the human-assisted task based on the task information.

4. The system according to claim 3, characterized in that: After the terminal robot performs the human-assisted task and obtains a task result, if the terminal robot is communicating with the human end, the terminal robot sends the task result to the human end; If the terminal robot is not communicating with the human end, the terminal robot sends the task result to the robot communicating with the terminal robot, and the robot communicating with the human end continues to transmit the task result along the direction from the terminal robot to the human end until the task result is sent to the human end.

5. The system according to claim 1, characterized in that The human end is also used to receive a new location selected by the operator; based on the task location, the new location and the communication plans of multiple robots performing map exploration in the current ring topology structure, re-plan the task and update the communication plans of each robot to obtain a communication relay link and an updated communication plan of each robot.

6. The system according to claim 1, characterized in that The human end is further used to determine the estimated completion time of the current human-assisted task; if the estimated completion time is less than the agreed threshold, determine whether there are any unprocessed human-assisted tasks; If yes, then plan for link transfer; if no, then plan for link demobilization.

7. The system according to claim 6, characterized in that The human end is specifically used to determine the operator position and the new task location corresponding to the unprocessed human-assisted task; based on the operator position and the new task location, determine the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, then the relay nodes in the current communication relay link are used as relay nodes for the new communication relay link; if the number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task, then select a first target robot from the robots used for map exploration, and the sum of the number of the first target robots and the number of relay nodes in the current communication relay link reaches the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task, then select a second target robot from the relay nodes in the current communication relay link as the relay node of the new communication relay link.

8. The system according to claim 6, characterized in that The human end is specifically used to obtain a third communication plan from the second messenger robot, where the third communication plan is a new communication plan for multiple robots currently performing map exploration; based on the third communication plan, the multiple robots currently performing map exploration and the robots in the current communication relay link are combined into a new ring topology structure, and the communication plan of the robots in the current communication relay link is updated to obtain a fourth communication plan, and the topology information of the new ring topology structure and the fourth communication plan are sent to the second messenger robot and the robots in the current communication relay link; The second messenger robot is used to locally update the topology information of the new ring topology structure and the fourth communication plan after receiving the topology information of the new ring topology structure and the fourth communication plan sent by the human end; According to the communication plan of the second courier robot in the fourth communication plan, arrive at a fourth communication location at a fourth communication time, and send topology information of a new ring topology structure and the fourth communication plan to a second successor robot of the second courier robot in the new ring topology structure; The second successor robot is used to locally update the topology information of the new ring topology structure and the fourth communication plan, and send the topology information of the new ring topology structure and the fourth communication plan to the subsequent robot of the second successor robot; The robot in the current communication relay link is used to locally update the topology information of the new ring topology structure and the fourth communication plan, and arrive at the agreed communication location according to the fourth communication plan; The robots that form the new ring topology are used to explore the map.

9. A human-machine collaborative working method, characterized in that: The human-machine collaborative work system according to any one of claims 1 to 8 comprises: a human end and a plurality of robots, and the method comprises: Receiving a human-assisted task request including a task location through the human terminal; obtaining link information according to the task location, the current location of the human terminal and a first communication plan and sending it to a first messenger robot, wherein the link information includes topology information of a communication relay link and a second communication plan; the first communication plan is a communication plan for each robot performing map exploration in the current ring topology structure, including a first communication time and a first communication location for communication between two robots; The link information is sent to the first successor robot through the first messenger robot in accordance with the current communication order, and arrives at the second communication location at the agreed second communication time; The link information is transmitted through the first successor robot and the robot behind the first successor robot, and each robot determines whether it is a robot on the communication relay link according to the link information, and if so, arrives at the second communication location at the agreed second communication time; if not, continues to perform map exploration; some robots among the multiple robots performing map exploration in the current ring topology structure serve as nodes of the communication relay link; Performing the human-assisted task through an end robot at an end of the communication relay link; The robots other than the terminal robot in the communication relay link receive data sent by the preceding node and transmit the received data to the subsequent node.

10. A robot, characterized in that: Includes multiple robots included in the human-machine collaborative work system as described in any one of claims 1 to 8.

11. A human terminal, characterized in that: The human end included in the human-machine collaborative work system according to any one of claims 1 to 8.

Images