A human-machine collaborative work system, method, robot and human terminal

Abstract

The embodiments of the present invention provide a human-machine collaborative work system, method, robot and human end, which relate to the field of robot technology, including: by selecting some robots from multiple robots performing map exploration to form a communication relay link for realizing human-assisted tasks, a part of the robots can perform distributed multi-machine collaborative exploration, and the other part of the robots form a stable communication relay link to perform human-assisted tasks, and each robot in the communication relay link can return to the exploration team to continue the map exploration task after completing the human-assisted task. Compared with the solution that can only realize multi-machine collaborative exploration, the embodiment of the present invention can make the two working modes of multi-machine collaborative exploration and human-machine collaborative execution of human-assisted tasks coexist, and can switch back and forth between the two working modes. In this way, not only collaborative exploration can be carried out in a communication-restricted environment, but also human-assisted tasks can be carried out, which can better meet user needs.

Description

Technical Field

[0001] The present invention relates to the field of robotics technology, and in particular to a human-machine collaborative working system, method, robot and human end. Background Art

[0002] In the field of robotics, related technologies often overlook the limitations of inter-robot communication, assuming full connectivity. In reality, in typical scenarios for underground exploration, reconnaissance, and search and rescue missions, such as caves, ruins, and mountain forests, inter-robot communication is often limited due to the vast and complex terrain and numerous obstacles. This poses a challenge to achieving these tasks in practical applications. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide a human-machine collaborative work system, method, robot, and human terminal to achieve not only collaborative exploration in communication-restricted environments, but also human-assisted tasks, which can better meet user needs. The specific technical solution is as follows:

[0004] In a first aspect, a human-robot collaborative working system is provided, comprising: a human terminal and multiple robots;

[0005] The human terminal is configured to receive a human-assisted task request including a task location; obtain link information based on the task location, the current location of the human terminal, and a first communication plan, and transmit the information to a first messenger robot, wherein the link information includes topology information of a communication relay link and a second communication plan; the first communication plan is a communication plan for each robot performing map exploration in the current ring topology, including a first communication time and a first communication location for communication between two robots;

[0006] The first messenger robot is configured to send the link information to the first successor robot according to the current communication order and arrive at the second communication location at the agreed second communication time;

[0007] The first successor robot and the robot following the first successor robot are configured to transmit the link information and, based on the link information, determine whether they are robots on the communication relay link. If so, they arrive at the second communication location at the agreed second communication time; if not, they continue to perform map exploration; some of the multiple robots performing map exploration in the current ring topology serve as nodes of the communication relay link;

[0008] The end robot at the end of the communication relay link is used to perform the human-assisted task;

[0009] The robots other than the terminal robot in the communication relay link are used to receive data sent by the preceding node and transmit the received data to the subsequent node.

[0010] Optionally, the first messenger robot is specifically used to receive the link information sent by the human end, wherein the first messenger robot is a robot that returns to the human end within the human-machine communication interval; based on the link information, locally updates its own stored communication plan and topology information of itself and a first other robot, wherein the first other robot is a robot other than the first messenger robot among multiple robots performing map exploration in the current ring topology structure; moves to a first communication location in the current ring topology structure that communicates with a first successor robot, and sends the link information to the first successor robot, and arrives at the second communication location at the agreed second communication time according to the communication plan of the first messenger robot in the second communication plan; the first successor robot is a robot that is located after the first messenger robot in the current communication order in the current ring topology structure;

[0011] The first successor robot is specifically used to locally update its own stored communication plan and topology information of itself and a second other robot. The second other robot is a robot other than the first successor robot among multiple robots exploring the map in the current ring topology structure, and sends the link information to the robot that is located after the first successor robot in the current ring topology structure according to the current communication order. The robot that is located after the first successor robot performs the same operation as the first successor robot until all robots in the current ring topology structure have obtained the topology information of the communication relay link and the second communication plan.

[0012] Optionally, the human end is further configured to confirm task information of the human-assisted task and send the task information to the robot communicating with the human end on the communication relay link, wherein the task information includes a task type;

[0013] If the robot communicating with the human end is an end robot, the robot communicating with the human end receives the task information and performs the human-assisted task based on the task information; if the robot communicating with the human end is not an end robot, the robot communicating with the human end continues to transmit the task information in the direction from the human end to the end robot until the task information is sent to the end robot, and the end robot receives the task information and performs the human-assisted task based on the task information.

[0014] Optionally, after the terminal robot performs the human-assisted task and obtains the task result, if the terminal robot is communicating with the human end, the terminal robot sends the task result to the human end; if the terminal robot is not communicating with the human end, the terminal robot sends the task result to the robot communicating with the terminal robot, and the robot communicating with the human end continues to transmit the task result along the direction from the terminal robot to the human end until the task result is sent to the human end.

[0015] Optionally, the human end is also used to receive a new location selected by the operator; re-plan the task and update the communication plan of each robot based on the task location, the new location and the communication plans of multiple robots exploring the map in the current ring topology structure, and obtain a communication relay link and an updated communication plan of each robot.

[0016] Optionally, the human end is also used to determine the estimated completion time of the current human-assisted task; if the estimated completion time is less than the agreed threshold, it is determined whether there are any unprocessed human-assisted tasks; if so, link transfer planning is performed; if not, link dismissal planning is performed.

[0017] Optionally, the human end is specifically used to determine the operator position and the new task location corresponding to the unprocessed human-assisted task; based on the operator position and the new task location, determine the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, then the relay nodes in the current communication relay link are used as relay nodes of the new communication relay link; if the number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task, then select a first target robot from the robots used for map exploration, and the sum of the number of the first target robots and the number of relay nodes in the current communication relay link reaches the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task, then select a second target robot from the relay nodes in the current communication relay link as the relay node of the new communication relay link.

[0018] Optionally, the human end is specifically configured to obtain a third communication plan from the second messenger robot, where the third communication plan is a new communication plan for the multiple robots currently performing map exploration; based on the third communication plan, the multiple robots currently performing map exploration and the robots in the current communication relay link are formed into a new ring topology structure, and the communication plans of the robots in the current communication relay link are updated to obtain a fourth communication plan; and the topology information of the new ring topology structure and the fourth communication plan are sent to the second messenger robot and the robots in the current communication relay link;

[0019] The second courier robot is configured to, after receiving the topology information of the new ring topology structure and the fourth communication plan sent by the human terminal, locally update the topology information and the communication plan of the new ring topology structure; arrive at a fourth communication location at a fourth communication time according to the communication plan of the second courier robot in the fourth communication plan, and send the topology information of the new ring topology structure and the fourth communication plan to a second successor robot of the second courier robot in the new ring topology structure;

[0020] The second successor robot is used for topology information of the new ring topology structure and the fourth communication plan, and sends the topology information of the new ring topology structure and the fourth communication plan to a subsequent robot of the second successor robot;

[0021] The robot in the current communication relay link is configured to locally update the topology information of the new ring topology structure and the fourth communication plan, and arrive at the agreed communication location according to the fourth communication plan;

[0022] The robots in the new ring topology are used to explore the map.

[0023] In a second aspect, a human-machine collaborative work method is provided, which is applied to the human-machine collaborative work system described in any one of the first aspects, wherein the human-machine collaborative work system includes: a human end and multiple robots, and the method includes:

[0024] Receiving, via the human terminal, a human-assisted task request including a task location; obtaining link information based on the task location, the current location of the human terminal, and a first communication plan, and sending the information to a first messenger robot, the link information including topology information of a communication relay link and a second communication plan; the first communication plan being a communication plan for each robot performing map exploration in the current ring topology, including a first communication time and a first communication location for communication between two robots;

[0025] The link information is sent to the first successor robot via the first messenger robot in accordance with the current communication order, and arrives at the second communication location at the agreed second communication time;

[0026] The link information is transmitted through the first successor robot and the robot located after the first successor robot. Based on the link information, each robot determines whether it is a robot on the communication relay link. If so, it arrives at the second communication location at the agreed second communication time; if not, it continues to perform map exploration; some of the multiple robots performing map exploration in the current ring topology structure serve as nodes of the communication relay link;

[0027] Performing the human-assisted task by an end robot at an end of the communication relay link;

[0028] The data sent by the preceding node is received by the robots other than the terminal robot in the communication relay link, and the received data is transmitted to the subsequent nodes.

[0029] In a third aspect, a robot is provided, comprising the multiple robots included in the human-machine collaborative work system described in any one of the first aspects.

[0030] In a fourth aspect, a human end is provided, comprising the human end included in the human-machine collaborative work system described in any one of the first aspects.

[0031] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of the method described in the second aspect are implemented.

[0032] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned human-computer collaborative working methods.

[0033] Beneficial effects of the embodiments of the present invention:

[0034] In the human-machine collaborative work system provided by an embodiment of the present invention, a communication relay link is formed by selecting some robots from multiple robots performing map exploration in the current ring topology structure. The topology information of the communication relay link and the second communication plan are first sent to each robot in the current ring topology structure based on the current ring topology structure. Then, the robot selected as a node on the communication relay link arrives at the second communication location at the agreed second communication time, forming a communication relay link from the human end to the terminal robot in a communication-restricted environment. The communication relay link can be used to realize interaction between the human end and the terminal robot in the communication-restricted environment to complete human-assisted tasks. In addition, the robot arrives at the agreed communication location at the agreed communication time to communicate, realizing communication in a communication-restricted environment. In this way, not only collaborative exploration can be carried out in a communication-restricted environment, but also human-assisted tasks can be carried out, which can better meet user needs.

[0035] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0037] Figure 1 A schematic diagram of a human-machine collaborative working system is provided for an embodiment of the present invention;

[0038] Figure 2 Schematic diagram of communication topology in an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of switching from exploration mode to relay mode in an embodiment of the present invention;

[0040] Figure 4A A schematic diagram of link transfer in an embodiment of the present invention;

[0041] Figure 4B Another schematic diagram of link transfer in an embodiment of the present invention;

[0042] Figure 4C This is another schematic diagram of link transfer in an embodiment of the present invention;

[0043] Figure 5A A schematic diagram of re-establishing a ring topology according to an embodiment of the present invention;

[0044] Figure 5B Another schematic diagram of re-establishing a ring topology according to an embodiment of the present invention;

[0045] Figure 5C Another schematic diagram of re-establishing a ring topology according to an embodiment of the present invention;

[0046] Figure 6 Schematic diagram of switching from relay mode to exploration mode in an embodiment of the present invention;

[0047] Figure 7 Schematic diagram of the interaction between a human terminal and a robot in an embodiment of the present invention;

[0048] Figure 8 Schematic diagram of a robot in relay mode according to an embodiment of the present invention;

[0049] Figure 9 Schematic diagram of a link processing flow in an embodiment of the present invention;

[0050] Figure 10 This is an overall diagram of the robot working only in "exploration" mode;

[0051] Figure 11 This is an overall schematic diagram of an embodiment of the present invention when a robot works in the "relay" mode. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention are within the scope of protection of the present invention.

[0053] Reference Figure 1 , an embodiment of the present invention provides a human-machine collaborative work system, comprising: a human terminal and multiple robots;

[0054] The human end is configured to receive a human-assisted task request including a task location; obtain link information based on the task location, the current location of the human end, and a first communication plan, and transmit the information to the first messenger robot. The link information includes topology information of the communication relay link and a second communication plan. The first communication plan is a communication plan for each robot performing map exploration in the current ring topology, including a first communication time and a first communication location for communication between two robots.

[0055] The first messenger robot is configured to send link information to the first successor robot according to the current communication sequence and arrive at the second communication location at the agreed second communication time;

[0056] The first successor robot and the robot following the first successor robot are used to transmit link information and, based on the link information, each determine whether it is a robot on the communication relay link. If so, it arrives at the second communication location at the agreed second communication time; if not, it continues to perform map exploration. Some of the multiple robots performing map exploration in the current ring topology serve as nodes of the communication relay link.

[0057] The terminal robot at the end of the communication relay link is used to perform human-assisted tasks;

[0058] The robots other than the terminal robot in the communication relay link are used to receive data sent by the preceding node and transmit the received data to the subsequent node.

[0059] In an embodiment of the present invention, a communication relay link is formed by selecting some robots from multiple robots performing map exploration in the current ring topology structure. The topology information of the communication relay link and the second communication plan are first sent to each robot in the current ring topology structure based on the current ring topology structure. Then, the robot selected as a node on the communication relay link arrives at the second communication location at the agreed second communication time, forming a communication relay link from the human end to the terminal robot in the communication restricted environment. The communication relay link can be used to realize interaction between the human end and the terminal robot in the communication restricted environment to complete human-assisted tasks. In addition, the robot arrives at the agreed communication location at the agreed communication time to communicate, realizing communication in the communication restricted environment. In this way, not only collaborative exploration can be carried out in the communication restricted environment, but also human-assisted tasks can be carried out, which can better meet user needs.

[0060] The human-assisted task request may be a request triggered by an operator. The human-assisted task request is used to request the establishment of a communication relay link for a target area on a map to perform a human-assisted task in the target area, which is also a task location.

[0061] The first messenger robot returns to the human side and sends the latest information in the current ring topology to the human side.

[0062] The human terminal performs task planning and updates the first communication plan according to the task location, the current location of the human terminal and the first communication plan, and obtains topology information of the communication relay link and the second communication plan.

[0063] The first messenger robot sends link information to the first successor robot according to the current communication order. Specifically, the information may include the following:

[0064] The first messenger robot is specifically used to receive link information sent by the human end, wherein the first messenger robot is a robot that returns to the human end within the human-machine communication interval; based on the link information, locally updates its own stored communication plan and topology information of itself and the first other robot, the first other robot being a robot other than the first messenger robot among multiple robots performing map exploration in the current ring topology structure; moves to a first communication location in the current ring topology structure that communicates with the first successor robot, and sends the link information to the first successor robot, and arrives at the second communication location at the agreed second communication time according to the communication plan of the first messenger robot in the second communication plan; the first successor robot is a robot that is located after the first messenger robot in the current ring topology structure according to the current communication order;

[0065] The first successor robot and the robots following the first successor robot are used to transmit link information and determine whether they are robots on the communication relay link based on the link information.

[0066] Among them, the first successor robot and the robot located after the first successor robot are used to transmit link information. Specifically, they can complete the originally agreed exploration task after obtaining the link information transmitted by the preceding robot, and then transmit the link information when communicating with their respective successor robots.

[0067] The first successor robot is specifically used to locally update its own stored communication plan and topology information of itself and the second other robot. The second other robot is a robot other than the first successor robot among the multiple robots exploring the map in the current ring topology structure, and sends the link information to the robot that is located after the first successor robot in the current ring topology structure according to the current communication order. The robot that is located after the first successor robot performs the same operation as the first successor robot until all robots in the current ring topology structure have obtained the topology information of the communication relay link and the second communication plan.

[0068] Among them, the robot located after the first successor robot according to the current communication order in the current ring topology structure can also be understood as other robots except the first messenger robot and the first successor robot.

[0069] The first successor robot and the robot located after the first successor robot, when determining that they are robots on the communication relay link, arrive at the second communication location at the agreed second communication time according to their respective communication plans in the second communication plan.

[0070] The robots other than the terminal robot in the communication relay link receive data sent by the preceding node and transmit the received data to the subsequent node. It can also be understood that the robots other than the terminal robot in the communication relay link are used for information transmission.

[0071] In the field of robotics, autonomous exploration refers to a robot's ability to perceive and move in an unknown environment in real time, relying on its own sensors and onboard processors to acquire environmental information from the unknown map. Autonomous robots are required for positioning, exploration, and mapping tasks such as underground exploration, reconnaissance, and search and rescue. These tasks typically require working in environments such as caves, ruins, and forests. These environments are often difficult to reach due to the complex terrain and numerous obstacles.

[0072] A common robotic exploration approach is frontier-based exploration, which guides exploration by identifying and utilizing the so-called "frontier." In this context, the "frontier" refers to the boundary between known and unknown areas, or the transition zone between them. This exploration approach allows robots to minimize the length of their exploration paths while ensuring efficient exploration. Simultaneous Localization and Mapping (SLAM) technology allows robots to incrementally build maps in unknown environments. The robot's initial position is unknown. During movement, it localizes itself based on its position and map, then incrementally builds a map, ultimately achieving autonomous localization and navigation. Due to the large scale and complex terrain of these tasks, the efficiency of a single robot is too low. Using multi-robot collaborative exploration can significantly improve efficiency. As the name suggests, multi-robot collaborative exploration involves multiple robots exchanging information and collaborating through a communication network, thereby dividing the work to explore the entire environment map. However, in extreme environments, such as caves and underground tunnels, communication between robots is severely limited, posing challenges for collaborative exploration.

[0073] In related technologies, there are solutions for distributed independent exploration by multiple robots, but in this solution, the robots cannot communicate with each other in real time; there are also solutions for distributed multi-UAV exploration, but this solution does not consider inter-machine communication, or assumes that the communication network is fully connected and ideal.

[0074] As can be seen, related technologies often overlook communication constraints between robots, assuming full connectivity. In reality, in typical scenarios such as underground exploration, reconnaissance, and search and rescue, robot communication is often limited, requiring only short, unobstructed distances to exchange data using ad hoc networks. Therefore, communication constraints should be considered in these multi-machine communication environments.

[0075] Furthermore, none of these technologies consider the interaction between robots and operators, neglecting the operator's reasonable needs and coordination capabilities during collaborative work, as well as the operator's ability to handle unexpected situations. The operator's role is crucial in human-machine collaboration tasks.

[0076] Related technologies often focus solely on maximizing exploration efficiency while neglecting the interaction between robots and operators, ignoring the crucial role of operators in the collaborative work of humans and multiple robots. In reality, operators have many legitimate needs, such as: 1. Keeping up-to-date on exploration progress and the status of the robot swarm; 2. Prioritizing specific areas or requiring robots to focus on specific areas based on transmitted maps and photos; and 3. If unexpected situations occur (such as a robot becoming stuck in a narrow area and unable to retreat autonomously, or the operator discovering a hidden, suspicious area unknown to the robot), the operator can make reasonable decisions and request intervention to assist the robot's work.

[0077] For example, for the second type of task mentioned above, the operator may need to focus on a specific area for real-time monitoring, requiring the robot to transmit real-time audio and video streams. If necessary, the operator may need to intervene in the robot's work process to provide human assistance and remotely operate the robot. This requires the operator to send real-time control commands to the remote robot. A prerequisite for fulfilling these requirements is a stable, high-quality communication relay link between the operator and the remote robot, capable of transmitting real-time audio and video streams and control signals. Furthermore, in complex or dangerous environments such as caves and disaster zones, it is difficult for humans to physically reach the remote robot. Therefore, multiple robots can only work in a "relay" manner, forming a stable, high-quality communication relay link with multiple robots acting as "relay" nodes to meet these requirements.

[0078] In addition, after a human-assisted task is completed, if there are no other human-assisted tasks, the communication relay link will be disbanded and the robot will switch from "relay" mode back to "exploration" mode; if there are multiple human-assisted tasks, the communication relay link can be directly transferred between multiple tasks without the need for mode switching.

[0079] In environments such as caves, ruins, and mountain forests, the vast and complex terrain and numerous obstacles often render communication signals unavailable, forcing robots to rely on wireless ad-hoc network technology. A wireless ad-hoc network is a multi-hop, mobile, peer-to-peer network composed of multiple nodes that utilizes wireless communication and supports dynamic networking. It lacks a central control center; all nodes are equal, effectively creating a peer-to-peer network. Multiple nodes coordinate their actions through a distributed algorithm, quickly and automatically forming an independent network upon power-up. Communication between nodes and more distant nodes requires multi-hop forwarding via intermediate nodes. Using wireless ad-hoc network equipment, multiple robots can form their own network, even in signal-deprived cave environments. As long as two robots are within range, they can communicate with each other.

[0080] Related technologies have not considered this new form of human-robot collaborative work, where human-assisted tasks are performed through relay links under communication constraints. The embodiments of the present invention implement a scheme for collaborative work between humans and multiple robots, relying on wireless ad hoc networking technology and centered around an operator, under communication constraints. Specifically, the scheme for collaborative work between humans and multiple robots involves humans and multiple robots performing distributed multi-robot collaborative exploration and human-assisted collaborative operations. This involves both multi-robot collaboration and human-robot collaboration, with some robots performing distributed multi-robot collaborative exploration and others forming stable communication relay links to perform human-assisted tasks.

[0081] In the embodiment of the present invention, the human end is an electronic device held by an operator, which may be a tablet, a laptop, etc., wherein the operator may also be referred to as a human operator.

[0082] Human-assisted tasks can also be understood as collaborative tasks assisted by humans. Alternatively, they can be understood as any task that requires human assistance to form a link. If the length of the link formed is 1, it can be understood that the human directly assists the end-user robot, without an intermediate robot. Specific tasks are determined based on actual needs.

[0083] For example, a human-assisted task can include: turning on a camera to capture audio and video of the surrounding environment and a target, and then returning the audio and video streams to the human end; or receiving remote control commands issued by the human end and executing corresponding actions, etc. Receiving remote control commands and executing corresponding actions can also be understood as implementing remote control. It should be noted that this is merely an example of a human-assisted task. In the embodiments of the present invention, a human-assisted task can specifically be any task that a robot can perform.

[0084] The human end receives data from the first messenger robot and can view information such as the integrated overall map, the current plans of all robots (time and location of each robot's meeting), and photos of suspicious areas on a human-machine interface, such as a display screen. Based on this information, the operator may have specific operational requirements, which can trigger corresponding task requests. For example, the operator may request that the robot establish a high-quality communication relay link for a specific area on the map and perform a human-assisted collaborative task in that area.

[0085] Based on the task location, the current location of the human terminal and the first communication plan, task planning is performed and the first communication plan is updated to obtain the topology information of the communication relay link and the second communication plan. Specifically, it can include performing task planning based on the task location and the current location of the human terminal to obtain the topology information of the communication relay link; and updating the first communication plan based on the topology information of the communication relay link to obtain the second communication plan.

[0086] The first communication plan is the communication plan of each robot exploring the map in the current ring topology, including the first communication time and the first communication location between two robots. The ring topology can also be called a ring communication topology. Figure 2 As shown on the left side, there is a ring communication topology in which robots i, j, k, l, and m are connected in sequence and communicate in the direction indicated by the arrows. Figure 2 The single arrows in the ring communication topology on the left side of the figure represent the order in which two-to-two communication events occur. The order of occurrence here can also be understood as the communication order of the robots in the ring topology.

[0087] The second communication plan includes a robot located on the communication relay link, a second communication time for performing the human-assisted task, and a second communication location.

[0088] Moreover, the topology information of the communication relay link can be understood as a chain communication topology, such as Figure 2 As shown on the right side, it includes a chain communication topology consisting of a human end q and robots p, o, n. Figure 2 The double arrows in the chain communication topology shown on the right indicate that data can be transmitted bidirectionally through the two robots o and p in the middle of the link.

[0089] The first messenger robot is the robot that returns to the human side within the human-machine communication interval.

[0090] The human-machine communication interval can be determined based on actual needs or experience, such as 300 seconds, 400 seconds, 500 seconds, etc.

[0091] In one implementation, the human-machine communication interval can be pre-set. In actual application, the human-machine communication interval can be changed, or can remain unchanged.

[0092] It should be noted that in the embodiments of the present invention, the messenger robot is dynamically determined to ensure that "a robot returns to the human side to update information at predetermined intervals (human-machine communication intervals)." This means that the messenger robot is not fixed; the messenger robot returning to the human side in each round may be different or the same as the one returning in the previous round, and this is not a limitation in the embodiments of the present invention. During a communication round, once a messenger robot completes communication with the human side and returns to the exploration team, its "messenger" status is terminated.

[0093] For example, Figure 1 The first messenger robot may be different in different rounds of communication.

[0094] Furthermore, in the embodiment of the present invention, the messenger robot is dynamically determined during the communication between two robots. For example, the first messenger robot to return to the human end is determined during the communication between two robots performing map exploration in the current ring topology.

[0095] After the first messenger robot sends the topology information of the communication relay link and the second communication plan to the first successor robot, the messenger identity of the first messenger robot is released.

[0096] All robots in the current ring topology have received the topology information and second communication plan for the communication relay link. The messenger robot (here, the first messenger robot) arrives at the second communication location at the agreed second communication time according to the first messenger robot's communication plan in the second communication plan. Furthermore, other robots in the current ring topology, excluding the messenger robot, upon identifying themselves as robots on the communication relay link, follow their respective communication plans in the second communication plan and arrive at the second communication location at the agreed second communication time. At this point, all robots in the current ring topology selected as part of the communication relay link have reached their designated communication nodes, forming a communication relay link. This allows human-side and end-user robots to communicate via this communication relay link to complete human-assisted tasks.

[0097] In the embodiment of the present invention, the communication sequence may be dynamically changed or fixed.

[0098] Among them, the terminal robot at the end of the communication relay link is used to perform human-assisted tasks. Specifically, it can turn on the camera to shoot audio and video of the surrounding environment and targets to return audio and video streams to the human end, receive remote control commands issued by the human end and perform corresponding actions, etc.

[0099] The terminal robot can be a robot located in the above-mentioned target area. For example, the human-assisted task is to shoot the environment of the target area and return the obtained audio and video stream to the human end.

[0100] The robots other than the terminal robot in the communication relay link are used to transmit information, receive data sent by the preceding node, and transmit the received data to the subsequent node.

[0101] For any robot other than the terminal robot, the preceding and succeeding nodes referred to here refer to the robots that communicate with the robot according to the direction of information transmission. If the information transmission direction is from the terminal robot to the human end, the preceding node of the robot is the robot that is close to the terminal robot and communicating with it, and the succeeding node of the robot is the robot that is close to the human end and communicating with it. If the information transmission direction is from the human end to the terminal robot, the preceding node of the robot is the robot that is close to the human end and communicating with it, and the succeeding node of the robot is the robot that is close to the terminal robot and communicating with it. The node close to the human end mentioned here may be the human end itself, and the node close to the terminal robot may be the terminal robot itself.

[0102] For example, for Figure 2 For the robot o shown on the right side, if the information transmission direction is from robot n (end robot) to human end q, then the predecessor node of robot o is robot n, and the predecessor node of robot o is robot p; if the information transmission direction is from human end q to end robot n, then the predecessor node of robot o is robot p, and the predecessor node of robot o is robot o.

[0103] In an embodiment of the present invention, a communication relay link is formed by selecting some robots from multiple robots exploring a map in a current ring topology structure. The topology information of the communication relay link and the updated second communication plan are first sent to each robot in the current ring topology structure based on the current ring topology structure. Then, the robot selected as a node on the communication relay link arrives at the second communication location at the agreed second communication time, forming a communication relay link from the human end to the terminal robot in a communication-restricted environment. The communication relay link can be used to realize interaction between the human end and the terminal robot in a communication-restricted environment to complete human-assisted tasks. Among them, some robots perform distributed multi-machine collaborative exploration, and the other part of the robots can form a stable communication relay link to perform human-assisted tasks.

[0104] like Figure 3 As shown on the left side of the arrow, assume that the communication order of the current exploration team is "-ijklmnoi-", and k is a messenger robot that receives the plan of "assigning 3 robots from the exploration team to the relay node". Courier k originally has two communication events, namely and , you should delete , in, Indicates the communication event between robot k and robot l, Indicates the communication event between robot j and robot k, and the new , which represents a special communication event. Position means that k is assigned to the position as a relay node, which means that k should go to the location of Position. So the messenger k first communicates with l, passes the information to l, and then deletes the original plan to communicate with j, and goes directly to the location of Position to act as a relay node for subsequent communications. For l and m, they can first receive the message that they have been selected from the communication with the predecessor node, and only after passing the message to the successor robot will they leave the exploration team and go to the location of the relay node. In order to ensure the communication loop, it is also necessary to modify the communication event of the messenger's predecessor robot j and the communication event of m's successor robot n. j originally planned to have and , now becomes and . nThe original plan was and Now add a new communication event, which becomes 、 、 , Indicates the communication event between robot m and robot m, Indicates the communication event between robot n and robot o. This represents a communication event between robots j and n. To put it simply, the ring network topology consisting of the communication order "-ijklmnoi-" in the current exploration team is split into: a ring network topology consisting of "-ijno-" and another chain network topology consisting of robots l, k, and m.

[0105] In an optional embodiment, the human end is further configured to confirm task information of the human-assisted task and send the task information to the robot communicating with the human end on the communication relay link, wherein the task information includes a task type;

[0106] If the robot communicating with the human end is an end robot, the robot communicating with the human end receives the task information and performs the human-assisted task based on the task information; if the robot communicating with the human end is not an end robot, the robot communicating with the human end continues to transmit the task information in the direction from the human end to the end robot until the task information is sent to the end robot, the end robot receives the task information, and performs the human-assisted task based on the task information.

[0107] Specifically, the human side can provide a human-machine interface to the operator, through which the human side can obtain task information. When a human-assisted task begins, the human side will ask the operator through the human-machine interface the type of task required, for example: (1) an inspection task, which requires the end robot of the link to turn on the camera to capture the surrounding environment and the target; (2) an operation task, which requires the end robot of the link to receive the operator's remote control instructions and perform the corresponding action.

[0108] In an optional embodiment, after the end robot performs a human-assisted task and obtains the task result, if the end robot is communicating with the human end, the end robot will send the task result to the human end; if the end robot is not communicating with the human end, the end robot will send the task result to the robot that the end robot is communicating with, and the robot that is communicating with the human end will continue to transmit the task result in the direction from the end robot to the human end until the task result is sent to the human end.

[0109] The task result may specifically include the audio and video stream obtained by shooting the surrounding environment, or may also include the result of executing the corresponding action according to the remote operation issued by the human end.

[0110] In general, the interaction between the human end and the terminal robot is realized according to the communication relay link and the information transmission direction, wherein the information transmission direction specifically includes the above-mentioned from the terminal robot to the human end or from the human end to the terminal robot.

[0111] In an optional embodiment, the human end is also used to receive the new location selected by the operator; based on the task location, the new location and the communication plans of multiple robots exploring the map in the current ring topology structure, the task is re-planned and the communication plans of each robot are updated to obtain a communication relay link and an updated communication plan of each robot.

[0112] In actual application, the operator is supported to move from the current position to another position.

[0113] For example, the human side initially determines that five robots, including the messenger, will serve as relay robots on the communication relay link to perform human-assisted tasks. The human side presents the planning results to the operator and asks the operator whether to choose to move.

[0114] The operator determines that the communication relay link for five robots is too long, and that only two robots remain in the exploration team, which will affect exploration efficiency. Therefore, the operator decides to work closer to the mission location. The operator selects their desired work location on the map (the new location selected by the operator above). The human client replans the communication relay link based on the planned mission location and the operator's selected work location. Because the operator is closer to the mission location, the communication relay link only requires three robots. The human client replans communication events for the three robots, including the messenger, and displays the planning results to the operator.

[0115] In the above Figure 3 After completing the calculation and task planning of the communication relay link, the planning results are displayed to the operator, and the operator is asked whether to choose to move. The planning results can specifically include information such as the position of each robot on the communication relay link and the communication time between each node on the communication relay link (including the human end and the robot).

[0116] The reason for asking the operator whether to move is that in collaborative work between humans and multiple robots, operators also have their own task requirements and sometimes need to change their work location, which requires replanning the communication relay link. On the other hand, if the operator chooses to move closer to the target point (i.e., the task location), fewer relay robots are needed for the human-assisted task, leaving more robots in "exploration" mode, improving overall exploration efficiency. Therefore, allowing the operator to move makes the planning process more flexible and achieves more satisfactory results.

[0117] In an optional embodiment, the human side is also used to determine the estimated completion time of the current human-assisted task; if the estimated completion time is less than the agreed threshold, it is determined whether there are any unprocessed human-assisted tasks; if so, the link transfer is planned; if not, the link dismissal is planned.

[0118] The agreed threshold may be preset or calculated on the spot based on efficiency and communication conditions.

[0119] The planning of link transfer can be carried out through the following process: the human side is specifically used to determine the operator position and the new task location corresponding to the unprocessed human-assisted task; based on the operator position and the new task location, the number of relay nodes required for the new task is determined; if the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, the relay nodes in the current communication relay link are used as relay nodes of the new communication relay link; if the number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task, the first target robot is selected from the robots used for map exploration, and the sum of the number of the first target robots and the number of relay nodes in the current communication relay link reaches the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task, the second target robot is selected from the relay nodes in the current communication relay link as the relay node of the new communication relay link.

[0120] Among them, the relay node can be understood as a robot on the communication relay link.

[0121] like Figure 4A As shown in the figure, the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, which is 3. This situation can also be understood as the case where the number of nodes in the new and old links is equal. In this case, the relay nodes (robots l, k, m) in the current communication relay link can be directly used as the relay nodes of the new communication relay link.

[0122] like Figure 4B As shown in the figure, the number of relay nodes in the current communication relay link is 3, while the number of relay nodes required for the new task is 4. The number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task. This situation can also be understood as the case where the number of nodes in the new link is greater. In this case, the first target robot, such as robot j, is selected from the robots used for map exploration. This can also be understood as robot j being seconded from the exploration team to form a relay node of the new communication relay link together with the relay nodes in the current communication relay link (robots l, k, and m).

[0123] like Figure 4C As shown in the figure, the number of relay nodes in the current communication relay link is 4, while the number of relay nodes required for the new task is 3. The number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task. This situation can also be understood as the case where the number of nodes in the new link is smaller. In this case, the second target robot (robots l, k, m) is selected from the relay nodes in the current communication relay link as the relay node of the new communication relay link. This can also be understood as returning robot j in the relay node of the current communication relay link to the exploration team.

[0124] Planning for link demobilization may include the following steps:

[0125] The human side is specifically used to determine the operator position and the new task location corresponding to the unprocessed human-assisted task; based on the operator position and the new task location, determine the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, then the relay nodes in the current communication relay link are used as relay nodes of the new communication relay link; if the number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task, then select a first target robot from the robots used for map exploration, and the sum of the number of the first target robots and the number of relay nodes in the current communication relay link reaches the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task, then select a second target robot from the relay nodes in the current communication relay link as the relay node of the new communication relay link.

[0126] The human end is specifically configured to obtain a third communication plan from the second messenger robot, the third communication plan being a new communication plan for the multiple robots currently performing map exploration; based on the third communication plan, the multiple robots currently performing map exploration and the robots in the current communication relay link are combined into a new ring topology, and the communication plans of the robots in the current communication relay link are updated to obtain a fourth communication plan; and the topology information of the new ring topology and the fourth communication plan are sent to the second messenger robot and the robots in the current communication relay link;

[0127] The second messenger robot is configured to, after receiving the topology information of the new ring topology structure and the fourth communication plan sent by the human terminal, locally update the topology information of the new ring topology structure and the communication plan; arrive at the fourth communication location at the fourth communication time according to the communication plan of the second messenger robot in the fourth communication plan, and send the topology information of the new ring topology structure and the fourth communication plan to a second successor robot of the second messenger robot in the new ring topology structure;

[0128] a second successor robot, configured to locally update topology information of the new ring topology structure and the fourth communication plan, and send the topology information of the new ring topology structure and the fourth communication plan to a subsequent robot of the second successor robot;

[0129] The robot in the current communication relay link is used to locally update the topology information of the new ring topology structure and the fourth communication plan, and arrive at the agreed communication location according to the fourth communication plan;

[0130] The robots in the new ring topology are used to explore the map.

[0131] One robot returns to the human side every human-machine communication interval. The second messenger robot and the first messenger robot return to the human side within the human-machine communication interval. They can be the same robot or different robots. For example, if the human-machine communication interval is 300 seconds, the first messenger robot returns to the human side every 300 seconds, and the second messenger robot returns to the human side every 600 seconds.

[0132] The second successor robot is the successor robot of the second messenger robot, that is, the robot that communicates with the second messenger robot according to the new ring topology structure, that is, the robot located after the second messenger robot according to the communication sequence of the fourth communication plan.

[0133] The subsequent robot of the second successor robot is a robot that communicates with the second successor robot according to the new ring topology structure, that is, a robot that is located after the second successor robot according to the communication sequence of the fourth communication plan.

[0134] After the second messenger robot arrives at the fourth communication location and sends the topology information of the new ring topology structure and the fourth communication plan to the second successor robot, the messenger identity of the second messenger robot is released.

[0135] The third communication plan is a new communication plan for the multiple robots currently exploring the map, and the fourth communication plan is a new communication plan obtained by updating the communication plans of the robots in the current communication relay link.

[0136] In this embodiment, the above process performed by the human end, the second messenger robot, and the robots in the current communication relay link can also be understood as a mode switch from "relay" to "exploration", and a process of re-establishing a ring topology structure between the robots.

[0137] In the process of establishing the ring topology structure, the robots that already know the fourth communication plan arrive at the communication location according to the communication time in the fourth communication plan, and the robots that do not know the fourth communication plan follow the communication time and communication location in the third communication plan.

[0138] In Example 1, if Figure 5A As shown, the current communication environment includes two independent links: the current communication relay link (including robots 5, 6, and 7) and the ring topology structure composed of multiple robots currently exploring the map (including robots 1, 2, 3, and 4). The second messenger robot is robot 1. In this embodiment, the robots in the current communication relay link are inserted into the ring topology structure to obtain a new ring topology structure, such as Figure 5AThe new ring topology shown on the right includes Robot 1, Robot 2, Robot 6, Robot 5, Robot 7, Robot 3, and Robot 4, in the order of communication. For ease of description, in this embodiment, the current communication relay link can be referred to as a straight link, the ring topology composed of multiple robots currently exploring the map can be referred to as a small ring, and the new ring topology to be formed can be referred to as a large ring.

[0139] In the description of this embodiment, the exploration process between communication events is ignored, and only how the robots re-establish the topology of the large ring is described.

[0140] In this case, it can be understood as a straight chain inserted in the middle of the ring topology.

[0141] It can be observed that the fourth communication plan is partially consistent with the third communication plan, that is, the arrows A represent the original communication events of the small ring and do not need to be changed. Therefore, the communication events represented by the arrows A are the same in the third communication plan and the fourth communication plan.

[0142] 1) Robot 1 (the second messenger robot) arrives at the agreed location at the agreed time according to the fourth communication plan and communicates with Robot 2. Robot 2 arrives at the agreed location at the agreed time according to the third communication plan and communicates with Robot 1. In this step, the third and fourth communication plans are consistent, so both can successfully meet. Afterward, Robot 1 waits for Robot 4 to communicate according to the fourth communication plan.

[0143] 2) After communicating with Robot 1, Robot 2 already knows the fourth communication plan, so it follows the fourth communication plan and arrives at the agreed location at the agreed time to wait for Robot 6 to communicate.

[0144] 3) On the other hand, Robots 5, 6, and 7 on the linear chain arrive at the agreed location at the agreed time according to the fourth communication plan after completing their mission (note: they do not necessarily depart at the same time as Robot 1, acting as the messenger; they must wait until the aforementioned human-assisted mission is complete. The waiting time here is the estimated completion time). Specifically, Robot 6 waits for Robot 2 to communicate, Robot 5 waits for Robot 6, and Robot 7 waits for Robot 5.

[0145] 4) According to the descriptions in 2) and 3), Robots 2 and 6 will meet and communicate according to the fourth communication plan. Afterward, Robot 6 will continue to wait for Robot 5 according to the fourth communication plan. As mentioned in 3), Robot 5 is waiting for Robot 6, so Robots 6 and 5 will also meet and communicate according to the fourth communication plan. Afterward, Robot 5 will continue to wait for Robot 7 according to the fourth communication plan. As mentioned in 3), Robot 7 is waiting for Robot 5, so Robots 5 and 7 will also meet and communicate according to the fourth communication plan.

[0146] 5) After 4) is complete, Robot 7 waits for Robot 3 according to the fourth communication plan. Note that the communication time and location between Robot 7 and Robot 3 in the fourth communication plan are identical to the communication time and location between Robot 3 and Robot 2 in the third communication plan. This is the key point: Robot 3 still waits for Robot 2 at the agreed time and location according to the third communication plan, but it is Robot 7 that arrives. Robot 7 communicates with Robot 3 under the guise of Robot 2, and during this communication, it informs Robot 2 of the fourth communication plan.

[0147] 6) After step 5), Robot 3 waits for Robot 4 according to the fourth communication plan, while Robot 4 waits for Robot 3 according to the third communication plan. In this step, the third and fourth communication plans are identical, so both can successfully meet. Afterward, Robot 4 waits for Robot 1 according to the fourth communication plan. According to step 1), Robot 1 is waiting for Robot 4, so communication is successful. At this point, all robots in the large ring are aware of the fourth communication plan, and the communication topology has been successfully transformed into a large ring.

[0148] In Example 2, if Figure 5B As shown, the straight chain is directly inserted into the beginning of the small ring, so the key lies in the communication between robot 7 and robot 2. This step is similar to 5) in Example 1.

[0149] In Example 3, if Figure 5C As shown, the straight link is directly inserted into the end of the small ring, so at the time when the straight link is added, the robots in the original small ring already know the fourth communication plan.

[0150] When it is necessary to dismiss the robot from the communication relay link and return it to the exploration team, the link dismissal process is involved, such as Figure 6 The specific contents are as follows:

[0151] The human client obtains the latest communication plan for the robots in the exploration team from the information brought back by the messenger. The human client then plans a suitable "return plan" for all relay robots that are about to be dismissed. In short, this involves finding a suitable communication event in the exploration team's current plan, ensuring that the robots to be dismissed arrive at that communication location before this communication event occurs, and then inserting these robots into the exploration team's ring communication topology to reintegrate them into the communication team. Specifically, the steps are as follows:

[0152] i) The human side will query the latest exploration team plan (including the time and location of a series of communication events) from the information sent back by the messenger, and then calculate the communication time for all robots to be dismissed to move from their current location to the communication location of each communication event after completing the human-assisted task.

[0153] ii) If all robots to be dismissed can reach the communication location of a communication event before the communication event occurs, it will be recorded as a candidate communication event.

[0154] iii) Finally, the earliest communication event is found from the candidate communication events. In other words, if the robots currently waiting to be dismissed depart immediately after completing their human-assisted tasks and head to the location of the communication event, all robots will arrive before the scheduled time of the communication event. After finding the target communication event, the communication topology order of each robot is determined based on the order in which the robots arrive at the target communication event, and the corresponding communication events are planned.

[0155] For example, in Figure 6 In the example, the target communication event is , the robots that arrive at the target communication event location are l, k, and m in order, then the original plans of robot i and robot j should be deleted. , add robot i , add to robot j For robot l, its predecessor robot is i and its successor robot is k, then add and Similarly, add robot k and , add robot m and , simply understand, adding a new communication edge. After the planning is completed, the human end passes the above "return to the team" plan to the messenger.

[0156] In this embodiment, multiple robots are used to conduct distributed, cyclical, multi-round autonomous exploration tasks and human-assisted collaborative tasks in a complex environment with limited communication. Each robot has two modes: "exploration" and "relay."

[0157] Robots in "exploration" mode will explore the environment in a distributed, cyclical manner (one round is when information is transmitted once in the ring communication topology) in a "communication-exploration-communication-exploration" manner. Each robot is equipped with self-organizing networking equipment. Through the ring communication topology, two robots communicate with each other and agree on the next plan (communication time and location), thus ensuring timely communication. The communication topology diagram is shown below. Figure 2 As shown on the left, the arrows in the figure indicate the order in which pairwise communication events occur.

[0158] The robot in the "relay" mode will form a high-quality communication relay link with the human end to perform human-assisted tasks. The communication topology diagram of the communication relay link is as follows: Figure 2 As shown on the right, consider n as the end robot and q as the human end. The operator interacts with end robot n through human end q. The double arrows no longer indicate the order of communication events, but instead emphasize that data can be transmitted bidirectionally through the two robots o and p in the middle of the link.

[0159] Therefore, within the collaborative robot system, there are two teams: an "exploration" team and a "relay" team. The communication topology of the robots in the "exploration" team is a ring structure, with information transmitted and updated within the ring. The "relay" team, on the other hand, has a single linear structure, with the human at the head and the working robots at the end, performing human-assisted tasks such as transmitting audio and video data to the human and receiving control commands from the human to execute tasks. In the middle of the team are relay robots, responsible for maintaining the communication relay link and transmitting data. Each robot can switch between "exploration" and "relay" modes based on the distributed algorithm and the operator's needs, meeting the complex requirements of human-multi-robot collaboration. In terms of human-robot interaction, a robot "returns" to the human at regular intervals to communicate with the operator. During this time, the operator can obtain the latest global environment map, the robot's plan, photos of suspicious areas, and more.

[0160] Depending on their needs, the operator may request (1) to continue the original task without any special processing; (2) to explore certain areas first; (3) to move to a certain area by themselves; or (4) to request the formation of a relay link to perform human-assisted tasks in a certain area.

[0161] For the first task, return to the communication location planned for this round. For the second task, return to the communication location planned for this round and give priority to the exploration area in the subsequent robot planning. For the third task, return to the communication location planned for this round and update the position of the human end.

[0162] The embodiment of the present invention mainly achieves the realization of the fourth task. Based on the existing multi-machine exploration method limited to multiple robots interacting with the operator, a new mode, the "relay" mode, is added to perform human-assisted tasks according to actual needs. It mainly includes the following parts:

[0163] A. When the robot is in "exploration" mode, it performs the following steps:

[0164] A1. The robots arrive at the agreed location at the designated time, forming a ring-shaped communication topology.

[0165] A2. Robots communicate with each other, such as robot i and robot j, exchanging information, including locally created maps of the environment, the current robot's plan transmitted by the previous robot (the time and location of communication between robots), and photos of suspicious areas.

[0166] A3. Plan the exploration path and agree on the next plan for robots (i, j). The planning process takes into account the time difference between the next communication meeting and the last human-machine meeting. If this time difference is greater than the preset human-machine communication interval, the current plan is discarded and replanned so that robot j returns before the next communication event with robot i, meets the operator, and interacts with the operator, proceeding to step C1. If this time difference is less than the preset human-machine communication interval, proceed to step A4.

[0167] A4. After planning, the robot completes the communication process and then explores the map along the planned path. It uses radar to collect map data and uses cameras and image recognition algorithms to capture environmental features and determine whether a suspicious area exists. After completing this round of exploration, the robot returns to step A1.

[0168] B. When the robot is in "Relay" mode, perform the following steps:

[0169] B1. The robot arrives at the designated location at the designated time, forming a node in the communication relay link. At this point, the robot remains in place, waiting for all robots or handheld operators in the communication relay link to arrive. After the last robot and operator in the communication relay link are in place, the following human-assisted tasks begin.

[0170] B2. If you are the robot at the end of the link (e.g. Figure 2The robot n is shown on the right. Based on human needs, it performs the following tasks: (1) uses a camera to capture the surrounding environment and target, and transmits the video and audio back to the previous relay node (e.g., robot o, the previous robot of robot n); (2) receives remote control commands from the human end and performs actions such as moving or grasping (if the end robot is a mobile robotic arm). If it is not the robot at the end of the link (e.g., robot or p), it is responsible for information transmission. Following the direction of information transmission, it receives information from the previous node (which may be a robot or a human, depending on the specific situation) and transmits it to the subsequent node.

[0171] B3. After receiving the end command from the receiver, perform the following actions based on the actual situation: (1) switch to the next human-assisted task, go to the next location, and return to step B1; (2) end the "relay" mode, enter the mode switching state, and proceed to step F3.

[0172] C. When the Courier in "exploration" mode returns to the human client, human-computer interaction occurs (which can also be understood as the interaction between the human client and the Courier, and between the human client and the operator when no human-assisted task is in progress), and the following steps are performed:

[0173] C1. A robot returns to the human side within the pre-set human-machine communication interval. This returning robot is called a "courier," referring to the aforementioned courier robot. The courier sends data to the human side, including a map of the environment, the current plans of all robots (the time and location of communication between robots) transmitted by the previous robot, and photos of the suspicious area. The courier then waits for the operator's interaction request.

[0174] C2. The human client receives the data delivered by the messenger and observes on the display information such as the integrated overall map, the current plans of all robots (the time and location of communication between robots), and photos of suspicious areas.

[0175] C3. The human side supports processing four types of interaction requests based on the operator's self-needs. The first three types of interaction requests are: (1) No special processing, allowing the robot to continue its own task; (2) Specifying a priority exploration area, allowing the robot to prioritize exploring this area in subsequent planning; (3) Specifying the area to go to, and the robot calculates the operator's reference movement path.

[0176] C4. The messenger receives the request from the human side and performs the following corresponding actions based on the content of the first three requests described in C3: (1) Return to the communication location planned for this round and return to step A1; (2) Return to the communication location planned for this round and prioritize the exploration area in the subsequent robot planning and return to step A1; (3) Return to the communication location planned for this round, update the location of the human side and return to step A1.

[0177] C5. The fourth type of interaction request received by the human side from the operator is a human-assisted task request: (4) The operator requires the robot to form a high-quality communication relay link for a certain area on the map and perform a human-assisted collaborative task in that area.

[0178] C6. The human side then plans and assigns tasks based on the planned task locations. Based on the communication events for the next round of robots in the current plan and the communication quality model in the environment, it plans a high-quality communication relay link, modifies the relevant communication events, and displays the planning results to the operator, asking the operator whether to choose to move.

[0179] Specifically:

[0180] i) The human client determines the task location and the human client's current location on the currently known fusion map.

[0181] ii) Then, referring to the communication quality model of the current environment (which can be derived from empirical formulas or learned through artificial intelligence algorithms), a search algorithm is used to find a high-quality communication relay link. This link connects the human end and the task location through multiple relay robots, with the human end at the beginning and the end robot at the end. This communication relay link minimizes the number of robots required while ensuring that the communication quality between adjacent nodes on the link exceeds a minimum threshold.

[0182] iii) After obtaining the communication relay link, the human side will arrange a sufficient number of robots from the exploration team as nodes on the communication relay link. Assuming that the communication relay link requires x robots, starting with the messenger robot, according to the order of their communication topology, a total of x robots, including the messenger, will be selected as nodes on the communication relay link.

[0183] iv) For the messenger, it has originally planned two communication events in the subsequent exploration mission. At this time, it is necessary to retain the originally planned first communication event, delete the second communication event, and add a special communication event for the relay task; for the subsequent x-1 robots selected, it is necessary to add a third special communication event for the relay task after the originally planned two communication events. In order to keep the original ring communication topology looped, it is also necessary to establish a forward and backward communication relationship between the messenger's original predecessor robot and the last selected relay robot, so the corresponding communication events of the two robots also need to be modified. The location of the special communication event mentioned above is set to the location of the relay node, and the time is calculated according to the time required for the robot to reach the relay node after the last communication. For example, the above Figure 3 shown.

[0184] C7. If the operator needs to move, they will select a new location on the human-side interface. The human-side will then re-plan the communication relay link to obtain the final communication relay link.

[0185] C8. The human client publishes the finalized human-assisted task information and the new communication plan and topology information, such as the allocation results of communication relay links, the planned and updated communication plan for each robot, and the new communication topology, to the messenger.

[0186] C9. The human-assisted task issued by the messenger recipient enters the mode switching state and proceeds to step E1.

[0187] D. When the human client performs a human-assisted task, the robot communicating with the human client will have human-machine interaction (interaction between the human client and the operator, interaction between the human client and the robot on the communication relay link, etc.), and perform the following steps:

[0188] D1. When a human-assisted task begins, the human side will ask the operator through the human-machine interface for the type of task required, for example: (1) an inspection task, which requires the end robot of the link to turn on the camera to capture the surrounding environment and target video and audio; (2) an operation task, which requires the end robot of the link to receive the operator's remote control instructions and perform the corresponding action.

[0189] D2. After the operator determines the task type, the operator transmits the information to the first robot on the communication relay link.

[0190] D3. The human receives real-time information (such as audio and video) from the first robot and displays it to the operator. It also receives remote control commands from the operator and transmits them to the first robot in the link.

[0191] D4. During the period when the human-side is performing the human-assisted task, if a messenger (from the exploration team) returns to the human-side, the human-side first executes and completes the human-computer interaction process C1-C4 of the "exploration" team according to the steps in C. Since the operator is currently in the process of the human-assisted task, there are only two interaction methods (1) and (2) in the execution of step C3. After completing the interaction content of (1) and (2) in C4, the messenger does not enter A1, but waits for instructions from the human side.

[0192] D5. The human client asks the operator for the current human-assisted task and its estimated completion time. If the operator's estimated completion time is too long or cannot be estimated (exceeding a certain threshold, which serves as the boundary for whether to terminate the link. The threshold can be determined based on actual needs or experience), the link processing is not performed in this round, and the messenger returns to step A1. If the estimated time is not too long (zero or less than a certain value), the human client enters the link processing process D6.

[0193] D6. After the human client enters the link processing process, it first determines whether there are any unfinished human-assisted tasks in the current task pool. If so, it plans to transfer the link based on the operator's selection (selecting an unfinished human-assisted task), and proceeds to step D7. If no human-assisted tasks exist or the operator refuses to continue with the human-assisted task, it plans to terminate the link, and proceeds to step D11.

[0194] D7. The human side first asks the operator to select the next operator position, then generates a new link based on the new human-assisted task position and the operator's position, and then compares the information of the current link and the new link.

[0195] D8. If the number of relays on the current link equals the number required for the new link, the current relay robot is directly assigned to the new link, and the time-optimal assignment is calculated to complete the planning. After the current human-assisted task is completed, the current link is transferred to the new link according to the assignment, and the messenger returns to step A1. If the number of relays on the current link is less than the number required for the new link, a sufficient number of robots will be "borrowed" from the exploration team in a subsequent step, and the process proceeds to step D9. If the number of relays on the current link is greater than the number required for the new link, only the number of robots required for the new link is assigned, and the time-optimal assignment is calculated. After the current task is completed, these robots will be transferred to the new link according to the assignment. Excess robots will be "returned" to the exploration team in a subsequent step, and the process proceeds to step D11.

[0196] D9. The human client plans the robots, communication events (including arrival time and location), and communication topology changes required to complete the "secondment" event. This planning process is similar to generating relay links in C6 and is equivalent to generating a new human-assisted task. The human client publishes the completed plan to the courier.

[0197] D10. The messenger receives the request for "secondment" from the human side, enters the mode switching state, and proceeds to step E1.

[0198] D11. The human client obtains the latest plan of the exploration team's robots from the information brought back by the messenger.

[0199] D12. The human client plans a suitable "return plan" for all relay robots about to be dismissed. In short, this involves identifying a suitable communication event within the exploration team's current plan, ensuring that the robots about to be dismissed arrive at that communication location before this communication event occurs, and reinserting them into the exploration team's ring communication topology. After planning is complete, the human client transmits the "return plan" to the messenger and robots on the communication relay link. This "return plan" specifically includes the topology information for the new ring topology and the fourth communication plan.

[0200] Specifically, it is divided into the following steps:

[0201] i) The human side will query the latest exploration team plan (including the time and location of a series of communication events) from the information sent back by the messenger, and then calculate the time it takes for all robots to be dismissed to move from their current location to the location of each communication event after completing the human-assisted task.

[0202] ii) If all robots to be dismissed can reach the location of a communication event before a communication event occurs, it will be recorded as a candidate communication event.

[0203] iii) Find the earliest communication event from the candidate communication events. That is, if the robots currently waiting to be dismissed depart immediately after completing their human-assisted tasks and head to the location of the communication event, all robots will arrive before the scheduled time of the communication event. After finding the target communication event, the communication topology order of each robot is determined based on the order in which the robots arrive at the target communication event, and the corresponding communication events are planned. This ensures that returning robots (robots currently on the communication relay link, which are waiting to return to the ring network topology used for map exploration) establish communication events in the order in which they are located in the new ring communication topology.

[0204] For example, in Figure 6 In the example, the target communication event is , the robots that arrive at the target communication event location are l, k, and m in order, then the original plans of robot i and robot j should be deleted. , add robot i , add to robot j For robot l, its predecessor robot is i and its successor robot is k, then add and Similarly, add robot k and , add robot m and After the planning is completed, the human client will pass the above “return to team” plan to the messenger. After the planning is completed, the human client will pass the “return to team” plan to the messenger.

[0205] D13. The messenger receives the "return to team" plan from the human side and proceeds to step F1.

[0206] D6-D13 can be understood as the link processing process, among which D7-D10 is the planning of link transfer, and D11-D13 is the planning of link demobilization.

[0207] E. To switch from "Exploration" to "Relay" mode, perform the following steps:

[0208] E1. The Messenger receives the request for the human-assisted task from the human side, updates its own and other robots' communication plans and topology information, and returns to the original communication location in the exploration team as planned.

[0209] E2. The Messenger meets the successor robot, delivers the human-assisted task information and the updated communication plan and topology information to the successor robot, then switches to "Relay" mode and proceeds to B1 at the time and location planned in C8.

[0210] E3. After receiving the courier's human-assisted task information and the updated communication plan and topology information, the courier's successor robot updates its own and other robots' communication plans and topology information and passes this information to the successor robot. It also determines whether it is the relay robot planned in C8. If so, it switches to "Relay" mode and proceeds to B1 according to the time and location planned in C8. If not, it continues with its original task.

[0211] E4. After the human-assisted task information and the updated communication plan and topology information have been passed through the exploration team, all robots in the exploration team are aware of this information. At this point, the communication topology has changed, and some robots in the exploration team are assigned to the "relay" team to form a communication relay link.

[0212] F. To switch from "Relay" to "Exploration" mode, perform the following steps:

[0213] F1. After the courier receives a reunion request, it updates the communication event, changes the stored communication topology, and alters the communication order of the exploration team, forming a new ring communication topology. It then returns to the communication location of the new ring communication topology according to the new communication plan. The communication location agreed upon in the new communication plan is the same as the courier's original communication location in the exploration team. The reunion request is then forwarded to the subsequent node. The reunion request can also be understood as the "reunion" plan described above.

[0214] A rejoin request is a request triggered when the human end completes a human-assisted task.

[0215] F2. The successor node reads the rejoin request and similarly changes the stored communication topology information, changing the communication order of the exploration team to form a new ring communication topology. It then passes this information to the successor node.

[0216] F3. After receiving the rejoin request message from the human client, the relay robot to be dismissed updates its communication event, changes its stored communication topology information, and alters the communication order of the exploration team, forming a new ring communication topology. It then returns to the communication location of the new ring communication topology according to the new communication plan.

[0217] F4. The dismissed relay robot (a robot on a communication relay link) will arrive at the communication location at the agreed communication time and communicate with the preceding robot (for a relay robot, the preceding robot is the robot that precedes the relay robot in the communication order after rescheduling) and the following robot (for a relay robot, the following robot is the robot that follows the relay robot in the communication order after rescheduling). It will then switch to "exploration" mode. When the last relay robot successfully rejoins the team, the mode switch for all robots is complete, forming a new exploration team. After information is passed through the exploration team once, all robots in the exploration team will be aware of the new communication topology.

[0218] Among them, step F3 may have no temporal sequence relationship with step F1 and step F2, and can be executed independently.

[0219] The embodiments of the present invention take into account communication limitations and can be applied to tasks involving multi-machine exploration in more complex large-scale environments. A new collaborative working mode between humans and multiple machines is proposed, which can form a stable communication relay link in complex geographical and communication environments to meet actual needs. In addition, it is people-centered, takes into account the importance of the operator's role, the complex needs of humans, and utilizes the advantages of humans in emergency response and overall planning, thereby strengthening the operator's supervision and control of multi-machine exploration tasks and improving the system robustness and task completion rate. In addition, it provides exploration priorities to meet more complex task requirements. In addition, it also points out the feasible domain of operator movement, which improves operator safety. And because the operator is mobile, the exploration range is significantly expanded compared to the fixed communication base station method.

[0220] Specifically, the robot may include the following processes when in "exploration" mode:

[0221] The robots arrive at the predetermined location at the designated time, forming a ring network topology;

[0222] Robots communicate with each other and exchange information such as environment maps and locations;

[0223] Plan the exploration path and agree on the time and location of the next communication meeting between the two robots, that is, the communication time and location for each robot to communicate;

[0224] Determine whether the meeting time is greater than the human-machine timestamp;

[0225] If yes, re-plan the next communication event, insert the human-side interaction event before the next communication, and then the robot and the human-side interact, specifically as follows Figure 7 If not, the robot performs distributed multi-machine map exploration according to the planned path and determines whether the exploration task and the assistance task are completed.

[0226] In the embodiment of the present invention, the interaction process between the human terminal and the robot is as follows: Figure 7 As shown:

[0227] The messenger began to return to the human side;

[0228] The messenger communicates with the human side and waits for the human side's interaction request;

[0229] The operator holds an electronic device and obtains the information sent back;

[0230] The operator issues an interaction request;

[0231] Interactive options may include: (1) no special processing, allowing the messenger to continue its mission; (2) specifying the priority exploration area for the next step; (3) the operator specifies the area to go to, and the human side calculates the safe feasible domain; (4) the operator requests a human-assisted collaborative task in a certain area, which is the human-assisted task mentioned above.

[0232] For the interactive operations (1), (2) and (3), the messenger returns to perform the exploration task. For the interactive selection (4), the following process is executed:

[0233] The operator selects the human-assisted task location, which is the task location mentioned above;

[0234] The human side obtains the communication relay link node (i.e. Figure 7 Communication link nodes in the network), and complete planning and allocation;

[0235] The operator chooses whether to move; if the operator chooses to move to a new location, the returning human side will find the communication relay link node based on the operator's location and the task location, and complete the planning and allocation steps; if the operator does not choose to move to a new location, the human side will publish the final result to the messenger, and then the messenger will return to the exploration team and enter the mode switching process from "exploration" to "relay".

[0236] The workflow in the relay mode in the embodiment of the present invention is as follows Figure 8 Shown, including:

[0237] The operator / human end and the associated robot arrive at their position on the communication relay link (also called relay link);

[0238] The end robot and the human end establish a real-time connection through a communication relay link;

[0239] The operator / human end issues inspection tasks or operation tasks to the remote robot;

[0240] The end robot and the operator / human end transmit information such as audio and video (audio and video information), control signals, etc. through the communication relay link;

[0241] Determine whether the human end initiates link processing;

[0242] If yes, then the link processing process is carried out. Specifically, the link processing process is as follows: Figure 9 If not, return to the step of issuing inspection-type tasks or operation-type tasks;

[0243] In addition, the messenger returns to the human side; the operator / human side and the messenger complete the human-computer interaction of the "exploration" team; the human side asks the operator for the estimated completion time of the human-assisted task; if the time is too long or cannot be estimated, the link processing is abandoned in this round and the messenger returns to the exploration team; if the time is short, such as less than a certain threshold, the above step of determining whether the human side initiates link processing is entered.

[0244] The link processing flow in the embodiment of the present invention is as follows: Figure 9 As shown:

[0245] The link processing process begins;

[0246] Determine whether there are any unfinished human-assisted tasks;

[0247] If not, the link begins to disband, and the messenger returns to the exploration team while the information is delivered;

[0248] If so, determine whether to switch to the next human-assisted task;

[0249] If the switch is not made to the next human-assisted task, the link begins to disband; if the switch is made to the next human-assisted task, the operator selects the task location for the next human-assisted task;

[0250] After the operator selects the task location for the next human-assisted task, the number of relay nodes in the new link is compared with the number of relay nodes in the old link;

[0251] If the two are equal, the link transfer planning is carried out directly, and then the messenger returns to the exploration team and transmits the new link information at the same time;

[0252] If the number of relay nodes in the new link is less than the number of relay nodes in the old link, the link transfer planning is carried out by "returning" the robot to the exploration team, and the robot to be dismissed begins to "return to the team", as follows: Figure 6 As shown, the messenger returned to the exploration team and delivered the "return to the team" message;

[0253] If the number of relay nodes in the new link is greater than the number of relay nodes in the old link, the link transfer planning is carried out by "borrowing" robots from the exploration team, and the human side carries out new human-assisted task planning, as follows: Figure 6 As shown, the messenger returns to the exploration team and enters the mode switching process from "exploration" to "relay", delivering the "secondment" information;

[0254] In this way, the robots on the original link (old link) start to transfer according to the plan, as shown below: Figure 4A 、 Figure 4B as well as Figure 4C shown.

[0255] The human-machine collaborative work system provided by the embodiment of the present invention is described in detail below using a specific example.

[0256] Assume that an operator holds an electronic device (the human end) and intends to use seven robots (numbered i, j, k, l, m, n, and o) to perform a human-robot collaborative task in an unknown cave environment. In this environment, there is no communication signal that covers the entire area, and due to terrain restrictions, when the robots are far apart, the communication quality will be very poor or even impossible. Therefore, the robots and the human end can only communicate temporarily through wireless ad hoc network devices. In this scenario, the robot behavior in exploration mode and the overall process of human-robot interaction are as follows: Figure 10 shown.

[0257] At the beginning of the mission, the operator and the robot have no map data. They determine the initial relative position through initial communication and build the initial map. At the same time, the operator sets the human-machine communication interval. , assuming , which means that every 300 seconds, at least one robot must serve as a messenger to return to the human side to communicate and respond to the human side's interaction needs. Then, the robot begins the first round of planning. According to the current map, the robot assigns different robots to different map "borders" for exploration tasks and follows the ring communication topology (refer to Figure 2 As shown on the left), the robots agree on the next round of communication events between each other, including the communication time and location.

[0258] After the first round of planning is complete, each robot begins its exploration mission based on the planned results. Each robot explores its assigned map boundary, updating its map in real time as it explores. If a robot encounters another robot during exploration, they quickly communicate, exchanging map data and other data, and locally integrating the maps. After completing the exploration process, the robot arrives at the communication location on time as previously agreed upon and communicates with the next robot in the order of the ring communication topology.

[0259] Take the communication event between robots i and j as an example. When they communicate with each other, robot i only needs to perform one stage of planning, while robot j needs to perform two stages of planning. This is because in a ring communication topology, robot i needs to communicate with both robots o and j. At the time it communicates with robot j, it has already completed communication with robot o and planned the time, location, and path planning for the next communication mission. Therefore, in this communication, robot i only needs to consider how to meet robot j after this. Robot j must first perform the planning between time and The first path planning between time and then consider the and The second path planning between the two paths. , it is necessary to determine whether the next human-machine communication occurs during the second segment of path planning. If so, replanning of the second segment is necessary. Before robots i and j can communicate, robot j must first travel to the human side as a messenger to communicate and interact. If not, robot j can proceed directly to the second segment of path planning without returning to the human side. These planning steps can be implemented using the principles of a greedy algorithm, as described in related art, and are not limited in the present embodiments.

[0260] According to the aforementioned setup, every 300 seconds, at least one robot must act as a messenger to return to the human client to communicate and respond to the human's interaction requests. After returning to the human client, the messenger sends the fused environmental map and other data to the human client and waits for the human client's interaction request. The human client (the operator's handheld electronic device) then receives and displays the map and other data returned by the robot, performing local map fusion. The human client then views the fused environmental map, environmental photos, and photos of suspicious areas on the display.

[0261] The operator makes upper-level decisions based on his or her own needs and the overall situation. At this point, the operator can issue four types of interaction requests: (1) No special processing, allowing the robot to continue its own task. At this point, the robot will directly complete the interaction with the human side and enter the next round of exploration tasks. (2) Specify a priority exploration area, which is the area of ​​interest to the human side. After receiving this instruction, the robot returns to the next round of exploration tasks and gradually transmits the human side's exploration requirements for the priority area to other robots in subsequent communication events. (3) Request the operator's feasible movement domain. The operator issues an interaction request for the feasible movement domain to allow the robot to calculate the current safe area where humans can move. The operator then specifies the destination to go to in the safe area, and the robot will calculate the operator's reference movement path. (4) Request the formation of a relay link to perform human-assisted collaborative tasks in a certain area.

[0262] For the first three interaction modes, after the messenger completes the interaction with the human end, the robot returns to the previously planned task and enters the execution of the next round of exploration tasks. For the fourth interaction mode, detailed description will be given later.

[0263] At this point, the robots have completed a cycle of "-exploration-communication-(interaction)-". The "interaction" phase is not required in every cycle, but is determined by the interaction time set by the human end (determined based on the human-machine communication interval). Each robot conducts distributed collaborative exploration of the environment through the "-exploration-communication-(interaction)-" cycle until the environment map is fully explored. Figure 10 As shown in the figure, multiple robots are distributed and working according to the planned tasks. Robots i, m, n, and o are performing exploration tasks, robots k and l are communicating, and robot j is returning to the human side, about to transmit its local map back to the operator. The operator continuously interacts with the returning robots, updating their own map in a timely manner and expressing their own interaction needs.

[0264] For the fourth interaction mode, the embodiment of the present invention is implemented in the following specific manner:

[0265] We still use the original cave scene to introduce the whole process. Figure 11 shown.

[0266] As the robot explores according to the above process, it often encounters suspicious areas. These areas require operator intervention for real-time audio and video monitoring, as well as human control of the robot. However, due to the unique terrain, narrow passages, or dangerous conditions in these areas, operators are not allowed to enter and operate them, so remote control is the only option. However, due to communication limitations in cave environments, communication over such long distances is impossible. Therefore, the only way to meet these requirements is to use multiple robots in a relay mode, forming a communication relay link between the operator and the remote robot.

[0267] The robot records the suspicious area's photos and location and transmits them to other robots via communication. When the messenger robot returns to the human user, the user receives the suspicious area information. At this point, the operator can initiate a fourth type of human-robot interaction: requesting a communication relay link to perform a human-assisted collaborative task in the suspicious area.

[0268] Based on the planned mission location and the operator's current location, the operator will plan a high-quality communication relay link using the communication quality model of the current environment. Assume that this communication relay link requires five relay nodes. Starting with the messenger robot, it will sequentially locate four successor robots according to the communication order within the exploration team and replan their communication events. This ensures that each selected robot receives its selection message first and relays the message to its successor robot before leaving the exploration team to head for the relay node. In this way, the five robots, including the messenger, will serve as relay robots on the communication relay link for subsequent human-assisted tasks. The operator will then be presented with the planning results and asked whether to move.

[0269] For example, an operator determines that the communication relay link for five robots is too long. This leaves only two robots in the exploration team, hindering exploration efficiency. Therefore, the operator decides to work closer to the task location. The operator selects their desired work location on the map, and the human client replans the communication relay link based on the planned task location and the operator's selected work location. Because the operator is closer to the task location, the communication relay link now only requires three robots. The human client replans communication events for the three robots, including the messenger, and displays the planning results to the operator.

[0270] The operator can see the following information on the interactive interface of the human side: audio and video information, map information, joystick and other information, such as the above-mentioned planning results.

[0271] The communication relay link can include robots k (the remote robot, also called the end robot), l, and j, and is used to complete human-assisted tasks such as transmitting audio and video or remote control commands. Robots i, m, n, and o collaborate to explore the map.

[0272] The operator expressed satisfaction with the link planning results and concluded the human-robot interaction. The human agent sent the planning results and the new communication plan to the messenger. The messenger received the request for the human-assisted task issued by the human agent, updated its own communication plan and that of other robots, and returned to the communication location in the original exploration team as planned. It then communicated with the successor robot, passing the human-assisted task information and the updated communication plan to the successor robot, and finally proceeded directly to its assigned node on the communication relay link. Similarly, all selected successor robots first received the human-assisted task information and the updated communication plan from their predecessor robot, then passed them to their successor robot, and finally proceeded directly to their assigned node on the communication relay link. Thus, all selected robots in the original exploration team eventually arrived at the corresponding location on the communication relay link, while unselected robots only transmitted the human-assisted task information and the updated communication plan information. This transmission continued until the information had been transmitted through the exploration team and all robots were aware of it. At this point, the communication topology has changed. Some robots in the exploration team are assigned to the "relay" team to form a link, and the original ring communication topology becomes an updated ring topology. This process is as follows Figure 3 shown.

[0273] Once all relay robots and operators are in place, the communication relay link is established. The operator can establish communication with the remote robot through the human-side interface to perform human-assisted tasks. The operator sends a command to the end-user robot to capture the surrounding environment. This command is relayed by the human-side interface along the relay link to the end-user robot. Upon receiving the command, the end-user robot uses its camera to capture the surrounding environment and the target. For example, robot k captures the surrounding environment and forwards the audio and video along the relay link to the human-side interface, which displays the audio and video on the interactive interface. The operator observes the audio and video information and initiates a remote control request. After receiving this request, the end-user robot begins receiving remote control commands. The operator then uses the virtual joystick and other buttons on the human-side interface to remotely control the end-user robot, causing it to move or grasp objects, completing the task.

[0274] During the human-assisted task, the exploration team is still conducting map exploration in parallel. Every scheduled human-machine communication event T, a messenger robot will return to the human side to communicate and exchange the latest map information, communication plan, new suspicious area photos, and other information. As described above for the multi-level collaborative exploration process, the operator can interact with the messenger through the human side. The difference is that the operator is doing the human-assisted task, so there is no need to move, nor is there a need to initiate the human-assisted task again. Therefore, there are only two interaction methods (1) and (2). After the operator issues the instruction (1) or (2), the messenger will not return to the exploration team immediately, but will wait for the next instruction from the human side.

[0275] The human client asks the operator for the current human-assisted task and its estimated completion time. If the operator's estimated completion time is too long or cannot be estimated (greater than a certain threshold, which serves as the boundary point for whether to terminate the link later), the link is not processed in this round and the messenger returns to the exploration team. If the estimated time is not too long (0 or less than a certain value), the human client will enter the link processing process.

[0276] The human side first determines whether the courier has discovered a new human-assisted task and asks the operator to make a decision. The operator can select one of the remaining human-assisted tasks or refuse to continue with the human-assisted task. If the operator selects a task, the link transfer process begins; if the operator refuses to continue with the task, the link dismissal process begins.

[0277] The link transfer process is as follows Figure 4A 、 Figure 4B and Figure 4CAs shown in the figure, the specific contents are as follows: the human side first asks the operator to select the next operator position, then generates a new link based on the position of the new human-assisted task and the operator's position, and then compares the information of the current link and the new link. There are three cases: (1) If the number of relays on the current link is equal to the number of relays required on the new link, the current relay robot is directly assigned to the new relay, and the time-optimal assignment is calculated to complete the planning. After the current human-assisted task is completed, the current link will be transferred to the new link according to the assignment result. This situation does not involve the exchange of robots between the relay team and the exploration team, and the messenger will return directly to the exploration team; (2) If the number of relays on the current link is less than the number of relays required on the new link, it is still necessary to "borrow" a sufficient number of robots from the exploration team in the subsequent steps. Based on the communication plan of the exploration team given by the messenger, the human side plans the robots required to complete the "borrowing" event, communication events (including arrival time and arrival location), and communication topology changes. This planning process is similar to the aforementioned process of generating relay links, which is equivalent to generating a new human-assisted task. The human side will publish the completed plan to the messenger. The messenger will receive the human side's "borrowing" request and enter the mode switching state. The subsequent steps are the same as the above. The messenger will pass the human-assisted task information, new communication events, and communication topology information to the robots in the exploration team. After one round of information transmission, the communication topology changes, and the robots in the exploration team are allocated enough individuals and "borrowed" to the predetermined position of the new relay link; (3) If the number of relays on the current link is greater than the number of relays required on the new link, only the number of robots required for the new link will be allocated. The optimal allocation is calculated in time. After the current task is completed, these robots will be transferred to the new link according to the allocation results. The excess robots will be "dismissed" back to the exploration team in the subsequent steps.

[0278] The specific link removal process is as described above. Figure 6 The content shown.

[0279] After the messenger receives a reunion request, it updates the communication event, changes the stored communication topology, and alters the communication order of the exploration team, forming a new ring communication topology. It then returns to the original communication location within the exploration team as planned. This essentially means returning to the communication location within the new ring communication topology according to the new communication plan, where the agreed-upon communication location is the same as the messenger's original communication location within the exploration team. The reunion request, new communication event, and communication topology are then passed back to the successor node. The successor node reads the reunion message and similarly updates the communication event, changing the stored communication topology and altering the communication order within the exploration team, forming a new ring communication topology. This information is then passed back to the successor node. The dismissed relay robot arrives at the communication location at the agreed-upon time, communicates with the preceding and following robots, and then switches to "exploration" mode. When the last relay robot successfully reunites, the mode switch for all robots is complete, forming a new exploration team. After the message has been passed through the exploration team once, all robots in the exploration team are aware of the new communication topology.

[0280] Through the above content, a human-centered system in which humans and multiple machines work together under communication constraints has been fully realized.

[0281] The embodiment of the present invention further provides a human-machine collaborative working method, which is applied to the human-machine collaborative working system in the above embodiment. The human-machine collaborative working system includes: a human terminal and multiple robots. The human-machine collaborative working method includes:

[0282] Receiving a human-assisted task request including a task location through a human terminal; obtaining link information based on the task location, the current location of the human terminal, and a first communication plan, and sending the information to a first messenger robot. The link information includes topology information of a communication relay link and a second communication plan. The first communication plan is a communication plan for each robot performing map exploration in the current ring topology, including a first communication time and a first communication location for communication between two robots.

[0283] The first messenger robot sends the link information to the first successor robot according to the current communication order, and arrives at the second communication location at the agreed second communication time;

[0284] The first successor robot and the robot behind the first successor robot transmit link information, and each robot determines whether it is a robot on the communication relay link based on the link information. If so, it arrives at the second communication location at the agreed second communication time; if not, it continues to perform map exploration; some of the multiple robots performing map exploration in the current ring topology structure serve as nodes of the communication relay link;

[0285] The end robot at the end of the communication relay link performs human-assisted tasks;

[0286] The robots other than the terminal robot in the communication relay link receive the data sent by the preceding node and transmit the received data to the subsequent node.

[0287] An embodiment of the present invention further provides a robot, including the multiple robots included in the human-machine collaborative work system in the above embodiment.

[0288] An embodiment of the present invention further provides a human terminal, including the human terminal included in the human-machine collaborative work system in the above embodiment.

[0289] In another embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned human-computer collaborative working methods are implemented.

[0290] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the human-computer collaborative working methods in the above embodiments.

[0291] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state disk (SSD)).

[0292] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0293] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences between other embodiments. In particular, the method, robot, human terminal, computer storage medium, and computer program product embodiments are generally similar to the system embodiments, so their descriptions are relatively simplified. For related portions, reference can be made to the descriptions of the system embodiments.

[0294] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A human-machine collaborative work system, characterized in that: include: Human side and multiple robots; The human end is used to receive a human-assisted task request including a task location; Based on the task location, the current location of the human terminal, and the first communication plan, link information is obtained and sent to the first messenger robot, the link information including topology information of the communication relay link and the second communication plan; the first communication plan is the communication plan of each robot performing map exploration in the current ring topology structure, including a first communication time and a first communication location for communication between two robots; The first messenger robot is configured to send the link information to the first successor robot according to the current communication order and arrive at the second communication location at the agreed second communication time; The first successor robot and the robot following the first successor robot are configured to transmit the link information and, based on the link information, determine whether they are robots on the communication relay link. If so, they arrive at the second communication location at the agreed second communication time; if not, they continue to perform map exploration; some of the multiple robots performing map exploration in the current ring topology serve as nodes of the communication relay link; The end robot at the end of the communication relay link is used to perform the human-assisted task; The robots other than the terminal robot in the communication relay link are used to receive data sent by the preceding node and transmit the received data to the subsequent node.

2. The system according to claim 1, wherein: The first messenger robot is specifically used to receive the link information sent by the human end, wherein the first messenger robot is a robot that returns to the human end within the human-machine communication interval; based on the link information, locally updates its own stored communication plan and topology information of itself and a first other robot, wherein the first other robot is a robot other than the first messenger robot among multiple robots performing map exploration in the current ring topology structure; moves to a first communication location in the current ring topology structure that communicates with a first successor robot, and sends the link information to the first successor robot, and arrives at a second communication location at the agreed second communication time according to the communication plan of the first messenger robot in the second communication plan; the first successor robot is a robot that is located after the first messenger robot in the current communication order in the current ring topology structure; The first successor robot is specifically used to locally update its own stored communication plan and topology information of itself and a second other robot. The second other robot is a robot other than the first successor robot among multiple robots exploring the map in the current ring topology structure, and sends the link information to the robot that is located after the first successor robot in the current ring topology structure according to the current communication order. The robot that is located after the first successor robot performs the same operation as the first successor robot until all robots in the current ring topology structure have obtained the topology information of the communication relay link and the second communication plan.

3. The system according to claim 1, wherein: The human terminal is further configured to confirm task information of the human-assisted task and send the task information to the robot communicating with the human terminal on the communication relay link, wherein the task information includes a task type; If the robot communicating with the human end is an end robot, the robot communicating with the human end receives the task information and performs the human-assisted task based on the task information; if the robot communicating with the human end is not an end robot, the robot communicating with the human end continues to transmit the task information in the direction from the human end to the end robot until the task information is sent to the end robot, and the end robot receives the task information and performs the human-assisted task based on the task information.

4. The system according to claim 3, characterized in that After the terminal robot performs the human-assisted task and obtains a task result, if the terminal robot is communicating with the human end, the terminal robot sends the task result to the human end; If the end robot is not communicating with the human end, the end robot sends the task result to the robot communicating with the end robot, and the robot communicating with the end robot continues to transmit the task result along the direction from the end robot to the human end until the task result is sent to the human end.

5. The system according to claim 1, wherein: The human end is also used to receive the new location selected by the operator; re-plan the task and update the communication plan of each robot based on the task location, the new location and the communication plan of multiple robots exploring the map in the current ring topology structure, and obtain the communication relay link and the updated communication plan of each robot.

6. The system according to claim 1, wherein: The human end is further configured to determine an estimated completion time for the current human-assisted task; if the estimated completion time is less than an agreed threshold, determine whether there are any unprocessed human-assisted tasks; If yes, then plan for link transfer; if no, then plan for link removal.

7. The system according to claim 6, characterized in that The human end is specifically used to determine the operator position and the new task location corresponding to the unprocessed human-assisted task; based on the operator position and the new task location, determine the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is equal to the number of relay nodes required for the new task, then the relay nodes in the current communication relay link are used as relay nodes of the new communication relay link; if the number of relay nodes in the current communication relay link is less than the number of relay nodes required for the new task, then select a first target robot from the robots used for map exploration, and the sum of the number of the first target robots and the number of relay nodes in the current communication relay link reaches the number of relay nodes required for the new task; if the number of relay nodes in the current communication relay link is greater than the number of relay nodes required for the new task, then select a second target robot from the relay nodes in the current communication relay link as the relay node of the new communication relay link.

8. The system according to claim 6, wherein: The human end is specifically configured to obtain a third communication plan from the second messenger robot, where the third communication plan is a new communication plan for the multiple robots currently performing map exploration; based on the third communication plan, the multiple robots currently performing map exploration and the robots in the current communication relay link are combined into a new ring topology structure, and the communication plans of the robots in the current communication relay link are updated to obtain a fourth communication plan; and the topology information of the new ring topology structure and the fourth communication plan are sent to the second messenger robot and the robots in the current communication relay link; The second messenger robot is configured to locally update the topology information of the new ring topology structure and the fourth communication plan after receiving the topology information of the new ring topology structure and the fourth communication plan sent by the human terminal; Arriving at a fourth communication location at a fourth communication time according to the communication plan of the second courier robot in the fourth communication plan, and sending topology information of the new ring topology and the fourth communication plan to a second successor robot of the second courier robot in the new ring topology; The second successor robot is configured to locally update topology information of the new ring topology structure and the fourth communication plan, and send the topology information of the new ring topology structure and the fourth communication plan to a subsequent robot of the second successor robot; The robot in the current communication relay link is configured to locally update the topology information of the new ring topology structure and the fourth communication plan, and arrive at the agreed communication location according to the fourth communication plan; The robots in the new ring topology are used to explore the map.

9. A human-machine collaborative working method, characterized in that: The human-machine collaborative work system according to any one of claims 1 to 8 comprises: a human terminal and multiple robots, and the method comprises: Receiving, via the human terminal, a human-assisted task request including a task location; obtaining link information based on the task location, the current location of the human terminal, and a first communication plan, and sending the information to a first messenger robot, the link information including topology information of a communication relay link and a second communication plan; the first communication plan being a communication plan for each robot performing map exploration in the current ring topology, including a first communication time and a first communication location for communication between two robots; The link information is sent to the first successor robot via the first messenger robot in accordance with the current communication order, and arrives at the second communication location at the agreed second communication time; The link information is transmitted through the first successor robot and the robot located after the first successor robot. Based on the link information, each robot determines whether it is a robot on the communication relay link. If so, it arrives at the second communication location at the agreed second communication time; if not, it continues to perform map exploration; some of the multiple robots performing map exploration in the current ring topology structure serve as nodes of the communication relay link; Performing the human-assisted task through an end robot at an end of the communication relay link; The robots other than the terminal robot in the communication relay link receive data sent by the preceding node and transmit the received data to the subsequent node.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of the method according to claim 9 are implemented.

11. A computer program product comprising instructions, characterized in that When the method is executed on a computer, the computer is enabled to execute the method according to claim 9 .

Images