Interaction method and system of multi-person collaborative bomb disposal robot based on mixed reality

Abstract

The present invention provides an interactive method and system for a multi-person collaborative bomb disposal robot based on mixed reality, the method comprising: using MR equipment and a computer to start up and run preset software; selecting a mode among an expert supervision mode, a multi-person collaborative information fusion operation, and a multi-person cooperative operation of multiple robots; setting a virtual object posture relationship; collecting operator data; data processing; sending a control instruction to control the bomb disposal robot; shutting down the device, and the task is completed. The present invention solves the technical problems of excessive redundancy of interactive equipment, dependence on manual operation experience due to complex operation process, and poor interactive control effect.

Description

Technical Field

[0001] The present invention relates to the field of robot control, and in particular to an interaction method and system of a multi-person collaborative bomb disposal robot based on mixed reality. Background Art

[0002] EOD robots are a type of robot that is specifically designed to handle explosives, bombs, and other dangerous objects. They play a key role in law enforcement, military, and security. These robots have a variety of roles and functions, including explosive disposal, explosive location, dangerous goods dismantling, search, rescue, reconnaissance, and intelligence gathering. With its efficiency, precision, and diversity, EOD robots have become an important tool for handling dangerous situations. They can significantly reduce the risk of operators in dangerous environments, improve the efficiency of task execution, and effectively ensure the safety of personnel. EOD robots use various sensors to obtain environmental information and target locations, thereby enhancing the safety of operators and improving the quality of task execution. In addition, the versatility of EOD robots allows them to be customized according to specific task requirements and adapt to various complex environments and task scenarios. This flexibility greatly enhances the operator's work efficiency. With the increasing complexity of the operating environment and the refinement of operating requirements, EOD robots and operators face higher challenges in order to effectively handle various dangerous objects. In this case, the role of the remote control system is particularly important. It ensures that the robot accurately executes the operator's instructions and provides real-time feedback so that the operator can understand the robot's status and surrounding environment in a timely manner.

[0003] Teleoperation systems usually consist of key parts such as operation terminals, interactive devices, sensors, communication systems, control algorithms and user interfaces, which work together to realize remote control of robots. First, the operation terminal is the key device used by operators to interact with the robot. The interactive device provides users with instructions and status monitoring between the robot, usually including control buttons, joysticks, touch screens, keyboards, etc. Secondly, sensors and are used to collect data and images of the robot's surrounding environment. They capture real-time information and transmit it back to the operation terminal for operators to analyze, judge and make decisions. The communication system is responsible for transmitting instructions and data between the control station and the robot, usually using wireless communication technologies such as radio, Wi-Fi, Bluetooth or satellite communication to ensure real-time data transmission and operation feedback. The control algorithm performs data processing and instruction conversion between the operation terminal and the robot. These algorithms translate the operator's instructions into the robot's actions to ensure that the robot responds in a predetermined manner. The user interface provides the operator with a window to interact with the system, displaying the robot's real-time status, environmental information and operation options, so that the operator can fully understand the robot's status.

[0004] For example, the existing invention patent application document "Explosive Ordnance Disposal Robot Control System" with publication number CN104942811A, the system includes a main control part with a main control microprocessor and a first image transmission module and a slave part with a slave microprocessor and a second image transmission module, the input end of the main control microprocessor is correspondingly connected to the output end of the real-time action information acquisition gloves, the manual joystick, the action information conversion button and the image conversion button, and its output end is connected to the input end of the wireless transmission module; the output end of the first image transmission module is correspondingly connected to the input end of the display screen and the 3D visual system; the input end of the slave microprocessor is connected to the output end of the wireless receiving module, and its output end is correspondingly connected to the input end of the robot arm drive module, the robot steering drive module and the video acquisition module; the input end of the second image transmission module is connected to the output end of the video acquisition module. It can be seen from the specific implementation content of the existing scheme that the interactive device of the remote control system is composed of data gloves, manual joysticks, etc., to realize the movement and steering of the mechanical arm of the explosive disposal robot, the closing of the gripper and other operations, thereby completing the explosive disposal task. However, the aforementioned interactive equipment in the prior art is too redundant and the operation process is complicated, requiring the operator to have a high level of proficiency and emergency response capabilities.

[0005] The existing invention patent application document "Explosive Ordnance Disposal Robot System, Control Method and Storage Medium Based on Virtual Reality" with publication number CN111452046A, the existing method includes: step 201, obtaining image information captured by the camera pan-tilt, imaging the image information in the VR head display; step 202, obtaining the user's hand motion information, generating the motion posture of the robot arm according to the user's hand motion information, and controlling the robot arm according to the motion posture; step 203, obtaining the angle information of the gyroscope, and controlling the camera pan-tilt to follow the rotation according to the angle information; step 204, repeating the above steps 201 to 203. From the specific implementation content of the existing method, it can be seen that the interactive device of the existing remote control system consists of a VR head display and Leap Motion, and the hand posture is recognized by Leap Motion, and the end of the robot arm is controlled by different gesture semantics to complete the explosive disposal task. However, the virtual reality is not real enough, and the recognition range of Leap Motion is limited. The operator must perform gesture operations in a small space. In addition, as an optical sensor, Leap Motion is highly sensitive to ambient lighting, and performance may be affected in backgrounds with strong lighting or light reflections.

[0006] The existing invention patent application document "AR immersive remote-operated bomb disposal robot system, control method and storage medium" with publication number CN116372954A, the existing system includes: a head-mounted AR helmet, an AR host, a cloud server, a mobile bomb disposal robot and a force feedback control handle, etc.; the head-mounted AR helmet controls the pan-tilt to drive the binocular camera through the AR host, the cloud server, the robot central processor, and the robot motion control board; the binocular camera transmits the collected environmental information to the head-mounted AR helmet through the robot central processor, the cloud server, and the AR host; the force feedback control handle controls the mechanical arm and hand grasping action through the AR host, the cloud server, the robot central processor, and the robot motion control board to realize the bomb disposal function. It can be seen from the specific implementation content of the prior art that the interactive device of the existing teleoperation system consists of an AR head display and a force feedback device, which displays the on-site image through AR, and the force feedback device realizes the operation of the mechanical arm to complete the bomb disposal task. However, the immersion of the AR head display may be insufficient. When the user wears the head display, the line of sight may be blocked, which may lead to misoperation when using the force feedback device.

[0007] Existing EOD robot teleoperation systems are mainly based on a single-operation mode. In this mode, the operator needs to invest a lot of time in skill training to become familiar with the interface and operation methods, otherwise operational errors may occur. In addition, during the execution of the task, the operator needs to continuously monitor the status of the equipment and the robot, which may cause operator fatigue and distraction, thus affecting the operation quality and task efficiency.

[0008] In summary, the existing technology has technical problems such as excessive redundancy of interactive equipment, complicated operation process leading to dependence on manual operation experience, and poor interactive control effect. Summary of the invention

[0009] The technical problem to be solved by the present invention is how to solve the technical problems in the prior art that interactive equipment is too redundant, the operation process is complicated resulting in dependence on manual operation experience, and the interactive control effect is poor.

[0010] The present invention adopts the following technical solutions to solve the above technical problems: The interactive method of multi-person collaborative bomb disposal robot based on mixed reality includes:

[0011] S1. Using the preset software to run the terminal device to run the preset software;

[0012] S2. Using the Hungarian algorithm to process the bomb disposal task allocation information, the operator's operation weight and current task status are obtained, the operator's ability data is collected and updated, and the bomb disposal robot interactive control mode is selected according to the current task complexity information. The bomb disposal robot interactive control mode includes: expert supervision mode, multi-person collaborative information fusion operation mode, and multi-person cooperative operation of multiple robots mode;

[0013] S3, according to the real scene, use the camera carried by the bomb disposal robot to perform SLAM reconstruction of the surrounding environment to obtain and generate a virtual environment based on the reconstruction model;

[0014] S4. Generate different weights for operators according to the current task status, and generate bomb disposal control instructions according to the Hungarian algorithm;

[0015] S5. Sending a bomb disposal control command to the bomb disposal robot to control the bomb disposal robot.

[0016] The present invention designs an interactive control mode for the bomb disposal robot, so that the operator can effectively control and cooperate with the bomb disposal robot. When encountering complex operations and unexpected situations during the bomb disposal mission, the teleoperation system can ensure that the robot accurately executes the operator's instructions and provides real-time feedback so that the operator can understand the robot's status and environment.

[0017] In the human-computer interaction of the remote operation system of the present invention, mixed reality (MR) is a new type of interaction platform, which perceives human natural behavior through multiple modalities, obtains multiple interaction channels such as human voice, gesture, and line of sight, and enables them to be selectively parallel and collaborative to achieve multi-channel, high-bandwidth, efficient and unrestricted location human-computer interaction.

[0018] In a more specific technical solution, in step S2, in the expert supervision mode, the steps of controlling the bomb disposal robot include:

[0019] S21, obtaining information on the nature of the task, the task target, and the execution process, and collecting information on the task area environment;

[0020] S22. Using Hololens 2 development tools, create a mixed reality environment based on task nature information, task target information, execution process information, and task area environment information;

[0021] S23. Establishing a remote communication connection between the operation end and the expert guidance end to realize real-time voice communication and share the mixed reality environment;

[0022] S24, using the expert guidance terminal to send the guidance command of the bomb disposal robot to the operation terminal, wherein the guidance command of the bomb disposal robot includes: gesture command and voice command:

[0023] S25. At the operation end, receiving and using the Hololens 2 development tool according to the guidance command of the explosive ordnance disposal robot, remotely controlling the explosive ordnance disposal robot to adjust the real-time status of the explosive ordnance disposal robot;

[0024] S26, using the operation terminal, monitoring and feeding back the real-time task progress information of the bomb disposal robot to the expert guidance terminal, and evaluating the real-time task progress information at the expert guidance terminal to obtain a task progress evaluation result;

[0025] S27. At the expert guidance end, based on the task progress assessment results, the task risk information is assessed, risk response strategy information is generated, and the risk assessment and response strategy information is sent to the operation end through virtual gesture information.

[0026] The present invention designs a dedicated remote control interactive device and control method for bomb disposal tasks, and performs complex bomb disposal tasks by sensing the interaction mode of human natural behavior through the interactive device. The present invention designs a combination of gesture and voice interaction to control the end of the mechanical arm of the bomb disposal robot, which can achieve a more natural and humanized interaction mode.

[0027] In a more specific technical solution, in step S2, in the multi-person collaborative information fusion operation mode, the steps of controlling the bomb disposal robot include:

[0028] S21', obtaining explosive ordnance disposal task information and setting a priority for each explosive ordnance disposal task;

[0029] S22', at the operation end, obtaining operator information, and evaluating and obtaining operator capability data based on the information;

[0030] S23', according to the priority and operator capability data, using the Hungarian algorithm, processing to obtain the bomb disposal task allocation information;

[0031] S24', dynamically update priority and operator capability data.

[0032] In view of the fact that the current single-person remote-operated bomb disposal robot may fail to respond to emergencies and may misoperate, the present invention designs an interactive system for multi-person collaborative bomb disposal robots to complete complex bomb disposal tasks. The present invention creates an environment that integrates virtual and real life to provide a richer collaborative experience, and can share information in real time. Multiple people can view the video stream and sensor data of real-world robots together, and can also observe real-time information about the robot's status, environmental information, and task progress in virtual reality. The present invention can jointly operate robots, jointly formulate plans, and collaboratively perform tasks to avoid collisions and conflicts. And based on the 5G communication protocol, multiple people can conduct real-time voice and video exchanges in the MR environment to discuss task progress and adjust strategies.

[0033] In a more specific technical solution, in step S21', the bomb disposal task information includes: task urgency, task importance information, and task execution period.

[0034] In a more specific technical solution, in step S23', the step of using the Hungarian algorithm to allocate the bomb disposal task also includes:

[0035] S231', initialize the matching information to empty;

[0036] S232', searching for an augmented path, wherein the augmented path is alternately provided with nodes corresponding to the bomb disposal task and the operator;

[0037] S233', when the augmented path is found, an additional matching operation is performed to reversely process all matching states on the augmented path;

[0038] S234', looping through steps S232' to S233' until no augmented path can be found, and taking the current match as the optimal match.

[0039] Based on the MR platform, the present invention designs a combination of gesture and voice interaction to control the end of the mechanical arm of the bomb disposal robot, which can achieve a more natural and humanized interaction method. An interactive system for multi-person collaborative bomb disposal robots is designed, which is mainly divided into three modes: expert supervision mode, multi-person collaborative information fusion operation, and multi-person cooperative operation of multiple robots. Multiple interactive modes are used to cope with the complex situation of bomb disposal tasks.

[0040] In a more specific technical solution, in step S232 ′, the starting point of the augmented path is set to an unmatched task node, and the ending node is set to no less than 2 unmatched operator nodes.

[0041] In a more specific technical solution, in step S232', the method of finding the augmenting path includes: depth-first search DFS and breadth-first search BFS.

[0042] In a more specific technical solution, in step S23', the objective function of the Hungarian algorithm is expressed using the following logic:

[0043]

[0044] In the formula, c ij represents the cost or weight of assigning task i to operator j, x ij is a binary variable indicating whether task i is assigned to operator j. Usually x ij A value of 1 indicates that task i is assigned to operator j.

[0045] In a more specific technical solution, in step S23', the constraints of the Hungarian algorithm are expressed using the following logic:

[0046]

[0047] In a more specific technical solution, the interactive system of a multi-person collaborative bomb disposal robot based on mixed reality includes:

[0048] A software running module is used to run the preset software on the terminal device using the preset software;

[0049] The mode selection and task allocation module is used to obtain the bomb disposal task allocation information by using the Hungarian algorithm, and obtain the operation weight and current task status of the operator, collect and update the operator's ability data, obtain and select the bomb disposal robot interactive control mode according to the current task complexity information, wherein the bomb disposal robot interactive control mode includes: expert supervision mode, multi-person collaborative information fusion operation mode and multi-person cooperative operation of multiple robots mode, and the mode selection and task allocation module is connected with the software operation module;

[0050] The environment generation module is used to perform SLAM reconstruction of the surrounding environment based on the real scene using the camera carried by the bomb disposal robot to obtain and generate a virtual environment based on the reconstruction model;

[0051] The explosive disposal instruction generation module is used to generate different weights for operators according to the current task status and generate explosive disposal control instructions according to the Hungarian algorithm. The explosive disposal instruction generation module is connected with the environment generation module and the mode selection and task allocation module;

[0052] The bomb disposal robot control module is used to send bomb disposal control instructions to the bomb disposal robot to control the bomb disposal robot. The bomb disposal robot control module is connected to the bomb disposal instruction generation module.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] The present invention designs an interactive control mode for the bomb disposal robot, so that the operator can effectively control and cooperate with the bomb disposal robot. When encountering complex operations and unexpected situations during the bomb disposal mission, the teleoperation system can ensure that the robot accurately executes the operator's instructions and provides real-time feedback so that the operator can understand the robot's status and environment.

[0055] In the human-computer interaction of the remote operation system of the present invention, mixed reality (MR) is a new type of interaction platform, which perceives human natural behavior through multiple modalities, obtains multiple interaction channels such as human voice, gesture, and line of sight, and enables them to be selectively parallel and collaborative to achieve multi-channel, high-bandwidth, efficient and unrestricted location human-computer interaction.

[0056] The present invention designs a dedicated remote control interactive device and control method for bomb disposal tasks, and performs complex bomb disposal tasks by sensing the interaction mode of human natural behavior through the interactive device. The present invention designs a combination of gesture and voice interaction to control the end of the mechanical arm of the bomb disposal robot, which can achieve a more natural and humanized interaction mode.

[0057] In view of the fact that the current single-person remote-operated bomb disposal robot may fail to respond to emergencies and may misoperate, the present invention designs an interactive system for multi-person collaborative bomb disposal robots to complete complex bomb disposal tasks. The present invention creates an environment that integrates virtual and real life to provide a richer collaborative experience, and can share information in real time. Multiple people can view the video stream and sensor data of real-world robots together, and can also observe real-time information about the robot's status, environmental information, and task progress in virtual reality. The present invention can jointly operate robots, jointly formulate plans, and collaboratively perform tasks to avoid collisions and conflicts. And based on the 5G communication protocol, multiple people can conduct real-time voice and video exchanges in the MR environment to discuss task progress and adjust strategies.

[0058] Based on the MR platform, the present invention designs a combination of gesture and voice interaction to control the end of the mechanical arm of the bomb disposal robot, which can achieve a more natural and humanized interaction method. An interactive system for multi-person collaborative bomb disposal robots is designed, which is mainly divided into three modes: expert supervision mode, multi-person collaborative information fusion operation, and multi-person cooperative operation of multiple robots. Multiple interactive modes are used to cope with the complex situation of bomb disposal tasks.

[0059] The present invention solves the technical problems existing in the prior art that interactive equipment is too redundant, the operation process is complicated resulting in dependence on manual operation experience, and the interactive control effect is poor. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a schematic diagram of an interactive method of a multi-person collaborative bomb disposal robot based on mixed reality according to Embodiment 1 of the present invention;

[0061] Figure 2 This is a schematic diagram of specific steps for controlling the bomb disposal robot in the expert supervision mode according to the following specific steps in Example 1 of the present invention;

[0062] Figure 3 This is a schematic diagram of data flow interaction processing in the expert supervision mode of embodiment 1 of the present invention;

[0063] Figure 4 This is a schematic diagram of specific steps of multi-person collaborative information fusion operation according to Embodiment 1 of the present invention;

[0064] Figure 5This is a schematic diagram of data flow interaction processing of multi-person collaborative information fusion operation according to Embodiment 1 of the present invention;

[0065] Figure 6 This is a schematic diagram of specific steps of using the Hungarian algorithm in Example 1 of the present invention;

[0066] Figure 7 This is a schematic diagram of data stream interaction processing of multiple robots operating in cooperation with each other according to Embodiment 1 of the present invention;

[0067] Figure 8 This is a schematic diagram of data interaction processing of an interactive system of a multi-person collaborative bomb disposal robot based on mixed reality according to Embodiment 2 of the present invention;

[0068] Fig. 9 This is a schematic diagram of the structure of a multi-person collaborative bomb disposal robot according to Example 2 of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Example 1

[0071] like Figure 1 As shown, the interactive method of a multi-person collaborative bomb disposal robot based on mixed reality provided by the present invention includes the following basic steps:

[0072] S1. Use the MR device and computer to start and run the pre-installed software;

[0073] S2. Select a mode among the expert supervision mode, the multi-person collaborative information fusion operation mode, and the multi-person cooperative operation of multiple robots mode, and control the bomb disposal robot to perform the task according to the selected mode;

[0074] like Figure 2 and Figure 3 As shown, in this embodiment, the bomb disposal robot is controlled according to the following specific steps in the expert supervision mode:

[0075] In this embodiment, the supervision mode of the teleoperated bomb disposal robot is a way of performing tasks through expert guidance to ensure the accuracy and safety of the tasks.

[0076] S21, Mission preparation and planning;

[0077] In this embodiment, before executing the mission, the nature, objectives and execution steps of the mission are determined. Information such as the terrain, location of explosives, obstacles, etc. of the mission area is collected to create an accurate mission scene in a virtual reality environment;

[0078] S22, virtual environment creation;

[0079] In this embodiment, a virtual reality environment is created using a development tool of Hololens 2. In this embodiment, the operation of creating the virtual environment involves using map data, models, and images to reproduce a virtual scene of the mission site so that the guidance expert can observe the mission area in Hololens 2;

[0080] S23, remote connection and communication;

[0081] In this embodiment, the operator establishes a remote communication connection with the guidance expert to achieve real-time voice calls, virtual environment sharing, etc. In this embodiment, the aforementioned remote communication connection can be carried out through the Internet or a local area network, allowing the operator and the guidance expert to communicate in real time;

[0082] S24. Experts provide guidance and advice through orders;

[0083] In this embodiment, based on the observation of the virtual environment, the guidance expert provides guidance and suggestions to the operator using gestures or voice commands of Hololens 2. In this embodiment, the expert can use gestures to indicate the direction in which the robot should move, or use voice commands to instruct the operator to perform specific operations;

[0084] S25, task execution;

[0085] Under the guidance of the expert in this embodiment, the operator uses Hololens 2 to teleoperate the robot to perform the task. In this embodiment, the operator can see the real-time status and position of the robot and operate it according to the guidance of the expert;

[0086] S26, real-time feedback;

[0087] In this embodiment, the guidance expert can evaluate the progress of the task in real time; in this embodiment, the guidance expert can use the gestures and voice commands of Hololens 2 to provide real-time feedback, such as marking the completion of the task through gestures or using voice commands to adjust the operation strategy;

[0088] S27, Risk assessment and response;

[0089] In this embodiment, the guidance expert can assess the risk of the task and provide corresponding risk assessment and response suggestions through virtual gestures. In this embodiment, the guidance expert can use gestures to indicate potential sources of danger and ways to deal with these risks;

[0090] like Figure 4 and Figure 5 As shown, in this embodiment, the multi-person collaborative information fusion operation includes the following specific steps:

[0091] S21', task priority setting;

[0092] In this embodiment, a priority is set for each task, and the priority can be set according to the urgency, importance, task time period and other factors of the task; in this embodiment, the priority can be expressed numerically, and the higher the numerical value, the higher the priority;

[0093] In this embodiment, the multi-person collaborative information fusion operation is different from the supervisory mode in task preparation, division of labor and task execution. In this embodiment, the content determined by the task division operation includes but is not limited to: task nature, objectives and execution steps. In this embodiment, the task is decomposed into different subtasks and each operator is assigned corresponding work. Task execution means that each operator performs specific tasks according to the tasks assigned to them. Operators need to cooperate with each other to ensure the collaborative execution of their respective tasks.

[0094] S22', operator competence assessment;

[0095] In this embodiment, the ability of each operator is evaluated, and the ability of the operator can be evaluated based on experience, skills, and professional background. In this embodiment, the ability of the operator can be expressed numerically, and the higher the value, the stronger the ability;

[0096] S23', task allocation decision;

[0097] In the task division of this embodiment, a task allocation algorithm is used. The implementation of the task allocation algorithm in multi-person collaborative control can adopt different methods, depending on the nature of the task and the ability of the operator. This embodiment is based on a priority-based dynamic task allocation algorithm. In this embodiment, each task has a priority, and the operator allocates tasks according to his own ability and task priority;

[0098] In the allocation of this embodiment, the Hungarian algorithm is used. The Hungarian algorithm is a classic algorithm for solving the maximum weight matching problem of bipartite graphs. The goal of the Hungarian algorithm is to find an optimal task allocation solution so that the total weight is maximized. The following is a detailed introduction to the Hungarian algorithm:

[0099] like Figure 6 As shown, in this embodiment, the specific steps of using the Hungarian algorithm include:

[0100] S231', initialization;

[0101] In this embodiment, the matching is initialized to empty, that is, no task is assigned to the operator;

[0102] S232', find augmenting paths;

[0103] In each round of this embodiment, an augmenting path is attempted to be found. An augmenting path is an alternating path, whose starting point is an unmatched task node and whose ending node is an unmatched operator node. Depth-first search (DFS) or breadth-first search (BFS) may be used; in this embodiment, the alternating path includes but is not limited to: alternating task and operator nodes;

[0104] S233', add matching;

[0105] In this embodiment, once an augmenting path is found, the matching is increased. That is, all matching states on the path are reversed. This will result in some tasks being assigned to operators while other tasks are contacted for assignment;

[0106] S234', repeat the above steps S232' to S233';

[0107] In this embodiment, steps S232' to S233' are repeated until no augmenting path is found. At this time, the current match is the optimal match;

[0108] In the Hungarian algorithm adopted in this embodiment, the objective function and constraint conditions are expressed using the following formula:

[0109] Objective function:

[0110]

[0111] Among them, c ij represents the cost or weight of assigning task i to operator j, x ij is a binary variable indicating whether task i is assigned to operator j. Usually x ij The value of 1 indicates that task i is assigned to operator j, and 0 indicates that it is not assigned;

[0112] Constraints:

[0113] Each task can be assigned to only one operator:

[0114]

[0115] Each worker can only accept one task:

[0116]

[0117] S24', dynamic update;

[0118] In this embodiment, during the execution of the task, the priority of the EOD task and the status and capability of the operator may change, so it is necessary to update the dynamics in real time.

[0119] like Figure 7 As shown, in this embodiment, in the process of multi-person cooperation to operate multiple robots, the information fusion operation of multi-person collaboration and the information fusion operation of multi-person collaboration are different in task scale and degree of collaboration, but there are still certain similarities in communication, collaboration, etc. In terms of task preparation and division of labor, the task needs to be subdivided into multiple subtasks, and each subtask requires one or more operators to collaborate to operate different robots. In terms of collaboration, it is necessary to consider not only the collaboration between operators, but also the collaborative path planning and mutual perception between bomb disposal robots.

[0120] S3, setting the virtual object posture relationship;

[0121] S4, collect and process operator data;

[0122] S5. Send control instructions to control the bomb disposal robot; turn off the device and the mission is completed.

[0123] Example 2

[0124] In this embodiment, the interactive system of a multi-person collaborative bomb disposal robot based on mixed reality provided by the present invention adopts an interactive method of controlling a real robotic arm. The depth sensing technology of HoloLens 2 can accurately track the movement and position of the hand and a single finger, so that the user can interact with the virtual object through natural gestures, thereby realizing the function of gesture interaction. In addition, the user can also interact with the virtual object through voice commands or natural language conversations to realize the function of voice interaction. The addition of voice interaction provides an alternative and supplementary method for gesture interaction. Therefore, in this embodiment, gestures are combined with voice interaction, so that the user can control the posture of the end of the robotic arm in the virtual environment, and finally map it to the real robotic arm to complete the bomb disposal task.

[0125] In this embodiment, the corresponding instructions are input through voice interaction to start gesture interaction; in this embodiment, gesture interaction includes but is not limited to: respectively, the position of the wrist corresponds to the position of the end of the robot arm, the three-dimensional position difference between the front and rear frames of the wrist bone is obtained through the camera, and the obtained three-dimensional position difference is mapped to the virtual robot arm end 1 in a certain proportion to make it move, and when it reaches the predetermined position, this part of the function is terminated. The posture of the wrist corresponds to the posture of the end of the robot arm, and the three-dimensional posture difference between the front and rear frames of the wrist bone is obtained through the camera, and the obtained posture difference is mapped to the virtual robot arm end to make it rotate, and when it rotates to the predetermined position, the rotation function of the robot arm end is terminated; the three-dimensional position distance between the middle finger and the thumb is obtained through the camera, and the three-dimensional position distance is used as a vector length, and then the aforementioned vector length is mapped to the virtual robot arm gripper. Because the gripper performs tasks by rotating, the vector between the grippers allows the gripper to rotate for judgment, and the specified vector is reached to complete the task.

[0126] like Figure 8 As shown, in this embodiment, the camera installed on the bomb disposal robot presents the on-site situation on the screen of HoloLens 2 through real-time image transmission, so that the operator can grasp the real-time situation on the scene, and there is a high-precision force sensor on the gripper at the end of the robotic arm that can feed back relevant information to the operator in real time, so that the operator can operate more accurately during the task.

[0127] like Fig. 9 As shown, in this embodiment, the chassis of the bomb disposal robot in the interactive system of the multi-person collaborative bomb disposal robot based on mixed reality adopts a double swing arm crawler chassis, which has excellent stability and high maneuverability, allowing the robot to move freely in various terrains and environments. Using a 6-degree-of-freedom robotic arm as the execution end of the bomb disposal task improves the operational accuracy and diversification capabilities of the task execution, and can complete complex tasks at the same time. It is also equipped with a depth camera and sensor to detect complex terrain and find targets, and the waist can be bent to deal with targets of different heights.

[0128] In this embodiment, the bomb disposal robot comprises: a robot crawler mobile chassis 1 and an upper task load system 2 .

[0129] In this embodiment, the robot crawler mobile chassis 1 includes but is not limited to: a crawler travel mechanism 11, a swing arm obstacle crossing mechanism 12 and a chassis body 13;

[0130] In this embodiment, the upper task load system 2 needs to be customized according to the specific task scenario requirements;

[0131] In this embodiment, the operations performed by the bomb disposal robot include, but are not limited to: identification, destruction, and transfer of explosives;

[0132] In this embodiment, the upper load task load system 2 includes but is not limited to: a robot control cabinet 21, two sets of 6-DOF robot arms 22, two sets of robot end grippers 23, a visual recognition camera 24, and a waist swing mechanism 25;

[0133] See also Fig. 9 In this embodiment, the crawler driving structure 11 is respectively installed on both sides of the chassis body 13 to drive the robot to move; the swing arm obstacle crossing mechanism 12 is respectively installed on the front and rear ends of the chassis body 13, and can swing back and forth with the driving wheel axle as the center;

[0134] In the present embodiment, the robot control cabinet 21 is installed at the center of the upper panel of the chassis body 13, and the installation method can be, for example, bolt connection; in the present embodiment, a waist swing mechanism 25 is provided at the front end of the robot control cabinet 21, and a 6-DOF robotic arm 22 is installed on both sides of the upper part of the waist swing mechanism 25, and a four-finger robotic arm end clamp 23 is installed at the end of each 6-DOF robotic arm 22, and a visual recognition camera 24 is installed between the robotic arm end clamp 23 and the robotic arm 22, and the visual recognition camera 24 can observe the current state of the explosive in real time.

[0135] In summary, the present invention designs an interactive control method for the bomb disposal robot, so that the operator can effectively control and cooperate with the bomb disposal robot. When encountering complex operations and unexpected situations during the bomb disposal task, the teleoperation system can ensure that the robot accurately executes the operator's instructions and provides real-time feedback so that the operator can understand the robot's status and environment.

[0136] In the human-computer interaction of the remote operation system of the present invention, mixed reality (MR) is a new type of interaction platform, which perceives human natural behavior through multiple modalities, obtains multiple interaction channels such as human voice, gesture, vision, etc., and enables them to be selectively parallel and collaborative to achieve multi-channel, high-bandwidth, efficient and unrestricted location human-computer interaction.

[0137] The present invention designs a dedicated remote control interactive device and control method for bomb disposal tasks, and performs complex bomb disposal tasks by sensing the interaction mode of human natural behavior through the interactive device. The present invention designs a combination of gesture and voice interaction to control the end of the mechanical arm of the bomb disposal robot, which can achieve a more natural and humanized interaction mode.

[0138] In view of the fact that the current single-person remote-operated bomb disposal robot may fail to respond to emergencies and may misoperate, the present invention designs an interactive system for multi-person collaborative bomb disposal robots to complete complex bomb disposal tasks. The present invention creates an environment that integrates virtual and real, provides a richer collaborative experience, and enables real-time information sharing. Multiple people can view the video stream and sensor data of the real-world robot together, and can also observe real-time information about the robot's status, environmental information, and task progress in virtual reality. The present invention can jointly operate robots, jointly formulate plans, and collaboratively perform tasks to avoid collisions and conflicts. And based on the 5G communication protocol, multiple people can conduct real-time voice and video exchanges in the MR environment to discuss task progress and adjust strategies.

[0139] Based on the MR platform, the present invention designs a combination of gesture and voice interaction to control the end of the mechanical arm of the bomb disposal robot, which can achieve a more natural and humanized interaction method. An interactive system for multi-person collaborative bomb disposal robots is designed, which is mainly divided into three modes: expert supervision mode, multi-person collaborative information fusion operation, and multi-person cooperative operation of multiple robots. Multiple interactive modes are used to cope with the complex situation of bomb disposal tasks.

[0140] The present invention solves the technical problems existing in the prior art that interactive equipment is too redundant, the operation process is complicated resulting in dependence on manual operation experience, and the interactive control effect is poor.

[0141] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive method for a multi-person collaborative bomb disposal robot based on mixed reality, characterized in that: The method comprises: S1. Using the preset software to run the terminal device to run the preset software; S2. Using the Hungarian algorithm to process the bomb disposal task allocation information, the operation weight and current task status of the operator are obtained, the capability data of the operator is collected and updated, and the bomb disposal robot interactive control mode is selected according to the complexity information of the current task, wherein the bomb disposal robot interactive control mode includes: expert supervision mode, multi-person collaborative information fusion operation mode and multi-person cooperative operation of multi-robot mode; S3, according to the real scene, using the camera carried by the bomb disposal robot to perform SLAM reconstruction on the surrounding environment, so as to obtain and generate a virtual environment based on the reconstruction model; S4, generating different weights for the operators according to the current task status, and generating bomb disposal control instructions according to the Hungarian algorithm; S5. Send the bomb disposal control instruction to the bomb disposal robot to control the bomb disposal robot.

2. The interactive method of a multi-person collaborative bomb disposal robot based on mixed reality according to claim 1, characterized in that: In step S2, in the expert supervision mode, the steps of controlling the bomb disposal robot include: S21, obtaining information on the nature of the task, the task target, and the execution process, and collecting information on the task area environment; S22, using the Hololens 2 development tool to create a mixed reality environment according to the task nature information, the task target information, the execution process information and the task area environment information; S23, establishing a remote communication connection between the operation terminal and the expert guidance terminal to achieve real-time voice communication and share the mixed reality environment; S24, using the expert guidance terminal to send the bomb disposal robot guidance command to the operation terminal, wherein the bomb disposal robot guidance command includes: gesture command, voice command: S25, receiving at the operating end and performing remote operation on the bomb disposal robot using the Hololens 2 development tool according to the bomb disposal robot guidance command, so as to adjust the real-time state of the bomb disposal robot; S26, using the operating terminal to monitor and feed back the real-time task progress information of the bomb disposal robot to the expert guidance terminal, and evaluating the real-time task progress information at the expert guidance terminal to obtain a task progress evaluation result; S27. At the expert guidance end, based on the task progress assessment result, the task risk information is evaluated and the risk response strategy information is generated. The risk information and the response strategy information are sent to the operation end through virtual gesture information.

3. The interactive method of a multi-person collaborative bomb disposal robot based on mixed reality according to claim 1, characterized in that: In step S2, in the multi-person collaborative information fusion operation mode, the steps of controlling the bomb disposal robot include: S21', obtaining explosive ordnance disposal task information and setting a priority for each explosive ordnance disposal task; S22', at the operation end, obtaining operator information, and evaluating and obtaining operator capability data based on the information; S23', according to the priority and the operator capability data, using the Hungarian algorithm, processing to obtain the bomb disposal task allocation information; S24', dynamically update the priority and the operator capability data.

4. The interactive method of a multi-person collaborative bomb disposal robot based on mixed reality according to claim 3, characterized in that: In the step S21', the bomb disposal task information includes: task urgency, task importance information, and task execution period.

5. The interactive method of a multi-person collaborative bomb disposal robot based on mixed reality according to claim 3, characterized in that: In the step S23', the step of allocating the bomb disposal task using the Hungarian algorithm further includes: S231', initialize the matching information to empty; S232', searching for an augmented path, wherein the augmented path includes nodes alternately arranged at the nodes corresponding to the bomb disposal task and the operator; S233', when the augmenting path is found, performing an additional matching operation to reversely process all matching states on the augmenting path; S234', looping through step S232' to step S233' until the augmented path cannot be found, and taking the current match as the optimal match.

6. The interactive method of a multi-person collaborative bomb disposal robot based on mixed reality according to claim 5, characterized in that: In the step S232', the starting point of the augmented path is set to an unmatched task node, and the ending node is set to no less than 2 unmatched operator nodes.

7. The interactive method of a multi-person collaborative bomb disposal robot based on mixed reality according to claim 5, characterized in that: In the step S232', the method of searching for the augmenting path includes: depth-first search DFS and breadth-first search BFS.

8. The interactive method of a multi-person collaborative bomb disposal robot based on mixed reality according to claim 5, characterized in that: In step S23', the objective function of the Hungarian algorithm is expressed using the following logic: In the formula, c ij represents the cost or weight of assigning task i to operator j, x ij is a binary variable indicating whether task i is assigned to operator j, usually x ij A value of 1 indicates that task i is assigned to operator j.

9. The interactive method of a multi-person collaborative bomb disposal robot based on mixed reality according to claim 5, characterized in that: In step S23', the constraints of the Hungarian algorithm are expressed using the following logic:

10. An interactive system of multi-person collaborative bomb disposal robots based on mixed reality, characterized in that: The system comprises: A software running module is used to run the preset software on the terminal device using the preset software; A mode selection and task allocation module is used to obtain bomb disposal task allocation information by using the Hungarian algorithm, and obtain the operation weight and current task status of the operator, collect and update the ability data of the operator, obtain and select the interactive control mode of the bomb disposal robot according to the complexity information of the current task, wherein the interactive control mode of the bomb disposal robot includes: expert supervision mode, multi-person collaborative information fusion operation mode and multi-person cooperative operation of multi-robot mode, and the mode selection and task allocation module is connected to the software operation module; An environment generation module, used to perform SLAM reconstruction of the surrounding environment using the camera carried by the bomb disposal robot, so as to obtain and generate a virtual environment based on the reconstruction model; An explosive disposal instruction generation module, used to generate different weights for the operators according to the current task status, and to generate explosive disposal control instructions according to the Hungarian algorithm, wherein the explosive disposal instruction generation module is connected to the environment generation module and the mode selection and task allocation module; The bomb disposal robot control module is used to generate and send the bomb disposal control instruction to the bomb disposal robot to control the bomb disposal robot. The bomb disposal robot control module is connected to the bomb disposal instruction generation module.

Images