Collaborative perception, decision-making and control method of virtual-reality fusion of multiple robots in underground coal mines

Abstract

The present invention discloses a method for collaborative perception, decision-making and control of virtual-reality fusion of multiple robots in coal mines. The method includes two overall cycle processes, a virtual perception decision-making control cycle and a physical perception decision-making control cycle, and three distributed cycle processes, a virtual-reality collaborative perception cycle, a virtual-reality collaborative control cycle, and a virtual-reality collaborative decision-making cycle. The virtual environment and the actual environment are combined, and the sensors of multiple robots are used to obtain real-time environmental information of the coal mine. The information is then integrated with the three-dimensional model and the virtual environment to achieve collaborative perception and path planning of multiple robots. The composition efficiency and accuracy are improved through multi-robot collaborative control and point cloud fusion. Virtual reality technology is used to generate a highly visualized digital twin scene in real time, and remote interaction with the robot is carried out through VR equipment based on the real-time virtual scene and video feedback.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and in particular to a method for collaborative perception, decision-making and remote control of multiple robots in underground coal mines based on digital twins. Background Art

[0002] With the growing demand for intelligent and unmanned underground mining, robots with diverse functions and shapes are increasingly being used in various underground scenarios. Based on their functions, underground coal mine robots can be categorized as tunneling robots, inspection robots, rescue robots, transport robots, and cleaning robots. These robots are capable of autonomous movement, positioning, and operations in coal mines, including mining, tunneling, inspection, inspection, rescue, transport, cleaning, and dust removal.

[0003] Based on their locomotion, underground coal mine robots can be categorized as tracked, wheeled, legged, and skateboard robots. Tracked robots utilize tracks for mobility, offering strong maneuverability and stability, enabling them to navigate the complex terrain of underground coal mines. Wheeled robots utilize wheels for mobility, offering high speed and maneuverability, making them suitable for navigating relatively flat terrain. Legged robots are designed using bionic principles to mimic human gait, offering strong adaptability and flexibility, enabling them to navigate the complex terrain of underground coal mines. Skateboard robots utilize skateboards for mobility, offering advantages such as high speed and compact size, making them suitable for relatively flat terrain. Depending on the specific underground coal mine environment and operational requirements, different types of robots can be selected to complete the task.

[0004] Coal mine environments are complex, closed, and volatile. Robots used in these environments require accurate and rapid perception capabilities, capable of acquiring real-time environmental information and status data. They also need intelligent, autonomous decision-making capabilities, capable of independently developing appropriate action plans based on perceived environmental information and mission requirements. They also need to be highly reliable and adaptable, able to adjust control strategies to accommodate the complex and changing conditions of underground coal mines.

[0005] Publication No. CN113485325A describes a method for a coal mine underground pump room inspection robot based on SLAM mapping and autonomous navigation. This method utilizes Kinect machine vision's terrain perception system to identify obstacles, employs a motion planning model to plan the shortest path, and uses iterative least squares methods to calibrate the odometry and optimize the graph to create a globally consistent map. This method enables unmanned inspection of pump room equipment without the need for tracks, cables, or GPS.

[0006] Publication No. CN114398455A discloses a heterogeneous multi-robot collaborative SLAM map fusion method applied to a cloud server. This method receives maps constructed by a first and second drone covering different areas at different times, identifies overlapping areas and their relative motion relationships, and then fuses the two maps into a single global map. This global map is then distributed to each unmanned vehicle and updated based on the updated data received by the vehicle.

[0007] Publication No. CN112394701A discloses a multi-robot cloud control system based on a cloud-edge-end hybrid computing environment. The system includes an execution module, a communication module, a knowledge base module, an intelligent algorithm module, and a master control module. The execution module is responsible for the acquisition, processing, and execution of control instructions of the robot and sensor equipment. The communication module forwards and processes real-time and dynamic data in the cloud-edge-end environment. The scenario-wide knowledge base module is used to store the general knowledge involved in the scenario and provides a unified calling interface. The intelligent algorithm module provides the intelligent perception and autonomous decision-making algorithms required in the scenario. The master control module orchestrates and schedules robots to collaboratively complete specified tasks.

[0008] Publication No. CN111596691B discloses a decision-making modeling and collaborative control method and system for a multi-robot system based on human-in-the-loop (HIL). This method involves obtaining output information values ​​from the robots after they perform a task and selecting the robot's position deviation information as the human's decision-making information. Using the human drift diffusion model as a modeling approach, the human's decision-making behavior is modeled based on the human's decision-making information. Furthermore, HIL tasks are designed to help the robots successfully complete their tasks.

[0009] The coal mine environment is complex, closed, and changeable. Traditional robot navigation and positioning methods are difficult to achieve high-precision and real-time requirements in coal mine environments. In addition, the existing technology lacks effective support for the collaborative work of multiple robots in coal mine environments. Therefore, it is of great significance to develop a coal mine dual-robot collaborative SLAM mapping and digital twin system and its control method. When danger occurs underground, facing the complex unknown environment, it can more accurately construct a real-time visual map, quickly and accurately locate trapped personnel and damaged equipment underground, ensure the safety of personnel and the safe operation of comprehensive mining equipment, realize the visual remote scheduling of underground robots, and improve the intelligence of coal mines.

[0010] The underground robot perception and multi-robot collaborative decision-making control studied above have the following defects:

[0011] 1) Existing SLAM mapping methods for underground mines are mostly based on a single inspection robot. Using a single sensor device, they acquire robot motion information, determine the robot's underground position, and then perform SLAM mapping. This single-source SLAM mapping strategy results in low mapping efficiency and insufficient map accuracy.

[0012] 2) The underground environment is more complex than the ground environment, with a lower degree of visualization and more difficult feature information extraction than the ground environment. The SLAM technology based on machine vision is difficult to be directly applied to SLAM in coal mines.

[0013] 3) Current multi-robot LiDAR-based SLAM map fusion technology involves constructing SLAM maps for each robot, then fusing these local maps to create a global map. This technology does not involve collaborative control or collaborative mapping, and the basic mapping method remains similar to that of a single robot. However, this SLAM mapping method struggles with accurate positioning in underground mine environments, resulting in poor mapping results and difficulty in constructing accurate underground maps. The mapping process for each robot is relatively decentralized, making real-time map fusion difficult, and thus, unable to generate a global map of the current state and exploration in real time.

[0014] 4) The lack of a visual dispatch center makes it difficult to provide timely feedback to operators on the actual underground situation. The complex environment of coal mines requires highly visual real-time maps to monitor the working status of underground robots in real time through remote display devices.

[0015] 5) Currently, there is a lack of remote human-robot interaction during underground coal mine robot operations. Relying solely on the robot's autonomous obstacle avoidance and tracking is insufficient to ensure safety during underground operations. Some work areas are inaccessible to personnel, so ensuring the safety of both personnel and equipment underground requires human monitoring and remote control of the robot's operations. Summary of the Invention

[0016] The purpose of this invention is to provide a multi-robot collaborative perception, decision-making and control method for virtual-reality fusion in coal mines, improve the composition efficiency and accuracy through multi-robot collaborative control and point cloud fusion, use virtual reality technology to generate highly visualized digital twin scenes in real time, and remotely interact with robots through VR equipment based on real-time virtual scenes and video feedback.

[0017] To achieve the above objectives, the present invention provides a multi-robot virtual-reality fusion collaborative perception, decision-making and control method for coal mines, including a virtual-reality collaborative perception loop, a virtual-reality collaborative decision-making loop and a virtual-reality collaborative control loop;

[0018] The virtual-reality collaborative perception loop includes a robot physical perception system and a robot virtual perception system. The robot physical perception system obtains underground environmental information, as well as the position information and motion parameter information of each physical collaborative robot in the underground environment at the physical level, and generates physical point cloud information. The robot virtual perception system enhances the physical environmental information at the virtual level, obtains virtual point cloud information, and performs virtual-reality point cloud fusion of the virtual point cloud information and the physical point cloud information.

[0019] The virtual-reality collaborative decision-making cycle includes a robot physical decision-making system and a robot virtual decision-making system; in the robot physical decision-making system, the physical collaborative robot generates a path to be selected according to the path planning algorithm, and enhances the robot's autonomous decision-making ability and collaborative decision-making ability in the face of emergencies through the cloud-edge-end collaboration mode; the decision results and information in the robot physical decision-making system are transmitted to the robot virtual decision-making system in real time, and the robot virtual decision-making system performs virtual planning and preview results in a virtual scene, obtains the optimal decision, and makes decision corrections to the decision results of the robot physical decision system. The corrected results are fed back to the robot physical decision system to achieve virtual-reality integrated decision-making;

[0020] The virtual-reality collaborative control loop includes a robot physical control system and a robot virtual control system; the robot physical control system collaboratively controls and forms a team with the physical collaborative robots, performs autonomous navigation and tracking based on the path planning results, and performs real-time mapping in Unity 3D through the digital twin virtual scene; a multi-module visual interactive interface is used to enhance the situational awareness of virtual operators and reduce the impact of latency issues, and in the virtual scene, remote operators control the digital twin robots to implement remote intervention in the physical collaborative robots.

[0021] Furthermore, in the perception cycle of virtual-reality collaboration, the robot's physical perception system includes a physical collaborative robot and various sensor devices integrated on the physical collaborative robot; the sensor equipment includes an airborne three-dimensional lidar, a UWB positioning system, an airborne camera, and an airborne infrared sensing device; the physical collaborative robot performs laser slam based on the three-dimensional lidar, and performs real-time map fusion through different map fusion algorithms; the UWB positioning system is combined with multi-robot collaborative positioning to realize multi-mode fusion positioning, the airborne camera calibrates and identifies features of underground obstacles, and obtains parameter information; the infrared sensing device identifies and perceives people entering the well.

[0022] Furthermore, the multi-mode fusion positioning combines SLAM real-time positioning, UWB positioning technology and multi-robot collaborative positioning to achieve fusion positioning of multiple positioning modes. Each robot is positioned by the laser radar SLAM, and the laser radar obtains environmental information and performs feature extraction on the scanning information to obtain feature points; the current position of the robot is matched with the feature point information in the laser radar scanning data; the robot position and posture information is preliminarily determined in combination with the odometer information; each robot is assisted in positioning by UWB positioning technology to obtain more accurate position information; after each robot estimates its own position and posture information, it sends the estimated position information to other robots, and fuses the position information with other robots to achieve global positioning of the entire environment.

[0023] Furthermore, within the collaborative virtual-real perception cycle, the robot's virtual perception system includes a digital twin virtual scene, a virtual radar system, a parametric model library module, and a priori information fusion module. Based on the physical perception system and information, a digital twin virtual scene is constructed in Unity 3D. Feature information is used to match parametric models in the parametric model library, and priori information is integrated to repair and enhance the digital twin virtual scene. The virtual radar system then acquires a virtual point cloud based on the digital twin virtual scene.

[0024] Furthermore, the digital twin virtual scene includes a virtual environment scene and a digital twin robot; through unity and ROS communication, the node in the ROS system is subscribed to in Unity 3D to obtain the point cloud map information of the three-dimensional radar after the fusion of the physical collaborative robot, and the coordinate system of Unity 3D and the ROS system is first aligned through the particle system in Unity3D, and the pcd file of the three-dimensional point cloud is displayed in real time in unity; the coordinates of the points are recorded using an XML file, and the virtual environment scene is generated through the mesh grid; the three-dimensional model of the robot is constructed in SolidWorks and imported into the Unity 3D virtual environment scene, and the position and posture of the digital twin robot in the virtual environment scene are determined through the position coordinates and posture information of the physical collaborative robot, and virtual perception systems such as virtual radar are integrated on the digital twin robot model.

[0025] Furthermore, in the virtual-reality collaborative decision-making cycle, the robot physical decision-making system includes multiple physical collaborative robots, industrial computers and cloud servers; each physical collaborative robot serves as a terminal device, and the physical collaborative robot is equipped with an industrial computer as an edge node. The industrial computer has a built-in ROS system, which analyzes environmental information and makes autonomous decisions through intelligent AI algorithms, and comprehensively processes and integrates the information of each terminal sensor through the cloud server to form a cloud-edge-end collaborative decision-making system.

[0026] Furthermore, the cloud-edge-end collaborative decision-making system uses multiple physical collaborative robots to collaboratively build SLAM maps, and uses the cloud server to comprehensively process and fuse the sensor information of each terminal device in real time, thereby obtaining more accurate and complete map information; the cloud server sends the map information to the terminal device in real time, helping the robot to update the map more quickly and improve the real-time performance of the map.

[0027] Furthermore, in the virtual-reality collaborative decision-making cycle, the robot virtual decision-making system includes virtual planning and result preview in the digital twin virtual scene; path planning for the virtual robot based on the digital twin virtual scene in Unity, and the planned path can be displayed in the virtual scene in real time; the result of the planned path is previewed in the virtual scene; taking into account safety and efficiency, a better path is selected in the digital twin virtual scene, and the path in the physical decision-making system is corrected in real time.

[0028] Furthermore, in the virtual-reality collaborative control loop, the robot physical control system includes multiple physical collaborative robots and an industrial computer as the host computer; each robot configures an autonomous control program and a control interface on the industrial computer according to its own size, moving speed, maximum turning radius and other parameter information, so that the host computer sends control instructions to control the physical collaborative robot; by configuring IP addresses and LAN communications under the ROS system, the control center controls multiple industrial computers to achieve distributed control and is responsible for the coordination and management of the entire system. Multiple industrial computers act as controlled nodes and are responsible for specific execution tasks and control operations.

[0029] Furthermore, in the virtual-reality collaborative control loop, the robot virtual control system includes a digital twin virtual scene, a multi-module visual interactive interface, and VR interactive equipment; based on the real-time digital twin virtual scene in Unity 3D, a visual interactive interface is designed in Unity 3D for the digital twin robot therein, and the corresponding control program is compiled; remote workers use VR interactive equipment to realize virtual monitoring, virtual remote scheduling, and VR interactive control of the digital twin robot.

[0030] The present invention is a method for collaborative perception, decision-making and control of multiple robots in a mine through virtual-reality fusion. Specifically, the method combines the virtual environment and the actual environment, uses the sensors of multiple robots to obtain real-time environmental information in the coal mine, and integrates it with the three-dimensional model and virtual environment to achieve collaborative perception and path planning of multiple robots.

[0031] The digital twin-based underground virtual-reality fusion collaborative perception, decision-making and control method of coal mines has the following beneficial effects compared with the existing technology.

[0032] 1) For underground robot positioning: This invention utilizes a method that fuses multiple positioning modes. Given the lack of GPS positioning underground, this method achieves higher accuracy than existing SLAM positioning for underground robots. This more precise positioning enables the robot to acquire more accurate underground point cloud information.

[0033] 2) Regarding SLAM mapping: The present invention sources point cloud information from multiple collaborative robots, enabling point cloud information to be acquired from multiple angles for the same scene. Multiple robots can communicate with each other, enabling real-time fusion of point cloud information. This point cloud information can be used to generate a real-time visualization scene in Unity 3D. Point cloud inpainting and information enhancement can be performed within the virtual scene. A virtual radar system acquires virtual point clouds and fuses them with physical point clouds. The SLAM map constructed by this invention integrates real-world environmental information, downhole equipment parameters, and prior geological and construction information, resulting in more accurate maps than existing robot-generated SLAM maps.

[0034] 3) Robot Decision-Making: The robot model of this invention, which utilizes cloud-edge-end collaboration, offers numerous advantages in decision-making, including strong data processing capabilities, optimized algorithms, high real-time performance, enhanced flexibility, and enhanced security. Cloud computing platforms possess powerful data processing capabilities, enabling centralized storage and analysis of robot data and optimizing decision-making algorithms, making robots more efficient and accurate in executing tasks. Edge computing enables local data processing and decision-making, reducing data transmission latency and improving the real-time performance and responsiveness of robots.

[0035] 4) Autonomous Robot Decision-Making: Leveraging intelligent AI algorithms, underground robots can dynamically adjust their decision-making strategies based on mission requirements and environmental changes, making them more flexible and adaptable to diverse scenarios. Furthermore, the cloud enables data encryption and secure transmission, ensuring the security and privacy of robot data. Furthermore, the virtual decision-making model of digital twins is integrated. By performing virtual planning and rehearsing results in a virtual scenario, the autonomous decisions of the physical robot can be corrected, achieving a decision-making model that integrates virtual and real-world scenarios.

[0036] 5) Robot Control: A digital twin model of the physical robot and a highly visual real-time virtual scene were constructed in Unity 3D. While each robot is capable of autonomous control, this highly visual digital twin virtual scene and real-time video information enable global control and scheduling by staff. VR interaction devices are also integrated, allowing staff to immersively remotely control the robot in the physical world through the digital twin virtual scene, while accounting for time delays. This system offers a richer range of control options than existing underground robots, combining autonomous robot control with the global control of experienced coal mine staff, ensuring greater safety for both coal mine staff and robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the overall framework diagram of the present invention;

[0038] Figure 2 This is the perception-decision-control framework diagram of the present invention;

[0039] Figure 3 This is a flow chart of virtual-reality fusion perception of the present invention;

[0040] Figure 4 It is the virtual-reality fusion decision flow chart of the present invention;

[0041] Figure 5 It is the virtual-reality fusion control flow chart of the present invention. DETAILED DESCRIPTION

[0042] A typical embodiment of the present invention provides a multi-robot collaborative perception, decision-making and control method based on digital twins that is applied to different working scenarios underground. Figure 1 , including two overall cycle processes: virtual perception decision-making and control cycle and physical perception decision-making and control cycle, and three distributed cycle processes: virtual-reality collaborative perception cycle, virtual-reality collaborative control cycle, and virtual-reality collaborative decision-making cycle.

[0043] Among them, the physical perception decision control loop physical collaborative robot can perceive and locate through a variety of sensors. The perception module will process and fuse the information obtained by the sensor, and generate a map and scene model of the robot's environment according to the control instructions. After fusing virtual information for information enhancement, the comprehensive perception information is passed to the decision and control loop. The robot based on the cloud-edge collaborative decision-making mode generates a map and scene model based on the comprehensive information obtained, and generates the robot's action strategy through path planning, target recognition, behavior planning and other technologies, and corrects it through virtual decision-making. Determine the map fusion mode and deliver the decision results. The robot will use the control instructions generated by the perception module and the decision module, combined with the remote control instructions to control the robot to complete the specified task, and generate control information and control instructions to pass to the decision loop and the perception loop.

[0044] The virtual perception decision-making control loop constructs a real-time digital twin scene based on physical point cloud information, integrating prior information with a parametric model. It also builds a virtual sensing device to acquire virtual scene point cloud information and integrate it with the physical point cloud for virtual-reality integration. Within the constructed digital twin virtual scene, the physical robot's decisions are mapped in real time, and virtual planning and results are previewed. Based on the comprehensive decision-making results, remote workers, through VR interactive devices, perceive the actual image in real time and the digital twin virtual scene in real time, allowing for remote global control of the robot while accounting for latency.

[0045] refer to Figure 2 , and gives a detailed explanation of the three aspects of perception, decision-making and control.

[0046] 1. Virtual-Real Collaborative Perception Cycle

[0047] The collaborative virtual-reality perception loop includes a physical robot perception system and a virtual robot perception system. The physical robot perception system acquires information about the underground environment, including the position and motion parameters of each robot within the environment, at the physical level. The virtual robot perception system enhances this physical environment information at the virtual level, acquires virtual point cloud information, and fuses the virtual and physical point clouds.

[0048] The robot physical perception system consists of a physical collaborative robot and various sensor devices integrated on the physical collaborative robot, including an airborne three-dimensional laser radar, a UWB positioning system, an airborne camera, and an airborne infrared sensing device. The physical collaborative robot performs laser slam based on the three-dimensional laser radar and performs real-time map fusion using different map fusion algorithms such as the DNT algorithm, the ICP algorithm, and the LOAM algorithm. The UWB positioning system is combined with multi-robot collaborative positioning to achieve multi-mode fusion positioning. The airborne camera calibrates and identifies features of underground obstacles and obtains parameter information. The infrared sensing device identifies and senses personnel entering the underground. The robot physical perception system obtains point cloud information of the underground environment, the position and motion parameter information of the physical collaborative robot, and the position information of underground obstacles and personnel.

[0049] The physical collaborative robots described here refer to wheeled robots in the physical world that integrate the various sensors mentioned above, enabling multi-robot collaborative control and information exchange between robots. Each robot is equipped with an industrial computer as the host computer, and each robot can be individually controlled through the ROS system, achieving distributed control of multiple physical robots.

[0050] Multi-mode fusion positioning combines SLAM real-time positioning, UWB positioning technology, and multi-robot collaborative positioning to achieve fusion positioning of multiple positioning modes, making underground robot positioning more accurate. Each robot is positioned using SLAM (LiDAR). The LiDAR acquires environmental information and extracts features from the scanned data to obtain feature points. The robot's current position is matched with the feature point information in the LiDAR scan data. Combined with odometry information, the robot's position and posture are preliminarily determined. UWB positioning technology is then used to assist in positioning each robot, obtaining even more accurate position information. After each robot estimates its own position and posture, it transmits this estimated position information to other robots, where it is integrated with the other robots to achieve global positioning of the entire environment.

[0051] The robot virtual perception system includes a digital twin virtual scene, a virtual radar system, a parametric model library module, and a prior information fusion module. Based on the physical perception system and information, a digital twin virtual scene is constructed in Unity 3D. Feature information is used to match parametric models in the parametric model library. Prior information is then integrated to repair and enhance the digital twin virtual scene. The virtual radar system then acquires a virtual point cloud based on the digital twin virtual scene.

[0052] The digital twin virtual scene consists of a virtual environment scene and a digital twin robot. Through Unity3D and ROS communication, the node in the ROS system is subscribed to in Unity 3D to obtain the point cloud map information of the three-dimensional radar after the physical collaborative robot is integrated. The coordinate system of Unity 3D and the ROS system is first aligned through the particle system in Unity 3D, and the pcd file of the three-dimensional point cloud is displayed in real time in Unity 3D. The coordinates of the points are recorded using an XML file, and the virtual environment scene is generated using a mesh grid. The three-dimensional model of the robot is constructed in SolidWorks and imported into the Unity 3D virtual environment scene. The position and posture of the digital twin robot in the virtual environment scene are determined by the position coordinates and posture information of the physical collaborative robot. Virtual perception systems such as virtual radar are also integrated into the digital twin robot model.

[0053] The parametric model library module constructs a model library for underground equipment based on real-world underground data. This library targets underground fully mechanized mining equipment, transportation equipment, and some auxiliary equipment. Using WebGL software, the underground equipment models are parameterized, and a data interface is provided to enable real-time feature point discovery and construction of the underground equipment model library. FBX-format model files are generated within the system and then imported into Unity 3D software. When feature point data is matched, a virtual model of the underground equipment is generated in real time within the Unity 3D virtual environment.

[0054] The prior information fusion module uses known geological and construction information as prior information, modifies the virtual scene in Unity 3D according to the virtual environment scene and the prior geological information and construction information, and performs information enhancement to generate a more accurate virtual map.

[0055] The virtual radar system adds a virtual radar script to the virtual radar model on the digital twin robot model, simulating the laser radar's ray emission. The virtual environment scene is set as a collision volume in Unity 3D. The virtual laser radar scans the virtual scene to obtain a virtual point cloud.

[0056] (2) Virtual-Real Collaborative Decision-Making Cycle

[0057] The virtual-reality collaborative decision-making loop includes a physical robot decision-making system and a virtual robot decision-making system. In the physical robot decision-making system, the physical collaborative robot generates a path to be selected based on a path planning algorithm. Through cloud-edge-end collaboration, the robot's autonomous and collaborative decision-making capabilities in emergency situations are enhanced. Decision results and information from the physical decision-making system are transmitted to the virtual decision-making system in real time. The virtual robot decision-making system performs virtual planning and previews the results in a virtual scene, obtaining the optimal decision and correcting the decision results of the physical decision-making system. The corrected results are then fed back to the physical robot decision-making system, achieving integrated virtual-reality decision-making.

[0058] The robotic physical decision-making system consists of multiple physical collaborative robots, industrial computers, and cloud servers. Each physical collaborative robot serves as a terminal device. Each robot is equipped with an industrial computer with sufficient computing power, serving as an edge node. The industrial computer has a built-in ROS system, which uses intelligent AI algorithms to analyze environmental information and make autonomous decisions. The cloud server then comprehensively processes and integrates information from various terminal sensors, forming a cloud-edge-end collaborative decision-making system.

[0059] The physical system robot and industrial computer are combined with intelligent AI algorithms to analyze and process the virtual and real integrated perception information to comprehensively evaluate the number of obstacles in the environmental information, the robot's movement ability, the navigation time and other requirements to select a suitable path planning algorithm to achieve efficient and safe navigation. Specifically, when there are fewer obstacles in the environmental information and the road conditions are good, the robot can use the A* algorithm to select a path with higher efficiency through the area. A* is a path planning algorithm based on heuristic search. It can select the optimal path by evaluating the path from the starting point to the end point to achieve efficient navigation. However, when there are many obstacles in the environmental information, the robot needs to choose a safer path planning algorithm. The virtual potential field algorithm is a path planning algorithm based on a physical model. It can calculate the robot's motion path by abstracting the robot and obstacles into physical particles and using the interaction between electric charge and magnetic field. Avoid collisions between the robot and obstacles, and optimize the robot's motion path to achieve safe and efficient navigation.

[0060] The cloud-edge-end collaborative decision-making system uses multiple physical collaborative robots to collaboratively build SLAM maps. Cloud servers process and fuse sensor information from each terminal device in real time, resulting in more accurate and complete map information. Furthermore, the cloud server sends map information to the terminal device in real time, enabling the robots to update the map more quickly and improving map real-time performance. During this process, the industrial computers onboard the physical collaborative robots can serve as edge nodes, offloading some data processing and analysis tasks to edge devices for edge computing. When the robots encounter personnel working underground or moving equipment on their planned paths while mapping underground, edge computing leverages the computing and storage resources of edge devices to achieve real-time data processing and analysis, reducing data transmission and latency while improving real-time performance. Edge computing leverages the computing and storage resources of edge devices to enable real-time data processing and analysis. Some decision-making tasks, such as emergency obstacle avoidance and parking, are performed on the edge devices, allowing them to make rapid decisions. Path planning results are then updated through cloud-based data collaboration and analysis.

[0061] The robot virtual decision-making system includes virtual planning and result previewing within a digital twin virtual scene. Path planning is performed for the virtual robot within the digital twin virtual scene in Unity, and the planned path is displayed in real time within the virtual scene. The results of the planned path are previewed within the virtual scene. Taking into account safety and efficiency, the optimal path is selected within the digital twin virtual scene, and the path in the physical decision-making system is corrected in real time.

[0062] (3) Virtual and Real Collaborative Control Cycle

[0063] The virtual-reality collaborative control loop includes a physical robot control system and a virtual robot control system. The physical robot control system coordinates and forms a team with the physical collaborative robots, autonomously navigates and tracks based on path planning results, and is mapped in real time within Unity 3D via the digital twin virtual scene. A multi-module visual interactive interface enhances the virtual operator's situational awareness and reduces latency. In the virtual scene, remote operators can control the digital twin robot through the multi-module visual interactive interface and VR interactive devices to remotely intervene in the physical collaborative robot.

[0064] The robot physical control system consists of multiple physical collaborative robots and an industrial computer (IPC) as the host computer. Each robot is configured with an autonomous control program and control interface on the IPC based on parameters such as its size, movement speed, and maximum turning radius. This allows the host computer to send control commands to control the physical robots. By configuring IP addresses and local area network communications under the ROS system, the control center controls multiple IPCs, achieving distributed control and responsible for the coordination and management of the entire system. The multiple IPCs act as controlled nodes, responsible for executing specific tasks and controlling operations. The IPCs can be distributed in different locations, providing high flexibility and scalability. A virtual force field algorithm is used under the distributed control mode to coordinate and form multiple robots in a coordinated formation.

[0065] The virtual force field algorithm in the distributed control mode establishes a virtual force field around each robot to achieve mutual attraction and repulsion between robots. In the virtual force field, each robot is modeled as a charged particle surrounded by interacting charge points, representing other robots or target points. The net force acting on each robot is calculated to achieve robot motion control. The net force can be divided into two parts: one is the attraction from other robots, and the other is the repulsion from obstacles. The formula for calculating the net force can be adjusted based on actual environmental information. The distributed control framework in ROS can be used to run the virtual force field algorithms of multiple robots as different nodes, and data exchange and coordination between nodes can be carried out through the ROS communication mechanism. Use the motion control library in ROS (such as MoveIt!) or write your own motion controller to achieve collaborative control of robot motion. Implement multi-robot formation control based on the virtual force field algorithm.

[0066] The robot virtual control system consists of a digital twin virtual scene, a multi-module visual interactive interface, and VR interactive equipment. Based on the real-time digital twin virtual scene in Unity 3D, a visual interactive interface is designed for the digital twin robot within it in the Unity 3D engine, and the corresponding control program is compiled. Remote workers can use VR interactive equipment such as HTC Vive to achieve virtual monitoring, virtual remote scheduling, and VR interactive control of the digital twin robot.

[0067] The multi-module visual interactive interface consists of four parts: a camera feed view interface, a virtual command status interface, a predicted status interface, and a time delay status interface. The camera feed view interface provides the option of viewing a dynamic floating window with a camera feed view and a graphic of the sensor output. During the collaborative operation of the robots, the cameras on each robot and the cameras in the underground environment obtain real-time image information of each robot and the robot group, and transmit the images in real time to the remote control center via 5G communication. In a long-distance teleoperation scenario, the operator can use this interface to view the working status of the underground robot in real time, which serves as visual information to verify the successful execution of the command task, thereby improving the real-time situation perception of remote personnel.

[0068] The virtual command status interface is displayed using a GUI within the Unity 3D engine and can be operated in real time in VR using motion controllers or operator-defined commands. After a command is issued, the robot's state in the predicted status interface is updated based on the constraints of the robot system, following the robot's movements in the virtual command status interface. Furthermore, the robot in the predicted status interface continuously verifies and corrects itself based on the robot's delay data in the time delay status interface, displaying it as a point cloud based on real-time data. When the robot view in the predicted status interface converges with the robot view in the time delay status interface, the operation is complete. The delay time can be customized in the delay status interface, fully accounting for operation delays. VR operators can switch between multiple visual interaction interfaces.

[0069] Virtual monitoring involves setting up corresponding virtual monitoring devices, such as virtual cameras and sensors, within the digital twin virtual scene. These virtual monitoring devices are configured with corresponding C# scripts in the Unity 3D engine. This captures the digital twin robot's operating status and sensor data, as well as information about the virtual environment. This data is visualized through the Unity 3D interactive interface, allowing remote workers to monitor this data in real time using VR devices.

[0070] Remote workers can use the digital twin virtual scene in Unity 3D to issue control commands to the digital twin robot from a global perspective through an interactive interface, enabling collaborative scheduling. Data transfer and communication between ROS and Unity 3D are achieved through two communication APIs, ROS# and ROS bridge. This enables remote scheduling of physical collaborative robots through the digital twin virtual scene.

[0071] like Figure 3The figure shows the SLAM mapping process for virtual-reality fusion. In an actual underground coal mine, multiple robots operate in a coordinated formation. LiDAR is used to obtain a three-dimensional point cloud map of the underground environment from multiple angles and sources. Due to the weak GPS signal underground, a fusion positioning method is used to accurately locate the underground robots. An onboard camera captures and transmits images and videos of the underground environment, identifying obstacles, personnel, and characteristic parameters of underground equipment. A fusion algorithm is used to fuse the point clouds collected by each robot in real time to generate a physical point cloud. A data transmission channel between ROS and Unity 3D is established as a virtual-reality interaction channel, mapping the real-time physical point cloud into the virtual scene. A parametric model library is constructed within the Unity 3D engine. When the characteristic parameters of underground equipment are obtained, equipment models are generated in the virtual scene based on their location information. The virtual scene is then modified in real time by combining underground construction and geological information as prior information. A virtual radar system is established on the digital twin robot to collect virtual point clouds for the virtual scene. The virtual point cloud and the physical point cloud are integrated, complementing each other and enhancing information to form a comprehensive environment information that integrates the virtual and the real. This can provide a higher-precision point cloud map for the physical robot and also build a more realistic virtual scene.

[0072] like Figure 4 Figure 2 shows the decision-making process of virtual-reality fusion. The physical collaborative robot identifies obstacles and road conditions by integrating environmental information. Based on this environmental information, the robot's intelligent AI algorithm, through a cloud-edge-end collaborative decision-making model, selects different path planning algorithms. The robot then autonomously determines the optimal path. When obstacles appear on the path, the decision results are updated in real time. Through virtual-reality interaction channels, decision results are presented in real time in the virtual scene. In this virtual scene, the digital twin robot can perform virtual path planning, preview the planning results, and compare them with the physical decision results. Path information is analyzed and mined in the virtual scene, serving as a historical dataset for information reinforcement, continuously optimizing the decision algorithm. Collaborative decision-making and correction are achieved through decision-making in both the physical and virtual dimensions. This improves the accuracy and safety of robot decisions at the physical level, optimizing robot operation. At the virtual level, historical data is used to train decision models, improving decision stability.

[0073] like Figure 5The figure shows the control process of virtual-reality fusion. The physical robot performs multi-machine collaborative control based on the virtual stance algorithm based on the path information after the decision. The robot's operating parameters are obtained from the host computer for fault-tolerant control. In the virtual scene, the remote operator can use VR equipment to virtually inspect the robot's digital twin. Considering the delay in information transmission, the operator wearing VR interactive equipment remotely controls the physical robot through the digital twin in the virtual scene, and performs virtual scheduling and task allocation. The physical robot and the digital twin in the virtual scene are mapped in real time, achieving two-way virtual-reality control. For physical robots, the intervention of remote workers can improve the fault tolerance of the robot control process, while providing a highly visual remote teleoperation method in the virtual dimension.

Claims

1. A multi-robot virtual-reality fusion collaborative perception, decision-making and control method for underground coal mines, characterized by: It includes virtual-reality collaborative perception loop, virtual-reality collaborative decision loop and virtual-reality collaborative control loop; The virtual-reality collaborative perception loop includes a robot physical perception system and a robot virtual perception system. The robot physical perception system obtains underground environmental information, as well as the position information and motion parameter information of each physical collaborative robot in the underground environment at the physical level, and generates physical point cloud information. The robot virtual perception system enhances the physical environmental information at the virtual level, obtains virtual point cloud information, and performs virtual-reality point cloud fusion of the virtual point cloud information and the physical point cloud information. The virtual-reality collaborative decision-making cycle includes a robot physical decision-making system and a robot virtual decision-making system; in the robot physical decision-making system, the physical collaborative robot generates a path to be selected according to the path planning algorithm, and enhances the robot's autonomous decision-making ability and collaborative decision-making ability in the face of emergencies through the cloud-edge-end collaboration mode; the decision results and information in the robot physical decision-making system are transmitted to the robot virtual decision-making system in real time, and the robot virtual decision-making system performs virtual planning and preview results in a virtual scene, obtains the optimal decision, and makes decision corrections to the decision results of the robot physical decision system. The corrected results are fed back to the robot physical decision system to achieve virtual-reality integrated decision-making; The virtual-reality collaborative control loop includes a robot physical control system and a robot virtual control system; the robot physical control system collaboratively controls and forms a team with the physical collaborative robots, performs autonomous navigation and tracking based on the path planning results, and performs real-time mapping in Unity 3D through the digital twin virtual scene; a multi-module visual interactive interface is used to enhance the situational awareness of virtual operators and reduce the impact of latency issues, and in the virtual scene, remote operators control the digital twin robots to implement remote intervention in the physical collaborative robots.

2. The method according to claim 1, wherein: In the perception cycle of virtual-reality collaboration, the robot's physical perception system includes a physical collaborative robot and various sensor devices integrated on the physical collaborative robot; the sensor equipment includes an airborne three-dimensional lidar, a UWB positioning system, an airborne camera, and an airborne infrared sensing device; the physical collaborative robot performs laser slam based on the three-dimensional lidar, and performs real-time map fusion through different map fusion algorithms; the UWB positioning system is combined with multi-robot collaborative positioning to achieve multi-mode fusion positioning, the airborne camera calibrates and identifies features of underground obstacles, and obtains parameter information; the infrared sensing device identifies and perceives people entering the well.

3. The method according to claim 2, wherein: The multi-mode fusion positioning combines SLAM real-time positioning, UWB positioning technology and multi-robot collaborative positioning to achieve fusion positioning of multiple positioning modes. Each robot is positioned by using the laser radar SLAM. The laser radar obtains environmental information and extracts features from the scanning information to obtain feature points. The current position of the robot is matched with the feature point information in the laser radar scanning data. The robot position and posture information are preliminarily determined by combining the odometer information. The UWB positioning technology is used to assist in positioning each robot, thereby obtaining more accurate position information. After each robot estimates its own position and posture information, it sends the estimated position information to other robots, and fuses the position information with other robots to achieve global positioning of the entire environment.

4. The method according to claim 1 or 3, characterized in that: In the perception cycle of virtual-reality collaboration, the robot virtual perception system includes a digital twin virtual scene, a virtual radar system, a parametric model library module, and a priori information fusion module. Based on the physical perception system and information, an underground digital twin virtual scene is constructed in Unity 3D, and the parametric model is matched in the parametric model library through feature information. Prior information is integrated to repair and enhance the digital twin virtual scene. The virtual radar system obtains a virtual point cloud based on the digital twin virtual scene.

5. The method according to claim 4, characterized in that: The digital twin virtual scene includes a virtual environment scene and a digital twin robot; through unity and ROS communication, the node in the ROS system is subscribed to in Unity 3D to obtain the point cloud map information of the three-dimensional radar after the physical collaborative robot is integrated; the coordinate system of Unity 3D and the ROS system is first aligned through the particle system in Unity3D, and the pcd file of the three-dimensional point cloud is displayed in real time in unity; the coordinates of the points are recorded in an XML file, and the virtual environment scene is generated through the mesh grid; the three-dimensional model of the robot is built in SolidWorks and imported into the Unity 3D virtual environment scene, and the position and posture of the digital twin robot in the virtual environment scene are determined through the position coordinates and posture information of the physical collaborative robot, and virtual perception systems such as virtual radar are integrated on the digital twin robot model.

6. The method according to claim 1 or 5, characterized in that: In the virtual-reality collaborative decision-making cycle, the robot physical decision-making system includes multiple physical collaborative robots, industrial computers and cloud servers; each physical collaborative robot serves as a terminal device, and the physical collaborative robot is equipped with an industrial computer as an edge node. The industrial computer has a built-in ROS system, which uses intelligent AI algorithms to analyze environmental information and make autonomous decisions. The cloud server comprehensively processes and integrates the information of each terminal sensor to form a cloud-edge-end collaborative decision-making system.

7. The method according to claim 6, characterized in that: The cloud-edge-end collaborative decision-making system uses multiple physical collaborative robots to collaboratively build SLAM maps, and uses cloud servers to comprehensively process and fuse the sensor information of each terminal device in real time to obtain more accurate and complete map information; the cloud server sends the map information to the terminal device in real time, helping the robot to update the map more quickly and improve the real-time performance of the map.

8. The method according to claim 1 or 7, characterized in that: In the virtual-reality collaborative decision-making cycle, the robot virtual decision-making system includes virtual planning and result preview in the digital twin virtual scene; path planning for the virtual robot based on the digital twin virtual scene in Unity, and the ability to display the planned path in the virtual scene in real time; previewing the results of the planned path in the virtual scene; selecting a better path in the digital twin virtual scene while comprehensively considering safety and efficiency, and correcting the path in the physical decision-making system in real time.

9. The method according to claim 8, characterized in that: In the virtual-reality collaborative control loop, the robot physical control system includes multiple physical collaborative robots and an industrial computer as the host computer; each robot is configured with an autonomous control program and control interface on the industrial computer based on its own size, moving speed, maximum turning radius and other parameter information, so that the host computer sends control instructions to control the physical collaborative robot; by configuring IP addresses and LAN communications under the ROS system, the control center controls multiple industrial computers to achieve distributed control and is responsible for the coordination and management of the entire system. Multiple industrial computers act as controlled nodes and are responsible for specific execution tasks and control operations.

10. The method according to claim 9, characterized in that: In the virtual-reality collaborative control loop, the robot virtual control system includes a digital twin virtual scene, a multi-module visual interactive interface, and VR interactive equipment. Based on the real-time digital twin virtual scene in Unity 3D, a visual interactive interface is designed in Unity 3D for the digital twin robot, and the corresponding control program is compiled. Remote workers use VR interactive equipment to achieve virtual monitoring, virtual remote scheduling, and VR interactive control of the digital twin robot.

Images