Three-level n-dimensional digital twin system for supporting coupled operation of a group of coal mine robots

Abstract

The application discloses a kind of three-level n-dimensional digital twin systems supporting coupling operation of coal mine robot group, including terminal layer, edge layer and cloud end layer three levels.The robot group in terminal physical layer generates single-machine behavior level twin in terminal virtual layer based on real-time data through terminal AI model, realizes virtual and real two-way control, and field operator obtains augmented information through AR equipment, feeds back decision-making opinion, and multi-modal control node;Edge virtual layer generates collaborative task level twin based on single-machine behavior level twin, and constructs n-dimensional edge twin system through edge AI model.The computer host of cloud end physical layer provides computing power support for n-dimensional cloud end twin system, and centralized control operator accesses cloud end twin system through VR equipment, to realize remote monitoring and control;Cloud end virtual layer generates global planning level twin based on edge side data, and parallel analysis of n-dimensional cloud end twin system is realized through cloud end AI model, and the optimal global planning is comprehensively obtained.

Description

Technical Field

[0001] This invention relates to the field of intelligent coal mining technology, and more specifically, to a three-level n-dimensional digital twin system that supports the coupled operation of multi-scenario robot groups in coal mines. Background Technology

[0002] With the advancement of intelligent coal mine construction, an increasing number of robots with diverse functions are being deployed, inevitably leading to the formation of various robot groups based on different work scenarios. The scientific and efficient management of these robot groups, and the achievement of human-machine collaboration among centralized control operators, field operators, and robot groups, have become crucial issues. Breakthroughs in emerging technologies such as big data, edge computing, cloud computing, blockchain, digital twins, and virtual reality provide powerful tools for solving this challenge. By integrating these emerging technologies, the level of intelligent coal mine construction can be further improved, and new models for smart mine operation can be explored.

[0003] The invention disclosed in CN113362037A is a coal mine intelligent management system and method based on edge cloud, including a ground system, a communication network module, and an underground system. The ground system is used to train a coal mine data model and transmit it to the underground system via cloud technology. The high-speed communication network provides a data transmission channel between the ground system and the underground system. The underground system receives the trained coal mine data model transmitted from the ground system and applies it. Step 1: In the coal mine ground system, the interaction module obtains a training request, and the training module develops and trains an algorithm model according to the training request. Step 2: The developed and trained algorithm model is sent to the central cloud. The central cloud obtains the corresponding hardware and software device interfaces through the high-speed communication network and transmits the developed and trained algorithm model to the underground edge cloud system. Step 3: The developed and trained algorithm model is deployed to the underground system for application.

[0004] The invention disclosed in CN116415816A provides an M-CPS intelligent mine management platform and system, belonging to the field of intelligent mine technology. It solves the problem that existing intelligent mines lack unified standards and cannot achieve overall mine safety management. The intelligent mine management platform follows a set of standard specifications, namely, a unified platform, a unified model, a unified architecture, and unified data, encompassing comprehensive perception, real-time interconnection, analysis and decision-making, autonomous learning, dynamic prediction, and intelligent control. It integrates information flow and control flow to construct an M-CPS model that enables interconnection between the physical world and information. The platform's interface connects to physical equipment, using big data technology as platform support, integrating mine operations and functions onto the entire platform, covering all aspects of mine production, and achieving the integration of various working face subsystems. This invention is applicable to intelligent mines.

[0005] The invention disclosed in CN116619360A presents a method for multi-robot virtual-real fusion collaborative perception, decision-making, and control in coal mines. This method includes two overall loop processes: a virtual perception-decision-control loop and a physical perception-decision-control loop; and three distributed loop processes: a virtual-real collaborative perception loop, a virtual-real collaborative control loop, and a virtual-real collaborative decision-making loop. It combines the virtual and real environments, utilizing sensors from multiple robots to acquire real-time environmental information from the coal mine, and then fusion this information with a 3D model and the virtual environment to achieve collaborative perception and path planning for multiple robots. Through multi-robot collaborative control and point cloud fusion, it improves mapping efficiency and accuracy. It uses virtual reality technology to generate highly visualized digital twin scenes in real time, and allows remote interaction with the robots via VR devices based on real-time virtual scenes and video feedback.

[0006] The invention disclosed in CN116305964A relates to a smart mining face system based on digital twin technology, belonging to the field of intelligent coal mine technology. It includes a physical layer, a twin layer, a perception layer, a decision layer, and a front-end layer. The physical layer represents the real-world scenario of the smart mining face. The perception layer collects real-time data from the physical layer using data acquisition equipment and sends it to the twin and decision layers. The twin layer constructs a virtual model of the physical layer based on its basic information and real-time data. The decision layer controls the physical and twin layers based on the real-time data and the virtual model. The front-end layer visualizes the virtual model constructed by the twin layer. This invention solves the problem of insufficient real-time data flow from fully mechanized mining equipment and mining scenarios, representing a significant step forward in smart mining faces. By establishing the physical, twin, perception, decision, and front-end layers, a relatively complete smart mining face system is proposed, providing a good approach for the application of digital twins in underground coal mines.

[0007] The above-mentioned existing technologies all involve using emerging technologies such as edge computing, digital twins, and virtual reality to improve the level of intelligence in coal mines, but they have the following drawbacks:

[0008] (1) At the terminal layer, the terminal device has a low level of data processing and can only collect data through sensors and then transfer it to other devices that specialize in processing complex data before it can build a twin and drive the twin to move. The real-time performance is very poor.

[0009] (2) At the edge layer, edge layer devices often only serve as data relay stations that receive data from the terminal layer, process the data, upload it to the cloud layer, and distribute data from the cloud layer to the terminal layer. They ignore the characteristics of being close to the terminal layer, which allows them to process data faster and apply it locally for decision-making. Digital twin technology is not integrated into the edge layer.

[0010] (3) At the cloud layer, many systems build digital twin systems based on the powerful computing capabilities of the cloud to process the complex data at the terminal layer, and then guide actual production through perception, decision-making, and control. However, this does not reflect the characteristics of multiple parallel clones of digital twins, and does not integrate with other systems to comprehensively evaluate the quality of decisions from multiple factors.

[0011] (4) In terms of data security, some systems do not emphasize how to ensure data security, while others improve data security by building a blockchain network. However, there is often only one level of network, and all node data is written and read out in a blockchain network. Data transmission is slow and cannot meet the real-time requirements of industrial production.

[0012] (5) Regarding access control, many systems only involve access control between physical assets, defining whether one device can communicate with another. However, in systems built using digital twin technology, in addition to physical assets, there are also a wide variety of virtual assets. Defining the communication permissions between virtual assets, as well as the communication permissions between virtual assets and physical assets, is equally important.

[0013] (6) In terms of human-machine collaboration, many systems only involve limited communication between a person and a single device. Operators can only receive information from a twin system and then return feedback on the information. This does not reflect the ability of people to integrate into the various details of the cloud, edge, and device, and directly control virtual and physical assets at different levels, which greatly limits the breadth and depth of human-machine collaboration. Summary of the Invention

[0014] The technical problem to be solved by this invention is to provide a three-level n-dimensional digital twin system that supports the coupled operation of coal mine robot groups. At the terminal layer, it improves the data processing capabilities of the equipment, rapidly generates twins based on data from multiple sensors, and enables the twins to control physical assets. At the edge layer, it enables efficient and rapid transmission of data between different nodes, ensuring the normal operation of collaborative tasks. At the cloud layer, it enables comprehensive analysis of the advantages and disadvantages of global planning and decision-making from multiple factors, derives the optimal global plan, and guides actual production.

[0015] To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0016] A three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot swarm includes three levels: terminal layer, edge layer, and cloud layer.

[0017] The terminal layer includes the terminal physical layer (TP) and the terminal virtual layer (TV).

[0018] The robot group in the terminal physical layer TP generates a single-machine behavior-level twin T in the terminal virtual layer TV based on real-time data through the terminal AI model, realizing two-way control between the virtual and physical worlds. On-site operators obtain augmented information through AR devices, provide feedback on decision-making, and control nodes in a multimodal manner.

[0019] The edge layer includes the edge physical layer (EP), the edge virtual layer (EV), and the edge blockchain network (EBN).

[0020] The edge computing gateway of the edge physical layer EP provides computing power support for collaborative tasks on the edge side and builds a bridge for communication between the cloud layer and the terminal layer; the edge virtual layer EV generates multiple collaborative task-level twins E based on the single-machine behavior-level twin T, and constructs an n-dimensional edge twin system including the collaborative mapping E1 system, the collaborative operation E2 system, and the edge twin En system through the edge AI model.

[0021] The cloud layer includes the cloud physical layer (CP), the cloud virtual layer (CV), and the cloud blockchain network (CBN).

[0022] The computer host of the cloud physical layer CP provides computing power support for the n-dimensional cloud twin system. The central control operator accesses the cloud twin system through VR devices to realize remote monitoring and control. The cloud virtual layer CV generates multiple global planning-level twins C based on edge side data. Through cloud AI models, it realizes parallel analysis of the n-dimensional cloud twin system, including virtual perception, virtual decision-making, virtual control, collision detection, and human-machine evaluation, and comprehensively derives the optimal global plan.

[0023] Furthermore, in the terminal physical layer TP, the robot group transmits the data detected by the sensors to the Unity3D software under the local Ubuntu system through the ROS-Unity bidirectional communication tool. Then, it constructs a virtual scene by comparing with the model library. At the same time, the robot's own motion state and pose information are mapped to the Unity3D virtual scene in real time to generate the corresponding single-machine behavior-level twin T.

[0024] Furthermore, the edge physical layer EP includes a collaborative mapping E1 edge computing gateway, a collaborative operation E2 edge computing gateway, an edge twin En edge computing gateway, and an edge access management E0 host. Among them, the collaborative mapping E1 edge computing gateway is responsible for handling the collaborative mapping tasks between robots P2, P3, and on-site operator P1; the collaborative operation E2 edge computing gateway is responsible for handling the collaborative operation tasks between robots P2, P3, and on-site operator P1; the edge twin En edge computing gateway is responsible for handling other collaborative tasks at the edge; and the edge access management E0 host is responsible for managing the communication permissions between all terminal nodes and edge nodes on this edge side.

[0025] Furthermore, in the edge virtual layer EV, the virtual humans E11, robots E12, E13, and other twins E1m in the collaborative mapping E1 system are corresponding collaborative task-level twins E generated based on the single-machine behavioral-level twin T data in the terminal virtual layer TV. First, the system receives the single-machine behavioral-level twin T data that has successfully passed permission comparison, generates the collaborative task-level twin E, and then uses an edge AI model to perform fusion analysis on the data of the collaborative task-level twin E to determine whether the current mapping data is complete, analyzes and decides on a preliminary repair plan, and then sends the preliminary repair plan to each single-machine behavioral-level twin T. Each single-machine behavioral-level twin T then controls each device to execute the preliminary repair plan. At the same time, the AR device also receives the current mapping data and the preliminary repair plan analyzed and decided. The on-site operator P1 provides feedback and controls the single-machine behavioral-level twin T or the physical robot to complete the current decision. Finally, the collaborative mapping E1 system uploads the complete mapping result to the cloud blockchain network CBN.

[0026] Furthermore, in the edge virtual layer EV, the virtual human E21, robot E22, E23, and other twins E2m in the collaborative operation E2 system are corresponding collaborative task-level twins E generated based on the single-machine behavior-level twin T data in the terminal virtual layer TV. First, the system receives the single-machine behavior-level twin T data that has successfully passed permission comparison, generates the collaborative task-level twin E, and obtains the virtual scene of the latest operation status. Then, based on the communication permission database, it sends the data of another device node that it needs to a certain device node to ensure rapid data sharing. Based on the current operation status and global planning, the collaborative operation E2 system judges the execution status of the current plan through the edge AI model. If the execution is not good, the system re-determines the allocation of human and machine tasks through the edge AI model, and then sends the decision results to each single-machine behavior-level twin and AR device. Each single-machine behavior-level twin T then controls each device to execute new operation tasks. At the same time, the on-site operator P1 also completes its own tasks based on the information received by the AR device. Finally, the collaborative operation E2 system uploads the collaborative operation results to the cloud blockchain network CBN.

[0027] Furthermore, in the edge virtual layer EV, the virtual human En1, robot En2, En3, and other twins Enm in the edge twin En system are corresponding collaborative task-level twins E generated based on the single-machine behavioral-level twin data T in the terminal virtual layer TV. By deploying multiple edge twin En systems in various subsystems of the coal mine, the rapid access to data between virtual and physical nodes at the edge is ensured, and the efficiency of completing collaborative tasks is improved by leveraging the analytical capabilities of edge AI.

[0028] Furthermore, the Edge Blockchain Network (EBN) is a blockchain network built by all virtual and physical nodes on a certain edge side of each subsystem of the coal mine. When a node needs to send or receive data, it first needs to send a request to the Edge Permission Management (E0) host. After the permission verification by the Edge Permission Management (E0) host, the node performs operations such as data download or data upload to the blockchain. Then, the Edge Permission Management (E0) host broadcasts and records the node's operation.

[0029] Furthermore, the cloud physical layer CP includes a virtual perception C1 host, a virtual decision-making C2 host, a virtual control C3 host, a collision detection C4 host, a human-machine evaluation C5 host, a cloud twin Cn host, VR devices, a centralized control operator, and a cloud permission management C0 host; the cloud permission management C0 host is responsible for managing the data transmission permissions between all virtual and physical nodes in the entire system's terminal layer, edge layer, and cloud layer.

[0030] Furthermore, the cloud-based virtual layer CV includes a global planning-level twin C and an n-dimensional cloud twin system comprising a virtual perception C1 system, a virtual decision-making C2 system, a virtual control C3 system, a collision detection C4 system, a human-machine evaluation C5 system, and a cloud twin Cn system. First, the virtual perception C1 system updates the latest and more complete scene information and then sends it to other cloud twin systems. The virtual decision-making C2 system generates the next stage of virtual decision based on the latest scene information and global tasks. Then, the virtual control C3 system executes the virtual decision. The operation process of virtual control is transmitted to the collision detection C4 system, the human-machine evaluation C5 system, and the cloud twin Cn system for relevant multi-parallel system analysis, comprehensively judging the quality of the virtual decision, adjusting the virtual decision, and finally generating the optimal global planning decision.

[0031] Furthermore, the Cloud Blockchain Network (CBN) is a blockchain network built by all virtual and physical nodes on the cloud side. When a node needs to send or receive data, it first needs to send a request to the cloud permission management C0 host. After the cloud permission management C0 host verifies the permissions, the node can perform operations such as downloading data or uploading data to the chain. Then, the cloud permission management C0 host will broadcast and record the node's operation. The Cloud Blockchain Network (CBN) packages and distributes the data from the cloud, transmitting it across chains to the Edge Blockchain Network (EBN).

[0032] Compared with existing technologies, the three-level n-dimensional digital twin system proposed in this invention, which supports the coupled operation of multi-scenario robot groups in coal mines, has the following beneficial effects:

[0033] (1) At the terminal layer, this invention deploys a terminal AI model and creates a single-machine behavioral twin. Through various sensors installed on the robot and in the work environment, high-value data such as the physical equipment's pose and the operator's body movements are directly extracted from the local machine and then mapped to the single-machine behavioral twin in real time. Furthermore, through an adapted bidirectional communication interface, the physical equipment can be controlled via the twin. The terminal AI model improves the data processing capabilities of the coal mine terminal layer equipment, resulting in a twin model with better real-time performance and realism. Simultaneously, the single-machine behavioral twin makes it easier to integrate terminal layer device nodes into the entire system, facilitating the transmission of control signals from the twin system to the physical equipment.

[0034] (2) At the edge layer, this invention deploys an edge AI model, creates a collaborative task-level twin, and builds an n-dimensional edge twin system. Through the collaborative mapping E1 system, collaborative operation E2 system, and other n-dimensional edge twin systems, information barriers between various edge devices are broken down, enabling rapid data sharing. Simultaneously, with the assistance of the edge AI model, the computational speed of intelligent devices in processing problems is improved, enabling rapid guidance of the terminal layer to handle specific collaborative tasks, greatly improving the efficiency of completing collaborative tasks at the edge.

[0035] (3) At the cloud layer, this invention deploys a cloud-based AI model, creates a global planning-level twin, and builds an n-dimensional cloud-based twin system. Through the virtual perception C1 system, and with the help of the cloud-based AI model, multi-source data from the edge is fused and processed to construct a more realistic twin system. Then, the n-dimensional cloud-based twin systems, including the virtual decision-making C2 system, virtual control C3 system, collision detection C4 system, and human-machine evaluation C5 system, perform related multi-twin parallel analysis to comprehensively judge the quality of the virtual decisions, adjust the virtual decisions, and finally generate the optimal global planning decision. This fully demonstrates that the cloud-based twin system can anticipate reality in advance and utilizes the parallel clones of the global planning-level twin to build multiple parallel twin systems to comprehensively analyze decisions and derive the optimal solution.

[0036] (4) Regarding data security, this invention establishes an edge blockchain network and a cloud blockchain network. This dual-layer blockchain network fully guarantees the security of data transmission and the speed of data transmission at the edge. The edge blockchain network is mainly responsible for the secure transmission and sharing of data between terminal nodes and edge nodes, offering stronger real-time performance. It can also package and upload edge-side data to the cloud blockchain network via an edge computing gateway. The cloud blockchain network is mainly responsible for the secure transmission and sharing of data between cloud nodes, as well as distributing decision results and other data from cloud nodes to the edge blockchain network. This achieves rapid and secure transmission of edge-side data and secure and efficient transmission of data between the cloud and edge during cloud-edge-device collaboration.

[0037] (5) Regarding access control, this invention deploys an edge access control E0 host and a cloud access control C0 host. This dual-layer access control ensures strict definition of communication permissions between all physical and virtual nodes in the system, enabling precise management of all physical and virtual assets. Each node can only communicate with the access control host and another node in the system through the identity certificate and license issued by the access control host, ensuring the closed and secure data transmission of the system.

[0038] (6) In terms of human-machine collaboration, this invention uses AR and VR devices to more closely connect field operators and central control operators to the n-dimensional edge twin system and the n-dimensional cloud twin system, respectively. Furthermore, based on the definition of communication permissions, field operators and central control operators can communicate with more virtual and physical asset nodes at different levels, such as directly controlling physical robots, single-machine behavior-level twins, collaborative task-level twins, and global planning-level twins. This provides more human intelligence to various detailed levels of the system's operation across the cloud, edge, and endpoint, truly realizing cloud-edge-end human-machine collaborative operation based on XR+ and the integration of virtual and physical elements. Attached Figure Description

[0039] Figure 1 This is a framework diagram of the three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group as described in this invention;

[0040] Figure 2 This is a communication initialization flowchart;

[0041] Figure 3 This is a communication flowchart when the recipient is a public object within the same blockchain network.

[0042] Figure 4 It is a communication flowchart when the recipient is a specific object within the same blockchain network;

[0043] Figure 5 This is a communication flowchart for a two-layer blockchain network;

[0044] Figure 6 This is a flowchart of the collaborative mapping edge side process;

[0045] Figure 7 This is a flowchart of the collaborative mapping cloud-based process;

[0046] Figure 8 This is a flowchart of the collaborative operation edge side;

[0047] Figure 9 This is a flowchart of the collaborative work process on the cloud side. Detailed Implementation

[0048] A typical embodiment of the present invention provides a three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group, such as... Figure 1 As shown, it includes three layers: the terminal layer, the edge layer, and the cloud layer.

[0049] (I) Terminal Layer

[0050] The terminal layer includes the terminal physical layer (TP) and the terminal virtual layer (TV).

[0051] The robot swarm in the terminal physical layer (TP) improves efficiency through the terminal AI model, and generates a single-machine behavioral twin (T) in the terminal virtual layer (TV) based on real-time data, enabling bidirectional control between the virtual and physical worlds. On-site operators obtain augmented information through AR devices, provide feedback for decision-making, and control nodes in a multimodal manner.

[0052] The terminal physical layer TP includes all physical assets, such as equipment, sensors, intelligent robots, and personnel, that make up the six major subsystems of a coal mine. This invention focuses on a specific operational scenario, using a robot group consisting of robot P2, robot P3, on-site operator P1, and other assets Pm as examples.

[0053] The six subsystems of the coal mine in the terminal physical layer TP include the coal mining subsystem, tunneling subsystem, electromechanical system, transportation subsystem, ventilation subsystem, and drainage subsystem.

[0054] The field operator P1 in the terminal physical layer TP refers to personnel working at the coal mine operation site, including various job types such as inspectors, maintenance workers, and operators. These field operators are equipped with intelligent mobile devices such as AR devices and smart bracelets.

[0055] The AR devices in the terminal physical layer (TP) refer to head-mounted AR devices, such as HoloLens glasses. On-site operators can obtain augmented information overlaid with virtual information through these devices, including their own work tasks and augmented action guidance, the current operating status of the equipment, work objectives, and system decision results. Furthermore, after successful authorization verification, they can utilize multimodal control to access physical assets such as on-site equipment and intelligent robots, as well as virtual assets such as stand-alone behavioral twins and collaborative task-level twins.

[0056] The Kinect device in the terminal physical layer TP is a 3D motion-sensing camera that can extract the limb movement information of the on-site operator through skeletal tracking, and then generate a virtual human twin driven by real-time data in Unity3D software. This allows the twin system to use the virtual human to perform virtual decision analysis such as collision detection and human-machine evaluation.

[0057] The robots P2 and P3 in the terminal physical layer TP refer to a group of intelligent robots used in coal mines. They are typically equipped with various sensors such as LiDAR, RTK-GPS, IMU, monocular cameras, and binocular cameras. They can run Ubuntu and ROS systems, as well as Unity3D software, and possess functions such as environmental detection, autonomous navigation, and machine vision. The robots can transmit sensor data to the Unity3D software on their local Ubuntu system via the ROS-Unity bidirectional communication tools, namely the ROS-TCP-Connector and ROS-TCP-Endpoint interfaces. A virtual scene is then constructed by comparing the data with a model library. Simultaneously, high-value data such as the robot's motion state and pose information are mapped in real-time into the Unity3D virtual scene, generating a corresponding single-machine behavioral twin.

[0058] Other assets Pm in the terminal physical layer TP refer to equipment, sensors, personnel, etc., other than the field operator P1, robot group robot P2, and robot P3 in the six subsystems of the coal mine. These physical assets can synchronously map their data information to the corresponding single-machine behavior-level twin in the Unity3D software through TCP / IP communication.

[0059] The terminal virtual layer TV is a virtual asset generated by all physical assets through sensors and other tools in Unity3D software under the Ubuntu system. It reflects the real-time data mapping of the physical assets and constructs a single-machine behavioral twin T, such as virtual human T1, robot T2, robot T3, and other twins Tm. At the same time, based on two-way communication tools, the virtual assets can also control the movement of physical assets in reverse.

[0060] The stand-alone behavioral twin T in the terminal virtual layer TV refers to the virtual asset generated in Unity3D software under the Ubuntu system by mapping the real-time data of the physical asset. The stand-alone behavioral twin can move synchronously based on the real-time data of the physical asset, or it can control the synchronous movement of the physical asset based on control scripts.

[0061] The virtual human T1 in the terminal virtual layer TV is a virtual twin generated in Unity3D from the limb movement information of the on-site operator P1 extracted by the Kinect device. It can only reflect the real-time dynamic data of the on-site operator P1 in one direction.

[0062] Robot T2 in the terminal virtual layer TV is a single-machine behavioral twin generated in real time by transmitting real-time data of robot P2 from the ROS end to the Unity3D software. Based on the ROS-Unity bidirectional communication tool, the movement of robot T2 is controlled in the Unity3D software, and its control signals can also be transmitted from the Unity3D software to the ROS end of robot P2, enabling the physical robot P2 to execute corresponding control commands.

[0063] Robot T3 in the terminal virtual layer TV is a single-machine behavioral twin generated in real time by mapping real-time data of robot P3 from the ROS end to the Unity3D software. Based on the ROS-Unity bidirectional communication tool, it is also possible to issue control commands to control the movement of robot T3 in the Unity3D software, and the ROS end of robot P3 will execute the corresponding control commands after receiving the control signals.

[0064] The other twins Tm in the terminal virtual layer TV are physical assets in the six subsystems of the coal mine that can synchronously map their data information to the corresponding single-machine behavior-level twins in the Unity3D software via TCP / IP communication. Through an adapted communication interface, it is also possible to control other physical assets Pm by controlling the twin Tm.

[0065] (II) Edge Layer

[0066] The edge layer includes the edge physical layer (EP), the edge virtual layer (EV), and the edge blockchain network (EBN).

[0067] The edge computing gateway of the edge physical layer EP provides computing power support for collaborative tasks on the edge side and builds a bridge for communication between the cloud and the terminal; the edge virtual layer EV generates multiple collaborative task-level twins E based on a single-machine behavior-level twin, and realizes the efficient operation of n-dimensional edge twin systems such as collaborative mapping and collaborative operations through the edge AI model.

[0068] The edge physical layer (EP) includes a collaborative mapping E1 edge computing gateway, a collaborative operation E2 edge computing gateway, an edge twin En edge computing gateway, and an edge access management E0 host. It provides computing power support for collaborative tasks at the edge, ensuring the real-time transmission of data during on-site operations, while also incorporating access management to guarantee the security of data transmission.

[0069] The collaborative mapping E1 edge computing gateway in the edge physical layer EP is responsible for handling the collaborative mapping tasks completed by robots P2 and P3, as well as the on-site operator P1. It connects to various devices and sensors in the terminal layer, ensuring the reception of data from terminal layers within the same local area network, and simultaneously writing data to the cloud blockchain network. It runs the collaborative mapping E1 system and is equipped with Unity3D software running on Ubuntu.

[0070] The collaborative operation E2 edge computing gateway in the edge physical layer EP is responsible for handling the collaborative tasks completed by robot P2, robot P3, and on-site operator P1. It enables one device to quickly acquire high-value data, such as the pose of another device, during collaborative operations, improving the speed and accuracy of collaborative operations. It runs the collaborative operation E2 system on a Unity3D system running on Ubuntu.

[0071] The edge twin En edge computing gateway in the edge physical layer (EP) is responsible for handling other collaborative tasks at the edge. In the six subsystems of a coal mine, there are many scenarios requiring collaborative tasks involving multiple devices simultaneously. By deploying the corresponding edge twin En edge computing gateway, information barriers between devices are broken down, enabling rapid data processing and sharing. Simultaneously, with the assistance of edge AI models, the computational speed of intelligent devices is improved, allowing for rapid processing of specific collaborative tasks and significantly increasing the efficiency of collaborative operations. It is equipped with Unity3D software running on Ubuntu and operates the edge twin En system.

[0072] The Edge Access Management (E0) host in the Edge Physical Layer (EP) refers to the host responsible for managing communication permissions between all terminal nodes and edge nodes on the edge side. When a new virtual or physical asset node needs to be introduced into each subsystem on the edge side, the new node information needs to be sent to the Edge Access Management (E0) host. The Edge Access Management (E0) host then generates a specific identity certificate based on the node information and issues it to the node. The certificate is the tool for the node to access the edge blockchain network and realize data transmission. The Edge Access Management (E0) host then sets up a communication fence for the new node, allowing it to communicate externally only through its certificate. Next, the Edge Access Management (E0) host defines the objects the new node can communicate with based on its functional purpose and writes this information into the communication permission database. Finally, the Edge Access Management (E0) host uploads the new node's node information, identity certificate, and updated communication permission database information to the Cloud Access Management (C0) host.

[0073] The terminal nodes managed by the edge permission management E0 host refer to all physical assets such as robots and AR devices at the terminal layer, as well as virtual assets such as stand-alone behavioral twins.

[0074] The edge nodes managed by the edge permission management E0 host refer to physical assets such as edge computing gateways in the edge layer, as well as virtual assets such as collaborative task-level twins and n-dimensional edge twin systems.

[0075] The edge virtual layer EV includes a collaborative task-level twin E and n-dimensional edge twin systems such as the collaborative mapping E1 system, the collaborative operation E2 system, and the edge twin En system. Each edge twin system establishes the communication bridges required for collaboration between multiple nodes on the edge side, ensuring efficient data utilization. Furthermore, different edge twin systems serve different collaborative tasks. Based on the database of each specific collaborative task, data mining through edge AI models can more efficiently guide the completion of collaborative tasks.

[0076] The collaborative task-level twin E in the edge virtual layer EV refers to multiple corresponding collaborative task-level twins E derived from the single-machine behavior-level twin T data in the terminal virtual layer TV. It mainly serves the construction of the n-dimensional edge twin system and is the basic building block of the n-dimensional edge twin system.

[0077] The collaborative mapping E1 system in the edge virtual layer EV refers to the edge twin system serving collaborative mapping tasks. The virtual human E11, robot E12, robot E13, and other twins E1m in this system are corresponding collaborative task-level twins E generated based on the single-machine behavioral-level twin data in the terminal virtual layer TV. The main functions of the collaborative mapping E1 system include: first, receiving single-machine behavioral-level twin data with successfully verified permissions and generating collaborative task-level twins; then, using an edge AI model, performing fusion analysis on the collaborative task-level twin data to determine the completeness of the current mapping data and analyzing and deciding on a preliminary repair plan; then, sending the preliminary repair plan to each single-machine behavioral-level twin, which in turn controls each device to execute the preliminary repair plan; simultaneously, the AR device also receives the current mapping data and the preliminary repair plan, and the on-site operator P1 provides feedback and controls the single-machine behavioral-level twins or physical robots to complete the current decision. Finally, the collaborative mapping E1 system uploads the complete mapping results to the cloud blockchain network.

[0078] The collaborative operation E2 system in the edge virtual layer EV refers to an edge twin system serving collaborative operation tasks. The virtual human E21, robot E22, robot E23, and other twins E2m in this system are corresponding collaborative task-level twins E generated based on the single-machine behavioral-level twin data in the terminal virtual layer TV. The main functions of the collaborative operation E2 system include: firstly, receiving single-machine behavioral-level twin data with successfully verified permissions, generating collaborative task-level twins, and obtaining the latest virtual scene of the current operation status. Then, based on communication permission database partitioning, it sends the data required by one device node to another device node, ensuring rapid data sharing. The collaborative operation E2 system also needs to determine the execution status of the current plan based on the current operation status and global planning, using an edge AI model. If the execution is not good, the edge AI model re-determines the allocation of human and machine tasks. The decision result is then sent to each single-machine behavioral-level twin and AR device. Each single-machine behavioral-level twin then controls each device to execute new operation tasks, while the on-site operator P1 also completes their own tasks based on the information received from the AR device. Finally, the E2 system uploads the collaborative work results to the cloud blockchain network.

[0079] The edge twin En system in the edge virtual layer EV refers to the edge twin system serving other collaborative tasks. The virtual human En1, robot En2, robot En3, and other twins Enm in this system are corresponding collaborative task-level twins E generated based on the single-machine behavioral-level twin data in the terminal virtual layer TV. By deploying multiple edge twin En systems in various subsystems of the coal mine, rapid data retrieval between virtual and physical nodes at the edge is ensured. Furthermore, the analytical capabilities of edge AI are leveraged to improve the efficiency of completing collaborative tasks.

[0080] The Edge Blockchain Network (EBN) refers to a blockchain network built by all virtual and physical nodes on the edge side of various subsystems in a coal mine. These virtual and physical nodes include physical assets such as robot groups and field operators equipped with AR devices in the terminal physical layer (TP); single-machine behavioral twins such as virtual human T1, robot T2, robot T3, and other twins Tm in the terminal virtual layer (TV); edge computing gateways and edge access management E0 hosts in the edge physical layer (EP); and collaborative task-level twins and n-dimensional edge twin systems such as the collaborative mapping E1 system, collaborative operation E2 system, and edge twin En system in the edge virtual layer (EV). Within the same edge blockchain network, secure data transmission between virtual and physical nodes can be achieved. When a node needs to send or receive data, it first needs to initiate a request to the edge access management E0 host. After the edge access management E0 host verifies the node's permissions, the node can perform operations such as downloading or uploading data to the blockchain. The edge access management E0 host then broadcasts and records the node's operation. The edge blockchain network can also package and upload data from the edge side, such as the latest scene data, and transmit it across chains to the cloud blockchain network CBN.

[0081] (III) Cloud Layer

[0082] The cloud layer includes the cloud physical layer (CP), the cloud virtual layer (CV), and the cloud blockchain network (CBN).

[0083] The cloud physical layer (CP) provides computing power to the n-dimensional cloud twin system. Centralized operators access the cloud twin system via VR devices for remote monitoring and control. The cloud virtual layer (CV) generates multiple global planning-level twins (C) based on edge data. Through cloud AI models, it performs parallel analysis of the n-dimensional cloud twin system, including virtual perception, virtual decision-making, virtual control, collision detection, and human-machine evaluation, ultimately deriving the optimal global plan.

[0084] The EBN and CBN dual-layer blockchain network ensures the security of data transmission between nodes and across chains within the same blockchain network. The edge access management E0 host and cloud access management C0 host ensure precise management of communication permissions for all virtual and physical asset nodes in the system.

[0085] The cloud physical layer CP includes a virtual perception C1 host, a virtual decision-making C2 host, a virtual control C3 host, a collision detection C4 host, a human-machine evaluation C5 host, a cloud twin Cn host, VR devices, a centralized control operator, and a cloud access control C0 host.

[0086] The virtual perception C1 host, virtual decision-making C2 host, virtual control C3 host, collision detection C4 host, human-machine evaluation C5 host, and cloud twin Cn host in the cloud physical layer CP are all equipped with Unity3D software under the Windows system and are responsible for running the cloud twin system, such as the virtual perception C1 system, virtual decision-making C2 system, virtual control C3 system, collision detection C4 system, human-machine evaluation C5 system, and cloud twin Cn system.

[0087] The centralized control operator in the cloud physical layer (CP) refers to the operator working in the centralized control center. Their main tasks include accessing the cloud twin system via VR devices after successful authorization verification, and immersively monitoring the system's operation. They then analyze the current state and provide their decision-making feedback to the cloud twin system. Furthermore, after successful authorization verification, they can control the twin's movement and experimentally test the decision results.

[0088] The VR devices in the cloud physical layer (CP) provide an immersive interactive experience of virtual 3D scenes, making the wearer feel completely integrated into the virtual environment. For example, the HTC Vive device. A complete HTC Vive setup mainly consists of a head-mounted display, two controllers, two SteamVR locators, a data cable, and a charging plug. After successful authorization verification, the central operator can access the cloud twin system through the VR device, immersively monitor its operation, and provide their own analytical and decision-making opinions. After successful authorization verification, they can also control the movement of the global planning-level twin within the cloud twin system to assist system operation.

[0089] The cloud-based access control (C0) host in the cloud physical layer (CP) refers to the host responsible for managing data transmission permissions between all virtual and physical nodes in the entire system's terminal layer, edge layer, and cloud layer. First, the edge access control (E0) hosts deployed in each coal mine subsystem upload the node information and communication permissions set for the edge side to the cloud access control (C0) host. Cloud nodes also need to send node information to the cloud access control (C0) host to complete registration. Then, the cloud access control (C0) host issues an identity certificate to each node and defines its communication permissions based on its function. This establishes a master database of communication permissions between all virtual and physical nodes in the entire system. The communication permissions of each node can be modified in the cloud access control (C0) host, and then updated in the communication permission databases of each edge side in each subsystem.

[0090] The cloud nodes managed by the cloud access control C0 host include physical assets such as computer hosts in the cloud layer and virtual assets such as global planning-level twins and n-dimensional cloud twin systems.

[0091] The cloud-based virtual layer (CV) includes a global planning-level twin C and n-dimensional cloud-based twin systems, such as a virtual perception C1 system, a virtual decision-making C2 system, a virtual control C3 system, a collision detection C4 system, a human-machine evaluation C5 system, and a cloud-based twin Cn system. First, the virtual perception C1 system updates the latest and more complete scene information and sends it to other cloud-based twin systems. The virtual decision-making C2 system generates the next stage's virtual decision based on the latest scene information and global tasks, and then the virtual control C3 system executes the virtual decision. The virtual control process is transmitted to the collision detection C4 system, the human-machine evaluation C5 system, and the cloud-based twin Cn system for related multi-parallel system analysis, comprehensively judging the quality of the virtual decision, adjusting the virtual decision, and ultimately generating the optimal global planning decision.

[0092] The global planning-level twin C in the cloud virtual layer CV refers to multiple global planning-level twins derived from the data of edge-side physical assets. It mainly serves the construction of the n-dimensional cloud twin system and is the basic building block of the n-dimensional cloud twin system.

[0093] The Virtual Perception C1 system in the cloud-based Virtual Layer (CV) refers to the cloud-based twin system serving virtual perception. The virtual human C11, robot C12, robot C13, and other twins C1m in this system are corresponding global planning-level twins generated based on the collaborative task-level twin data in the EV (Virtual Reality Layer). The main functions of the Virtual Perception C1 system include: first, downloading the latest mapping results and the latest status information of each physical asset from the cloud blockchain network; then, updating the data of the global planning-level twins in the system; and second, using a cloud-based AI model to perform fusion analysis on the global planning-level twin data to determine whether the current mapping data is complete. If complete, it notifies other cloud-based twin systems to update the mapping results. If incomplete, it analyzes and decides on a secondary repair plan, and after successful authorization verification, transmits the secondary repair plan to the collaborative mapping E1 system.

[0094] The virtual decision-making C2 system in the cloud-based virtual layer (CV) refers to a cloud-based twin system serving virtual decision-making. The virtual human C21, robot C22, robot C23, and other twins C2m in this system are corresponding global planning-level twins generated based on the latest twin data from the virtual perception C1 system. The main functions of the virtual decision-making C2 system include: first, receiving the latest and improved scene data from the virtual perception C1 system and updating the global planning-level twins in the virtual decision-making C2 system; then, based on the global task objectives, generating virtual decisions for the next stage through a cloud-based AI model, and rationally planning the target tasks that all equipment and personnel in the entire system should complete; and finally, sending the generated virtual decisions to the virtual control C3 system. Simultaneously, the virtual decision-making C2 system will also comprehensively adjust the virtual decisions based on subsequent parallel analysis of the cloud twin system's testing of the virtual decisions, until the optimal global planning decision is generated.

[0095] The virtual control C3 system in the cloud-based virtual layer CV refers to a cloud-based twin system serving virtual control. The virtual human C31, robot C32, robot C33, and other twins C3m in this system are corresponding global planning-level twins generated based on the latest twin data from the virtual perception C1 system. The main functions of the virtual control C3 system include: first, receiving the latest and improved scene data from the virtual perception C1 system and updating the global planning-level twins in the virtual control C3 system; then, receiving virtual decisions generated by the virtual decision-making C2 system, and autonomously generating virtual control scripts for each global planning-level twin based on the virtual decisions using a cloud-based AI model to achieve the requirements of the virtual decisions; then, running the scripts to control the global planning-level twins in the virtual control C3 system to execute the movements corresponding to the virtual decisions; and finally, uploading the virtual control scripts to the cloud blockchain network for analysis by other cloud-based twin systems.

[0096] The collision detection C4 system in the cloud-based virtual layer CV refers to a cloud-based twin system serving collision detection. The virtual human C41, robot C42, robot C43, and other twins C4m in this system are corresponding global planning-level twins generated based on the latest twin data from the virtual perception C1 system. The main functions of the collision detection C4 system include: first, receiving the latest and improved scene data from the virtual perception C1 system and updating the global planning-level twins in the collision detection C4 system; then, receiving virtual control scripts issued by the virtual control C3 system to control the movement of the global planning-level twins in the collision detection C4 system; then, each twin executes the collision detection analysis script to determine whether there are dangerous behaviors such as collisions based on the current virtual decision-making plan; and finally, feeding back the analysis results to the virtual decision-making C2 system.

[0097] The Human-Machine Evaluation C5 system in the cloud-based virtual layer CV refers to a cloud-based twin system serving human-machine evaluation. The virtual human C51, robot C52, robot C53, and other twins C5m in this system are corresponding global planning-level twins generated based on the latest twin data from the virtual perception C1 system. The main functions of the Human-Machine Evaluation C5 system include: first, receiving the latest and improved scene data from the virtual perception C1 system and updating the global planning-level twins in the Human-Machine Evaluation C5 system; then, receiving virtual control scripts issued by the virtual control C3 system to control the movement of the global planning-level twins in the Human-Machine Evaluation C5 system; then, each twin executes the human-machine evaluation analysis script, mainly judging whether the planning based on the current virtual decision is reasonable, whether the task allocation between operators and devices is reasonable, and whether efficient collaborative work is possible. The analysis results are then fed back to the virtual decision C2 system.

[0098] The cloud-based virtual twin Cn system in the cloud-based virtual layer CV refers to a cloud-based twin system that serves other global planning analyses. The virtual human Cn1, robot Cn2, robot Cn3, and other twins Cnm in this system are corresponding global planning-level twins generated based on the latest twin data from the virtual perception C1 system. The main functions of the cloud-based twin Cn system include: first, receiving the latest and improved scene data from the virtual perception C1 system and updating the global planning-level twins in the cloud-based twin Cn system; then, receiving virtual control scripts published by the virtual control C3 system to control the movement of the global planning-level twins in the cloud-based twin Cn system; and then, utilizing the rich communication interfaces of Unity3D software, connecting to other simulation analysis software or platforms to achieve more detailed simulation analysis, judge the quality of virtual decisions, and provide feedback on adjustments to the virtual decision-making C2 system. Simultaneously, the cloud-based twin Cn system fully embodies the characteristics of digital twin technology—surpassing reality through pre-calculation. Before physical space movement, the virtual decisions have been fully verified and improved through the n-dimensional cloud-based twin system, truly achieving optimal global planning.

[0099] The Cloud Blockchain Network (CBN) refers to a blockchain network built by all virtual and physical nodes on the cloud side of the system. These nodes include the computer hosts in the CP (Content Provider) and the cloud access management C0 host, as well as centralized control operators equipped with VR devices; and the global planning-level twin and n-dimensional cloud twin systems in the CV (Computer Controller) such as the virtual perception C1 system, virtual decision-making C2 system, virtual control C3 system, collision detection C4 system, human-machine evaluation C5 system, and cloud twin Cn system. Secure data transmission between nodes is possible within the cloud blockchain network. When a node needs to send or receive data, it first sends a request to the cloud access management C0 host. After authorization by the C0 host, the node can perform operations such as downloading or uploading data to the blockchain. The C0 host then broadcasts and records the node's operation. The cloud blockchain network can also package and distribute cloud data, such as generated optimal global plans, across blockchains to the edge blockchain network.

[0100] The communication in this invention system mainly includes three types: First, communication where, within the same blockchain network, data from a node can be received by any node in that blockchain, i.e., the receiver is a public object. Second, communication where, within the same blockchain network, data from a node can only be received by another node in that blockchain, i.e., the receiver is a specific object. Third, communication between the edge blockchain networks built on the edge sides of each coal mine subsystem and the cloud blockchain network built on the cloud side of the central control center.

[0101] As attached Figure 2 The diagram illustrates the initialization process before communication begins between nodes in both edge and cloud blockchain networks. First, all terminal nodes, edge nodes, cloud nodes, edge access management E0 hosts, and cloud access management C0 hosts generate private and public key pairs based on the RSA algorithm. The private key is kept secret by the node, while the public key is made public. Data encrypted with a node's public key can only be decrypted using the private key stored by that node.

[0102] All terminal nodes and edge nodes need to encrypt their own node information using the public key of the edge access management E0 host and send it to the edge access management E0 host. The edge access management E0 host decrypts the information using its own private key, registers the node information, and generates a unique identity certificate. This identity certificate is then encrypted using each node's public key and issued to the terminal and edge nodes. Each node then decrypts the certificate using its own private key to obtain its own identity certificate. Similarly, all cloud nodes also encrypt their own node information using the public key of the cloud access management C0 host and send it to the cloud access management C0 host. The cloud access management C0 host decrypts the information using its own private key, registers the node information, and generates a unique identity certificate. This certificate is then encrypted using each node's public key and issued to the cloud nodes. Each node then decrypts the certificate using its own private key to obtain its own identity certificate.

[0103] The edge access management E0 host sets up communication fences for all terminal nodes and edge nodes, while the cloud access management C0 host sets up communication fences for all cloud nodes. The communication fence means that after a node connects to the blockchain network and downloads the communication fence, it can only communicate externally based on a certificate issued by the access management host. There are two types of certificates: one is an identity certificate used for communication with the access management host; the other is license certificates a and b, issued by the access management host to the data sender and receiver respectively, after authorizing communication between any two nodes. The sender encrypts and sends data using license certificate a, and the receiver decrypts the data using license certificate b.

[0104] The edge access management E0 host defines the communication permissions of each node based on its functional purpose, specifying which nodes the node can send data to and receive data from. This information is then written into the communication permission database. The edge access management E0 host then uploads the node information registered on the edge side and the defined communication permission database to the cloud access management C0 host. The cloud access management C0 host also defines the communication permissions of each cloud node based on its functional purpose and writes this information into the overall communication permission database. In this way, the cloud access management C0 host builds an overall communication permission database containing all virtual and physical nodes of the system, and can modify the communication permissions of each node, updating the communication permission databases of each edge access management E0 host.

[0105] After completing the communication initialization settings, secure communication can be achieved. The specific procedures for the three communication types in this invention's system are described below.

[0106] As attached Figure 3The diagram illustrates the communication process within the same blockchain network when the recipient is a public object, using the example of terminal node A needing to send data to a public object, and edge node B wanting to receive that data. First, node A initiates a request, the content of which is encrypted using the E0 public key. This request includes A's identity certificate, the recipient being a public object, a data digest, and a timestamp, and is added to the request sequence of the edge access management E0 host. Then, the edge access management E0 host receives requests sequentially and decrypts the data using its own private key. It then checks if node A's identity certificate is correct. If the identity certificate comparison fails, A's request is deleted. If the identity certificate comparison succeeds, and after determining that A is the sender and the recipient is a public object, the edge access management E0 host broadcasts node A's request throughout the blockchain network.

[0107] After receiving a broadcast request, other nodes in the blockchain network determine whether they need to receive data from node A based on their own requirements. If edge node B needs to receive this data, it will send a request to the edge access management E0 host. The request content, encrypted with the E0 public key, includes B's identity certificate, the recipient being B, and a timestamp indicating that it is receiving data from A. This request is then added to the edge access management E0 host's request sequence. The edge access management E0 host then receives requests sequentially and decrypts the data using its private key. It then checks if node B's identity certificate is correct. If the identity certificate verification fails, B's request is deleted. If the identity certificate verification succeeds, B is determined to be the recipient, and A is the sender.

[0108] The edge access management E0 host determines the communication permissions between A and B based on the communication permission database, i.e., whether B can receive data from A. If the permission comparison fails, B's request is deleted. If the permission comparison succeeds, the edge access management E0 host generates a pair of key pairs with expiration dates: license certificate a and license certificate b. It then sends license certificate b, encrypted with B's public key, with an expiration date and a timestamp indicating it will receive data from A; and sends license certificate a, encrypted with A's public key, with an expiration date, B as the recipient, and a timestamp indicating it will receive data from A. A and B then decrypt the data using their respective private keys to obtain the content.

[0109] Node A sends data encrypted with license certificate 'a' to Node B. Node B then decrypts the data using license certificate 'b'. After receiving the data, Node B sends a request to the edge management host E0, encrypted with E0's public key, containing B's identity certificate and a timestamp. The edge management host E0 then decrypts the data and deletes the timestamp request. Finally, the edge management host E0 broadcasts B's receipt of A's timestamp request to the blockchain network, creating a record.

[0110] As attached Figure 4The diagram illustrates the communication process within the same blockchain network when the recipient is a specific object, taking the example of terminal node A needing to send data to edge node C, a specific object. First, node A initiates a request, the content of which is encrypted using the E0 public key. This request includes A's identity certificate, the recipient being node C, a data digest, and a timestamp, and is added to the request sequence of the edge permission management E0 host. Then, the edge permission management E0 host receives the requests in sequence and decrypts the data using its own private key. It then checks if node A's identity certificate is correct; if the certificate verification fails, the request is deleted. If the certificate verification succeeds, and A is confirmed as the sender and the recipient as the specific object node C, the edge permission management E0 host searches for node C in the communication permission database. If the search fails, A's request is deleted. If the search succeeds, it begins to determine the communication permissions from A to C, i.e., whether A can send data to C. If the permission verification fails, A's request is deleted. If the permission verification succeeds, the edge permission management E0 host generates a pair of key pairs with a limited-time license certificate 'a' and license certificate 'b'. Then, a license certificate 'a' encrypted with A's public key, specifying its expiration date and a timestamp from A, is sent to A; a license certificate 'b' encrypted with C's public key, specifying its expiration date and a timestamp from A, is also sent to C. A and C then each decrypt the license using their own private keys to obtain the content.

[0111] Node A sends data encrypted with license certificate 'a' to Node C. Node C then decrypts the data using license certificate 'b'. After receiving the data, Node C sends a request to the edge management host E0, containing its identity certificate encrypted with E0's public key, along with a timestamp from A, indicating successful data reception. The edge management host E0 then decrypts the data and deletes the request from A's timestamp. Finally, the edge management host E0 broadcasts Node C's receipt of A's timestamp request to the blockchain network, creating a record.

[0112] As attached Figure 5 The diagram illustrates the communication process between the edge blockchain network and the cloud blockchain network. First, data from the edge blockchain network is used to create transactions via the edge computing gateway. Then, the data from the edge side is packaged and uploaded to the cloud blockchain network. The cloud blockchain network verifies permissions based on the communication permission database information in the cloud permission management C0 host. If verification fails, the transaction is deleted. If verification succeeds, the packaged data is written to the cloud blockchain network. A data digest is then broadcast to all cloud nodes. If a cloud twin system needs certain data, it can download it from the cloud blockchain network after permission verification.

[0113] Conversely, data from the cloud-based blockchain network is used to create transactions via the cloud host. The cloud-side data is then packaged and distributed to the edge blockchain network. The edge blockchain network verifies permissions based on the communication permission database information in the edge permission management E0 host. If verification fails, the transaction is deleted. If verification succeeds, the packaged data is written to the edge blockchain network. A data digest is then broadcast to all edge nodes and terminal nodes. If an edge node or terminal node needs certain data, it can download it from the edge blockchain network after permission verification.

[0114] After establishing the communication network, the following example illustrates the specific operational flow of the system of this invention, using a coal mine operation scenario as an example: the on-site operator P1, the detection robot P2, and the work robot P3 collaborate to complete mapping and operational tasks, while the central control operator remotely monitors the process. The on-site operator P1 and the central control operator are equipped with AR and VR devices, respectively. The detection robot P2 is a four-wheeled, differentially driven, independently driven wheeled intelligent robot. It is equipped with multiple sensors, including LiDAR, RTK-GPS, IMU, monocular camera, and binocular camera, and runs Ubuntu 18.04, ROSMelodic (Full Desktop), and Unity3D software. The work robot P3 is also a four-wheeled, differentially driven, independently driven wheeled intelligent robot. It consists of a wheeled chassis and a six-axis robotic arm, which can perform various tasks through an end effector mounted at the end of the six-axis robotic arm. Its other software and hardware configurations are consistent with the detection robot, and it also possesses functions such as environmental detection, autonomous navigation, and machine vision.

[0115] As attached Figure 6 As shown, at the edge of the collaborative mapping process, the detection robot P2 and the operation robot P3 first execute mapping tasks according to the initial mapping target. The detection data from their onboard sensors and their own status data are processed by the terminal AI model and mapped to the detection robot T2 and the operation robot T3. After successful permission verification, the data from the detection robot T2 and the operation robot T3 are transmitted to the collaborative mapping E1 system, where they are virtually mapped to generate the detection robot E12, the operation robot E13, and so on.

[0116] In the collaborative mapping E1 system, an edge AI model integrates and analyzes current detection data, mapping tasks, and secondary repair plans from the cloud to determine the completeness of the current mapping data. If complete, the E1 system uploads the collaborative mapping results to the cloud blockchain network after successful authorization verification. If incomplete, the E1 system analyzes and makes decisions, generating a preliminary repair plan. Simultaneously, the preliminary mapping results and the preliminary repair plan are transmitted to the on-site operator P1 via an AR device. Operator P1 then analyzes the mapping results and provides feedback to help the E1 system generate a final decision. Finally, the decision result, after successful authorization verification, is transmitted to the detection robot T2, the operation robot T3, and the AR device.

[0117] Furthermore, based on the decision-making results, the detection robot T2 and the operation robot T3 will generate control signals through the terminal AI model to control the detection robot P2 and the operation robot P3 to perform corresponding mapping tasks. If the detection robot P2 and the operation robot P3 encounter difficulties, the on-site operator P1 can use AR equipment, after successful authorization verification, to transmit control signals to the detection robot T2 and the operation robot T3 through multimodal control. Alternatively, after successful authorization verification, the on-site operator P1 can directly control the detection robot P2 and the operation robot P3 to help them solve the problem. Then, the new sensor data will be transmitted to the E1 system for another assessment of the mapping data completeness until the assessment result is complete, and then the collaborative mapping result will be uploaded to the cloud blockchain network.

[0118] As attached Figure 7 As shown, on the cloud side of the collaborative mapping process, after successful permission verification, the Virtual Perception C1 system downloads the collaborative mapping results from the cloud blockchain network and updates the data of the global planning-level twin. Through a cloud-based AI model, the data of the global planning-level twin is fused and analyzed to determine whether the current mapping data is complete. If complete, other cloud-based twin systems are notified to update the mapping results. If incomplete, the cloud-based AI model analyzes and decides on a secondary repair plan, and after successful permission verification, the secondary repair plan is transmitted to the collaborative mapping E1 system. Throughout the process, the central control operator can access the Virtual Perception C1 system through a VR device that has successfully verified permissions, immersively monitoring and analyzing the mapping results and secondary repair plans; after permission verification, they can control the global planning-level twin to experiment with how to control the robot to optimize the mapping effect; and then provide feedback to the Virtual Perception C1 system to assist in its decision-making.

[0119] As attached Figure 8As shown, at the edge of the collaborative operation process, the detection robot P2 and the operation robot P3 first execute their respective tasks according to their initial task objectives. The operation data and their own status data detected by their onboard sensors are processed by the edge AI model and mapped to the detection robot T2 and the operation robot T3. After successful authorization verification, the data from the detection robot T2 and the operation robot T3 are transmitted to the collaborative operation E2 system, updating the data of the detection robot E22 and the operation robot E23. At the same time, the on-site operator P1 also receives their current task through the AR device, completes the relevant task, and then the Kinect device extracts the body movement information of the on-site operator P1 to generate a virtual human T1. After successful authorization verification, the data of the virtual human T1 is transmitted to the collaborative operation E2 system.

[0120] Furthermore, in the collaborative operation E2 system, an edge AI model is used to fuse and process data uploaded from various devices, updating the state of their respective collaborative task-level twins to obtain the latest virtual scene of the current operation status. After the update is complete, if nodes such as on-site operator P1, detection robot T2, and operation robot T3 require high-value data such as the pose of another node, the E2 system will distribute the required data based on communication permission-based database partitioning. Then, the node will adjust its own pose and other states according to the data obtained from the other node to complete the collaborative operation task.

[0121] The collaborative operation E2 system also uses an edge AI model to assess the execution status of the current plan based on the current operation status and global planning. If the execution is unsatisfactory, the edge AI model makes a preliminary decision on the allocation of tasks between humans and machines. Simultaneously, the operation status and decision results are transmitted to the on-site operator P1 via an AR device. After analyzing the status, operator P1 provides feedback to help the collaborative operation E2 system generate a final decision. Finally, after successful authorization verification, the decision result is sent to the detection robot T2, the operation robot T3, and the AR device.

[0122] Based on the decision-making results, the detection robot T2 and the work robot T3 will generate control signals through the terminal AI model to control the detection robot P2 and the work robot P3 to execute corresponding new work tasks. The on-site operator P1 will also complete their assigned tasks based on the augmented information provided by the AR device. If the detection robot P2 or the work robot P3 encounters difficulties, the on-site operator P1 can use the AR device, after successful authorization verification, to transmit control signals to the detection robot T2 and the work robot T3 through multimodal control. Alternatively, after successful authorization verification, the on-site operator P1 can directly control the detection robot P2 and the work robot P3 to help them solve the problem. Then, the new work data will be transmitted to the collaborative work E2 system. After updating the data, the collaborative work E2 system, if it determines that the execution is successful, will upload the collaborative work results to the cloud blockchain network after successful authorization verification.

[0123] As attached Figure 9 As shown, on the cloud side of the collaborative operation process, after successful permission verification, the virtual perception C1 system downloads the latest collaborative operation results from the cloud blockchain network and updates the data of the global planning-level twin. It then sends this data to other cloud twin systems. The virtual decision-making C2 system, after successful permission verification, generates the next stage's virtual decision based on the latest scenario information and global tasks. Then, the virtual control C3 system, after receiving the virtual decision, autonomously generates a virtual control script through a cloud AI model, ensuring that the global planning-level twin in the virtual control C3 system strictly executes the virtual decision.

[0124] The entire virtual control process involves inputting data into several cloud-based twin systems, including the collision detection C4 system (which has successfully passed permission verification), the human-machine evaluation C5 system, and the cloud twin Cn system. These systems perform parallel analysis, using a cloud-based AI model to comprehensively assess the quality of the virtual decisions. If any unreasonable factors are detected in the virtual decision-making process, feedback is sent to the virtual decision-making C2 system, which then adjusts the virtual decision using the cloud-based AI model. This process continues until the optimal global planning decision is generated. The global plan is then transmitted to the collaborative operation E2 system, which has passed permission verification, to guide the collaborative operation E2 system in completing the collaborative task.

[0125] Throughout the process, the central control operator can access cloud twin systems such as the Virtual Perception C1 system, Virtual Decision C2 system, Virtual Control C3 system, Collision Detection C4 system, Human-Machine Evaluation C5 system, and Cloud Twin Cn system through VR devices that have successfully passed permission verification. This allows for immersive monitoring and analysis of the current operational status and decision-making results. After permission verification, the operator can control the global planning-level twin to experiment with how to control the robot to optimize virtual decisions. The operator can then provide feedback on their decision-making to each system, offering human analytical and decision-making wisdom to the operation of each cloud twin system and helping the Virtual Decision C2 system achieve optimal global planning decisions.

Claims

1. A three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot swarm, characterized in that: It includes three layers: the terminal layer, the edge layer, and the cloud layer; The terminal layer includes the terminal physical layer (TP) and the terminal virtual layer (TV). The robot group in the terminal physical layer TP generates a single-machine behavior-level twin T in the terminal virtual layer TV based on real-time data through the terminal AI model, realizing two-way control between the virtual and physical worlds. On-site operators obtain augmented information through AR devices, provide feedback on decision-making, and control nodes in a multimodal manner. The edge layer includes the edge physical layer (EP), the edge virtual layer (EV), and the edge blockchain network (EBN). The edge computing gateway of the edge physical layer EP provides computing power support for collaborative tasks on the edge side and builds a bridge for communication between the cloud layer and the terminal layer; the edge virtual layer EV generates multiple collaborative task-level twins E based on the single-machine behavior-level twin T, and constructs an n-dimensional edge twin system including the collaborative mapping E1 system, the collaborative operation E2 system, and the edge twin En system through the edge AI model. The collaborative mapping E1 system in the edge virtual layer EV refers to the edge twin system that serves the collaborative mapping task. The virtual human E11, robot E12, robot E13, and other twins E1m in this system are corresponding collaborative task-level twins E generated based on the single-machine behavior-level twin data in the terminal virtual layer TV. The collaborative operation E2 system in the edge virtual layer EV refers to the edge twin system that serves collaborative operation tasks. The virtual human E21, robot E22, robot E23, and other twins E2m in this system are corresponding collaborative task-level twins E generated based on the single-machine behavior-level twin data in the terminal virtual layer TV. The edge twin En system in the edge virtual layer EV refers to the edge twin system that serves other collaborative tasks. The virtual human En1, robot En2, robot En3, and other twins Enm in this system are corresponding collaborative task-level twins E generated based on the single-machine behavior-level twin data in the terminal virtual layer TV. The cloud layer includes the cloud physical layer (CP), the cloud virtual layer (CV), and the cloud blockchain network (CBN). The computer host of the cloud physical layer CP provides computing power support for the n-dimensional cloud twin system. The central control operator accesses the cloud twin system through VR devices to realize remote monitoring and control. The cloud virtual layer CV generates multiple global planning-level twins C based on edge side data. Through cloud AI models, it realizes parallel analysis of the n-dimensional cloud twin system, including virtual perception, virtual decision-making, virtual control, collision detection and human-machine evaluation, and comprehensively derives the optimal global plan. The cloud-based virtual layer (CV) is an n-dimensional cloud twin system, including a virtual perception C1 system, a virtual decision-making C2 system, a virtual control C3 system, a collision detection C4 system, a human-machine evaluation C5 system, and a cloud twin Cn system. First, the virtual perception C1 system updates the latest and more complete scene information and then sends it to other cloud twin systems. The virtual decision-making C2 system generates the next stage of virtual decision based on the latest scene information and global tasks. Then, the virtual control C3 system executes the virtual decision. The operation process of virtual control is transmitted to the collision detection C4 system, the human-machine evaluation C5 system, and the cloud twin Cn system for relevant multi-parallel system analysis, comprehensively judging the quality of the virtual decision, adjusting the virtual decision, and finally generating the optimal global planning decision. The cloud twin Cn system in the cloud virtual layer CV refers to the cloud twin system that serves other global planning analyses; The virtual perception C1 system in the cloud virtual layer CV refers to the cloud twin system that serves virtual perception. The virtual human C11, robot C12, robot C13, and other twins C1m in this system are corresponding global planning level twins generated based on the collaborative task-level twin data in the edge virtual layer EV.

2. The three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group according to claim 1, characterized in that: In the terminal physical layer (TP), the robot group transmits the data detected by the sensors to the Unity3D software under the local Ubuntu system through the ROS-Unity bidirectional communication tool. Then, it constructs a virtual scene by comparing with the model library. At the same time, the robot's own motion state and pose information are mapped to the Unity3D virtual scene in real time to generate the corresponding single-machine behavior-level twin (T).

3. The three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group according to claim 1 or 2, characterized in that: The edge physical layer (EP) includes a collaborative mapping E1 edge computing gateway, a collaborative operation E2 edge computing gateway, an edge twin En edge computing gateway, and an edge access management E0 host. The collaborative mapping E1 edge computing gateway handles the collaborative mapping tasks between robot P2, robot P3, and on-site operator P1. The collaborative operation E2 edge computing gateway handles the collaborative operation tasks between robot P2, robot P3, and on-site operator P1. The edge twin En edge computing gateway handles other collaborative tasks at the edge. The edge access management E0 host manages the communication permissions between all terminal nodes and edge nodes on this edge side.

4. The three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group according to claim 3, characterized in that: In the edge virtual layer (EV), the system first receives the data of a single-machine behavioral twin T that has successfully passed permission verification, generates a collaborative task-level twin E, and then uses an edge AI model to fuse and analyze the data of the collaborative task-level twin E to determine whether the current mapping data is complete. It then analyzes and decides on a preliminary repair plan and sends the preliminary repair plan to each single-machine behavioral twin T. Each single-machine behavioral twin T then controls each device to execute the preliminary repair plan. At the same time, the AR device also receives the current mapping data and the preliminary repair plan. The on-site operator P1 provides feedback and controls the single-machine behavioral twin T or the physical robot to complete the current decision. Finally, the collaborative mapping E1 system uploads the complete mapping result to the cloud blockchain network CBN.

5. The three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group according to claim 4, characterized in that: In the edge virtual layer EV, the system first receives the data of the single-machine behavior-level twin T that has successfully passed permission comparison, generates a collaborative task-level twin E, and obtains the virtual scene of the latest work status. Then, based on the communication permission database, it sends the data of another device node that it needs to a certain device node to ensure rapid data sharing. The collaborative operation E2 system, based on the current work status and global planning, judges the execution status of the current plan through the edge AI model. If the execution is not good, it re-determines the allocation of human and machine tasks through the edge AI model, and then sends the decision results to each single-machine behavior-level twin and AR device. Each single-machine behavior-level twin T then controls each device to execute new work tasks. At the same time, the on-site operator P1 also completes its own tasks based on the information received by the AR device. Finally, the collaborative operation E2 system uploads the collaborative operation results to the cloud blockchain network CBN.

6. The three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group according to claim 5, characterized in that: In the edge virtual layer (EV), multiple edge twin En systems deployed in various subsystems of the coal mine ensure rapid access to data between virtual and physical nodes at the edge, and improve the efficiency of completing collaborative tasks by leveraging the analytical capabilities of edge AI.

7. The three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group according to claim 6, characterized in that: The Edge Blockchain Network (EBN) is a blockchain network built by all virtual and physical nodes on the edge side of each subsystem of the coal mine. When a node needs to send or receive data, it first needs to send a request to the Edge Permission Management (E0) host. After the E0 host verifies the permissions, the node performs the data download or data upload operation. Then the E0 host broadcasts and records the node's operation.

8. The three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group according to claim 1 or 7, characterized in that: The cloud physical layer CP includes a virtual perception C1 host, a virtual decision-making C2 host, a virtual control C3 host, a collision detection C4 host, a human-machine evaluation C5 host, a cloud twin Cn host, VR devices, a centralized control operator, and a cloud permission management C0 host; the cloud permission management C0 host is responsible for managing the data transmission permissions between all virtual and physical nodes in the entire system's terminal layer, edge layer, and cloud layer.

9. The three-level n-dimensional digital twin system supporting the coupled operation of a coal mine robot group according to claim 8, characterized in that: The Cloud Blockchain Network (CBN) is a blockchain network built by all virtual and physical nodes on the cloud side. When a node needs to send or receive data, it first needs to send a request to the cloud permission management C0 host. After the cloud permission management C0 host verifies the permissions, the node performs the data download or data upload operation. Then, the cloud permission management C0 host broadcasts and records the node's operation. The Cloud Blockchain Network (CBN) packages and distributes the data from the cloud to the Edge Blockchain Network (EBN) for cross-chain transmission.

Images