Digital twinning system of strip mine drilling machine based on WEBGL

Abstract

The invention discloses a digital twin system of a strip mine drilling machine based on WEBGL, which is constructed based on Web technology, adopts a three-dimensional visual engine taking WebGL as a bottom graphic interface, comprises mainstream open source frameworks such as Cesium and Three.js, and performs real-time butt joint with data of a front-end visual platform and a rear-end drilling machine sensor. Key operation information such as the geographic position, the drill boom posture and the operation track of the drilling machine can be visually displayed in a common browser, and the transparency and controllability of equipment operation are greatly enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to a WEBGL-based digital twin system for a drill rig in an open-pit mine, belonging to the field of three-dimensional visualization and digital twin technology. BACKGROUND

[0002] With the rapid development of digital twin technology in the industrial background, its application in the fields of equipment manufacturing, smart mines, intelligent operation and maintenance, etc. is continuously deepening, and has gradually become one of the core means to improve the intelligent level of equipment and the efficiency of operation and maintenance. Especially in the management of open-pit mine mechanical equipment, the digital twin system can realize the virtual-real mapping and state visualization of mine operation equipment by means of three-dimensional modeling and real-time data driving, providing strong support for operation scheduling, remote monitoring and intelligent decision-making.

[0003] In the actual production process of an open-pit mine, the drill rig, as a key hole-distributing equipment, directly affects the efficiency and safety of subsequent blasting, loading and dumping, etc. However, at present, the monitoring of the drill rig in most mines still mainly relies on manual patrol, video monitoring or recording systems, which has several significant problems: such as the monitoring blind area caused by limited view angle, the lack of real-time caused by communication delay, the response delay caused by lagging data acquisition and analysis methods, and the lack of unified and intuitive three-dimensional information display means. These problems are more prominent in complex terrain and harsh environment, which seriously restricts the improvement of the intelligent level of open-pit mines. SUMMARY

[0004] The technical problem to be solved by the present application is to overcome the defects of the prior art and provide a WEBGL-based digital twin system for a drill rig in an open-pit mine. The present application has become a development trend to build a three-dimensional digital twin visualization system highly consistent with the real open-pit mine scene. The system is built based on Web technology, uses a three-dimensional visualization engine with WebGL as the underlying graphics interface, includes mainstream open-source frameworks such as Cesium and Three.js, and combines the front-end visualization platform with the back-end drill rig sensor data for real-time connection. The system can intuitively present the key operating information of the drill rig, such as the geographical position, drill arm posture and operating trajectory, etc. in a normal browser, which greatly enhances the transparency and controllability of equipment operation.

[0005] Preferably, the present application provides a WEBGL-based digital twin system for a drill rig in an open-pit mine, comprising:

[0006] The operating data of the drill rig chassis is received in real time by the back end;

[0007] The operating data of the drill arm and its components is received in real time to obtain multi-source sensor data;

[0008] The multi-source sensor data is classified and packaged uniformly by the back end to obtain structured real-time data;

[0009] pushing the structured real-time data stream to the front-end visualization platform built based on Vue.js through WebSocket by using the back-end;

[0010] The back-end comprises a Socket communication module, a data analysis module and a WebSocket service module.

[0011] The Socket communication module establishes a connection with the controller deployed on the drilling rig through TCP protocol, continuously receives the operation data of the drilling rig chassis and the operation data of the drilling boom and its components; wherein the operation data of the drilling rig chassis is collected by a Python program, establishes a TCP connection with the CAN gateway unit through the native socket library, and obtains the CAN frame data forwarded by the gateway in real time, encapsulates the CAN frame data into a standard JSON format and pushes it to the back-end in real time through TCP protocol; the operation data of the drilling boom and its components is received in real time and pushed to the back-end through TCP protocol.

[0012] The data analysis module is used to classify and extract the received operation data of the drilling rig chassis and the operation data of the drilling boom and its components according to the pre-defined data protocol, obtain the key fields and encapsulate them into the structured real-time data in the JSON structure conforming to the Web transmission format.

[0013] The WebSocket service module is used to establish a full-duplex communication channel with the front-end visualization platform by using WebSocket protocol, and push the structured real-time data to the front-end visualization platform based on the event-driven mechanism.

[0014] Preferably, the front-end visualization platform loads the three-dimensional terrain model based on the Cesium engine, and loads the drilling rig three-dimensional model based on the Three.js engine.

[0015] The front-end visualization platform converts the local coordinate system of the drilling rig model drilling rig three-dimensional model into the Cesium geographic coordinate system, adjusts the position, rotation angle and zoom ratio of the drilling rig three-dimensional model, and superimposes the drilling rig three-dimensional model and the three-dimensional terrain model.

[0016] The front-end visualization platform synchronizes the Three.js camera parameters and the Cesium camera view angle.

[0017] Preferably, the front-end visualization platform continuously obtains the structured real-time data pushed by the back-end through the WebSocket connection.

[0018] After receiving the structured real-time data by the front-end visualization platform, the corresponding parameters are extracted, and the corresponding parameters are bound to the position and quaternion attributes of the model in the Three.js engine.

[0019] The structured real-time data is parsed by the front-end visualization platform and is bound to the corresponding drilling rig and components, so that the drilling rig and components respond to the control logic of the drilling boom controller and perform corresponding actions.

[0020] Preferably, the structured real-time data pushed by the back-end is continuously acquired by the front-end visualization platform through a WebSocket connection, including:

[0021] Based on the Vue.js framework, the front-end visualization platform constructs a visualization interface, establishes a full-duplex communication connection with the back-end through the WebSocket protocol, and receives and parses the transmitted structured real-time data in real time.

[0022] Preferably, the structured real-time data is parsed by the front-end visualization platform and is bound to the corresponding drilling rig and components, so that the drilling rig and components respond to the control logic of the drilling boom controller and perform corresponding actions, including:

[0023] The structured real-time data is used to drive the dynamic translation and rotation of the drilling rig three-dimensional model in the three-dimensional space, as well as the action changes of the drilling boom and its components.

[0024] Preferably, the operation data of the drilling rig chassis includes latitude, longitude, heading angle, speed, attitude angle, and running mode.

[0025] Preferably, the present application provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to realize the steps of any one of the methods.

[0026] Preferably, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to realize the steps of any one of the methods.

[0027] The present application has the following beneficial effects:

[0028] The present application provides a WebGL-based open-pit mine drilling rig digital twin visualization system, which combines Cesium and Three.js lightweight three-dimensional engines to construct a real-time three-dimensional dynamic display system for open-pit mine drilling rig operation scenarios. Through the front-end and back-end separation architecture, combined with three-dimensional terrain models and high-precision drilling rig three-dimensional models, the system realizes dynamic visualization mapping of drilling rig movement, drilling boom action and other operation processes, effectively improving the accuracy and intuitiveness of equipment state perception and remote supervision in complex operation scenarios.

[0029] Compared with the traditional way of relying on a heavy graphics engine, the present application is directly run based on a browser end, does not need to install additional plug-ins or clients, has good cross-platform compatibility and rapid deployment capability, and is especially suitable for mine industrial scenes that need flexible deployment, remote access and lightweight maintenance. The system performs excellently in data communication efficiency, three-dimensional model loading speed and user interaction response, can realize high real-time and low-delay data-driven visualization effects, fully meets the comprehensive needs of complex industrial environments for system lightweight, response speed and stability, and has good practicability and popularization value. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, other drawings can also be obtained by those skilled in the art without any creative effort on the premise of not paying any creative effort.

[0031] Figure 1 is a whole architecture diagram of the open-pit mine drill digital twin visualization system;

[0032] Figure 2 is a flow chart of the open-pit mine drill digital twin model construction;

[0033] Figure 3 is an interface diagram of the experimental table digital twin system. DETAILED DESCRIPTION

[0034] Referring to Figure 1 , the present application is a whole architecture diagram of the open-pit mine drill digital twin visualization system based on WebGL, fuses Cesium and Three.js engines, and realizes unified rendering of the open-pit mine terrain and the drill three-dimensional model. Through cooperation of the front end and the back end and real-time data transmission, the system can restore the motion state and component action of the drill. The key lies in spatial coordinate unification, data transmission real-time and action control accuracy. The system includes a back end constructed based on Spring Boot and a front end visualization platform constructed based on Vue.js, supports access and three-dimensional display of multi-source sensor data such as positioning and attitude of the open-pit mine drill.

[0035] To realize three-dimensional visualization display, as shown in Figure 2 , the system needs to construct a digital model including the mine terrain and the drill structure. The three-dimensional terrain model serves as the basis of the space scene, is used for carrying and positioning the drill; the drill three-dimensional model serves as the core carrier of dynamic display, is used for reflecting the motion trajectory and component action state of the drill. The above models are loaded and rendered by Cesium and Three.js engines in the system respectively, and through structured real-time data driving, dynamic restoration and interactive response in vision are realized.

[0036] Firstly, in terms of the construction of the terrain environment, the unmanned aerial vehicle oblique photography technology is used to obtain image data of the open-pit mine terrain, and the three-dimensional reconstruction software is used for automatic modeling processing. Through the generated triangular mesh and texture map file, the standard 3D Tiles format data is exported, so that the terrain has high fidelity spatial restoration capability. The 3D Tiles format data is loaded by the Cesium engine in the front-end visualization platform, and serves as the spatial basement of the overall three-dimensional visualization scene.

[0037] In the production of the three-dimensional model of the drill rig in the open-pit mine, firstly, a complete geometric model is established using SolidWorks software according to the shape and size characteristics of the drill rig, and is exported in STEP format. Considering the requirements of Three.js rendering on the model format, it needs to be converted into a GLB format model, which is completed by an online tool. In order to realize the independent animation control of the drill arm and its components, the initially generated GLB format model is imported into Blender software, and the movable structures (such as drill arm and drill bit) are split and optimized. Mainly including: separating components, setting rotation axis, uniform naming and assigning materials, ensuring that the GLB format model has good animation controllability and visual consistency in the front-end visualization platform. Finally, the optimized GLB file model is exported for loading by the front-end visualization platform. The invention uses the combination of SolidWorks modeling and Blender optimization to split the structure of the drill arm and its components, set the rotation axis, name and assign materials, realize the independent animation control of the drill rig and its components, and improve the dynamic response capability and display precision of the model.

[0038] In order to realize the real-time perception and visualization driving of the digital twin system on the running state of the drill rig, the system needs to obtain the real-time operation data of the drill rig. The data acquisition work is divided into two main modules: the CAN bus module of the drill rig chassis and the drill arm controller.

[0039] Firstly, the CAN bus module of the drill rig chassis receives the operation data of the drill rig chassis in real time, such as the latitude and longitude of the drill rig, the heading angle, the speed, the running direction and the current operation mode, etc. The operation data of the drill rig chassis is transmitted to the back end through the TCP protocol. The back end analyzes, classifies and formats the received operation data of the drill rig chassis, and finally encapsulates it in JSON format to obtain structured real-time data; and pushes the structured real-time data to the front-end visualization platform through WebSocket connection, realizing the dynamic position and state visualization of the drill rig chassis.

[0040] Secondly, the drill arm controller is specifically responsible for acquiring real-time operational data of the drill arm and its components. Specifically, it monitors parameters such as the drill arm's movement angle, displacement changes, and execution mode, and transmits this data to the backend via a TCP-based communication connection. The backend parses and structures this operational data, merging it with real-time operational data from the CAN bus module of the drilling rig chassis acquired through a CAN gateway. Finally, it sends this data to the front-end visualization platform in a unified format (such as JSON). Upon receiving the structured real-time data, the front-end visualization platform dynamically updates the data via WebSocket, thereby precisely controlling the 3D model movements of the drilling rig and its components, and synchronously displaying the drilling rig's real-time operational status on the 3D visualization platform.

[0041] To achieve real-time perception and visualization of device operating status in a digital twin system, real-time data from the experimental platform vehicles needs to be acquired. The backend functionality can be divided into the following parts:

[0042] (1) Socket communication module.

[0043] The system's Socket communication module establishes TCPSocket connections with the CAN bus module of the drilling rig chassis and the drill arm controller, continuously receiving work data sent by both. The data acquisition of the CAN bus module is implemented in Python, establishing a TCP connection with the CAN gateway unit through a native socket library. It receives CAN frame data (work data from the drilling rig chassis) forwarded by the gateway in real time, encapsulates the raw CAN frame data into standard JSON format, and continuously pushes it to the backend system via Socket for further processing.

[0044] The drill arm controller is responsible for receiving real-time operation data of the drill arm and its components, mainly including parameters such as movement angle, displacement change and execution mode. The operation data of the drill arm and its components is pushed to the backend via TCP protocol.

[0045] (2) Data parsing module.

[0046] For the drilling rig chassis's operational data, different types of CAN frame data are parsed according to preset CAN ID mapping rules to extract key parameters such as latitude and longitude, heading angle, speed, attitude angle, and operating mode, obtaining multi-source sensor data. For the drilling arm and its components' operational data, parameters such as motion angle, displacement change, and execution mode are parsed and structured according to predefined data protocols. The parsed multi-source sensor data is encapsulated into standard JSON format data and pushed to the front-end visualization platform via WebSocket.

[0047] (3) WebSocket service module.

[0048] The backend uses the Spring Boot framework to implement WebSocket communication. The WebSocket service module is responsible for establishing a full-duplex communication channel with the frontend visualization platform and pushing the real-time parsed drilling rig operation data to the frontend visualization platform based on an event-driven mechanism. Through the WebSocket service module, the frontend visualization platform can receive the drilling rig's operation data in real time and dynamically update the drilling rig's 3D model, including the drill arm's posture and movement trajectory.

[0049] To achieve dynamic interaction between backend sensor data and the 3D model displayed on the frontend visualization platform, the frontend visualization platform is developed based on the Vue.js framework and is responsible for the construction and interactive control of the 3D scene. The 3D terrain model is loaded with 3D Tiles through Cesium to achieve high-precision spatial base rendering; the drilling rig 3D model is loaded with an optimized GLB format model through Three.js, supporting independent animation control of various components of the drilling rig (such as the drill arm and drill bit).

[0050] The front-end visualization platform receives drilling rig operation data in real time from the back-end in JSON format via a WebSocket connection. The data includes key fields such as the drilling rig's longitude, latitude, altitude, and heading angle. This location and direction information transmitted from the back-end is based on the internationally recognized WGS84 geographic coordinate system: latitude and longitude represent the angular coordinates of the geographical location, altitude represents the height relative to sea level, and the heading angle is the angle of the drilling rig's clockwise rotation relative to true north, ranging from 0° to 360°.

[0051] To accurately map the drilling rig's location information in real geographic space onto the 3D terrain model, the system first performs geographic coordinate transformation on the longitude, latitude, and elevation data provided by the backend. This transformation is completed by the front-end visualization platform calling the Cartesian3.fromDegrees interface provided by the Cesium engine, converting the geographic coordinates into points in a 3D Cartesian coordinate system, that is, the X, Y, Z spatial positions with the Earth's center as the origin.

[0052] However, due to the different coordinate axis definitions used by Cesium and Three.js, there are directional differences between them. For example, Cesium's Z-axis typically points vertically to the Earth's surface, while in Three.js, the Y-axis is usually used as the vertical axis. To establish a unified spatial reference frame across different rendering engines, the system further calls Cesium's Transforms.eastNorthUpToFixedFrame interface to construct a local coordinate transformation matrix based on the drilling rig's current position, using "east-north-up" as the directional reference.

[0053] The system then decomposes the transformation matrix to extract key parameters: position vector and rotation quaternion. These parameters are directly applied to the 3D drilling rig model loaded in Three.js, thereby achieving an accurate mapping from geographic spatial coordinates to rendering spatial coordinates at the visual level.

[0054] For the motion control of drilling rig components, such as the raising and lowering and rotation of the drill arm, the front-end visualization platform receives structured operation data pushed from the back-end, parses the drill arm's motion state from it, and uses it as animation driving parameters to control the rotation behavior of the drill arm nodes in the 3D model. The `getObjectByName` method provided by Three.js is used to obtain the drill arm model nodes, and their `rotation` property is modified to control rotation around the axis. For the raising and lowering of the drill arm, the `position` property of the drill arm model is modified to achieve translation. Through these steps, the system can achieve real-time perception, data transmission, and visualization of the drilling rig's operational status, improving the transparency and intelligent management level of equipment operation. Finally, the front-end visualization platform is developed based on the Vue.js framework, and Vite is used to complete the project compilation and packaging to generate the front-end page files.

[0055] like Figure 3 As shown, the page displays a dynamic effect of the experimental platform vehicle on an open lawn, serving as a demonstration platform for the system's functions. Users can directly access this page through a browser to view the dynamic display of the experimental platform vehicle in a virtual environment in real time. The page supports free mouse control of the viewing angle, enabling real-time observation and interaction of the vehicle's operating status. Although Figure 3 The scenario showcased is an experimental platform, but the system can be flexibly applied to similar industrial environments, such as monitoring and operating equipment like open-pit mine drilling rigs. The entire system runs in a standard browser, requiring no plugins or additional software, and is compatible with multiple devices and platforms, ensuring an intuitive, smooth, and easy-to-use digital twin effect.

[0056] In this embodiment of the application, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0057] In this application embodiment, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0058] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0059] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not invented herein. The specification and embodiments are to be considered exemplary only.

[0060] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A digital twin system for open-pit mine drilling rigs based on WebGL, characterized in that, include: The backend receives real-time operation data from the drilling rig chassis; It receives real-time operational data from the drill arm and its components, and obtains multi-source sensor data. The backend classifies and encapsulates multi-source sensor data to obtain structured real-time data. The backend is used to push structured real-time data streams to a front-end visualization platform built on Vue.js via WebSocket; The backend includes a Socket communication module, a data parsing module, and a WebSocket service module; The Socket communication module establishes a connection with the controller deployed on the drilling rig via the TCP protocol, continuously receiving operation data from the drilling rig chassis, drill arm, and its components. The drilling rig chassis operation data is collected by a Python program, which establishes a TCP connection with the CAN gateway unit through the native socket library, acquires CAN frame data forwarded by the gateway in real time, encapsulates the CAN frame data into standard JSON format, and pushes it to the backend in real time via the TCP protocol. The module also receives operation data from the drill arm and its components in real time and pushes it to the backend via the TCP protocol. The data parsing module is used to classify and extract the received drilling rig chassis operation data, drilling arm and component operation data according to a predefined data protocol, obtain key fields and encapsulate them into structured real-time data in JSON format that conforms to the Web transmission format. The WebSocket service module is used to establish a full-duplex communication channel with the front-end visualization platform using the WebSocket protocol, and to push structured real-time data to the front-end visualization platform based on an event-driven mechanism.

2. The web-based digital twin system for open-pit mine drilling rigs according to claim 1, characterized in that, Based on the Cesium engine, a front-end visualization platform is used to load the 3D terrain model, and the Three.js engine is used to load the 3D model of the drilling rig. The local coordinate system of the drilling rig 3D model is converted to the Cesium geographic coordinate system using a front-end visualization platform. The position, rotation angle and scaling ratio of the drilling rig 3D model are adjusted, and the drilling rig 3D model is superimposed on the 3D terrain model. Synchronize Three.js camera parameters with Cesium camera view using a front-end visualization platform.

3. The web-based digital twin system for open-pit mine drilling rigs according to claim 1, characterized in that, Utilize a front-end visualization platform to continuously acquire structured real-time data pushed from the back-end via WebSocket connection; After receiving structured real-time data using a front-end visualization platform, extract the corresponding parameters and bind them to the position and quaternion properties of the model in the Three.js engine. After parsing the structured real-time data using a front-end visualization platform, the data is bound to the corresponding drilling rig and its components, enabling the drilling rig and its components to respond to the control logic of the drill arm controller and execute corresponding actions.

4. The web-based digital twin system for open-pit mine drilling rigs according to claim 1, characterized in that, Utilizing a front-end visualization platform, structured real-time data pushed from the backend is continuously acquired via WebSocket connection, including: Based on the Vue.js framework, a visual interface is built using a front-end visualization platform. A full-duplex communication connection is established with the back-end via the WebSocket protocol to receive and parse the transmitted structured real-time data in real time.

5. A web-based digital twin system for open-pit mine drilling rigs according to claim 4, characterized in that, After parsing structured real-time data using a front-end visualization platform, the data is bound to the corresponding drilling rig and its components. This enables the drilling rig and its components to respond to the control logic of the drill arm controller and execute corresponding actions, including: Structured real-time data is used to drive the dynamic translation and rotation of the drilling rig's 3D model in 3D space, as well as the movement changes of the drill arm and its components.

6. The WEBGL-based digital twin system for open-pit mine drilling rigs according to claim 1, characterized in that, The operating data of the drilling rig chassis includes latitude and longitude, heading angle, speed, attitude angle, and operating mode.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5.

Images