Rotary drilling rig data monitoring method and system based on augmented reality

Abstract

The invention discloses a rotary drilling rig data monitoring method and system based on augmented reality, and the method comprises the steps: employing an EPEC controller and sensor data real-time collection and transmission, outputting real-time operation data and pre-constructed virtual model matching data to a HoloLens device and a remote device worn by an operator through a communication module, carrying out the assistance and marking of a remote expert, and carrying out the real-time monitoring of the data of the rotary drilling rig. And virtual-real fusion and display are realized. Real-time monitoring of the operation state of the rotary drilling rig is guaranteed, accurate recognition of the operation state of the rotary drilling rig component is remarkably improved by combining an attention mechanism and a feature fusion structure, the operation efficiency is improved and the error rate is reduced through accurate superposition of virtuality and reality, and meanwhile operation safety and accuracy are improved. On the whole, the operation reliability of the rotary drilling rig is improved, the operation efficiency is also improved, and the operation management of the rotary drilling rig is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data monitoring of rotary drilling rigs, and particularly relates to a data monitoring method and system for rotary drilling rigs based on augmented reality. Background Art

[0002] Drilling rigs play a crucial role in fields such as mining, construction, and geological exploration. However, in traditional two-dimensional observation interfaces, data association is often prone to confusion and the interactivity is poor, resulting in low data processing efficiency. In addition, the operation status and maintenance management of drilling rigs usually rely on manual judgment, which makes the early warning and maintenance of equipment failures more dependent on experienced technicians. For newbies, it is difficult to understand and process complex equipment information. In actual operation, due to long-term high-intensity work, maintenance personnel are prone to fatigue, leading to missed inspections or misinspections, increasing the risk of safety accidents. Summary of the Invention

[0003] Aiming at the above problems, the purpose of the present invention is to provide a data monitoring method and system for rotary drilling rigs based on augmented reality, which can realize the recognition of operation data and real-time matching with a virtual model, enabling operators to see the intuitive superposition of real-time operation data and the virtual model on a real rotary drilling rig, and further improving the operation efficiency and reducing the error rate through remote expert assistance.

[0004] The technical solution adopted by the present invention to solve the above technical problems is as follows:

[0005] The data monitoring method for rotary drilling rigs based on augmented reality of the present invention includes the following specific steps:

[0006] S1 Real-time data collection and transmission: The operation data of the rotary drilling rig is collected and transmitted in real time through a data collection and transmission module, providing accurate dynamic data support for the subsequent augmented reality environment.

[0007] S2 Real-time matching of data with a 3D virtual model: The data collection and transmission module (specifically, the EPEC controller of the data collection and transmission module) identifies the operation status of each component of the rotary drilling rig, realizes the matching of real-time operation data with a pre-constructed virtual model, and outputs the matching data of the real-time operation data and the pre-constructed virtual model to the HoloLens device (optical see-through head-mounted display) worn by the operator and the remote device through a communication module.

[0008] S3 Remote expert assistance and annotation: The remote expert communicates with the operator through a communication module. The remote expert views the superposition result of the real-time operation data and the virtual model, and interacts with the operator through video, voice, data feedback, and / or remote annotation methods to guide on-site operation, thereby improving efficiency and reducing the error rate.

[0009] S4 Virtual-Reality Fusion and Display: The operator uses the HoloLens device to determine the world coordinate position of the components of the real rotary drilling rig in the HoloLens device, and superimposes the real-time operation data, virtual models, and data feedback from remote experts and / or remote annotation data at the position of the components to achieve virtual-reality fusion.

[0010] Furthermore, it also includes step S5, specifically as follows: When the operator needs to interact with the remote expert again, return to step S3.

[0011] Furthermore, in the above-mentioned step S1, the data collection and transmission module is an EPEC controller integrated sensor, which collects the operation data of the rotary drilling rig in the real application scenario, and uses the 5G network and Modbus technology to transmit the operation data to the communication module in real time.

[0012] Furthermore, in the above-mentioned step S2, the communication module realizes real-time remote communication through the WebRTC technology, ensuring the efficient interconnection between the EPEC controller and the remote device, and supporting the transmission of low-latency and high-quality video and data streams.

[0013] Furthermore, in the above-mentioned step S2, the improved YOLOv7-tiny image recognition algorithm is used to recognize the operation status of each component of the rotary drilling rig.

[0014] Furthermore, the improved YOLOv7-tiny image recognition algorithm is as follows: An attention mechanism is added to the original backbone framework, the original ELAN module is replaced with an ELAN-ECA module (or ELAN-E module, E-ELAN module), a Bifpn structure is used for feature fusion in the feature fusion area, and an ELAN-ECA module is also used to strengthen the extraction of features during the feature fusion process. Finally, a cross-stage local network under the CSPAECA structure is used for feature output detection at the output layer.

[0015] Furthermore, in the above-mentioned step S3, the operator wears the HoloLens device to communicate with the remote expert using the remote device.

[0016] Furthermore, in the above-mentioned step S3, the operator uses the camera integrated on the HoloLens device to capture real-time images and video streams in the real scene, and interacts with the remote expert through gesture interaction and voice interaction.

[0017] Furthermore, in the above-mentioned step S4, after the operator uses the camera of the HoloLens device to obtain the component image of the real rotary drilling rig, the HoloLens device uses SLAM and anchor point technology to determine the world coordinate position of the component in the HoloLens device.

[0018] The data monitoring system for a rotary drilling rig based on augmented reality of the present invention includes a data collection and transmission module, a communication module, a HoloLens device, and a remote device; the data collection and transmission module collects the operation data of the rotary drilling rig in real time, identifies the operation states of each component of the rotary drilling rig, realizes the matching of the real-time operation data with a pre-constructed virtual model, and outputs the matching data of the real-time operation data and the pre-constructed virtual model to the HoloLens device worn by the operator and the remote device through the communication module. The HoloLens device communicates with the remote device through the communication module, specifically: the operator uses the camera integrated on the HoloLens device to capture real-time images and video streams in the real scene, and interacts with remote experts through gesture interaction and voice interaction; the remote expert views the superposition result of the real-time operation data and the virtual model, and interacts with the operator through video, voice, data feedback, and / or remote annotation methods to guide on-site operations; the operator uses the HoloLens device to determine the world coordinate position of the components of the real rotary drilling rig in the HoloLens device, and superimposes the real-time operation data, the virtual model, and the data feedback and / or remote annotation data of the remote expert at the position where the components are located.

[0019] The beneficial effects of the present invention are as follows:

[0020] The present invention uses an EPEC controller and sensors to collect data in real time, and realizes efficient transmission through 5G network and Modbus technology, ensuring real-time monitoring of the operation state of the rotary drilling rig. By using an improved YOLOv7-tiny image recognition algorithm and combining an attention mechanism and a feature fusion structure, the accurate recognition of the operation states of the components of the rotary drilling rig is significantly improved. The operator wears a HoloLens device and can interact with remote experts in real time. Through the precise superposition of virtual and real, the operation efficiency is improved and the error rate is reduced. At the same time, the application of SLAM and anchor point technology enables the virtual model to be accurately superimposed on the positions of the components of the real rotary drilling rig, enhancing the safety and accuracy of the operation. Generally speaking, the present invention not only improves the operation reliability of the rotary drilling rig, but also improves the operation efficiency and optimizes the operation management of the rotary drilling rig. Description of the Drawings

[0021] Figure 1 It is a flowchart of the data monitoring method for a rotary drilling rig based on augmented reality of the present invention.

[0022] Figure 2 It is a structural block diagram of the data monitoring system for a rotary drilling rig based on augmented reality of the present invention. Detailed Embodiments

[0023] The present invention will be further described in detail below with reference to the drawings and embodiments.

[0024] Example 1:

[0025] As Figure 1 shown, the data monitoring method for a rotary drilling rig based on augmented reality includes the following specific steps:

[0026] S1 Real-time data collection and transmission: The operation data of the rotary drilling rig is collected and transmitted in real time through the data collection and transmission module, providing accurate dynamic data support for the subsequent augmented reality environment. Among them: The data collection and transmission module is an EPEC controller integrated with sensors, which collects the operation data of the rotary drilling rig in the real application scenario, and uses 5G network and modbus technology to transmit the operation data to the communication module in real time. The communication module realizes real-time remote communication through WebRTC technology, ensuring the efficient interconnection between the EPEC controller and the remote device, and supporting the transmission of low-latency and high-quality video and data streams. The operation data includes: current drilling depth, working hoist speed, pressurized hoist speed, mud or clean water flow rate, bus voltage, motor current, power head motor speed, motor temperature, hydraulic oil temperature, electricity consumption per single pile, total electricity consumption, main pump pressure, pilot oil pressure, mast inclination and alarm data; The alarm data includes controller communication failure, remote controller communication failure, remote controller handle signal failure, handle signal failure, frequency converter communication failure, rotary encoder failure, wireless communication module communication failure and background controller communication failure; The bus voltage, motor current, electricity consumption per single pile and total electricity consumption can be calculated by detecting the signals of each channel of the EPEC controller, or can be detected by using a voltmeter, ammeter and multimeter. The alarm data is detected through the signals of each channel of the EPEC controller, and the rest of the operation data can be detected by the integrated sensors.

[0027] The data collection and transmission module is specifically: The EPEC controller integrates multiple sensors for distance (detecting drilling depth), speed, temperature, pressure, vibration, flow rate, speed, and inclination, and ensures high-precision data collection through the standard serial port interface RS-485. The data is encoded through the modbus protocol. The modbus protocol is based on the master-slave architecture. The EPEC controller, as the master device, requests the data of each sensor regularly or on demand, and the sensor, as the slave device, provides the measured values. The EPEC controller will convert these physical data into a standardized format internally, transmit the real-time data through modbus TCP / IP, and send it to the communication module through the 5G network.

[0028] The communication module is specifically as follows: A signaling server based on Node.js is established through WebRTC and sent to the remote device for the remote expert. The remote expert receives the offer, sets the remote description, responds with an Answer, and sends it back to the operator through the signaling server. Subsequently, both ends cooperate with the STUN / TURN server to collect and exchange ICE candidates, and these candidate information includes the network connection method. After completing the ICE candidate confirmation, a peer-to-peer connection is established between the operator and the remote expert, allowing the media stream to be directly transmitted between the operator and the remote expert, realizing real-time audio and video communication between the remote expert and the operator. Data communication such as text and pictures is realized through the Data Channel function of WebRTC.

[0029] S2 Real-time matching of data and 3D virtual model: The data collection and transmission module identifies the operating states of each component of the rotary drilling rig, realizes the matching of real-time operating data with the pre-constructed virtual model, and outputs the real-time operating data and the pre-constructed virtual model matching data to the HoloLens device worn by the operator and the remote device through the communication module.

[0030] Among them, the improved YOLOv7-tiny image recognition algorithm is used to identify the operating states of each component of the rotary drilling rig. An attention mechanism is added to the original backbone framework, the original ELAN module is replaced with the ELAN-ECA module, the Bifpn structure is adopted for feature fusion in the feature fusion area, and the ELAN-ECA module is also used for enhanced feature extraction during the feature fusion process. Finally, the cross-stage local network under the CSPAECA structure is adopted in the output layer for feature output detection.

[0031] Specifically: In the data collection and transmission module, the real-time operating data is input into the improved YOLOv7-tiny model. The introduction of the attention mechanism and the ELAN-ECA module in this model can effectively improve the feature extraction accuracy of each component of the rotary drilling rig. Then the model uses the Bifpn structure to fuse the extracted features. The Bifpn structure allows bidirectional fusion of features at multiple scales, ensuring the reasonable integration of multi-scale features and enhancing the recognition ability for components of different sizes. The information after feature fusion is further optimized through the ELAN-ECA module, improving the recognition effect of components under small targets or complex backgrounds. Finally, the output layer combines the cross-stage local network through the CSPAECA structure for object detection and status judgment, matches the operating data of each component of the identified rotary drilling rig with the pre-constructed virtual model, and feeds it back to the HoloLens device and the remote device in real time, so as to provide the operator with the real-time device operating status and accurate component information.

[0032] S3 Remote Expert Assistance and Annotation: The remote expert communicates with the operator through the communication module. The remote expert views the superposition result of the real-time operation data and the virtual model, and interacts with the operator through video, voice, data feedback, and / or remote annotation methods to guide on-site operations, thereby improving efficiency and reducing error rates.

[0033] Specifically: First, after the communication module establishes stable communication through WebRTC, the operator uses the HoloLens device to superimpose and display the real-time operation data and the pre-constructed virtual model matching data, providing intuitive operation feedback. In the Unity 3D development platform, through the fusion of the virtual model and the on-site real-time operation data, the remote expert can clearly see the current operation status of the working components and discover potential problems. The remote expert can communicate with the operator in real time through video and voice to provide operation guidance or technical support. In the Unity environment, the remote expert first selects 2D coordinates and shapes on the Canvas for annotation, and these annotation information are then transmitted to the HoloLens device in real time through the Data Channel function of WebRTC. After receiving these 2D coordinates, the HoloLens device calculates the camera position C1 at the annotation moment by parsing the data and the camera space matrix data at the annotation time point, and calculates the position C2 of the projection plane annotation point according to the length-width ratio on the Canvas. After completing the positioning of position C2, a ray is emitted from the camera position C1 to C2, and the collision detection function of the HoloLens device is used to search for the intersection point (position C3) of the ray and the virtual model or the actual component. Once C3 is determined, the HoloLens device will perform annotation registration at this position. By performing real-time annotation on the virtual model (such as in the form of arrows, box selections, or text annotations), the remote expert can accurately guide the on-site personnel to operate, and at the same time provide improvement suggestions through real-time feedback data of remote annotation.

[0034] S4 Virtual-Reality Fusion and Display: The operator uses the HoloLens device to determine the world coordinate position of the components of the real rotary drilling rig in the HoloLens device, and superimposes the real-time operation data, the virtual model, and the data feedback and / or remote annotation data of the remote expert at the position where the components are located to achieve virtual-reality fusion.

[0035] Specifically: First, the operator uses the camera of the HoloLens device to obtain the component images of the real rotary drilling rig. Then, using the SLAM technology of the HoloLens device, by scanning the surrounding environment in real time, a three-dimensional map of the environment is constructed, and the position and pose of the HoloLens device are determined simultaneously. Combining with the spatial anchor technology of the HoloLens device, the components are identified through object detection and the specific 2D coordinates of the components in the component images are determined. The 2D coordinates are converted into the camera coordinates of the HoloLens device, and then the camera coordinates are converted into the world position coordinates of the HoloLens device. The collision detection function is used to search for the virtual anchors generated by the rays on the components of the real rotary drilling rig, so that the virtual model can be accurately aligned with the components of the real rotary drilling rig. After the operator wears the HoloLens device, the virtual model will be displayed in their field of vision through the HoloLens device and synchronously displayed with the components of the real rotary drilling rig. Using the SLAM and anchor technologies, the virtual model will update its position in real time, maintain an accurate docking with the real rotary drilling rig, and ensure that the virtual model can be correctly superimposed on the real rotary drilling rig at any angle. The operator can directly interact with the virtual model during the operation, such as rotating, zooming in, zooming out, or viewing the specific operating status of the rotary drilling rig.

[0036] Embodiment 2:

[0037] As Figure 2 shown, the rotary drilling rig data monitoring system based on augmented reality includes a data collection and transmission module, a communication module, a HoloLens device, and a remote device; the data collection and transmission module collects the operation data of the rotary drilling rig in real time, identifies the operation status of the rotary drilling rig and its various components, realizes the matching of the real-time operation data with the pre-constructed virtual model, and outputs the real-time operation data to the HoloLens device worn by the operator through the communication module, and outputs the matching data of the real-time operation data and the pre-constructed virtual model to the remote device through the communication module. The HoloLens device communicates with the remote device through the communication module. Specifically: The operator uses the integrated camera on the HoloLens device to capture the real-time images and video streams in the real scene, and interacts with the remote experts through gesture interaction and voice interaction; the remote experts view the superimposed results of the real-time operation data and the virtual model, and interact with the operator through video, voice, data feedback, and / or remote annotation methods to guide the on-site operation; the operator uses the HoloLens device to determine the world coordinate position of the components of the real rotary drilling rig in the HoloLens device, and superimposes the real-time operation data, the virtual model, and the data feedback and / or remote annotation data of the remote experts at the position where the components are located.

Claims

1. A rotary drilling rig data monitoring method based on augmented reality, characterized in that: The following steps are involved: S1 collects and transmits the operating data of the rotary drilling rig in real time through the data collection and transmission module; The S2 data collection and transmission module identifies the operating status of each component of the rotary drilling rig, matches the real-time operating data with the pre-built virtual model, and outputs the real-time operating data and the pre-built virtual model matching data to the HoloLens device worn by the operator and the remote device through the communication module; S3 remote experts communicate with operators via the communication module. Remote experts view the superposition results of real-time operation data and virtual models, and interact with operators through video, voice, data feedback and / or remote annotation to guide on-site operations; The S4 operator uses the HoloLens device to determine the world coordinate position of the components of the real rotary drilling rig on the HoloLens device, and superimposes the real-time operation data, virtual model, and data feedback and / or remote annotation data of remote experts on the location of the components to achieve virtual-real integration.

2. The method for monitoring rotary drilling rig data based on augmented reality according to claim 1, characterized in that: The method further includes step S5: when the operator needs to interact with the remote expert again, the method returns to step S3.

3. The method for monitoring rotary drilling rig data based on augmented reality according to claim 1 or 2, characterized in that: In step S1, the data collection and transmission module is an integrated sensor of the EPEC controller, which collects the operating data of the rotary drilling rig in the real application scenario, and uses the 5G network and modbus technology to transmit the operating data to the communication module in real time.

4. The method for monitoring rotary drilling rig data based on augmented reality according to claim 1 or 2, characterized in that: In step S2, the communication module implements real-time remote communication through WebRTC technology.

5. The method for monitoring rotary drilling rig data based on augmented reality according to claim 1 or 2, characterized in that: In step S2, the operating status of each component of the rotary drilling rig is identified by using an improved YOLOv7-tiny image recognition algorithm.

6. The method for monitoring rotary drilling rig data based on augmented reality according to claim 5, characterized in that: The improved YOLOv7-tiny image recognition algorithm is as follows: an attention mechanism is added to the original backbone network framework, the original ELAN module is replaced by an ELAN-ECA module, a Bifpn structure is used for feature fusion in the feature fusion area, and an ELAN-ECA module is also used for enhanced feature extraction during the feature fusion process; finally, a cross-stage local network under a CSPAECA structure is used in the output layer for feature output detection.

7. The method for monitoring rotary drilling rig data based on augmented reality according to claim 1 or 2, characterized in that: In step S3, the operator wears the HoloLens device to communicate with the remote expert using the remote device.

8. The method for monitoring rotary drilling rig data based on augmented reality according to claim 7, characterized in that: In step S3, the operator uses the camera integrated in the HoloLens device to capture real-time images and video streams in the real scene, and interacts with the remote expert through gesture interaction and voice interaction.

9. The method for monitoring rotary drilling rig data based on augmented reality according to claim 7, characterized in that: In step S4, the operator uses the camera of the HoloLens device to obtain the component image of the real rotary drilling rig, and then uses the SLAM and anchor point technology through the HoloLens device to determine the world coordinate position of the component on the HoloLens device.

10. A rotary drilling rig data monitoring system based on augmented reality, comprising a data collection and transmission module, a communication module and a remote device, characterized in that: It also includes a HoloLens device; the data collection and transmission module collects the operating data of the rotary drilling rig in real time, identifies the operating status of each component of the rotary drilling rig, matches the real-time operating data with a pre-built virtual model, and outputs the real-time operating data and the pre-built virtual model matching data to the HoloLens device and the remote device via the communication module; the HoloLens device communicates with the remote device via the communication module, specifically: using the camera integrated on the HoloLens device to capture real-time images and video streams in real scenes, and interacting with the remote device through gesture interaction and voice interaction; viewing the superposition result of the real-time operating data and the virtual model through the remote device, and interacting with the HoloLens device through video, voice, data feedback and / or remote annotation to guide on-site operations; using the HoloLens device to determine the world coordinate position of the components of the real rotary drilling rig on the HoloLens device, and superimposing the real-time operating data, virtual model, and data feedback and / or remote annotation data of the remote device at the location of the component.

Images