Virtual reality-based remote unmanned operation system and method for coal mining face

Abstract

The application provides a virtual reality-based remote unmanned operation system and method for a coal mining face, and relates to the technical field of mining engineering. The system comprises a real-time coal mining face unit, a working control room unit, a robot field operation unit, a cloud data processing unit, a virtual working unit, a precise positioning unit, a data acquisition unit, a data transmission unit, a VR virtual imaging unit, a registration and login unit, a working protection unit, a motion capture unit, an instruction transmission unit, an operation feedback unit, an information acquisition unit, a monitoring and alarm unit, and real-time image, sound, stress and air information of a coal mining face construction site is uploaded to the cloud data processing unit. The virtual working unit uses virtual reality VR technology to build a platform, the robot field operation unit synchronizes user actions, the virtual working unit monitors the operation of the equipment and performs remote control. The system realizes remote coal mining and improves the coal mining working environment.

Description

Technical Field

[0001] This invention relates to the field of mining engineering technology, and in particular to a remote unmanned operating system and method for coal mining faces based on virtual reality. Background Technology

[0002] With the increasing demand for energy resources, a significant amount of human and material resources are required. However, the safety and working environment of underground coal mines limit their development due to human resource constraints. The working environment in coal mines, characterized by high temperatures, high humidity, low oxygen levels, dust, toxic and harmful gases, and water spray, significantly impacts the safety and health of operators. With the continuous development of artificial intelligence, the construction of digital and intelligent mines has become a key focus for coal mining enterprises. Existing coal mine production systems and methods suffer from limitations such as restricted work locations, inability to achieve remote operation, numerous on-site safety hazards, and operators' inability to accurately perceive the on-site production situation. Summary of the Invention

[0003] To address the limitations of working locations and conditions in coal mining, improve the intelligence, digitalization, and unmanned operation of coal mines, and provide a better mining environment, this invention provides a remote unmanned operating system and method for coal mining faces based on virtual reality. The specific technical solution is as follows.

[0004] A virtual reality-based remote unmanned operating system for coal mining faces includes a real-time coal mining face unit, a work control room unit, a robot field operation unit, a cloud data processing unit, a virtual work unit, a precise positioning unit, a data acquisition unit, a data transmission unit, a VR virtual imaging unit, a registration and login unit, a work protection unit, a motion capture unit, an instruction transmission unit, an operation feedback unit, an information acquisition unit, and a monitoring and alarm unit. The coal mining face unit includes construction equipment and on-site data collection equipment, which performs coal mining operations and collects image, sound, stress, and air data from the construction site, uploading the data to the cloud data processing unit. The work control room unit includes VR glasses and a virtual studio, establishing a data connection with the coal mining face unit. The robot field operation unit includes a data receiving station and a virtual digital robot, which maintains synchronization with the user's actions. The cloud data processing unit performs data calculation, organization, and classification. The virtual work unit utilizes virtual reality (VR)... The system comprises a central control platform, an information communication processing platform, a mine accident handling platform, and a virtual working face platform. The precise positioning unit acquires personnel and equipment information within the target area. The data acquisition unit uses real-time 3D map technology for positioning and panoramic acquisition equipment for data collection at the working face. The data transmission unit transmits data via wireless broadband. The VR virtual imaging unit imports the data into the 3D mine modeling software Surpac. The motion capture unit uses infrared cameras and infrared sensors for positioning and motion capture. The work protection unit controls personnel access to the virtual workspace. The registration and login unit connects the user and the virtual working face. The instruction transmission unit transmits the information collected by the motion capture unit to the virtual digital robot. The operation feedback unit synchronizes user operation instructions and the virtual digital robot's actions. The information acquisition unit acquires data information from the work site, and the monitoring and alarm unit monitors and issues warnings regarding the working face construction status, equipment operation status, and user operation status.

[0005] Preferably, the construction equipment includes a coal mining machine, hydraulic supports, scraper conveyors, transfer conveyors, crushers, and belt conveyors, and the on-site data information collection equipment includes a panoramic acquisition device, sensors, signal transmitters, and virtual digital robots.

[0006] Preferably, the virtual workspace is equipped with VR glasses, a haptic bodysuit, an omnidirectional treadmill, force feedback haptic gloves, and a host computer. The host computer receives data from the panoramic acquisition device, the visual information of the virtual digital robot, and the motion commands captured by the sensors.

[0007] Preferably, the data receiving station receives the instruction signal issued by the user and sends the signal to the virtual digital robot. The real-time images captured by the virtual digital robot at the working face, the received sound, and the real-time data on the working status of the construction equipment are sent to the cloud data processing unit.

[0008] Preferably, the cloud data processing unit includes a server, a storage device, and a network connection device. The cloud data processing unit connects the coal mining face unit, the robot field operation unit, the work control room unit, and the virtual work unit. The cloud data processing unit constructs a virtualization layer and performs resource slicing. The cloud data processing unit constructs a cloud resource layer and a cloud service layer. The cloud resource layer integrates and pools the cloud operating system. The cloud service layer provides cloud products and interfaces with the cloud resource layer.

[0009] Preferably, the central control platform of the virtual working unit sends task instructions and monitors the working face, the information communication processing platform monitors communication data transmission, the mine accident handling platform provides accident solutions, and the virtual working face platform collects real-time data and builds models, synchronizing the coal mining face and virtual working face data.

[0010] Preferably, the precise positioning unit calculates and measures the coal seam's mining height and area information, captures and replays the movement trajectories of personnel and construction equipment, and counts the number of personnel and equipment within the target area; the data acquisition unit collects the geometric dimensions of the working face and the outline information of the construction equipment, and also calls up geographic information systems and geological exploration information.

[0011] Preferably, the data transmission unit uses wireless broadband to transmit data, and the data transmission unit transmits the data from the panoramic acquisition device, the user operation commands captured by the sensors, and the information collected by the virtual digital robot to the cloud data processing unit; the working protection unit includes a face information recognition device.

[0012] Preferably, the registration and login unit is developed based on a web front-end, using HTML to build a framework, cascading style sheets to decorate the webpage, and JavaScript to set up pop-up dialog boxes; the VR virtual imaging unit constructs a 3D model in a virtual reality text format supported by VRML using Surpac software, imports the model into VRML software, and uses Hypertext Markup Language to program and realize virtual scene interaction in the webpage; the virtual digital robot and the construction equipment are wirelessly connected and exchange data, and the data of the construction equipment includes motor power, dimensions, voltage, vibration rate, traction force, and speed.

[0013] A remote unmanned operation method for coal mining faces based on virtual reality is disclosed. Utilizing the aforementioned remote unmanned operating system for coal mining faces based on virtual reality, the coal mining construction process includes coal cutting and loading by the coal mining machine, coal transportation by the scraper conveyor, crushing of large coal pieces by the crusher, coal transportation by the transfer conveyor, and coal transportation by the belt conveyor. The sequence of hydraulic support movement is: retracting side guard plates, retracting side protection plates, lowering the support, pulling the support, raising the support, extending the side protection plates, and extending the side guard plates. The virtual digital robot synchronizes with the user's actions and shares data information.

[0014] The beneficial effects of the virtual reality-based remote unmanned operating system and method for coal mining faces provided by this invention include:

[0015] (1) In this system, the virtual working face can be used to operate or check the operation status of each production equipment to ensure the normal operation of the equipment. The virtual digital robot performs the corresponding operation, realizing the goal of remote control, avoiding manual operation and the operation location is not restricted, improving the working environment of coal mine workers, and greatly enhancing the interactivity and immersion of employees at work.

[0016] (2) The data information collected by the virtual digital robot is based on the user's senses. It can obtain data information from the coal mining face from the perspectives of seeing, hearing and smelling. With the help of force feedback tactile gloves, it can establish a connection with the target equipment to realize the exchange of equipment information. The virtual digital robot can also receive various sounds from the working face and obtain the operating status of each piece of equipment. At the same time, it can work synchronously according to the user's instructions and can operate various buttons to ensure the normal operation of production.

[0017] (3) The method of mining using this system can ensure the equipment control of the working face, and complete the production task efficiently and safely. As the panoramic acquisition equipment moves, it automatically and accurately records the geographical location information of the working face, generates a 360-degree panoramic image, and clearly displays some small details such as equipment status and specific equipment layout on the map. It can create a real all-round stereoscopic map to help staff understand the information of the working face. Attached Figure Description

[0018] Figure 1 This is a working block diagram of a remote unmanned operating system for coal mining faces based on virtual reality.

[0019] Figure 2 This is a schematic diagram of the equipment layout at the coal mining face;

[0020] Figure 3 This is a schematic diagram of the layout of the work control room unit;

[0021] Figure 4 This is a flowchart of the cloud data processing unit's workflow;

[0022] Figure 5 This is a schematic diagram of the structure of VR virtual reality glasses;

[0023] Figure 6 This is a schematic diagram of the structure of an infrared camera and an infrared sensor;

[0024] Figure 7 This is a schematic diagram of the force feedback haptic glove structure;

[0025] Figure 8 This is a schematic diagram of the structure of the tactile bodysuit;

[0026] Figure 9 This is a schematic diagram of the structure of an all-around treadmill;

[0027] Figure 10 This is a schematic diagram of the structure of a virtual digital robot;

[0028] Figure 11 This is a schematic diagram of the panoramic acquisition device.

[0029] Figure 12 This is a schematic diagram of the oblique cutting feed of the end of the coal mining machine for cutting triangular coal.

[0030] In the diagram: 11-track level roadway, 12-end hydraulic support, 13-scraper conveyor, 14-coal mining machine, 15-transfer conveyor, 16-transport level roadway, 17-belt conveyor, 18-goaf, 19-hydraulic support;

[0031] 21- All-around treadmill; 22- Infrared camera and infrared sensor; 23- Screen.

[0032] 31-Plastic housing, 32-Glasses switch, 33-Lens, 34-Vision conversion switch, 35-Skeleton sensor, 36-Connector frame, 37-Adjustable strap;

[0033] 41-Infrared lens, 42-Infrared sensor, 43-Plastic housing, 44-Connector, 45-Base;

[0034] 51-Glove body, 52-IMU sub-sensor, 53-Connecting cable, 54-IMU main sensor and power supply assembly;

[0035] 61-Main body of clothing, 62-Bioelectric signal circuit, 63-Information receiver, 64-Waist belt, 65-Processor and battery;

[0036] 71-Support arm, 72-Main bracket, 73-Base, 74-Motion platform, 75-Protective vest, 76-Metal connector;

[0037] 81-High-precision camera, 82-Odor sensor, 83-Temperature sensor, 84-Command receiver, 85-Information transmitter, 86-Processor, 87-Force sensor, 88-Auxiliary scanner, 89-Positioning sensor;

[0038] 91-Fiber optic sensor module, 92-Laser rangefinder, 93-Infrared sensor, 94-“AI Eagle Eye” panoramic camera. Detailed Implementation

[0039] Combination Figures 1 to 12 The following describes a specific implementation of a virtual reality-based remote unmanned operating system and method for coal mining faces provided by the present invention.

[0040] A virtual reality-based remote unmanned operating system for coal mining faces includes a real-time coal mining face unit, a work control room unit, a robot on-site operation unit, a cloud data processing unit, a virtual work unit, a precise positioning unit, a data acquisition unit, a data transmission unit, a VR virtual imaging unit, a registration and login unit, a work protection unit, a motion capture unit, a command transmission unit, an operation feedback unit, an information acquisition unit, and a monitoring and alarm unit. In this system, panoramic acquisition devices and various sensors collect real-time data from the actual coal mining face. This data is transmitted in real-time to the cloud data processing unit using 6G information transmission technology. The cloud data processing unit calculates, organizes, and classifies the information. A digital model of the coal mining face is constructed using the mining modeling software Surpac. This model is then imported into VRML software, where HTML (Hypertext Markup Language) programming is used to implement virtual scene interaction on a webpage. The user, wearing VR glasses, force feedback haptic gloves, and a haptic bodysuit, becomes immersed in the virtual work face. Simultaneously, a virtual digital robot, operated underground by a robot on-site operation unit, begins operation, synchronizing its movements with the user's.

[0041] The system's various units enable remote control of the operation of equipment at the coal mining face and complete corresponding tasks according to instructions. The coal mining face unit includes construction equipment and on-site data collection equipment, which performs coal mining operations and collects image, sound, stress, and air data from the construction site, uploading the data to the cloud data processing unit. The work control room unit includes VR glasses and a virtual studio, establishing a data connection with the coal mining face unit. The robot field operation unit includes a data receiving station and a virtual digital robot, which maintains synchronization with the operator's movements. The cloud data processing unit performs data calculation, organization, and classification. The virtual work unit utilizes virtual reality (VR) to construct a central control platform, an information communication processing platform, a mine accident handling platform, and a virtual work face platform. The precision positioning unit acquires personnel and equipment information within the target area; the data acquisition unit uses 3D real-time map technology for positioning and panoramic acquisition equipment to collect data at the work face. The data transmission unit transmits data information wirelessly; the VR virtual imaging unit imports the data information into the 3D mine modeling software Surpac. The motion capture unit uses infrared cameras and infrared sensors for positioning and motion capture. The work protection unit controls personnel access to the virtual studio; the registration and login unit connects the user and the virtual work unit. The command transmission unit transmits information collected by the motion capture unit to the virtual digital robot. The operation feedback unit synchronizes user operation commands with the virtual digital robot's actions. The information acquisition unit acquires data from the work site, and the monitoring and alarm unit monitors and issues warnings regarding the construction status of the work surface, the operating status of equipment, and the user's operation status.

[0042] Coal mining face unit such as Figure 2 As shown, its construction equipment includes a coal mining machine, hydraulic supports, scraper conveyors, transfer conveyors, crushers, and belt conveyors. The on-site data collection equipment includes a panoramic acquisition device, sensors, signal transmitters, and virtual digital robots, which are used to collect real-time images, collect sound, collect the stress status of equipment during construction, monitor air composition, and send the relevant data to the cloud data processing unit to provide a data foundation for building a virtual coal mining face.

[0043] The enclosed space of the virtual work unit is equipped with VR glasses, a haptic bodysuit, an omnidirectional treadmill, force feedback haptic gloves, and a host computer. The host computer receives data from the panoramic acquisition device, visual information from the virtual digital robot, and motion commands captured by sensors. The VR glasses immerse the user in the virtual workstation, receiving and displaying images of the real coal mining face acquired by the virtual digital robot. The haptic bodysuit simulates human force sensing, providing feedback on the robot's vibrations, touches, and various force sensations during on-site operation. The force feedback haptic gloves grasp and control equipment in the virtual scene, acquiring relevant data and enabling corresponding commands based on the user's wishes. The omnidirectional treadmill restricts the user's spatial movement, ensuring safety. A speaker receives sound. The office also includes Surpac and VRML software for building virtual workstations, auxiliary equipment and tools, a host computer for transmitting and receiving data, and video screens for monitoring users. The host unit has three functions: First, it receives data from the coal mining face sent by the panoramic acquisition device and uses Surpac and VRML software to build a virtual workstation. Second, it receives sensory information from the virtual digital robot. The user's VR glasses, haptic bodysuit, force feedback haptic gloves, and speakers are connected to the host unit via Wi-Fi, synchronizing the user's sensory information with that of the virtual digital robot. Third, it receives user action commands captured by sensors and sends them to the virtual digital robot, synchronizing the user's actions with those of the virtual digital robot. The data collected by the panoramic acquisition device is mainly used to construct the virtual coal mining face. The collected data focuses on building a framework model, requiring high precision in the outlines and dimensions of equipment, coal seams, and personnel, while less emphasis is placed on detailed information such as coal quality, coal wall roughness, and equipment color. The data collected by the virtual digital robot, however, is based on the user's senses, acquiring data from the coal mining face through sight, hearing, and smell. Furthermore, it can establish a connection with the target equipment using force feedback haptic gloves, enabling the exchange of equipment information.

[0044] The data receiving station receives command signals from the user and sends them to the virtual digital robot, which can synchronize its actions with the user. Real-time images captured by the virtual digital robot at the work site, received audio data, and real-time data on the operating status of construction equipment are sent to the cloud data processing unit. The virtual digital robot and its operator capture image data through sensors and video data through cameras. Sensors also receive audio data. The virtual digital robot can also control equipment buttons according to instructions, ensuring safe production at the work site.

[0045] The cloud data processing unit plays a central role in the system, connecting the coal mining face unit, the robot field operation unit, the work control room unit, and the virtual space station. It calculates, organizes, and classifies the data received by each unit, quickly filtering out the data information needed by each platform, and simultaneously sending it to the corresponding platform in conjunction with the data transmission system. The cloud data processing unit includes a server, storage, and network connection devices. It connects the coal mining face unit, the robot field operation unit, the work control room unit, and the virtual work unit. The cloud data processing unit constructs a virtualization layer and performs resource slicing. It also constructs a cloud resource layer and a cloud service layer. The cloud resource layer integrates and pools the cloud operating system, while the cloud service layer provides cloud products and interfaces with the cloud resource layer. The cloud data processing unit architecture is divided into three layers. The first is the hardware infrastructure layer, which includes various hardware facilities such as servers, networks, and storage. The second is the virtualization layer, built on top of the hardware infrastructure layer. Virtualization can extract computing, storage, and networking functions to create resource slices. Resource slices refer to the process of allowing multiple virtual machines to share resources, allocating limited memory to other virtual machines, and integrating these resources. The virtualization layer and the hardware infrastructure layer are interdependent. The third is the cloud layer, which is further divided into the cloud resource layer and the cloud service layer. The cloud resource layer integrates all resources through the cloud operating system and further pools them, that is, it unifies and integrates various virtualization resource pools into a single resource pool, and adds a user-friendly graphical interface on top of the resource pool. The cloud service layer mainly provides cloud products, interfaces with the cloud resource layer, and provides a web interface for users to access resources in the resource pool.

[0046] The virtual work unit's central control platform sends task instructions and monitors the working face's operation. The information and communication processing platform monitors communication data transmission, the mine accident handling platform provides accident solutions, and the virtual working face platform collects real-time data and builds models, synchronizing data between the coal mining face and the virtual working face. The virtual work unit utilizes virtual reality (VR) to construct the central control platform, information and communication processing platform, mine accident handling platform, and virtual working face platform. The central control platform primarily sends task instructions to various departments based on management requirements and monitors the working face's condition in real time. When accidents occur, it makes immediate decision-making instructions for appropriate personnel to handle them. The information and communication processing platform is responsible for managing the normal transmission of communication data from various mechanical equipment, sensors, receivers, cameras, robots, and other devices. The mine accident handling platform primarily develops solutions based on the accidents that occur, sends the solutions to the central control platform for review, and then sends instructions to the relevant personnel for accident handling after approval. The virtual working face, as the main component, mainly uses advanced technology and equipment to collect real-time data from the site to build a model, ensuring that the data of the real coal mining face and the virtual working face are synchronized, and that the virtual digital robot is synchronized with the user's virtual image. Based on the work tasks issued by the central control platform, the equipment of the working face is adjusted to complete the production tasks efficiently and safely.

[0047] The precise positioning unit calculates and measures the coal seam's mining height and area, captures and replays the movement trajectories of personnel and construction equipment, and counts the number of personnel and equipment within the target area. Specifically, the precise positioning unit uses the BeiDou positioning system combined with technologies such as 3D real-time maps to obtain the position and distance information of all personnel and equipment across the entire working face. It can calculate and measure information such as the coal seam's mining height and area, capture and replay the movement trajectories of personnel and equipment, count the number of personnel and equipment within the target area, and ultimately obtain comprehensive information about the target in real time.

[0048] The data acquisition unit collects the geometric dimensions of the working face and the outline information of the construction equipment. It also accesses geographic information systems (GIS) and geological exploration information. Precise positioning is achieved using 3D real-time mapping technology, and data is collected from the working face using panoramic acquisition equipment. The collected data focuses on the geometric dimensions of the working face and the outline of the equipment, constructing a framework model of the entire working face. Details such as coal quality, coal face roughness, and equipment color are less critical. Furthermore, the data acquisition system can access other data from the GIS and geological exploration departments, such as geological conditions (strata composition, coal seam occurrence conditions, roof and floor characteristics, geological structure), hydrological conditions (surface and underground hydrogeological conditions, aquifers, mine water inrush), roadway layout, mining methods, and mining processes.

[0049] The data transmission unit utilizes wireless broadband to transmit data. It transmits data from the panoramic acquisition device, user operation commands captured by sensors, and information collected by the virtual digital robot to the cloud data processing unit. Based on 6G wireless broadband transmission technology, the data transmission unit has three main data transmission paths: First, it sends data collected from the actual coal mining face via the panoramic acquisition device to the cloud data processing unit. After cloud data processing, calculation, classification, and transformation, the data required to construct the virtual working unit is sent to the work control room to complete the construction of the virtual coal mining face. Second, it sends user action command information captured by sensors to the cloud data processing unit. After cloud data processing, classification, and transformation, the command information is sent to the robot's on-site operation center, which then sends it to the virtual digital robot, ensuring synchronization between the virtual digital robot and user actions. Third, it sends data collected by the virtual digital robot to the cloud data processing unit. After cloud data processing, classification, and transformation, it sends the data to the work control room, updating the user's sensory information in real time.

[0050] The work protection unit includes a facial recognition device. Before a user enters the control room, a facial recognition device is installed outside the control room to prevent unauthorized personnel from entering, thus protecting the entire control room and system. Upon initial entry, clicking "Initial Login" initiates facial data collection. After collection, the system uploads the information to the backend database. Clicking "Start Facial Recognition" on the screen then compares the recognized image with images in the database using a coordinate axis digital matrix corresponding to the image's pixel matrix. If the comparison is successful, entry is granted; otherwise, entry is denied. It is important to note that if someone is inside the control room, the screen displays "Working, Do Not Disturb," and no one is allowed to enter. This ensures a safe working environment and maintains the quality of work for the personnel inside the control room.

[0051] The registration and login unit is developed based on web front-end technology. It utilizes HTML (Hypertext Markup Language) to build the framework, Cascading Style Sheets (CSS) to style the webpage, and JavaScript to create pop-up dialog boxes. Specifically, it integrates the user into a virtual workstation. Once the user puts on VR glasses, a haptic bodysuit, and force feedback gloves, a login interface automatically appears in front of them. Upon first login, account registration is required. After registration, the username and password are automatically stored in the database. When the user logs in again, the entered username and password are compared with the database records. If the comparison is correct, login is successful; otherwise, an incorrect password message is displayed, and login fails. Furthermore, users can select "Remember Password" and "Automatic Login" buttons after registration to save time during subsequent logins. If a user forgets their password, they can click the "Forgot Password" button, enter the correct verification information, and the database will automatically delete the corresponding password, thus resetting the password for the user.

[0052] The VR virtual imaging unit constructs a 3D model in VRML-supported WRL (Virtual Reality Text) format using Surpac software, imports the model into VRML, and then uses HTML (Hypertext Markup Language) programming to implement virtual scene interaction on a webpage. Specifically, the VR virtual imaging unit imports data from the cloud data processing center into the 3D mining modeling software Surpac, constructs a 3D model in VRML-supported WRL format using Surpac, imports the model into VRML, and then uses HTML programming to implement virtual scene interaction on a webpage. Users can then immerse themselves in the constructed virtual scene and interact with it by wearing VR glasses.

[0053] The motion capture unit uses infrared cameras and infrared sensors for positioning and motion capture, especially the user's forward, backward, left, and right movements, and monitors whether the user's position has deviated from the omnidirectional treadmill to ensure the user's safety; the IMU sensor is responsible for capturing the user's limb movements, especially arm movements.

[0054] The command transmission unit first packages the motion commands captured by the infrared camera and sensors using a signal transmitter, then sends them to the cloud data processing center via 6G information transmission technology. After processing, the commands are sent to the robot's on-site operation center. Finally, the robot operation center decompresses the operation commands and sends them to the virtual digital robot.

[0055] The operation feedback unit is carried out by a virtual digital robot. The virtual digital robot receives operation commands from the on-site robot operation center and performs corresponding actions accordingly, ensuring real-time synchronization with the user's actions. During operation, it can simulate real-world operating scenarios, providing feedback on the magnitude of force applied to contact between the virtual robot and equipment, button controls, etc. The virtual digital robot then sends its tactile feedback and force information to the control room, where a tactile bodysuit simulates the touch and force, recreating the realism of the work environment and enhancing the user's immersion and interactivity.

[0056] The information acquisition unit primarily uses a virtual digital robot as its carrier. Through the robot's high-precision camera, it possesses the flexibility of the human eye, capable of zooming in on targets up to 50 times. The virtual digital robot is equipped with various sensors, including sound sensors, odor sensors, and infrared sensors, to acquire relevant information about the working environment. It can establish a Wi-Fi connection with the target equipment for data exchange, obtaining basic parameters such as motor power, dimensions, and voltage, as well as real-time operating status such as vibration rate, traction force, and speed. This system differs from a data acquisition system in that the data acquisition system focuses on building a model of the entire working face, with lower requirements for detailed information such as coal quality, coal face roughness, and equipment color. The information acquisition system, on the other hand, primarily focuses on the user's sensory experience, acquiring data from the coal mining face through sight, sound, and smell.

[0057] The monitoring and alarm unit is primarily responsible for monitoring and overseeing all aspects of the entire system, including the construction status at the work site, equipment operation, user actions, and potential accidents. Specifically, it monitors data received from various devices and the cloud data processing center, comparing this data with data in the large database and expert system library. If an anomaly is detected, an alarm is issued to the user, the central control platform, and various auxiliary management platforms based on the accident level. The central control platform analyzes the accident category and issues instructions to the corresponding departments, providing overall guidance. Each department then handles the accident upon receiving the instructions. Once all accidents are resolved, a signal is sent back to the central control platform, which then sends work instructions to each platform. Upon receiving these instructions, each platform begins normal operation.

[0058] The VR headset includes a plastic shell, a power switch, lenses, a vision switching switch, a skeletal sensor, a connecting frame, and adjustable straps. Each lens displays two independent virtual images. The power switch is located above the lenses, and a vision switching button is on the side, allowing users to freely switch between real and virtual scenes. The connecting frame houses the skeletal sensor for human skeletal scanning, and the straps are adjustable for user comfort. The VR headset connects to the host computer via Wi-Fi, receiving decompressed 3D video images. Users establish a connection with the virtual workstation by wearing the VR headset, while the skeletal sensor scans and renders the user's entire skeleton, creating a 3D virtual human model.

[0059] The infrared camera and infrared sensor, along with their mounting structure, include an infrared lens, an infrared sensor, a plastic housing, connectors, and a base. Integrating the infrared camera and sensor into a single unit allows for computer-controlled operation and a Wi-Fi connection to the main unit. Captured motion commands are transmitted to the main unit, and video images are displayed on an office computer screen. The infrared camera and sensor perform positioning and motion capture, particularly monitoring the user's forward, backward, left, and right movements, and ensuring the user remains within the omnidirectional treadmill, thus guaranteeing user safety.

[0060] The force feedback haptic glove consists of a glove body, IMU sub-sensors, connecting cables, a main IMU sensor, and a power supply unit. The IMU sub-sensors primarily capture the movements of each finger. The connecting cables transmit the finger movement commands to the main IMU sensor. The main IMU sensor, by simultaneously capturing arm movements and combining the finger movements captured by the sub-sensors, optimizes the overall arm movement to ensure the accuracy of each action. Finally, the optimized movement commands are sent to the host computer. The force feedback haptic glove is a tool for users to complete production tasks in a virtual workspace. It can grasp target equipment, obtain necessary data, and control the equipment via buttons, enabling remote operation.

[0061] The haptic bodysuit comprises the main body of the garment, bioelectric signal circuitry, a receiver, a waistband, a processor, and a battery. The main body is constructed from a special smart fabric with numerous tiny nodes and temperature sensors, allowing the skin to directly perceive sensations through pulsed currents. The central control unit is a smart waistband equipped with a quad-core processor and a battery. Its haptic feedback system is based on technologies widely used in the medical field—transcutaneous electrostimulation (TENS) and electromuscular stimulation (EMS). Utilizing electromuscular pulse technology, it transmits sensations from the nervous system to the brain in the form of bioelectricity. The bodysuit connects to the main unit via Wi-Fi. The receiver receives the feedback force information, which is then analyzed and processed by the processor, converting it into bioelectric signals to stimulate the user and provide sensory feedback to the brain. Users can experience realistic sensations such as wind flow, device cooling, and the impact of an explosion in a virtual world, enhancing their immersion.

[0062] An omnidirectional treadmill includes support arms, a main frame, a base, an exercise platform, a protective vest, and metal connectors. The omnidirectional treadmill is primarily designed to restrict the user's spatial position. Wearing the protective vest and standing on the exercise platform, the user can naturally walk in a 360° range, satisfying their exercise experience while ensuring their safety.

[0063] The virtual digital robot and construction equipment communicate and exchange data wirelessly. The construction equipment data includes motor power, dimensions, voltage, vibration rate, traction force, and speed. The virtual digital robot specifically includes a high-precision camera, odor sensor, temperature sensor, command receiver, information transmitter, processor, force sensor, auxiliary scanner, and positioning sensor. The external material of the virtual digital robot can reflect the magnitude of contact force, simulating human touch, much like human skin. It is equipped with a high-precision camera with a wide and flexible camera angle and the ability to zoom up to 50 times. The odor sensor captures specific odors from the site, the temperature sensor captures the ambient temperature, the command receiver receives user commands, the information transmitter transmits data collected by the virtual digital robot, the processor integrates and classifies the transmitted and received data, and manages the operation of the virtual digital robot itself, the force sensor reports the magnitude of the applied force, the auxiliary scanner works with the high-precision camera to collect data, and the positioning sensor displays the robot's position on the work surface, ensuring that the robot's distance from its surroundings is accurate. Virtual digital robots primarily act as user surrogates in real coal mining operations. Users operate the robots from a control room, and the robot receives these commands and mimics the user's actions, completing the corresponding tasks under the user's control. Simultaneously, the virtual digital robot collects on-site data through high-precision cameras and various sensors, transmitting the data to the user via an information transmitter. This real-time updating of the received data allows the user to experience the process as if they were actually there.

[0064] The panoramic acquisition equipment specifically includes a fiber optic sensor module, a laser rangefinder, an infrared sensor, and an "AI Eagle Eye" panoramic camera. The fiber optic sensor converts the scanned surrounding environment into measurable light signals, which are then sent via fiber optic cable to a modulator. Within the modulator, the light signals interact with the external parameters being measured, causing a change in their properties. These changes are then transmitted via fiber optic cable to an optoelectronic device and demodulated to obtain the measured data. The rangefinder sets a distance threshold and uses laser ranging to determine the distance between the device and the actual object, comparing it to the threshold to select different lasers and display different scenes. The infrared sensor measures the composition and temperature of the air in the work area, enabling the detection of abnormal equipment and monitoring the work environment. The "AI Eagle Eye" panoramic camera uses 10MP and 25MP panoramic eagle eye cameras as its primary and secondary cameras, respectively, providing panoramic close-up images that balance panoramic views with detailed shots. Its unique design consists of five high-definition cameras fixed horizontally in different directions, forming a special monitoring structure. In panoramic display mode, four secondary cameras monitor the scene to achieve a panoramic monitoring effect. The main camera is responsible for the device display mode, providing a high-definition view of the device's specific status. It offers 32x optical zoom and 24x digital zoom, allowing for quick focusing by simply clicking on any point in the panoramic view. This device can automatically and accurately record the geographical location information of the work area as it moves with the panoramic acquisition equipment, generating a 360-degree panoramic image. It clearly displays small details such as device status and layout on the map, creating a realistic, all-around stereoscopic map to help workers understand the work area.

[0065] In addition, the panoramic acquisition equipment mainly utilizes laser point cloud technology to scan the surrounding environment and collect reflected laser information. Based on the time difference of the reflected laser light, it presents a panoramic image. It employs four 360° "AI Eagle Eye" panoramic cameras combined with laser range sensors, using dual-laser panoramic technology. Before operation, a distance threshold is set between the equipment and the actual workpiece. If the distance between the equipment's current location and the workpiece is greater than the set threshold, the first laser is used to activate the panoramic display mode, showing the overall panoramic layout of the three-dimensional surroundings of the workpiece. If the distance is less than the set threshold, the second laser is used to activate the equipment display mode, showing the detailed layout of specific equipment within the workpiece. The panoramic acquisition equipment is equipped with fiber optic sensors, laser range sensors, and infrared sensors to collect data on the coal mining face equipment and scene. It can perceive the workpiece condition from multiple angles and distances, and uses an infrared sensing module to measure air composition and temperature, monitoring the workpiece environment in real time.

[0066] A remote unmanned operation method for coal mining faces based on virtual reality is disclosed. Utilizing the aforementioned remote unmanned operating system for coal mining faces based on virtual reality, the coal mining construction process includes coal cutting and loading by the coal mining machine, coal transportation by the scraper conveyor, crushing of large coal pieces by the crusher, coal transportation by the transfer conveyor, and coal transportation by the belt conveyor. The sequence of hydraulic support movement is: retracting side guard plates, retracting side guard plates, lowering the support, pulling the support, raising the support, extending the side guard plates, and extending the side guard plates. The virtual digital robot synchronizes with the user's actions and shares data information.

[0067] Taking the fully mechanized coal mining process as an example, it consists of five processes: coal cutting, coal loading, coal transportation, support, and goaf treatment. The process involves: coal mining machine (coal cutting and loading) → scraper conveyor (coal transportation) → crusher (breaking large pieces) → transfer conveyor (coal transportation) → belt conveyor (coal transportation). The coal mining machine is a double-drum type, and the hydraulic support is a shield-type support. The mining method is the inclined long-arm mining method, and the cutting method is end-cutting. The support shifting sequence is: retracting side guard plates → retracting side protection plates → lowering the support → pulling the support → raising the support → removing side protection plates → removing side guard plates. The hydraulic support and the working face scraper conveyor are connected by pushing jacks, serving as fulcrums to achieve alternating forward movement of the scraper conveyor and the support. When moving the support, the support is unloaded, and the top beam is separated from the roof or not completely separated from the roof. Using the conveyor as a fulcrum, the push jack is contracted to move the support forward, and then the support is re-erected. When moving the scraper conveyor, the push jack uses the support as a fulcrum to push the scraper conveyor towards the coal face.

[0068] Users enter the work control room, put on VR glasses and a haptic bodysuit, and then log in through the registration and login unit to access the constructed virtual space station. They select and enter the virtual coal mining face unit, at which point the virtual digital robot of the real coal mining face is activated. Once inside the virtual face, any user movement can be captured by sensors, and the virtual face has a 1:1 geometric scale with the real coal mining face. The virtual digital robot can maintain synchronization with the user's movements, and the user can also share the data information acquired by the virtual digital robot in real time.

[0069] In the control room, a virtual human simulating the user can grasp the target equipment with their hand, after which a virtual information control panel will appear. The user can then use a haptic glove to click the equipment connection option. The virtual digital robot then establishes a signal connection with the equipment. Each piece of equipment is equipped with a data transceiver, transmitting data wirelessly. After establishing the connection, the virtual human can click the equipment information option, displaying various data information of the target equipment. Taking a coal mining machine as an example, it will display parameters such as cutting depth, drum diameter, drum speed, rocker arm length, traction force, traction type, installed power, and voltage. The virtual human can also click the fault detection option. The virtual digital robot will then send instructions to the target equipment. Upon receiving the instructions, the equipment will enter self-test mode, performing various functional tests. After the tests are complete, it will send the results to the robot. If everything is normal, the virtual human will see "Equipment Normal"; otherwise, it will see "Equipment Error, Please Repair". Work can only begin after all production equipment, including the coal mining machine, hydraulic supports, scraper conveyor, transfer conveyor, and belt conveyor, has been inspected and the central control platform issues a "Start Work" command.

[0070] The virtual operator controls the emulsification pump station via an interface, and then sequentially starts the belt conveyor, transfer conveyor, crusher, and scraper conveyor. The coal mining machine then begins operation. After the virtual operator controls the coal mining machine to cut coal to the head end, it controls the left drum to lower to cut the bottom coal, and controls the right drum to slowly raise to cut the top coal. It then returns to perform a diagonal cutting advance. Figure 12 As shown in (a), when the coal mining machine lags behind the hydraulic support by three supports, another virtual operator controls the hydraulic support to move along with the machine until the coal mining machine completes the curved section and enters the straight section of the scraper conveyor. Figure 12 As shown in (b), then control the scraper conveyor from the coal mining machine to the head section to straighten it. Continue to control the coal mining machine to raise the left drum and lower the right drum to cut triangular coal along the head direction of the scraper conveyor. Figure 12 As shown in (c), after cutting the triangular coal seam, the coal mining machine is controlled to raise the right drum and lower the left drum. Then the coal mining machine returns, with the right drum cutting the top coal and the left drum cutting the bottom coal for normal coal cutting. Figure 12(d) As shown. During operation, pay close attention to changes in the mining height and adjust the height of the coal mining machine drum in a timely manner. If the cutting teeth are severely damaged, work must be stopped for repair. The feeding method of the coal mining machine at the tail is the same as that at the head, and the functions of the left and right drums are interchangeable. After the production task is completed, the virtual robot stops the coal mining machine → scraper conveyor → crusher → transfer machine → belt conveyor in the order required by the equipment shutdown sequence, and then cuts off the power and locks the machine. When shutting down, the coal mining machine should be stopped in a section with better roof conditions. The scraper conveyor, transfer machine, and belt conveyor should be emptied of coal before stopping. After the working face is shut down, the virtual robot begins to thoroughly clean the working face and wash the floating coal. After the working face is cleaned up, the virtual robot receives the "work completed" command and can exit the virtual space station. The virtual digital robot then navigates back to the designated position on its own. The entire work process is recorded with images and the work progress data is integrated. After the work is completed, the data is automatically transmitted to the cloud database for storage.

[0071] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A remote unmanned operating system for coal mining faces based on virtual reality, characterized in that, It includes a real-time coal mining face unit, a work control room unit, a robot field operation unit, a cloud data processing unit, a virtual work unit, a precise positioning unit, a data acquisition unit, a data transmission unit, a VR virtual imaging unit, a registration and login unit, a work protection unit, a motion capture unit, an instruction transmission unit, an operation feedback unit, an information acquisition unit, and a monitoring and alarm unit. The coal mining face unit includes construction equipment and on-site data collection equipment, which performs coal mining operations and collects image, sound, stress, and air data from the construction site, uploading the data to the cloud data processing unit. The work control room unit includes VR glasses and a virtual studio, establishing a data connection with the coal mining face unit. The robot field operation unit includes a data receiving station and a virtual digital robot, which keeps synchronized with the operator's movements. The cloud data processing unit performs data calculation, organization, and classification. The virtual work unit utilizes virtual reality (VR) to construct a central control platform, an information communication processing platform, a mine accident handling platform, and a virtual work face platform. The precise positioning unit acquires personnel and equipment information within the target area. The data acquisition unit uses 3D real-time map technology for positioning and panoramic acquisition equipment for data collection at the work face. The data transmission unit transmits data via wireless broadband. The VR virtual imaging unit imports the data into the 3D mine modeling software Surpac. The motion capture unit uses infrared cameras and infrared sensors for positioning and motion capture. The work protection unit controls personnel access to the virtual studio; the registration and login unit connects the user and the virtual work unit; the instruction transmission unit transmits the information collected by the motion capture unit to the virtual digital robot. The operation feedback unit synchronizes user operation commands and the actions of the virtual digital robot; the information acquisition unit acquires data information from the work site; and the monitoring and alarm unit monitors and issues warnings about the construction status of the work surface, the operating status of the equipment, and the user's operation status. The virtual workroom is equipped with VR glasses, a haptic bodysuit, an all-around treadmill, force feedback haptic gloves, and a host computer. The host computer receives data information from the panoramic acquisition device, visual information from the virtual digital robot, and motion commands captured by the sensors. The registration and login unit is developed based on a web front-end, using HTML to build a framework, cascading style sheets to decorate the webpage, and JavaScript to set up pop-up dialog boxes. The VR virtual imaging unit uses Surpac software to build a 3D model in VRML-supported virtual reality text format, imports the model into VRML software, and uses Hypertext Markup Language to program and realize virtual scene interaction in the webpage. The virtual digital robot and the construction equipment are wirelessly connected and exchange data. The data of the construction equipment includes motor power, dimensions, voltage, vibration rate, traction force, and speed.

2. The remote unmanned operating system for coal mining faces based on virtual reality according to claim 1, characterized in that, The construction equipment includes a coal mining machine, hydraulic supports, scraper conveyors, transfer conveyors, crushers, and belt conveyors. The on-site data collection equipment includes a panoramic acquisition device, sensors, signal transmitters, and virtual digital robots.

3. The remote unmanned operating system for coal mining faces based on virtual reality as described in claim 1, characterized in that, The data receiving station receives the command signals sent by the user and sends the signals to the virtual digital robot. The virtual digital robot then sends the real-time images captured on the work surface, the received sounds, and the real-time data on the working status of the construction equipment to the cloud data processing unit.

4. The remote unmanned operating system for coal mining faces based on virtual reality according to claim 1, characterized in that, The cloud data processing unit includes a server, a storage device, and a network connection device. The cloud data processing unit connects the coal mining face unit, the robot field operation unit, the work control room unit, and the virtual work unit. The cloud data processing unit constructs a virtualization layer and performs resource slicing. The cloud data processing unit constructs a cloud resource layer and a cloud service layer. The cloud resource layer integrates the cloud operating system and performs pooling. The cloud service layer provides cloud products and interfaces with the cloud resource layer.

5. The remote unmanned operating system for coal mining faces based on virtual reality according to claim 1, characterized in that, The central control platform of the virtual working unit sends task instructions and monitors the working face. The information communication processing platform monitors communication data transmission. The mine accident handling platform provides accident solutions. The virtual working face platform collects real-time data and builds models, synchronizing coal mining face and virtual working face data.

6. The remote unmanned operating system for coal mining faces based on virtual reality according to claim 1, characterized in that, The precise positioning unit calculates and measures the coal seam's mining height and area information, captures and replays the movement trajectories of personnel and construction equipment, and counts the number of personnel and equipment within the target area; the data acquisition unit collects the geometric dimensions of the working face and the outline information of the construction equipment, and also calls up geographic information systems and geological exploration information.

7. The remote unmanned operating system for coal mining faces based on virtual reality according to claim 1, characterized in that, The data transmission unit uses wireless broadband to transmit data, and transmits the data from the panoramic acquisition device, the user operation commands captured by the sensors, and the information collected by the virtual digital robot to the cloud data processing unit; the working protection unit includes a face information recognition device.

8. A method for remote unmanned operation of a coal mining face based on virtual reality, utilizing the remote unmanned operating system for a coal mining face based on virtual reality as described in any one of claims 1-7, characterized in that, The coal mining construction process includes coal cutting and loading by the coal mining machine at the working face, coal transportation by the scraper conveyor, crushing of large coal pieces by the crusher, coal transportation by the transfer machine, and coal transportation by the belt conveyor; the sequence of hydraulic support movement is: retracting the side guard plate, retracting the side guard plate, lowering the support, pulling the support, raising the support, removing the side guard plate, and removing the side guard plate; the virtual digital robot synchronizes with the user's actions and shares data information.

Images