Tunneling cutting surface video stitching method based on laser point cloud

Abstract

The invention discloses a tunneling cutting surface video stitching method based on laser point cloud, and the method comprises the following steps: S1, scanning a tunnel environment through a laser radar, generating point cloud data, and carrying out the primary processing; s2, modeling the point cloud data to generate a three-dimensional roadway model; s3, using a camera to shoot the roadway environment in real time, and splicing the shot video data into a seamless panoramic video; s4, dynamically fitting the spliced panoramic video to the surface of the three-dimensional roadway model to realize fusion of the panoramic video and the three-dimensional roadway model; s5, generating a virtual three-dimensional model on a display according to the fused three-dimensional roadway model, and displaying the virtual three-dimensional model; s6, the laser radar and the camera detect obstacles, personnel and equipment in the roadway environment in real time, and alarm prompt is carried out in the virtual three-dimensional model; and S7, combining the UWB positioning system with sensor data of the heading machine, and displaying the current position and attitude of the heading machine in real time in the virtual three-dimensional model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of coal mining, in particular to a tunneling cutting face video stitching method based on laser point cloud. BACKGROUND

[0002] Coal mine tunneling operation is a crucial link in the mining process, but the operation environment is complex and involves many factors. For example, the space of the roadway is narrow, and dust and light changes make it impossible for the tunneling machine driver to directly observe the working face with the naked eye. In addition, the roadway form is complex, and obstacles or personnel may appear at any time, making it difficult for traditional safety management methods to meet the needs of real-time dynamic monitoring. Existing technologies usually use electronic fences or AI cameras to manage the tunneling face, but these technologies have limitations and cannot provide comprehensive and accurate operation guidance for the tunneling machine driver.

[0003] Therefore, the existing technology still has the following shortcomings in the process of coal mine tunneling operation: 1. Visual blind area problem; In the existing technology, the tunneling machine driver cannot fully observe the cutting face, and there is a visual dead zone. Especially in the case of winding roadways and limited visibility, the operation of the tunneling machine driver is often restricted by insufficient information, which leads to inaccurate operation of the tunneling machine and increases the risk of misoperation; 2. Poor real-time environmental perception; Existing environmental monitoring devices, such as AI cameras and electronic fences, can usually only provide single-angle or regional monitoring, and lack panoramic perception capabilities. For rapidly changing tunneling environments, these devices have poor response time and real-time performance, and cannot provide accurate operation information for the driver in a timely manner; 3. Environmental perception and safety monitoring problem; Currently, most safety monitoring systems for tunneling machines rely on traditional static sensors or simple video monitoring methods, and lack real-time perception and intelligent feedback of dynamic environments. SUMMARY

[0004] The present application aims to solve the above problems by providing a tunneling cutting face video stitching method based on laser point cloud that can obtain panoramic views in real time, clearly see the surrounding environment, improve operation efficiency and reduce safety hazards.

[0005] To achieve the above purpose, the technical solution of the present application is: The tunneling cutting face video stitching method based on laser point cloud includes the following steps: S1, use a laser radar to scan the roadway environment 360 degrees, generate point cloud data containing all obstacles and roadway walls, and perform preliminary processing; S2, after the point cloud data is preliminarily processed, the point cloud data is modeled by using a Poisson equation to generate a three-dimensional roadway model; S3, six cameras are used to shoot the roadway environment from six different angles of the heading machine in real time, and a panoramic splicing algorithm is used to splice the video data shot by the six cameras into a seamless panoramic video; S4, the panoramic video after splicing is dynamically attached to the surface of the three-dimensional roadway model through an image mapping technology, so that the panoramic video and the three-dimensional roadway model are fused; S5, the fused three-dimensional roadway model is generated into a virtual three-dimensional model and displayed on a display in the heading machine cockpit; S6, the laser radar and the camera detect obstacles, personnel and equipment in the roadway environment in real time, and alarm prompts are given in the virtual three-dimensional model; S7, the virtual three-dimensional model is combined with the sensor data of the heading machine through an integrated UWB positioning system, and the current position and attitude of the heading machine are displayed in the virtual three-dimensional model in real time.

[0006] Further, the step S1 specifically includes the following steps: S11, the laser radar collects point cloud data of the surrounding roadway environment in real time during the driving of the heading machine, and synchronizes the scanning sensor with the inertial measurement unit to ensure the accuracy of the point cloud data; S12, noise points in the point cloud data are removed and redundant points in the point cloud data are merged through a filtering algorithm; S13, the point cloud data is converted into the same coordinate system through an ICP algorithm, so that the registration of scanning data at different positions is realized, thereby facilitating subsequent data processing operations.

[0007] Further, the step S2 specifically includes the following steps: S21, normal vectors of the point cloud data are calculated and the point cloud data is converted into a voxel grid; S22, based on the local neighborhood information of the point cloud data, a weight matrix is calculated to provide input data for solving the Poisson equation; S23, a surface grid is generated by solving the Poisson equation to obtain a preliminary three-dimensional roadway model; S24, the preliminary three-dimensional roadway model is further optimized, and a three-dimensional roadway model is obtained after removing redundant information.

[0008] Further, the step S3 specifically includes the following steps: S31, a feature matching algorithm is used to register images at different angles shot by the six cameras, so that the images shot by each camera can be seamlessly connected; S32, a real-time rendering technology is used to convert the spliced images into an interactive panoramic video.

[0009] Further, the step S5 specifically comprises the following steps: S51, using a rendering engine in graphics to generate a virtual three-dimensional model of the fused three-dimensional tunnel model on the display of the tunneling machine cockpit; S52, according to the tunnel environment changes detected by the laser radar scanning, the virtual three-dimensional model is updated in real time, so that the driver can obtain the new tunnel environment change information in the shortest time; S53, the driver controls the virtual three-dimensional model through the touch screen on the display to zoom, rotate and adjust the viewing angle.

[0010] Further, the step S6 specifically comprises the following steps: S61, using the point cloud data collected by the laser radar, applying a clustering algorithm to identify the obstacles in the tunnel environment, and by analyzing the spatial distribution of the point cloud data, distinguishing the obstacles from other non-obstacle areas, and feeding back the detected obstacle information in the virtual three-dimensional model in real time; S62, applying computer vision algorithm for personnel identification and tracking in the image captured by the camera, and when detecting that the personnel enter the warning area, displaying the personnel position in the virtual three-dimensional model; S63, when the laser radar and the camera detect the obstacle blocking condition or the personnel entering the warning area condition, the virtual three-dimensional model immediately issues a danger alarm to the driver.

[0011] Further, in the step S63, the way of issuing a danger alarm to the driver includes sound, image or vibration.

[0012] Further, the step S7 specifically comprises the following steps: S71, deploying a UWB positioning system inside the tunneling machine to obtain the accurate position of the tunneling machine in the tunnel in real time; S72, combining the UWB positioning system with the sensor data of the tunneling machine to monitor the attitude information of the tunneling machine in real time; S73, mapping the accurate position and attitude information of the tunneling machine to the virtual three-dimensional model in real time, and displaying the trajectory line and arrow mark of the accurate position and attitude information of the tunneling machine in the virtual three-dimensional model through the graphic rendering engine, so as to enable the driver to quickly understand the current state and attitude change of the tunneling machine.

[0013] Further, in the step S72, the attitude information of the tunneling machine includes the inclination angle, rotation angle and acceleration of the tunneling machine.

[0014] Compared with the prior art, the present application has the advantages and positive effects: 1. Panoramic Perception: This invention acquires omnidirectional video data through multiple cameras and combines it with a 3D model generated by laser scanning, eliminating visual blind spots in traditional technologies and providing complete environmental perception.

[0015] 2. Real-time obstacle and personnel alarm: Compared with traditional safety monitoring methods, this invention combines lidar and computer vision technology to achieve real-time detection and alarm of obstacles and personnel. In complex environments, it can identify dangers and issue warnings to drivers in the shortest possible time, significantly improving the safety of coal mining operations.

[0016] 3. Enhanced safety: This invention integrates obstacle detection and personnel recognition functions, which can promptly detect personnel entering the warning zone and alert to potential safety hazards, significantly reducing the risk of accidents.

[0017] 4. Real-time and accurate operation feedback: This invention uses a virtual 3D model to display and update the external environment and working status of the tunneling machine in real time, providing the driver with more intuitive operation feedback and effectively improving operation efficiency and safety. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art to all other embodiments obtained without creative effort should be included within the protection scope of the present invention.

[0019] This invention proposes a video stitching method for tunneling cutting faces based on laser scanning. By combining laser scanning technology and multi-camera video stitching technology, it comprehensively improves the safety and accuracy of tunneling operations. Through this technical solution, tunneling machine operators can obtain a panoramic view in real time, clearly see the surrounding environment, especially the cutting face, thereby enabling better precision operation, improving work efficiency and reducing safety hazards.

[0020] The core of this invention is the fusion of laser scanning and video stitching technologies, combined with real-time environmental perception. Its specific implementation steps are as follows: 1. LiDAR data acquisition and processing; LiDAR scans the tunnel environment 360 degrees to generate point cloud data containing all obstacles, tunnel walls, and their details; this process mainly includes the following steps: (1) Point cloud acquisition and registration: The lidar collects point cloud data of the surrounding environment in real time during the tunneling machine's operation, and synchronizes with the inertial measurement unit (IMU) through scanning sensors to ensure the accuracy of the data.

[0021] (2) Noise removal and data deduplication: Point cloud data often contains noisy points or duplicate data, especially in complex environments; noise points are removed and redundant points from multiple scans are merged by filtering algorithms (such as Voxel Grid filter and Statistical Outlier Removal algorithm).

[0022] (3) Coordinate system registration: All collected point cloud data must be converted to the same coordinate system to facilitate subsequent data processing; the ICP (Iterative Closest Point) algorithm is used to register scanned data at different locations to ensure the consistency of all point cloud data.

[0023] 2. Three-dimensional model construction; After point cloud data processing, an advanced Poisson Surface Reconstruction algorithm is used to model the data and generate an accurate 3D tunnel model; the specific steps include: (1) Normal estimation and voxelization: Calculate the normal vector of the point cloud and convert the point cloud data into a voxel mesh, which helps to improve the accuracy and detail of the model.

[0024] (2) Constructing the weight matrix: Based on the local neighborhood information of the point cloud, the weight matrix is ​​calculated to provide input data for solving the Poisson equation.

[0025] (3) Triangular mesh generation: By solving the Poisson equation, a surface mesh is generated to further refine the model and ensure a true reflection of the tunnel morphology.

[0026] (4) Model optimization: Further optimize the 3D model through algorithms (such as mesh refinement and surface reconstruction), remove redundant information, and ensure computational efficiency.

[0027] 3. Video stitching and fusion; Six high-definition cameras capture real-time footage of the tunnel environment from six different angles of the tunneling machine. The video data is then stitched together into a seamless panoramic video using panoramic stitching algorithms (such as multi-view image registration and stereo image stitching algorithms). This process requires the application of the following technologies: (1) Image registration and stitching: Image registration is performed on images from different perspectives using feature matching algorithms (such as SIFT and ORB) so that images captured by each camera can be stitched together seamlessly.

[0028] (2) Real-time video rendering: Real-time rendering technology (such as OpenGL and DirectX) is used to convert the stitched images into an interactive panoramic video stream.

[0029] 4. Fusion of 3D models and video data; The stitched panoramic video is then fused with the generated 3D model. Through image mapping technology (such as UV mapping), the video data is dynamically applied to the surface of the 3D model, ensuring that every detail of the video is accurately reflected in the 3D scene.

[0030] 5. Display and interaction of virtual 3D scenes; The fused 3D tunnel model and real-time video data are displayed on an explosion-proof monitor inside the tunneling machine's cab. This process involves not only graphics rendering technology but also the implementation of dynamic environment updates and interactive feedback functions. To ensure the visualization and interactivity of the information, the following technical solutions are adopted: (1) Virtual Reality Scene Construction: A virtual 3D environment is generated on the cockpit display. A rendering engine in graphics (such as Unity3D or Unreal Engine) is used for scene construction and rendering. GPU acceleration rendering technology is used to improve the real-time performance and smoothness of the scene, and to ensure seamless integration of the displayed 3D model and video.

[0031] (2) Dynamic environment update: When the lidar scan detects changes in the roadway (such as the appearance of obstacles or personnel entering the warning zone), the three-dimensional scene will be updated immediately to present the newly detected information to the driver; by calculating the changes in the graphical interface in real time, the driver can obtain new environmental data in the shortest possible time.

[0032] (3) User interaction and control: To enhance the user experience, the system also provides interactive functions with the three-dimensional scene; for example, the driver can zoom, rotate, and adjust the viewing angle through the touch screen in the cockpit or other control methods to better view the cutting surface and the surrounding environment. This function is implemented through virtual reality technology, ensuring that the system is easy to operate and responds quickly.

[0033] 6. Obstacle and personnel recognition and alarm system; LiDAR uses high-precision scanning to detect obstacles, personnel, and equipment in the tunnel in real time. Combined with video data, it provides the tunneling machine with a more comprehensive environmental perception. This system, through intelligent algorithms and hardware combinations, can quickly identify and display potential hazards. Its specific implementation steps are as follows: (1) Obstacle recognition of lidar: Using the point cloud data of lidar, clustering algorithms (such as DBSCAN or K-Means) are applied to identify obstacles. By analyzing the spatial distribution of the point cloud data, obstacles are automatically distinguished from other non-obstacle areas, and the detected obstacle information is fed back in real time in the virtual three-dimensional model.

[0034] (2) Personnel detection and tracking: By combining with the video surveillance system, the system can use computer vision algorithms (such as background modeling, motion detection, YOLO object detection, etc.) to identify and track personnel in the images captured by the camera; when the system detects that a person has entered the warning area, it immediately marks the person's location in the virtual three-dimensional model and triggers an alarm.

[0035] (3) Alarm System and Safety Warnings: When the lidar or camera detects a hazard, the system will issue a warning to the tunneling machine operator through sound, graphics, or vibration. The alarm content may include obstacles, personnel intrusion, equipment failure, etc., to enhance operational safety.

[0036] 7. Real-time position and attitude display of the tunneling machine; To accurately reflect the current operating status of the tunneling machine, it is necessary to display the machine's position and attitude in real time. This function can be achieved by integrating UWB (Ultra-Wideband) technology with the tunneling machine's sensor data; the specific technical implementation process is as follows: (1) UWB positioning and location tracking: By deploying a UWB positioning system, the precise location of the tunneling machine in the roadway can be obtained in real time; UWB technology can provide high-precision positioning in the underground environment. Combined with multi-point positioning technology, UWB can effectively solve the positioning problem in the complex underground environment and ensure that the position information of the tunneling machine remains accurate.

[0037] (2) Sensor Data and Attitude Estimation: By combining sensor data from the tunnel boring machine (such as IMU, accelerometer, gyroscope, etc.), the attitude of the tunnel boring machine is monitored in real time, including information such as tilt angle, rotation angle, and acceleration. The IMU (Inertial Measurement Unit) system can accurately sense the dynamic changes of the tunnel boring machine and provide accurate attitude estimation without external positioning signals. The sensor data is used in combination with UWB positioning information to more accurately calculate the position and attitude of the tunnel boring machine in three-dimensional space.

[0038] (3) Dynamic display in the three-dimensional scene: The real-time position and attitude information of the tunneling machine will be mapped to the virtual three-dimensional tunnel model in real time. The driver can intuitively see the specific position and movement trajectory of the tunneling machine in the tunnel. Through the graphics rendering engine, this information is presented in the form of trajectory lines, arrow marks, etc., so that the driver can clearly understand the current status and dynamic changes of the tunneling machine.

[0039] The beneficial effects of this invention are as follows: 1. This invention acquires accurate three-dimensional point cloud data of the tunnel using lidar and combines it with video data from multiple high-definition cameras to achieve panoramic perception of the environment surrounding the tunneling machine. This eliminates blind spots in traditional technologies and enhances the tunneling machine operator's comprehensive understanding of the surrounding environment. 2. This invention seamlessly integrates the three-dimensional tunnel model generated by LiDAR with real-time stitched video data, and uses texture mapping technology to map the video image onto the surface of the three-dimensional tunnel model, displaying a more intuitive and clear cut surface view, which can provide drivers with accurate feedback on the working environment. 3. This invention uses the coordinated operation of lidar and cameras to detect obstacles and personnel in the tunnel in real time. When an obstacle is detected or personnel enter the warning zone, an alarm will be automatically triggered and the dangerous location will be displayed in the virtual 3D model, prompting the driver to take appropriate safety measures, which effectively reduces the safety risks of the tunneling machine in coal mining operations.

Claims

1. A video stitching method for tunnel cutting surfaces based on laser point clouds, characterized in that: Includes the following steps: S1. Use LiDAR to scan the alleyway environment 360 degrees, generate point cloud data containing all obstacles and alleyway walls, and perform preliminary processing. S2. After the initial processing of the point cloud data, the Poisson equation is used to model the point cloud data and generate a three-dimensional tunnel model. S3. Use six cameras to capture the tunnel environment in real time from six different angles of the tunneling machine, and use a panoramic stitching algorithm to stitch the video data captured by the six cameras into a seamless panoramic video. S4. Using image mapping technology, the stitched panoramic video is dynamically attached to the surface of the 3D tunnel model to achieve the fusion of panoramic video and 3D tunnel model. S5. Generate a virtual 3D model on the monitor in the tunneling machine's cab after the fusion of the 3D tunnel model and display it. S6, LiDAR, and cameras detect obstacles, personnel, and equipment in the tunnel environment in real time and provide alarm prompts in the virtual 3D model; S7. By integrating the virtual 3D model with the sensor data of the tunneling machine through the UWB positioning system, the current position and attitude of the tunneling machine are displayed in real time in the virtual 3D model.

2. The video stitching method for tunnel cutting surfaces based on laser point clouds as described in claim 1, characterized in that: Step S1 specifically includes the following steps: S11. During the tunneling machine's operation, the lidar collects point cloud data of the surrounding tunnel environment in real time and synchronizes the scanning sensor with the inertial measurement unit to ensure the accuracy of the point cloud data. S12. Remove noise points from the point cloud data and merge redundant points from multiple scans in the point cloud data using a filtering algorithm. S13. The point cloud data is converted into the same coordinate system through the ICP algorithm to achieve registration of scanned data at different locations, so as to facilitate subsequent data processing operations.

3. The video stitching method for tunnel cutting surfaces based on laser point clouds as described in claim 2, characterized in that: Step S2 specifically includes the following steps: S21. Calculate the normal vector of the point cloud data and convert the point cloud data into a voxel mesh; S22. Based on the local neighborhood information of point cloud data, calculate the weight matrix to provide input data for solving the Poisson equation; S23. By solving the Poisson equation, a surface mesh is generated to obtain a preliminary three-dimensional tunnel model; S24. Further optimize the preliminary three-dimensional tunnel model and remove redundant information to obtain the three-dimensional tunnel model.

4. The video stitching method for tunnel cutting surfaces based on laser point clouds as described in claim 3, characterized in that: Step S3 specifically includes the following steps: S31. The images captured by the six cameras from different perspectives are registered using a feature matching algorithm, so that the images captured by each camera can be seamlessly connected. S32. Real-time rendering technology is used to convert the stitched images into interactive panoramic videos.

5. The video stitching method for tunnel cutting surfaces based on laser point clouds as described in claim 4, characterized in that: Step S5 specifically includes the following steps: S51. Use a rendering engine in computer graphics to generate a virtual 3D model on the monitor in the tunnel boring machine's cab after the fusion of the 3D tunnel model. S52. Based on the changes in the tunnel environment detected by the lidar scan, the virtual 3D model is updated in real time, allowing the driver to obtain new tunnel environment change information in the shortest possible time. S53. The driver controls the virtual 3D model by zooming, rotating, and adjusting the viewing angle through the touch screen on the display.

6. The video stitching method for tunnel cutting surfaces based on laser point clouds as described in claim 5, characterized in that: Step S6 specifically includes the following steps: S61. Using point cloud data collected by lidar, clustering algorithms are applied to identify obstacles in the alleyway environment. By analyzing the spatial distribution of point cloud data, obstacles are distinguished from other non-obstacle areas, and the detected obstacle information is fed back in real time in the virtual 3D model. S62. Apply computer vision algorithms to identify and track people in the images captured by the camera. When a person is detected entering the restricted area, immediately display the person's location in the virtual 3D model. When the S63 system, lidar, or camera detects obstacles or people entering the restricted area, the virtual 3D model immediately issues a danger warning to the driver.

7. The video stitching method for tunnel cutting surfaces based on laser point clouds as described in claim 6, characterized in that: In step S63, the method of issuing a danger warning to the driver includes sound, image, or vibration.

8. The video stitching method for tunnel cutting surfaces based on laser point clouds as described in claim 7, characterized in that: Step S7 specifically includes the following steps: S71. Deploy a UWB positioning system inside the tunneling machine to obtain the precise location of the tunneling machine in the roadway in real time; S72. Combine the UWB positioning system with the sensor data of the tunneling machine to monitor the attitude information of the tunneling machine in real time; S73. The precise position and attitude information of the tunneling machine are mapped into the virtual 3D model in real time. The graphics rendering engine displays the trajectory lines and arrows of the precise position and attitude information of the tunneling machine in the virtual 3D model, so that the driver can quickly understand the current status and attitude changes of the tunneling machine.

9. The video stitching method for tunnel cutting surfaces based on laser point clouds as described in claim 8, characterized in that: In step S72, the attitude information of the tunneling machine includes the tilt angle, rotation angle, and acceleration of the tunneling machine.