A coal mine underground radar and visual fusion device and three violation identification method

Abstract

This invention discloses a laser radar-visual fusion device and a method for identifying violations in coal mines. The device comprises a base with a vertically mounted height adjustment component along its upper edge. A pitch angle adjustment component is located at the top of the height adjustment component, upon which a laser radar and a camera component are sequentially mounted. The height, horizontal azimuth, and pitch angles of the laser radar and camera component at the top can be flexibly adjusted through the height and pitch angle adjustment components. The modular structure and tool-free locking design of this invention allow for installation, attitude adjustment, equipment maintenance, and replacement by a single person, reducing installation and commissioning time by more than 60% compared to traditional supports. It is suitable for the confined working spaces and rapid construction requirements of underground mines. Furthermore, the standardized hole design ensures compatibility with different specifications of laser radars and explosion-proof cameras, offering strong versatility.

Description

Technical Field

[0001] This invention relates to the field of coal mine safety production monitoring technology, specifically to a coal mine underground radar-visual fusion device and a method for identifying three violations. Background Technology

[0002] Underground gas extraction, water exploration and drainage, and drilling are core construction processes in coal mines. Drilling sites are densely populated with personnel and operate equipment frequently, making them high-risk areas for coal mine safety accidents. Intelligent safety management in these areas is of paramount importance for safe underground production. Currently, underground drilling site monitoring mainly uses a single sensor solution, coupled with traditional mounting brackets. This solution suffers from two major problems: insufficient sensing capabilities and poor hardware compatibility.

[0003] (a) Deficiencies in existing monitoring technologies

[0004] Pure vision cameras are significantly degraded in image quality due to the complex environment underground, such as low light, dust, and water mist. They also suffer from high rates of missed detection and false detection of small, distant, or occluded targets, and cannot acquire information such as 3D spatial location or personnel movement trajectory. RFID / locator cards can only track personnel location and cannot identify the wearing status of personal protective equipment such as helmets, miners' lamps, and self-rescue devices. They also suffer from problems such as lost tags, signal obstruction, and poor reliability. Single lidar offers advantages such as all-weather detection, strong anti-interference, and high-precision 3D positioning, and is unaffected by light or dust. However, point clouds lack semantic information such as texture and color, making it impossible to identify human features and the wearing status of safety equipment.

[0005] (ii) Defects of traditional installation and fixing structures The existing lidar and camera mounting brackets are of a general-purpose structure and are not specifically designed for the conditions of strong vibration, high dust, and high humidity in downhole drilling sites. Specific problems are as follows: Poor vibration stability: High-frequency vibrations generated during drilling operations can easily cause loosening of support bolts and displacement of the equipment's line of sight, resulting in image jitter, point cloud distortion, severely reducing monitoring accuracy, and even damaging the equipment. Insufficient adjustment capability: The support lacks a multi-degree-of-freedom adjustment mechanism, with fixed installation height, horizontal orientation, and pitch angle, making it impossible to flexibly adjust the viewing angle according to the drilling rig position and drill rod trajectory, resulting in numerous monitoring blind spots. Difficulty in ensuring radar-camera coaxiality: The radar and camera are installed on independent supports without a rigid connection reference, making them prone to relative displacement under vibration and environmental interference, causing point cloud and image spatial registration failure and distorted fused data. Weak protection capability: Lack of dustproof, moisture-proof, and rust-proof structures, allowing dust and moisture to easily penetrate the adjustment mechanism and the equipment's interior, causing component jamming, corrosion, and shortening the equipment's lifespan. Cumbersome installation and maintenance: Low degree of structural modularity, complex disassembly, assembly, debugging, and secondary calibration procedures, low efficiency in on-site deployment and subsequent maintenance, and inability to adapt to different specifications of drilling rigs and monitoring equipment.

[0006] In summary, existing downhole drilling site monitoring equipment adopts a fixed installation method, which has a single perspective and poor adaptability to working conditions, and cannot meet the current intelligent drilling operations' requirements for multi-dimensional, blind-spot-free, and highly stable monitoring. Therefore, developing a detection technology that integrates a multi-dimensional adjustable vibration-resistant fixed structure, a downhole-specific radar-visual fusion calibration algorithm, a lightweight intelligent recognition model, and a comprehensive judgment of violations has become an urgent technical challenge to be solved in this field. Summary of the Invention

[0007] This invention addresses the challenges of underground drilling conditions and existing technological shortcomings by providing a coal mine underground radar-visual fusion device and a method for identifying violations. It solves problems such as weak vibration resistance of traditional supports, lack of multi-degree-of-freedom attitude adjustment, difficulty in maintaining coaxiality of radar-visual equipment, poor dust, moisture, and rust prevention capabilities, low drilling rig compatibility, and cumbersome installation, debugging, and maintenance. Furthermore, it optimizes the target association algorithm to improve target matching accuracy in densely populated and mutually obstructed scenarios, avoiding target ID confusion.

[0008] To solve the above problems, the technical solution adopted by the present invention is as follows: A coal mine underground radar-visual fusion device includes a base with a height adjustment component vertically mounted on the upper edge of the base. A pitch angle adjustment component is mounted at the top of the height adjustment component. A lidar and a camera component are installed sequentially on the lidar and the camera component mounted at the top. The height, horizontal azimuth, and pitch angle of the lidar and camera component mounted at the top can be adjusted by the height adjustment component and the pitch angle adjustment component.

[0009] Optionally, the height adjustment assembly includes a sleeve and an adjustment rod inserted into the sleeve. The side wall of the adjustment rod is provided with multiple adjustment grooves, and adjustment screws are installed on the side wall of the sleeve corresponding to the adjustment grooves. The horizontal orientation angle is adjusted by rotating the adjusting rod inside the sleeve.

[0010] Optionally, the pitch angle adjustment component is installed at the top of the height adjustment component, including a U-shaped groove component and a support rod. A sliding block is provided at the lower end of the support rod, and the sliding block is locked on the U-shaped groove of the U-shaped groove component. A fastening screw is provided in the middle of the support rod, and a laser radar and camera component are installed at the top of the support rod.

[0011] Optionally, the lidar is an intrinsically safe lidar for mining, which is fixedly mounted on a support rod.

[0012] Optionally, the camera assembly includes a first connector and a second connector disposed on the top of the support rod, with a spherical camera disposed at the top of the second connector; Both the first and second connectors are disc-shaped components; The first connector is located at the bottom, and the second connector is located at the top. The diameter of the second connector is larger than the diameter of the first connector.

[0013] A method for identifying illegal operations in coal mines using a combined radar and video surveillance system, employing any of the coal mine radar and video surveillance fusion devices described in this invention, specifically includes: Step 1, LiDAR-Camera Joint Calibration: Calibration involves three types of coordinate system transformations: LiDAR coordinate system {L}, camera coordinate system {C}, and image pixel coordinate system {P}. It is divided into camera intrinsic parameter calibration, LiDAR-camera extrinsic parameter calibration, coordinate system transformation, and accuracy verification. Step 2: The lidar-camera system performs data acquisition and processing; Step 3: Use the fusion confidence score to identify and determine the three violations in the processing results of Step 2. The score is obtained by weighting the four dimensions of visual recognition confidence, radar positioning confidence, spatial matching consistency and historical trajectory continuity.

[0014] Optionally, step one specifically includes: Camera intrinsic parameter calibration: The lidar coordinate system {L} is a three-dimensional rectangular coordinate system with the lidar optical center as the origin, outputting the three-dimensional coordinates of the point cloud. Camera coordinate system {C}: A three-dimensional rectangular coordinate system with the camera's optical center as the origin; World coordinate system {W}: A local spatial coordinate system for the downhole drilling site, established based on fixed feature points on site, replacing the outdoor GPS coordinate system; Image pixel coordinate system {P}: A two-dimensional image pixel row and column coordinate system, outputting the target pixel coordinates. ; Solving for the intrinsic parameter matrix: The camera intrinsic parameter matrix K is calculated as follows: ; In the formula: , These are the equivalent focal lengths (in pixels) in the x and y directions of the image, respectively. , The coordinates of the camera's optical center in the pixel coordinate system; Radar-camera extrinsic parameter calibration: At least six high-reflectivity feature control points are deployed at the drilling site. The three-dimensional coordinates of these control points in the radar coordinate system and the image pixel coordinates are simultaneously acquired. The extrinsic parameter matrix [R|t] is solved using the PnP algorithm to minimize the reprojection error and optimize the parameters. The projection formula is: ; In the formula: s is the scale factor, and p is the pixel coordinate. Let R be the homogeneous coordinates of the radar's three-dimensional point, R be the rotation matrix, and t be the translation vector. Coordinate system transformation: Converting radar 3D points to image pixel coordinates: ; This represents the depth value of the point in the camera coordinate system. Calibration accuracy verification: The accuracy is evaluated using the reprojection error (RPE), which requires RPE < 2.0 pixels. Once the standard is met, all calibration parameters are fixed to the embedded device.

[0015] Optionally, step two specifically includes: The lidar sampling frequency is 10Hz, and the camera frame rate is 25-30fps; based on the lidar timestamp, image frames with the smallest time difference are selected for pairing, with a synchronization tolerance threshold. =50ms; if the deviation exceeds the limit, linear interpolation is performed on the image frame to ensure frame-level time correspondence. Judgment conditions: ; Represents: the timestamp corresponding to the LiDAR data collected in the k-th frame; Represents: the timestamp corresponding to the acquisition of the j-th frame of the camera image; Spatial matching DIoU target association: The radar 3D bounding box is projected onto the image plane through extrinsic parameters. An improved range intersection-union (DIoU) algorithm is used to associate radar targets with visual targets, taking into account bounding box overlap, center point distance, and scale difference. The formula is as follows: + ); In the formula: ρ is the Euclidean distance of the center point, d is the length of the diagonal of the minimum closure, and the scale penalty coefficient α is taken as 0.1 to 0.3; The projected target bounding box is obtained by projecting the 3D bounding box of the radar point cloud onto the image plane using the camera's intrinsic and extrinsic parameters. The bounding box of the target in the image is directly output by the visual image detection algorithm. represent and The intersection / union area of ​​two bounding boxes measures the degree of overlap between the two boxes, with a value of [0,1]. This represents the width of the bounding box of the image after the radar 3D frame is projected. The width of the bounding box of the visually detected image; The height of the bounding box in the visually detected image; The height of the bounding box of the image after the radar 3D frame is projected. The acquired video images were processed using an improved YOLOv8s-WS object detection model in the visual channel. The overall loss calculation formula is as follows: ; Based on the operational needs of underground violation detection, the weight values ​​were optimized through multiple experiments: Bounding box loss weights, Classification loss weights, Distributed focal loss weights; Perfect intersection-union loss Binary cross-entropy loss Distribution focal loss .

[0016] Optionally, the YOLOv8s-WS network architecture consists of an input layer, a backbone network, a neck network, and a head detection network, adapted to downhole image feature extraction and target detection tasks. Input layer: Receives RGB images output from explosion-proof cameras in underground coal mines, performs size normalization, and then sends them to the backbone network; Backbone Network: The main body retains the native YOLOv8 Conv convolutional layers and C2f feature extraction module, and replaces the traditional max pooling in the terminal SPPF module with two-dimensional discrete wavelet pooling. Wavelet pooling decomposes the input feature map into four types of subbands through discrete wavelet transform: low-frequency approximate component XLL, horizontal high frequency XLH, vertical high frequency XHL, and diagonal high frequency XHH. Only the low-frequency component is downsampled, and the high-frequency detailed features are fully preserved. The neck network employs an FPN+PAN dual-fusion structure, including convolution, upsampling, and feature concatenation units. It fuses and enhances the multi-scale features output by the backbone network to generate three types of feature maps with different resolutions: P3: 80×80 small-scale feature map; P4: 40×4 medium-scale feature map; P5: 20×4 large-scale feature map. Head detection network: A shuffling attention module is embedded in all three detection heads, P3, P4, and P5; the internal process of the shuffling attention module is: feature grouping → channel attention calculation → feature shuffling → spatial attention weighting → feature fusion output; the detection head simultaneously outputs the target category, detection box coordinates, and confidence score, and is equipped with a triple loss function of CIoU+BCE+DFL to complete training optimization.

[0017] Optionally, step three specifically includes: ; For visual confidence weights, The target detection confidence score output by the YOLOv8s-WS model. For radar confidence weights, The calculation is based on a combination of personnel point cloud density, cluster integrity, and trajectory smoothness. For spatial matching weights, Match values ​​to the target bounding box. For trajectory consistency weight, To improve the success rate of target association across multiple consecutive frames; Weight constraints: All weights have been measured and optimized in multiple downhole scenarios to adapt to all working conditions at the drilling site.

[0018] Confidence level assessment rules: Based on the fusion confidence level, three judgment levels are established to distinguish between valid targets, suspected targets, and invalid targets, and differentiating treatment is applied accordingly. Highly credible targets: With high matching accuracy and stable recognition results from dual channels, the detection results are directly accepted and the process for judging violations is initiated. Suspected Target: For targets in scenes with obstruction, distance, low light, and high dust, violations are not immediately determined. The results are output only after three consecutive frames are checked to reduce the false alarm rate. Invalid target: If it is determined to be noise, reflective interference, or equipment false detection, it will be directly rejected and will not be included in the subsequent behavior judgment. Secondary fusion analysis of unmatched targets: For single-channel targets detected only by radar or only by vision, supplementary analysis rules are established: Detected by LiDAR only: Vision may miss detection due to obstruction, strong light, or dense dust. If the target remains in the work area for an extended period and its movement characteristics match those of a person, it will be temporarily marked as a suspected person and continuously tracked. Detected by vision only: If radar fails to cluster the personnel due to them being too close to the equipment or insufficient point cloud data, the visual wear status will be accepted if the category is stable and multiple frames of visual recognition results are combined. Typical violation judgment rules: If a person is detected as not wearing complete personal protective equipment, but any one of the following is not worn properly: safety helmet, miner's lamp, self-rescue device, protective gloves, or work clothes, it is directly judged as a violation; a binary alarm code is generated based on the missing item: missing safety helmet 0x01, missing miner's lamp 0x02, missing self-rescue device 0x04, missing protective gloves 0x08, and multiple missing items are combined by bitwise OR operation; Dangerous area intrusion: Dangerous areas such as drilling rig operation area, belt conveyor area, and unsupported area are pre-demarcated; if the three-dimensional centroid of personnel falls into the dangerous area and the distance from the boundary exceeds the intrusion threshold, it is judged as a violation of dangerous area intrusion. Alarm linkage and data upload: Alarm information format: Generates a standard JSON message, including alarm number, timestamp, on-site location, violation type, personnel 3D coordinates, fusion confidence level, and on-site captured screenshot; On-site linkage: Triggers underground audible and visual alarms and voice broadcasts, providing immediate on-site alerts to workers; Remote upload: Uploads alarm data, video, and trajectory data to the surface dispatch platform in real time via the underground industrial Ethernet ring network, enabling remote monitoring, data storage, and accident tracing.

[0019] The advantages of this invention are: Long-term coaxial accuracy maintenance: The lidar and explosion-proof camera are rigidly connected via a dedicated adapter disk, strictly ensuring coaxial optical centers, parallel front planes, and overlapping fields of view. Unlike split mounting brackets, this structure avoids relative displacement caused by vibration from the hardware source, ensuring long-term stability of the lidar spatial registration benchmark and eliminating the need for frequent secondary calibration. The slotted guide rail adapts to the drilling rig frame, eliminating the need for on-site drilling. The modular structure combined with a tool-less locking design allows a single person to complete installation, attitude adjustment, equipment maintenance, and replacement. Installation and commissioning time is reduced by more than 60% compared to traditional brackets, making it suitable for confined downhole working spaces and rapid construction needs. Furthermore, the standardized hole design is compatible with different specifications of lidar and explosion-proof cameras, offering strong versatility.

[0020] The combined calibration and spatiotemporal synchronization technology of the laser-guided radar achieves the following results: Employing a timestamp alignment strategy with a 50ms time tolerance, coupled with linear interpolation compensation, it perfectly adapts to the heterogeneous sampling characteristics of the 10Hz LiDAR and 25~30fps camera, achieving near 100% frame-level temporal matching accuracy. Based on an improved DIoU spatial association algorithm, the target tracking rate reaches 92.6%, and the correct tracking rate reaches 95.8%, effectively solving the ID confusion problem under conditions of dense crowds and target occlusion, laying a solid foundation for multimodal fusion. By integrating wavelet pooling and shuffling attention mechanisms on the YOLOv8s platform, it specifically optimizes the detection capabilities for low-light, high-dust, and small targets in downhole environments, balancing accuracy with the real-time performance of embedded devices. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the underground radar-visual fusion device for coal mines according to the present invention; 1-Base; 2-Height adjustment component, 21-Sleeve, 211-Sliding block, 22-Adjusting screw, 23-Adjusting rod, 231-Adjusting groove; 3-Pitch angle adjustment component, 31-U-shaped groove component, 32-Sliding block, 33-Fasting screw, 34-Support rod; 4-LiDAR; 5-Camera component, 51-First connector, 52-Second connector, 53-Spherical camera; Figure 2 This is an ablation experiment diagram of the YOLOv8s-WS of this invention; Figure 3 A bar chart comparing AP@0.5 for each category of this invention; Figure 4 This is a graph showing the change in the training loss function of the YOLOv8s-WS model of this invention. Figure 5This is a graph showing the changes in precision and recall during training of the YOLOv8s-WS model of this invention. Figure 6 This is a graph showing the mAP variation of the YOLOv8s-WS model of this invention. Figure 7 This is a graph showing the variation of mAP@0.5 radar-visual fusion according to the present invention. Figure 8 Curves showing the changes in precision and recall during the training of the Ravis fusion model. Detailed Implementation

[0022] The technical solution of the present invention will be further illustrated below through specific embodiments. These embodiments are merely illustrative of the invention and should not be considered as specific limitations thereof.

[0023] The coal mine underground radar-visual fusion device of the present invention includes a base 1, with a height adjustment component 2 vertically arranged on the base 1. A pitch angle adjustment component 3 is located at the top of the height adjustment component 2. A lidar 4 and a camera component 5 are sequentially mounted on the lidar 4 and camera component 5, respectively. Through the height adjustment component 2 and the pitch angle adjustment component 3, the height, horizontal azimuth angle, and pitch angle of the lidar 4 and camera component 5 can be flexibly adjusted. For example, the pitch angle adjustment range is 0° to 90°, which can cover the top, bottom, and side blind spots of the drilling rig's working area, achieving blind-spot-free monitoring. The horizontal azimuth angle adjustment range is 0° to 360°.

[0024] In this invention, the base 1 is the bottom supporting component of the height adjustment component 2, and adopts a grooved guide rail structure. A sliding block 211 can be connected and set at the bottom of the height adjustment component 2. For example, the specific structure can be a cross-shaped bracket. The cross-shaped bracket is locked in the grooved guide rail structure and can move along the guide rail to realize fine adjustment of the left and right positions. It can also be installed and fastened by the set bolts. The base 1 can be installed on the drilling rig body and move with the drilling rig body.

[0025] The height adjustment assembly includes a sleeve 21 and an adjusting rod 23 inserted into the sleeve 21. For example, the sleeve 21 is a cylindrical seamless steel pipe made of stainless steel; the adjusting rod 23 is a cylindrical rod made of stainless steel. The adjusting rod 23 is slidably inserted into the sleeve 21 with a clearance of 0.5mm, allowing for smooth extension and retraction without jamming or offset. The side wall of the adjusting rod 23 has multiple adjusting grooves 231, which slide against guide blocks on the inner wall of the sleeve 21 to prevent rotation during extension and retraction. Adjusting screws 22 are installed on the threaded holes in the adjusting grooves 231 and the side wall of the sleeve 21. Rubber anti-slip pads are adhered to the ends of the screws, and after tightening, they fit tightly against the surface of the adjusting rod 23, ensuring a secure lock on the height of the adjusting rod 23 and preventing bolt loosening or rod offset due to downhole vibration. The horizontal azimuth angle is adjusted by the rotation of the adjusting rod 23 within the sleeve 21.

[0026] The pitch angle adjustment component 3 is installed at the top of the height adjustment component 2, specifically at the top of the adjustment rod 23. It includes a U-shaped groove component 31 and a support rod 34. A sliding block 32 is provided at the lower end of the support rod 34. The sliding block 32 is engaged in the U-shaped groove of the U-shaped groove component 31 and can slide along the groove. A fastening screw 33 is provided in the middle of the support rod 34. The fastening screw 33 passes through the U-shaped groove component 31 and the support rod 34 and connects and locks the two together for locking the tilt angle adjustment position. A lidar 4 and a camera component 5 are installed at the top of the support rod 34.

[0027] The lidar 4 is an intrinsically safe lidar for mining, which is fixedly installed on the support rod 34. For example, a mounting base can be set on the support rod. The lidar 4 is set on one side of the mounting base by screws and buffer pads to protect the internal components of the equipment and prevent the equipment from being damaged by vibration.

[0028] The camera assembly 5 includes a first connector 51 and a second connector 52 located at the top of the support rod 34. A spherical camera 53 is installed at the top of the second connector 52. The spherical camera 53 is an intrinsically safe spherical camera for mining. Its positional relationship with the lidar 4 is such that their field of view completely overlaps. The first connector 51 is located at the bottom and has a smaller diameter. The second connector 52 is located at the top and has a larger diameter. Both are disc-shaped components and are fixed by matching bolts. The bottom of the spherical camera 53 is also padded with a buffer pad to enhance the fixation and also to provide a buffering effect.

[0029] System integration and data transmission stability verification: The spherical camera 53 and the lidar 4 were connected to the underground monitoring system via a dedicated mining communication cable. Equipment and system adaptation and debugging were completed to ensure smooth communication. During the test, video images and lidar point cloud data were continuously collected for 72 hours to verify the clarity, frame rate (not less than 25fps), and absence of stuttering or screen tearing in the video images, ensuring the integrity of the point cloud data. The real-time transmission performance of the lidar-visual fusion data was tested to ensure that the fusion data transmission latency was ≤150ms, with no data loss or transmission interruption, meeting the real-time processing requirements of the underground monitoring system for multi-source sensing data.

[0030] The method for identifying illegal activities in coal mines using a combined radar and video surveillance system of the present invention is implemented using the same device and specifically includes: Step 1, LiDAR-Camera Joint Calibration: Calibration involves three types of coordinate system transformations: LiDAR coordinate system {L}, camera coordinate system {C}, and image pixel coordinate system {P}. It is divided into camera intrinsic parameter calibration, LiDAR-camera extrinsic parameter calibration, coordinate system transformation, and accuracy verification. Step 2: The lidar-camera system performs data acquisition and processing; Step 3: Use the fusion confidence score to identify and determine the three violations in the processing results of Step 2. The score is obtained by weighting the four dimensions of visual recognition confidence, radar positioning confidence, spatial matching consistency and historical trajectory continuity.

[0031] The specific process is as follows: I. Radar-Vision Fusion Calibration and Spatiotemporal Data Matching: For working conditions in mines with no GPS signal and confined spaces, a joint calibration method based on feature point triangulation is designed, along with a timestamp alignment strategy, to achieve high-precision spatiotemporal synchronization of point cloud data and image data. LiDAR-camera joint calibration: The calibration involves three types of coordinate system transformations: LiDAR coordinate system {L}, camera coordinate system {C}, and image pixel coordinate system {P}. It is divided into four steps: camera intrinsic parameter calibration, LiDAR-camera extrinsic parameter calibration, coordinate system transformation, and accuracy verification. The camera intrinsic parameter calibration adopts the improved Zhang Zhengyou calibration method, using a 9×7 checkerboard calibration board with a grid side length of 30mm. At least 20 images from different angles are collected under an explosion-proof light source in the well, and the camera intrinsic parameter matrix K is solved. At the same time, the radial and tangential distortion caused by the wide-angle lens is corrected.

[0032] Intrinsic parameter matrix: The lidar coordinate system {L} is a three-dimensional rectangular coordinate system with the lidar optical center as the origin, outputting the three-dimensional coordinates of the point cloud. Camera coordinate system {C}: A three-dimensional rectangular coordinate system with the camera's optical center as the origin; World coordinate system {W}: A local spatial coordinate system for the downhole drilling site, established based on fixed feature points on site, replacing the outdoor GPS coordinate system; Image pixel coordinate system {P}: A two-dimensional image pixel row and column coordinate system, outputting the target pixel coordinates. .

[0033] Camera Intrinsic Parameter Calibration (Improved Zhang Zhengyou Calibration Method): Calibration Equipment: A 9×7 checkerboard calibration board is used, with each square having a side length of 30mm; a stable light source is provided in conjunction with an explosion-proof underground supplemental light to avoid low illumination affecting calibration accuracy. Image Acquisition: Within the underground monitoring area, the calibration board is held handheld, and the distance, angle, and posture are varied to continuously acquire no fewer than 20 valid calibration images, covering the entire field of view of the camera. Intrinsic Parameter Matrix Solution: The camera intrinsic parameter matrix K is calculated as follows: ; In the formula: , These are the equivalent focal lengths (in pixels) in the x and y directions of the image, respectively. , The coordinates of the camera's optical center in the pixel coordinate system.

[0034] Lens distortion correction: Downhole cameras mostly use wide-angle lenses, which have obvious radial and tangential distortion. A distortion model is established to complete pixel coordinate correction and eliminate calibration errors caused by lens distortion.

[0035] Extrinsic parameter calibration (rotation matrix R + translation vector t): At least six high-reflectivity feature control points are set up at the drilling site. The three-dimensional coordinates of these control points in the radar coordinate system and the image pixel coordinates are simultaneously acquired. The extrinsic parameter matrix [R|t] is solved using the PnP algorithm to minimize the reprojection error and optimize the parameters. Projection formula: ; In the formula: s is the scale factor, and p is the pixel coordinate. Let R be the homogeneous coordinates of a radar point in three dimensions, R be the rotation matrix, and t be the translation vector.

[0036] Complete coordinate system transformation: Radar 3D points are converted to image pixel coordinates. ; This represents the depth value of the point in the camera coordinate system.

[0037] Calibration accuracy verification: The accuracy is evaluated using the reprojection error (RPE), which requires RPE < 2.0 pixels. Once the standard is met, all calibration parameters are fixed to the embedded device.

[0038] II. Data Acquisition and Processing by LiDAR-Camera Spatiotemporal data matching: Time matching (timestamp alignment), LiDAR sampling frequency 10Hz (period 100ms), camera frame rate 25-30fps (period 33-40ms). Based on the LiDAR timestamp, image frames with the smallest time difference are selected for pairing, with a synchronization tolerance threshold. =50ms, if the deviation exceeds the limit, linear interpolation is performed on the image frame to ensure frame-level time correspondence. Judgment condition: ; Represents the timestamp corresponding to the LiDAR data collected in the k-th frame. Represents the timestamp corresponding to the acquisition of the j-th frame image by the camera.

[0039] Spatial matching DIoU target association: The radar 3D bounding box is projected onto the image plane through extrinsic parameters. An improved range intersection-union (DIoU) algorithm is used to associate radar targets with visual targets, taking into account bounding box overlap, center point distance, and scale difference. The formula is as follows: + ); In the formula: ρ is the Euclidean distance between the center points, d is the length of the diagonal of the minimum closure, and the scale penalty coefficient α is taken as 0.1 to 0.3; an association threshold is set. =0.3, combined with the Hungarian algorithm to find the optimal match; The projection target bounding box is obtained by projecting the 3D bounding box of the radar point cloud onto the image plane using the camera's intrinsic and extrinsic parameters. Represents: The two-dimensional bounding box of the target in the image directly output by the visual image detection algorithm. Indicates: Represents and The intersection / union area of ​​two bounding boxes measures the degree of overlap between the two boxes, with values ​​ranging from [0,1]. The more overlap there is, the larger the value. Represents: the width (in pixels) of the bounding box of the image after the radar 3D frame is projected. express The width (in pixels) of the bounding box of the visually detected image. Indicates the height (in pixels) of the bounding box of the visually detected image. Indicates the height (in pixels) of the bounding box of the image after the radar 3D frame is projected.

[0040] Multimodal target detection algorithm: It is divided into a visual detection channel and a LiDAR point cloud processing channel, with parallel operation of the two channels, taking into account both semantic recognition and 3D localization capabilities.

[0041] The acquired video images are processed using an improved YOLOv8s-WS target detection model in the visual channel. Two lightweight improvements are made to the standard YOLOv8s model to adapt it to low-light, high-dust, and small-target detection scenarios in wells: Wavelet pooling replaces max pooling: The MaxPool of the backbone network SPPF module is replaced with two-dimensional discrete wavelet transform pooling, which decomposes the feature map into low-frequency approximate components and horizontal / vertical / diagonal high-frequency components. While downsampling, the edges and texture details of small targets such as safety helmets and miners' lamps are fully preserved, solving the problem of information loss in traditional pooling.

[0042] Shuffle Attention (SA) Mechanism Embedded: A lightweight SA attention module is embedded in the three detectors P3 / P4 / P5 to fuse channel attention and spatial attention, suppress interference from complex backgrounds such as coal walls and pipelines, and enhance the extraction of foreground target features. The module adds only 0.05M parameters and does not affect real-time performance.

[0043] Loss function and detection categories, total loss function: ; Indicates the bounding box loss weights. Indicates the loss of complete intersection and union. Represents the classification loss weights. This represents the binary cross-entropy loss. Indicates the distribution focus loss weights. This represents the distribution focus loss.

[0044] The model defines 10 categories of detection targets: personnel, wearing a safety helmet correctly, not wearing a safety helmet, miner's lamp, not wearing a miner's lamp, self-rescue device, not wearing a self-rescue device, protective gloves, work clothes, and safety belt for working at heights, directly distinguishing between compliance and violation status.

[0045] Loss Function Design and Optimization: Addressing the challenges of low illumination, high dust levels, target occlusion, irregular personnel postures, and a high proportion of small targets such as safety helmets and miners' lamps in underground coal mines, this invention performs specific optimizations based on traditional loss functions. It improves the general dual-loss architecture by introducing Distributed Focus Loss (DFL) and constructs a triple composite loss system of CIoU+BCE+DFL, enhancing target localization and classification accuracy from the model training perspective. The overall loss calculation formula is as follows: ; Based on the operational needs of underground violation detection, the weight values ​​were optimized through multiple experiments: (Bounding box loss weights) (Classification loss weights) (Distribution focus loss weights). Functional and scenario adaptation of each loss term: (1) Complete intersection-union loss Taking into account the overlapping area of ​​the detection box, the center offset, and the aspect ratio, it makes up for the shortcomings of the traditional IoU loss in its ability to regress irregular targets, and adapts to the various postures of downhole workers such as bending over, raising their hands, and turning to the side, so as to ensure the fit of the human target box. (2) Binary cross-entropy loss : Used for multi-target, multi-state classification tasks, this model sets up 10 types of detection labels. This loss can effectively distinguish between personnel, various types of labor protection products, and the violation status of wearing / not wearing, reducing the classification false detection rate. (3) Distribution focus loss This is the core optimization module of the present invention. Traditional solutions only predict a single coordinate value, which can easily lead to frame drift when dust, water mist, or partial occlusion of the target cause image blurring. This solution transforms continuous coordinate regression into discrete interval probability distribution prediction, significantly improving the positioning accuracy of small targets such as safety helmets, miners' lamps, and protective gloves; at the same time, it automatically increases the training weight of difficult samples, allowing the model to focus on learning the features of complex underground scenes.

[0046] The weight allocation prioritizes ensuring the accuracy of target positioning, followed by ensuring the accuracy of multi-category classification, making it suitable for fine-grained tasks of recognizing the compliance / violation status of personal protective equipment.

[0047] The YOLOv8s-WS network adopts the classic four-layer architecture of YOLOv8: input layer, backbone network, neck network, and head detection network, which is suitable for downhole image feature extraction and target detection tasks.

[0048] Input layer: Receives RGB images output from explosion-proof cameras in coal mines, performs size normalization, and then sends them to the backbone network. To address issues such as image jitter and dust blurring in underground scenes, a front-end image preprocessing module (frame smoothing, local contrast enhancement) is provided as a pre-optimization step before network input.

[0049] Backbone: The main body retains the native YOLOv8 Conv convolutional layers and C2f feature extraction module. The core improvement is that the traditional max pooling in the terminal SPPF (fast spatial pyramid pooling) module is uniformly replaced with two-dimensional discrete wavelet pooling.

[0050] The YOLOv8s-WS network works by using wavelet pooling, which decomposes the input feature map into four sub-bands through Discrete Wavelet Transform (DWT): low-frequency approximation component XLL, horizontal high-frequency component XLH, vertical high-frequency component XHL, and diagonal high-frequency component XHH. Only the low-frequency component is downsampled, preserving high-frequency detail features. For small targets such as underground safety helmets, miners' lamps, and protective gloves, traditional max pooling loses high-frequency features like edges and textures during downsampling, leading to missed detections of small targets. Wavelet pooling solves this problem. Furthermore, this invention only modifies the SPPF module, unlike the global replacement of the pooling layer in file comparison, accurately locating the optimization area, controlling computational overhead, and adapting to embedded edge devices used in mining.

[0051] Neck Network: Employs a dual-fusion structure of FPN (Feature Pyramid Network) + PAN (Path Convergence Network), including convolutional, upsampling, and feature concatenation units. It fuses and enhances the multi-scale features output from the backbone network, generating three different resolution feature maps: P3 (80×80): small-scale feature map, corresponding to extremely small targets such as protective gloves and miners' lamps; P4 (40×4): medium-scale feature map, corresponding to the human torso and work clothes; P5 (20×4): large-scale feature map, corresponding to a complete worker. Head Detection Network: Embeds a mixed-attention (SA) module within all three detection heads (P3, P4, and P5), unlike the design in the comparison file which only deploys it in a single P5 head.

[0052] The SA module's internal workflow involves feature grouping, channel attention calculation, feature shuffling, spatial attention weighting, and feature fusion output. Channel attention filters effective feature channels, while spatial attention focuses on the target region. This dual mechanism suppresses interference from complex backgrounds such as underground coal walls, metal supports, drilling rigs, and pipelines. Underground targets are distributed across the full-scale feature map. The SA module is deployed across all detection heads, enabling layered suppression of background noise at different scales and comprehensively improving the recognition accuracy of various targets. The detection head simultaneously outputs the target category, bounding box coordinates, and confidence score, and is equipped with a triple loss function (CIoU+BCE+DFL) for training optimization.

[0053] Combination Figure 2-4This study verifies the rationality, training stability, and generalization ability of the YOLOv8s-WS network structure and triple loss function. The model employs a composite loss function: LCIoU (Bounding Box Loss): constrains the position and size accuracy of the detection boxes; LBCE (Classification Loss): constrains the accuracy of target category and violation status recognition; LDFL (Distributed Focus Loss): a new module in this invention, optimizing the box regression for occlusion, blurriness, and small targets. After 300 epochs of training, the three loss values ​​were initially high, but decreased continuously with increasing iterations, gradually stabilizing after 200 epochs, indicating model convergence. The DFL loss curve shows stable convergence, proving that the newly added distributed focus loss function works effectively, optimizing the positioning accuracy of irregular human bodies and small protective equipment in underground mines. Combination Figure 8 The model performance iteration curve includes precision and recall. Precision and recall rise rapidly in the early stage of training, and stabilize and remain at high values ​​after 200 rounds. Ultimately, both indicators are at high levels, indicating that the improved model effectively controls false positives and false negatives in complex underground scenarios. Combined with the underground violation detection scenario, low false positives and low false negatives can avoid invalid alarms and missed hazards, meeting the hard requirements of coal mine safety production management.

[0054] Combination Figure 6 The YOLOv8s-WS model's mAP variation curve and average accuracy iteration curve, including mAP@0.5 and mAP@0.95, show that both curves rise synchronously and stabilize in the later stages, with mAP@0.5 eventually reaching approximately 82.2%. The high IoU performance is excellent, proving that YOLOv8s-WS has excellent detection capabilities in the single visual dimension.

[0055] LiDAR Channel: Point Cloud Processing and Trajectory Tracking. Performs full-process processing on point cloud data, outputting the 3D position, dimensions, and motion trajectory of personnel. Process: Pass-through filtering, voxel downsampling, RANSAC ground point removal, DBSCAN-CFSFDP two-stage clustering, 3D bounding box calculation, and Kalman filter trajectory tracking.

[0056] Preprocessing: The monitoring area was limited by direct filtering, and the voxel grid was downsampled to 0.05m; RANSAC plane fitting was used to remove ground points with a distance threshold of 0.1m.

[0057] Two-stage clustering: The first stage is DBSCAN coarse clustering (neighborhood radius 0.5m, minimum number of points 5); the second stage is CFSFDP fine clustering (cutoff distance 0.3m), which selects cluster centers to distinguish between personnel and equipment, coal blocks and other interfering objects.

[0058] Trajectory tracking: Using Kalman filtering, with state vector X=[x,y,z,vx,vy,vz]T, the system calculates the three-dimensional centroid, velocity, and dwell time of the person in real time, and outputs a continuous motion trajectory.

[0059] Combination Figure 6 The mAP variation curve of the radar-visual fusion model, compared with the pure vision YOLOv8s-WS, shows that the radar-visual fusion curve is higher than the pure vision curve throughout the process, and the final mAP of the fusion solution is stable at 92.1%. The lidar is not affected by light or dust and can output stable three-dimensional position information, making up for the shortcomings of vision in harsh environments. The decision-level multimodal fusion architecture can realize the complementary advantages of visual semantics and radar spatial information, and continuously improve the overall accuracy of the system.

[0060] Combination Figure 5 and 6 The precision and recall curves of the radar-vision fusion model were compared with those of the native YOLOv8s, pure vision YOLOv8s-WS, and the radar-vision fusion solution. The radar-vision fusion solution outperformed the other two solutions, achieving a final precision of 97.7% and a recall of 93.3%, with optimal control over false positives and false negatives. The fusion system can still stably detect targets even in scenarios with dense personnel, mutual obstruction, and high dust levels in underground mines.

[0061] The test site was used for underground gas extraction / water exploration drilling in a coal mine, covering four typical working conditions: normal illumination, low illumination, ordinary dust, and high dust. The testing equipment included the integrated laser-visual anti-vibration support and a laser radar + explosion-proof camera. A total of 5000 sets of images and point cloud data were collected on-site, including samples of single-person, multi-person, occlusion, and compliance / violation of work safety attire. mAP@0.5: Original model 76.3% → Improved single-vision 82.2% → Laser-visual fusion 92.1%, a cumulative improvement of 15.2%, significantly enhancing overall detection capabilities. Precision and recall rates both exceeded 90% in the fusion solution, significantly reducing the risk of false alarms and missed alarms related to violations. The fusion solution achieved a frame rate of 30 FPS. Finally, the results show that the algorithm improvement and multimodal fusion dual empowerment resulted in a step-by-step improvement in the system's overall performance. See Table 1 for details. Figure 7 and 8 As shown.

[0062] Table 1 Performance Comparison of Drilling Site Target Detection Algorithms

[0063] Table 2 Comparison of Core Indicators

[0064] As shown in Table 2, the accuracy under extreme working conditions is as follows: in the worst underground environment (illuminance ≤5 lux, high dust), the traditional solution achieves only 41.2%, while this solution reaches 79.5%, demonstrating environmental robustness as a core technological barrier. Regarding target tracking rate, relying on the improved DIoU association algorithm + DBSCAN-CFSFDP point cloud clustering, there is no ID confusion in densely populated multi-person scenarios, improving tracking stability. The 92.9% detection rate of violations effectively prevents safety accidents and meets the core needs of coal mine safety supervision. The alarm delay of 0.96s provides a second-level response, enabling immediate warnings of violations. The practical application of the integrated vibration-resistant, dust-proof, and multi-adjustable support system has been verified, demonstrating the practical application of hardware innovation. Finally, this invention achieves a complete technological upgrade across the entire chain, from hardware structure and calibration algorithm to visual model, point cloud clustering, multimodal fusion, and behavior determination, offering comprehensive advantages over traditional solutions.

[0065] III. Multi-dimensional Fusion Confidence Calculation Model Fusion confidence score is a core indicator for determining target validity and behavioral credibility. It is calculated by weighting four dimensions: visual recognition credibility, radar positioning credibility, spatial matching consistency, and historical trajectory continuity, thus avoiding misjudgments caused by the failure of a single sensor. The fusion confidence score calculation formula is as follows: ; =0.40, visual confidence weight: This represents the target detection confidence score output by the YOLOv8s-WS model. The visual channel is responsible for semantic recognition of personal protective equipment and wearing status, and is the core basis for determining violations; therefore, it is given the highest weight. Radar confidence weights: The calculation is based on a combination of personnel point cloud density, cluster integrity, and trajectory smoothness. Radar, unaffected by lighting or dust, is responsible for 3D positioning and motion state assessment, serving as a crucial auxiliary basis. =0.20, Spatial matching weight: The bounding box matching value reflects the spatial consistency between the 2D image and the 3D point cloud. The higher the matching degree, the stronger the credibility that the target is the same person.

[0066] Trajectory consistency weight: The success rate of target association across multiple consecutive frames is denoted by [0,1], which represents the stability of target tracking and filters out false targets detected momentarily. Weight constraints: All weights have been measured and optimized in multiple downhole scenarios to adapt to all working conditions at the drilling site.

[0067] Confidence level assessment rules: Based on the fusion confidence level, three judgment levels are established to distinguish between valid targets, suspected targets, and invalid targets, and differentiating treatment is applied accordingly. Highly credible targets: With high matching accuracy and stable recognition results from dual channels, the detection results are directly accepted and the process for judging violations is initiated. Suspected Target: For targets in scenes with obstruction, distance, low light, and high dust, violations are not immediately determined. The results are output only after three consecutive frames are checked to reduce the false alarm rate. Invalid target: If it is determined to be noise, reflective interference, or equipment false detection, it will be directly rejected and will not be included in the subsequent behavior judgment.

[0068] Secondary fusion analysis of unmatched targets: For single-channel targets detected only by radar or only by vision, supplementary analysis rules are established: Detected by LiDAR only: Vision missed detection due to obstruction, strong light, or dense dust. Combining radar trajectory and regional location judgment, if the target remains in the work area for an extended period and its movement characteristics match those of a person, it is temporarily marked as a suspected person and continuously tracked. Detected by vision only: Radar failed to cluster the personnel due to them being too close to the equipment or insufficient point cloud data. Combining continuous multi-frame visual recognition results, if the category is stable, the visual wear status judgment is adopted.

[0069] (19) Typical three violations judgment rules, combined with coal mine safety regulations, divide into four categories of violations, the judgment logic is clear, quantifiable and can be automatically executed: Incomplete wearing of labor protection equipment (high frequency three violations) If a personnel target is detected, but any one of the safety helmet, mine lamp, self-rescue device, protective gloves or work clothes is not worn properly, it is directly judged as a violation; Binary alarm code is generated according to the missing item: missing safety helmet (0x01), missing mine lamp (0x02), missing self-rescue device (0x04), missing protective gloves (0x08), and multiple missing items are combined by bit OR operation code.

[0070] Dangerous area intrusion: Dangerous areas such as drilling rig operation area, belt conveyor area, and unsupported area are pre-demarcated; if the three-dimensional centroid of personnel falls into the dangerous area and the distance from the boundary exceeds the intrusion threshold, it is judged as a violation of dangerous area intrusion.

[0071] Alarm linkage and data upload: Alarm information format: Generates a standard JSON message, including alarm number, timestamp, on-site location, violation type, personnel 3D coordinates, fusion confidence level, and on-site captured screenshot; On-site linkage: Triggers underground audible and visual alarms and voice broadcasts, providing immediate on-site warnings to workers; Remote upload: Uploads alarm data, video, and trajectory data to the surface dispatch platform in real time via the underground industrial Ethernet ring network, enabling remote monitoring, data storage, and accident tracing.

[0072] Technical Effects and Experimental Data Verification: This invention was tested in downhole field trials, examining hardware structure performance, algorithm recognition accuracy, environmental robustness, system real-time performance, and violation detection rate. All indicators outperformed traditional solutions. Multi-dimensional Adjustment Capability: It enables full adjustment in lateral, height, 360° horizontal, and 0°~90° pitch, completely eliminating blind spots in drilling site monitoring. Vibration Resistance and Stability: Under long-term strong vibration conditions, the equipment's line of sight remained unchanged, the structure remained stable, and image and point cloud data remained consistently stable. Protection Capability: The equipment operated continuously for months in high-dust and high-humidity environments without jamming, corrosion, or malfunctions. Installation and Maintenance: The modular structure and tool-free locking allow for installation and commissioning by a single person, improving maintenance efficiency by over 60%.

[0073] At underground coal mine construction sites, the NVC fusion recognition system was used for verification. The system draws target detection boxes in real time, labeling categories such as personnel, safety helmets, miner's lamps, self-rescue devices, and protective gloves, while also distinguishing between compliant and non-compliant wearing (three violations). The overall average recognition accuracy of the entire solution for personal protective equipment (PPE) reached 90.7%, an improvement of 9.1% compared to a purely visual solution; the largest improvement was seen for smaller targets such as miner's lamps and protective gloves, with the highest improvement reaching 11.8%.

[0074] Target association and violation detection indicators: DIoU target tracking rate 92.6%, correct tracking rate 95.8%, no ID confusion in multi-person dense scenarios; comprehensive detection rate of violations 92.9%, overall false alarm rate only 3.8%, average detection latency 0.96s, meeting the requirements for real-time early warning.

[0075] This invention addresses the complex working conditions of underground coal mine drilling sites by integrating a multi-dimensional vibration-resistant and protective fixing structure, a dedicated underground radar-visual joint calibration system, an improved deep learning model, an optimized point cloud clustering algorithm, decision-level multimodal fusion, and a graded system for determining violations. After on-site deployment and multi-condition comparative testing at underground drilling sites, the overall technical effectiveness is as follows: Integrated fixed structure technology benefits: Comprehensive, blind-spot-free monitoring. The support bracket supports four adjustment capabilities: lateral sliding of the drilling rig, height adjustment from 0 to 500 mm, 360° horizontal rotation, and pitch angle adjustment from 0° to 90°. Combined with standard installation heights of 2.5–3.5 m and monitoring tilt angles of 15°–30°, the monitoring angle can be flexibly adjusted according to different drilling rig models, drilling site layouts, and operating conditions, completely eliminating traditional blind spots such as those on the sides, bottom, and rear of the drilling rig, achieving full coverage of the drilling site. Strong vibration resistance and posture stability: The entire structure is made of 304 stainless steel, featuring a three-stage vibration reduction structure including base washers, height adjustment rubber pads, and equipment buffer pads, along with all-point anti-loosening bolts and a butterfly-shaped tool-free locking mechanism. Under long-term high-frequency vibration conditions of the drilling rig, the equipment's line of sight remains unchanged, the support bracket remains stable without deformation, the lidar point cloud remains distortion-free, and the camera image remains jitter-free, ensuring long-term stability of the sensed data. Long-term coaxial accuracy of the laser radar: The laser radar and the explosion-proof camera are rigidly connected via a dedicated adapter disk, strictly ensuring coaxial optical centers, parallel front planes, and overlapping fields of view. Unlike split mounting brackets, this structure avoids relative displacement caused by vibration from the hardware source, ensuring long-term stability of the laser radar's spatial registration benchmark and eliminating the need for frequent secondary calibration.

[0076] Ease of installation and maintenance: The slotted guide rails are compatible with the drilling rig frame, eliminating the need for on-site drilling. The modular structure, combined with a tool-less locking design, allows a single person to complete installation, attitude adjustment, equipment inspection, and replacement. Installation and commissioning time is reduced by more than 60% compared to traditional supports, making it suitable for confined downhole working spaces and rapid construction needs. Furthermore, the standardized hole design ensures compatibility with different specifications of lidar and explosion-proof cameras, offering strong versatility.

[0077] The combined calibration and spatiotemporal synchronization technology of the radar and vision system has the following effect: By adopting the improved Zhang Zhengyou calibration method and the on-site feature point PnP calibration technology, calibration can be completed using only a small checkerboard grid and the inherent corner points and high-reflection targets on site. It is not limited by downhole space or signal, and can quickly carry out calibration work in any drilling site or roadway.

[0078] Precise and reliable spatiotemporal synchronization: Employing a timestamp alignment strategy with a 50ms time tolerance, coupled with linear interpolation compensation, it perfectly adapts to the different frequency sampling characteristics of LiDAR at 10Hz and cameras at 25~30fps, achieving a frame-level temporal matching accuracy close to 100%. Based on the improved DIoU spatial association algorithm, the target tracking rate reaches 92.6% and the correct tracking rate reaches 95.8%, effectively solving the ID confusion problem under dense crowds and target occlusion, laying a solid foundation for multimodal fusion.

[0079] Based on YOLOv8s, wavelet pooling and mixed-wash attention mechanisms are integrated to optimize the detection capabilities of small targets in low-light, high-dust, and downhole environments, while balancing accuracy and real-time performance of embedded devices.

[0080] Combination Figure 2-4 The YOLOv8s-WS network structure replaces traditional max pooling with wavelet pooling, fully preserving the edge and texture features of small targets smaller than 16×16 pixels, such as safety helmets, miner's lamps, and protective gloves. Real-world testing data shows that the detection accuracy for extremely small targets like miner's lamps and protective gloves is improved by 22.7% and 21.0% respectively, completely resolving the problem of large-area missed detections of small targets in pure vision solutions. A lightweight mixed-wash attention mechanism is embedded to automatically suppress background interference from coal walls, metal supports, pipelines, etc., significantly reducing the model's false detection rate. Compared to native YOLOv8s, YOLOv8s-WS's mAP@0.5 increases from 76.3% to 82.2%, with simultaneous improvements in precision and recall. The total number of model parameters only increases from 11.2M to 11.6M, with only 0.05M new parameters added, and the inference frame rate only slightly decreases, achieving a single-path inference frame rate of 30FPS. While improving detection accuracy, it fully meets the real-time operation requirements of underground embedded equipment.

[0081] Improved DBSCAN-CFSFDP point cloud clustering and trajectory tracking technology: A two-stage combined clustering algorithm is adopted, combined with Kalman filter trajectory tracking, to solve the problems of sparse point clouds, high noise, and easy target adhesion in underground mines, and to achieve accurate three-dimensional positioning and motion state perception of personnel.

[0082] Decision-level multimodal fusion and violation judgment system technology: Integrating visual semantic recognition capabilities with radar all-weather detection and 3D positioning capabilities, the overall system mAP@0.5 reaches 91.5%, an improvement of 9.3 percentage points compared to the pure YOLOv8s-WS visual solution; the comprehensive recognition accuracy of personal protective equipment wear reaches 90.7%. Under the most severe working conditions of extremely low illumination and high dust, the pure visual detection accuracy is only 41.2%, while this fusion solution can still maintain a recognition accuracy of 79.5%, demonstrating extremely significant environmental robustness advantages.

[0083] The judgment rules are standardized and automated. All violations are judged using quantitative thresholds and logical judgments, requiring no manual intervention. It can automatically distinguish between various violations such as incomplete protective equipment, area intrusion, and abnormal movement, and generate standardized codes. At the same time, it sets up a three-level warning system to distinguish between high-risk, routine, and warning risks, which facilitates differentiated handling by the dispatch center.

[0084] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0085] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0086] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A coal mine underground radar-visual fusion device, characterized in that, Set a base (1), and set a height adjustment component (2) vertically along the upper edge of the base (1). Set a pitch angle adjustment component (3) at the top of the height adjustment component (2). Install a laser radar (4) and a camera component (5) on it in sequence. Adjust the height, horizontal azimuth angle and pitch angle of the laser radar (4) and the camera component (5) set at the top by setting the height adjustment component (2) and the pitch angle adjustment component (3).

2. The underground radar-visual fusion device for coal mines according to claim 1, characterized in that, The height adjustment component (2) is provided with a sleeve (21) and an adjustment rod (23) inserted on the sleeve (21). The side wall of the adjustment rod (23) is provided with multiple adjustment grooves (231). Adjustment screws (22) are installed on the side wall of the sleeve (21) corresponding to the adjustment grooves (231). The adjustment rod (23) rotates inside the sleeve (21) to adjust the horizontal orientation angle.

3. The underground radar-visual fusion device for coal mines according to claim 1 or 2, characterized in that, The pitch angle adjustment component (3) is installed at the top of the height adjustment component (2), including a U-shaped groove component (31) and a support rod (34). A sliding block (32) is provided at the lower end of the support rod (34), and the sliding block (32) is locked on the U-shaped groove of the U-shaped groove component (31). A fastening screw (33) is provided in the middle of the support rod (34), and a laser radar (4) and a camera component (5) are installed at the top of the support rod (34).

4. The underground radar-visual fusion device for coal mines according to claim 1 or 2, characterized in that, The laser radar (4) is an intrinsically safe laser radar for mining and is fixedly installed on the support rod (34).

5. The underground radar-visual fusion device for coal mines according to claim 1 or 2, characterized in that, The camera assembly (5) includes a first connector (51) and a second connector (52) disposed on the top of the support rod (34), and a spherical camera (53) is disposed on the top of the second connector (52). Both the first connector (51) and the second connector (52) are disc-shaped components; The first connector (51) is located at the bottom, and the second connector (52) is located at the top. The diameter of the second connector (52) is larger than the diameter of the first connector (51).

6. A method for identifying illegal operations in coal mines using a combination of radar and video surveillance, characterized in that... The coal mine underground radar-visual fusion device according to any one of claims 1-5 is used, specifically including: Step 1, LiDAR-Camera Joint Calibration: Calibration involves three types of coordinate system transformations: LiDAR coordinate system {L}, camera coordinate system {C}, and image pixel coordinate system {P}. It is divided into camera intrinsic parameter calibration, LiDAR-camera extrinsic parameter calibration, coordinate system transformation, and accuracy verification. Step 2: The lidar-camera system performs data acquisition and processing; Step 3: Use the fusion confidence score to identify and determine the three violations in the processing results of Step 2. The score is obtained by weighting the four dimensions of visual recognition confidence, radar positioning confidence, spatial matching consistency and historical trajectory continuity.

7. The method for identifying illegal operations in coal mines using a combination of radar and video surveillance, as described in claim 6, is characterized in that... Step one specifically includes: Camera intrinsic parameter calibration: The lidar coordinate system {L} is a three-dimensional rectangular coordinate system with the lidar optical center as the origin, outputting the three-dimensional coordinates of the point cloud. Camera coordinate system {C}: A three-dimensional rectangular coordinate system with the camera's optical center as the origin; World coordinate system {W}: A local spatial coordinate system for the downhole drilling site, established based on fixed feature points on site, replacing the outdoor GPS coordinate system; Image pixel coordinate system {P}: A two-dimensional image pixel row and column coordinate system, outputting the target pixel coordinates. ; Solving for the intrinsic parameter matrix: The camera intrinsic parameter matrix K is calculated as follows: ； In the formula: , These are the equivalent focal lengths (in pixels) in the x and y directions of the image, respectively. , These are the coordinates of the camera's optical center in the pixel coordinate system. Radar-camera extrinsic parameter calibration: At least six high-reflectivity feature control points are deployed at the drilling site. The three-dimensional coordinates of these control points in the radar coordinate system and the image pixel coordinates are simultaneously acquired. The extrinsic parameter matrix [R|t] is solved using the PnP algorithm to minimize the reprojection error and optimize the parameters. The projection formula is: ； In the formula: s is the scale factor, and p is the pixel coordinate. Let R be the homogeneous coordinates of the radar's three-dimensional point, R be the rotation matrix, and t be the translation vector. Coordinate system transformation: Converting radar 3D points to image pixel coordinates: ； This represents the depth value of the point in the camera coordinate system. Calibration accuracy verification: The accuracy is evaluated using the reprojection error (RPE), which requires RPE < 2.0 pixels. Once the standard is met, all calibration parameters are fixed to the embedded device.

8. The method for identifying illegal activities in coal mines using a combination of radar and video surveillance, as described in claim 6 or 7, is characterized in that... Step two specifically includes: The lidar sampling frequency is 10Hz, and the camera frame rate is 25-30fps; based on the lidar timestamp, image frames with the smallest time difference are selected for pairing, with a synchronization tolerance threshold. =50ms; if the deviation exceeds the limit, linear interpolation is performed on the image frame to ensure frame-level time correspondence. Judgment conditions: ； Represents: the timestamp corresponding to the LiDAR data collected in the k-th frame; Represents: the timestamp corresponding to the acquisition of the j-th frame of the camera image; Spatial matching DIoU target association: The radar 3D bounding box is projected onto the image plane through extrinsic parameters. An improved range intersection-union (DIoU) algorithm is used to associate radar targets with visual targets, taking into account bounding box overlap, center point distance, and scale difference. The formula is as follows: + )； In the formula: ρ is the Euclidean distance of the center point, d is the length of the diagonal of the minimum closure, and the scale penalty coefficient α is taken as 0.1 to 0.3; The projected target bounding box is obtained by projecting the 3D bounding box of the radar point cloud onto the image plane using the camera's intrinsic and extrinsic parameters. The bounding box of the target in the image is directly output by the visual image detection algorithm. represent and The intersection / union area of ​​two bounding boxes measures the degree of overlap between the two boxes, with a value of [0,1]. This represents the width of the bounding box of the image after the radar 3D frame is projected. The width of the bounding box of the visually detected image; The height of the bounding box in the visually detected image; The height of the bounding box of the image after the radar 3D frame is projected. The acquired video images were processed using an improved YOLOv8s-WS object detection model in the visual channel. The overall loss calculation formula is as follows: ； Based on the operational needs of underground violation detection, the weight values ​​were optimized through multiple experiments: Bounding box loss weights, Classification loss weights, Distributed focal loss weights; Perfect intersection-union loss Binary cross-entropy loss Distribution focal loss .

9. The method for identifying illegal operations in coal mines using a combination of radar and video surveillance, as described in claim 8, is characterized in that... The YOLOv8s-WS network architecture consists of an input layer, a backbone network, a neck network, and a head detection network, which is adapted to downhole image feature extraction and target detection tasks. Input layer: Receives RGB images output from explosion-proof cameras in underground coal mines, performs size normalization, and then sends them to the backbone network; Backbone Network: The main body retains the native YOLOv8 Conv convolutional layers and C2f feature extraction module, and replaces the traditional max pooling in the terminal SPPF module with two-dimensional discrete wavelet pooling. Wavelet pooling decomposes the input feature map into four types of subbands through discrete wavelet transform: low-frequency approximate component XLL, horizontal high frequency XLH, vertical high frequency XHL, and diagonal high frequency XHH. Only the low-frequency component is downsampled, and the high-frequency detailed features are fully preserved. Neck network: adopts FPN+PAN dual fusion structure, including convolution, upsampling and feature concatenation units; the multi-scale features output by the backbone network are fused and enhanced to generate three types of feature maps with different resolutions: P3: 80×80 small-scale feature map; P4: 40×4 mesoscale feature map; P5: 20×4 large-scale feature map; Head detection network: A mixed-shuffle attention module is embedded inside all three detection heads, P3, P4, and P5; The internal process of the shuffling attention module is as follows: feature grouping → channel attention calculation → feature shuffling → spatial attention weighting → feature fusion output; the detection head simultaneously outputs the target category, detection box coordinates, and confidence score, and completes training optimization with the CIoU+BCE+DFL triple loss function.

10. The method for identifying illegal activities in coal mines using a combination of radar and video surveillance, as described in claim 6 or 7, is characterized in that... Step three specifically includes: ； For visual confidence weights, The target detection confidence score output by the YOLOv8s-WS model. For radar confidence weights, The calculation is based on a combination of personnel point cloud density, cluster integrity, and trajectory smoothness. For spatial matching weights, Match values ​​to the target bounding box. For trajectory consistency weight, To improve the success rate of target association across multiple consecutive frames; Weight constraints: All weights have been measured and optimized in multiple downhole scenarios to adapt to all working conditions at the drilling site. Confidence level assessment rules: Based on the fusion confidence level, three judgment levels are established to distinguish between valid targets, suspected targets, and invalid targets, and differentiating treatment is applied accordingly. Highly credible targets: With high matching accuracy and stable recognition results from dual channels, the detection results are directly accepted and the process for determining violations is initiated. Suspected Target: For targets in scenes with obstruction, distance, low light, and high dust, violations are not immediately determined. The results are output only after three consecutive frames are checked to reduce the false alarm rate. Invalid target: If it is determined to be noise, reflective interference, or equipment false detection, it will be directly rejected and will not be included in the subsequent behavior judgment. Secondary fusion analysis of unmatched targets: For single-channel targets detected only by radar or only by vision, supplementary analysis rules are established: Detected by LiDAR only: Vision may miss detection due to obstruction, strong light, or dense dust. If the target remains in the work area for an extended period and its movement characteristics match those of a person, it will be temporarily marked as a suspected person and continuously tracked. Detected by vision only: If radar fails to cluster the personnel due to them being too close to the equipment or insufficient point cloud data, the visual wear status will be accepted if the category is stable and multiple frames of visual recognition results are combined. Typical violation judgment rules: If a person is detected as not wearing complete personal protective equipment, but any one of the following is not worn properly: safety helmet, miner's lamp, self-rescue device, protective gloves, or work clothes, it is directly judged as a violation; a binary alarm code is generated based on the missing item: missing safety helmet 0x01, missing miner's lamp 0x02, missing self-rescue device 0x04, missing protective gloves 0x08, and multiple missing items are combined by bitwise OR operation; Dangerous area intrusion: Dangerous areas such as drilling rig operation area, belt conveyor area, and unsupported area are pre-demarcated; if the three-dimensional centroid of personnel falls into the dangerous area and the distance from the boundary exceeds the intrusion threshold, it is judged as a violation of dangerous area intrusion. Alarm linkage and data upload: Alarm information format: Generates a standard JSON message, including alarm number, timestamp, on-site location, violation type, personnel 3D coordinates, fusion confidence level, and on-site captured screenshot; On-site linkage: Triggers underground audible and visual alarms and voice broadcasts, providing immediate on-site alerts to workers; Remote upload: Uploads alarm data, video, and trajectory data to the surface dispatch platform in real time via the underground industrial Ethernet ring network, enabling remote monitoring, data storage, and accident tracing.

Images