A laser and vision-based system and method for detecting misalignment in coal mine vertical shaft guideways.

Abstract

This invention belongs to the field of safety monitoring and fault diagnosis technology for mining equipment. Specifically, it discloses a laser and vision-based system and method for detecting misalignment in coal mine vertical shaft cages. The system includes a linear laser emitter, a high-speed industrial camera, an adaptive lighting unit, an inertial measurement unit, and an edge computing server. The edge computing server receives the image stream acquired by the high-speed industrial camera and the cage attitude information obtained by the inertial measurement unit. It uses the data from the inertial measurement unit to perform spatial consistency correction on the visual geometric features and outputs the cage misalignment amount after decoupling from cage vibration interference. The system adopts a technical path of establishing an absolute geometric benchmark through active laser projection, capturing dynamic scenes with high-speed vision, and intelligent decoupling and decision-making through a deep learning model. This effectively isolates cage vibration interference, improves the imaging quality of the joint area, and integrates multimodal features for accurate judgment. Ultimately, it achieves stable and accurate identification of 2mm-level cage misalignment in a strong vibration environment.

Description

Technical Field

[0001] This invention belongs to the field of safety monitoring and fault diagnosis technology for mining equipment, and specifically discloses a misalignment detection system and method for coal mine vertical shaft guideways based on laser and vision. Background Technology

[0002] The integrity of the geometry of the hoisting cage in a coal mine vertical shaft directly affects the safety of the entire hoisting system. The hoisting cage is composed of multiple sections of steel rails fixed together by connectors. Under long-term alternating loads, corrosion, and potential impacts, minor misalignments, deformations, or wear can easily occur at the joints. These initial, millimeter-level defects are potential precursors to major safety accidents such as cage jamming or falling, and must be identified at an early stage. Currently, existing detection methods mainly consist of periodic manual inspections and automated inspections based on traditional machine vision.

[0003] Manual periodic inspections utilize tools such as rulers and feeler gauges for measurement, relying on the experience of maintenance personnel. This method is inefficient, has a long inspection cycle, cannot achieve full wellbore coverage, and is limited by subjective factors and the wellbore environment, making it difficult to detect and quantify millimeter-level gradual defects. Automated inspection based on traditional machine vision uses industrial cameras to capture images of the cage passageway, analyzing the passageway contour through edge detection and template matching algorithms. However, this method faces fundamental challenges in practical applications in vertical shafts: 1. Poor image quality: Multidimensional vibrations generated during high-speed cage operation lead to severe motion blur in images; uneven lighting and dust interference within the wellbore result in low image contrast and high noise; the texture features of the passageway joints are inherently weak, making clear imaging even more difficult under blurry and low-light conditions. 2. Insufficient algorithm robustness: Traditional image processing algorithms are extremely sensitive to the aforementioned image degradation. Slight jitter or changes in lighting can lead to edge detection failures or mismatched feature points, generating numerous false alarms that misjudge normal joints, oil stains, and shadows as misalignments, and missed alarms due to the inability to identify genuine minute misalignments. 3. Limited accuracy: Due to limitations in pixel resolution and feature extraction methods, traditional methods are unable to achieve quantitative detection accuracy better than 5mm, which cannot meet the early warning requirements for safety hazards at the 2mm level.

[0004] Therefore, there is an urgent need to design a method for sensing the health status of coal mine shafts with sub-millimeter accuracy, high reliability, and real-time online operation under the harsh working conditions of coal mine shafts. Summary of the Invention

[0005] To address the problems in the background technology, this invention discloses a laser and vision-based system and method for detecting misalignment in coal mine vertical shaft cages. The system employs a technical approach that combines active laser projection to establish an absolute geometric benchmark, high-speed vision to capture dynamic scenes, and deep learning models for intelligent decoupling and decision-making. This approach effectively isolates cage vibration interference, improves imaging quality in joint areas, and integrates multimodal features for accurate judgment. Ultimately, it achieves stable and accurate identification of 2mm-level cage misalignment under strong vibration environments.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: A laser and vision-based misalignment detection system for coal mine vertical shaft cages includes a linear laser emitter mounted on the cage, a high-speed industrial camera, an adaptive lighting unit, an inertial measurement unit, and an edge computing server. The laser emitter projects bright, fine linear light stripes onto the cage surface. The high-speed industrial camera acquires image sequences containing both the natural texture of the cage and the laser stripes superimposed on it, at a frame rate higher than the cage's vibration frequency. The adaptive illumination unit adjusts the brightness based on ambient light feedback to ensure uniform illumination in the target area; the inertial measurement unit acquires the cage's attitude information; the edge computing server receives the image stream acquired by the high-speed industrial camera and the cage's attitude information acquired by the inertial measurement unit, uses the inertial measurement unit data to perform spatial consistency correction on the visual geometric features, and outputs the cage track fusion misalignment amount after decoupling from cage vibration interference. .

[0007] A method for detecting misalignment of coal mine vertical shaft guideways based on laser and vision, utilizing the coal mine vertical shaft guideway misalignment detection system based on laser and vision as described in claim 1, specifically includes the following steps: S1: Multi-source collaborative sensing data acquisition: A laser projects a beam of bright, fine, straight light stripes onto the surface of the cage. A high-speed camera acquires an image sequence containing both the natural texture of the cage and the laser stripes superimposed on it, at a frame rate higher than the dominant vibration frequency of the cage. The supplementary lighting unit adjusts its brightness based on ambient light feedback to ensure uniform illumination of the target area; S2: Image stabilization and enhancement preprocessing based on a differentiable physical model: preprocessing the jitter-blurred original image sequence acquired in S1. Input a pre-trained image stabilization and enhancement fusion model to obtain stable features that reflect the consistency of the image's geometric structure. and enhanced features Regarding the aforementioned stable features With enhanced features The system performs fusion processing to generate fused feature representations, which are then input into the decoding and reconstruction module. To obtain a clear image : ; S3: Laser line subpixel extraction and seam area positioning; S4: Intelligent misalignment quantification and determination based on multi-feature fusion, outputting the cage track fused misalignment amount decoupled from cage vibration interference. and will The system is compared with a preset threshold to determine whether the tank passage is in a normal, warning, or alarm state. S5: Model training and system deployment. The two deep learning models mentioned above are trained using labeled datasets collected in the experimental platform and real wellbore, covering different levels of vibration, illumination and misalignment.

[0008] Furthermore, the specific steps of step S2 in the laser and vision-based coal mine vertical shaft misalignment detection method are as follows: S2.1 Physical Model Construction: Modeling inter-frame motion as a dense optical flow field V t ,in , representing a pixel The displacement vector from frame t to the reference frame; S2.2 Differentiable Motion Compensation and Stabilized Image Generation: Based on the Dense Optical Flow Field V t Using a differentiable bilinear sampling operator For the current frame image Pixel-level remapping is performed to obtain a stable image that has been initially freed from the effects of jitter. Since the bilinear sampling operator is a differentiable operator, the model can adjust the motion estimation results in reverse according to the error information of the output results during the training phase, thereby realizing the joint optimization of motion modeling and subsequent image processing. S2.3 Stabilizing and Enhancing Feature Extraction: The model includes a stabilizing feature extraction module and an enhancing feature extraction module, stabilizing the image. The input is fed into the stable feature extraction module, which extracts stable features that reflect the consistency of the image's geometric structure. Meanwhile, the original input image The input is fed into the enhanced feature extraction module for feature encoding and reconstruction processing, in order to complete noise suppression, contrast adjustment, and detail information recovery within the feature domain, thereby obtaining enhanced features. ; S2.4 Multi-scale Feature Fusion and Image Reconstruction: The multi-scale feature fusion module performs image reconstruction on the stable features... With enhanced features The system performs fusion processing to generate fused feature representations, which are then input into the decoding and reconstruction module. To obtain a clear image : ; During the training phase, by minimizing the sharp image With corresponding clear truth image The model is optimized using a loss function that includes at least pixel-level L1 loss and perceptual loss to improve the overall performance of the reconstructed image in terms of structural consistency and subjective visual quality.

[0009] Furthermore, in the laser and vision-based method for detecting misalignment in coal mine vertical shaft guideways, step S3 involves performing laser line sub-pixel extraction and joint area localization operations on the clear image obtained in step S2. The specific steps are as follows: Sub-pixel extraction of S3.1 laser lines: First, based on the brightness threshold and gradient magnitude constraints, the clear image is... A coarse screening of laser stripes is performed to construct a candidate pixel set. Within this candidate pixel set, the first and second partial derivatives of the grayscale value of any pixel (x, y) are calculated, and the corresponding Hessian matrix H is constructed. ; Perform eigenvalue decomposition on the Hessian matrix and extract the eigenvector corresponding to the smallest eigenvalue. The normal direction of the laser stripe at that pixel is used to characterize the local geometry of the laser stripe. Based on the constraint relationship between the first and second derivatives of the grayscale function, the sub-pixel offset of the laser center in the neighborhood of this pixel is obtained by solving the following system of linear equations: ; This yields the sub-pixel coordinates of the laser center. Traverse all candidate pixels to obtain a set of laser center points with sub-pixel precision. ; S3.2 Seam ROI Delineation Based on Target Detection: To limit the spatial range of laser line analysis and avoid interference from non-seam areas, the lightweight target detection network YOLOv5 is used. Position the tank runner joint in the middle and output the bounding box. Using the bounding box as the center, and extending outward from it with a preset safety boundary, the final region of interest R is constructed.

[0010] Furthermore, in the laser and vision-based coal mine vertical shaft guide misalignment detection method, in step S4, to achieve highly robust quantification and intelligent determination of guide joint misalignment, based on the laser center point set C and the region of interest R of the joint obtained in step S3, the following multi-feature fusion analysis is performed: S4.1 Dynamic Attitude Compensation and Physical Reference Alignment: The edge computing server extracts the attitude data of the inertial measurement unit synchronized with the current image frame in real time. Based on the attitude data, a rotation matrix is ​​constructed, and dynamic coordinate transformation is performed on the image feature points and laser point set C to eliminate the geometric projection bias caused by swinging and yaw during the operation of the cage. At the same time, combined with the preset camera intrinsic parameters, the matrix and the installation position calibration matrix are used to establish the mapping relationship between the pixel coordinate system and the actual physical coordinate system, so as to realize the unified dimension conversion of pixel displacement to actual physical length. S4.2 Multimodal Feature Construction of Seam Region: Under the unified coordinate system after pose compensation, extract the image patch corresponding to the region of interest of the seam. and a subset of laser center points located within that region. And construct multimodal features for misalignment analysis; S4.2.1 Laser geometric distortion feature extraction: A subset of laser center points at the seam. Sort by horizontal coordinate, assuming the seam is located at... At this point, the laser center point subset Divided into left-side point sets and the right-side point set Two groups were used, and the lines were fitted using the least squares method for each group. and , yes , yes ,in and Through The slope and intercept obtained from point set estimation and Through The slope and intercept obtained from point set estimation; Based on the above fitting results, the transverse step of the laser line at the joint was calculated. and the bend in the laser line at the seam , , The above two features are used to characterize the geometric discontinuity of the laser line at the joint; S4.2.2 Visual Texture Misalignment Feature Extraction: In the image patch after pose compensation The above method uses deep learning feature descriptors to extract feature points on both sides of the seam and performs matching to obtain a set of matching point pairs. Based on the vertical displacement of the matching point, the visual misalignment estimate is calculated. , This feature is used to reflect the degree of relative misalignment between the left and right sides of the seam; S4.3 Adaptive Weighted Feature Fusion and Decision: Integrating Laser Geometric Features and visual misalignment features As input, the model is a misalignment intelligent judgment model, which learns a nonlinear mapping function. Adaptive weighted fusion is performed based on the quality confidence of each feature to output the fusion misalignment in physical space. , Unit: millimeters; Among them, parameters During training, the system automatically learns weight allocation strategies for each feature under different imaging conditions: when the laser line is clear, it assigns... Higher weighting; when the seam texture contrast is high, it is increased. The weight, ultimately, will Compared with a preset threshold, the preset threshold includes a warning threshold. and alarm threshold : (1) If < This is considered normal. (2) If ≤ < It was determined to be a warning; (3) If ≥ The system is identified as an alarm and an alarm message containing location, image, and quantification data is reported.

[0011] Furthermore, in the laser and vision-based coal mine vertical shaft cage misalignment detection method, in step S5, the above two deep learning models are trained using labeled datasets collected in the experimental platform and the real shaft, covering different vibration, illumination and misalignment levels. During deployment, the trained models are integrated into the edge computing device at the cage end to realize real-time data acquisition, processing, analysis and alarm.

[0012] Furthermore, in the laser and vision-based coal mine vertical shaft misalignment detection method, in step S1, the frame rate of the high-speed camera is not less than 500 fps.

[0013] Furthermore, in the laser and vision-based method for detecting misalignment in coal mine vertical shaft guideways, step S4.3 involves an alarm threshold. Set to 2.0mm.

[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) It fundamentally solves the problem of image jitter and blur: By introducing a physical anti-shake model based on differentiable optical flow estimation and embedding motion compensation into a deep learning framework for end-to-end optimization, it can explicitly and accurately counteract the influence of multidimensional vibration of the cage on imaging and obtain stable and clear images from the source. (2) Sub-millimeter-level high-precision detection is achieved: Combining the absolute geometric reference provided by the laser and the sub-pixel laser center extraction algorithm based on the Hessian matrix, the detection accuracy breaks through the pixel limit and can theoretically reach the 0.1 pixel level. After calibration, it can easily achieve quantitative identification of misalignment of 2mm or even smaller. (3) It has extremely high robustness and extremely low false alarm rate: It pioneered an adaptive weighted fusion decision mechanism of "laser geometric features + visual texture features". The laser features are not sensitive to surface stains and lighting; the visual features can verify the authenticity of the geometric shape. The two complement each other and verify each other, effectively overcoming the defect that single features are easily interfered with, and significantly reducing false alarms caused by environmental factors such as oil stains, shadows, and rust. (4) High level of intelligence and strong adaptability: The entire analysis process is driven by a deep learning model, without the need for complex manual parameter adjustment. The model can learn and adapt to the image features of different mines, different tank tunnel models and different degradation levels. It has strong generalization ability and greatly reduces deployment and maintenance costs. (5) Real-time online safety monitoring around the clock: The system is deployed in the operating cage, changing the periodic spot check to real-time monitoring throughout the process. It can detect and locate safety hazards in time, provide accurate data support for preventive maintenance, and greatly improve the inherent safety level of the vertical shaft hoisting system. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method for detecting misalignment in coal mine vertical shaft guideways based on laser and vision, according to the present invention. Detailed Implementation

[0016] To better understand the present invention, the following embodiments further illustrate the content of the invention, but the scope of protection of the present invention is not limited to the following embodiments. Numerous specific details are set forth in the following description to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without one or more of these details.

[0017] Combined with appendix Figure 1This invention details a laser and vision-based misalignment detection system for coal mine vertical shaft cages. It includes a 650nm linear laser emitter, a high-speed CMOS camera with a global shutter, a high-brightness LED array, an inertial measurement unit (IMU), and an edge computing server, all mounted on a rigid, vibration-damped platform welded to the top of the cage. The system adjusts the laser emitter and camera orientation to project the laser line vertically onto the center area of ​​the cage's front rail, with the camera's optical axis aligned with the laser illumination area. The frame rate is set to 1000 fps, and the exposure time is less than 1ms to freeze the dynamics. The brightness of the LED array is controlled by a closed-loop photosensitive sensor, adjusting based on ambient light feedback to ensure uniform illumination of the target area. Images captured by the camera are transmitted in real-time to the reinforced edge computing server inside the cage via a Camera Link interface. The IMU acquires the cage's attitude information. The edge computing server receives the image stream from the high-speed industrial camera and the cage's attitude information from the IMU, and uses this attitude information to perform geometric correction on the images. It incorporates an NVIDIA Jetson AGX processor. The Orin computing module runs the image stabilization and enhancement fusion model and the intelligent cage misalignment determination model to output the cage misalignment amount after removing cage vibration interference. During system operation, as the cage moves up and down along the shaft, the high-speed CMOS camera continuously acquires a sequence of frontal images of the cage passage, forming the original time-series image data. Meanwhile, the linear laser emitter forms a continuous structured laser line on the surface of the track rail to reflect the geometric changes of the track. In addition, the system synchronously acquires cage attitude information aligned with the image frame timestamp. The attitude information is obtained by an inertial measurement unit (IMU) installed on the cage. Image data, laser structured light information, and attitude information are synchronously aligned through a unified timestamp to form multi-source collaborative sensing data, providing a data foundation for subsequent image stabilization, feature extraction, and misalignment determination.

[0018] To balance the computing load on the edge side with real-time requirements, the system adopts the following operator scheduling strategy: Lightweight monitoring mode: By default, the system only calculates the image grayscale statistical features and the mean of motion optical flow in real time to quickly determine whether there are structural abrupt changes or suspected seam areas in the current image; Full-precision inference mode: When the lightweight monitoring results detect a suspected seam area in the image or receive a position trigger signal from the synchronous depth encoder, the system triggers the full-precision inference thread and calls the TensorRT acceleration operator to perform image stabilization and misalignment quantization. Dynamic operator allocation: The inference step size Stride is adjusted according to the running speed of the cage. When passing through the critical area containing the seam, the full frame rate calculation is performed at 1000 fps. Frame skipping is used in the straight section without seam to ensure that the overall system response latency is stable within 100 milliseconds. A method for detecting misalignment of coal mine vertical shaft guideways based on laser and vision, utilizing the aforementioned laser and vision-based coal mine vertical shaft guideway misalignment detection system, specifically includes the following steps: S1: Multi-source collaborative sensing data acquisition: A laser projects a beam of bright, fine, straight light stripes onto the surface of the cage. A high-speed camera acquires an image sequence containing both the natural texture of the cage and the laser stripes superimposed on it, at a frame rate higher than the dominant vibration frequency of the cage. The supplementary lighting unit adjusts its brightness based on ambient light feedback to ensure uniform illumination of the target area; S2: Image stabilization and enhancement preprocessing based on a differentiable physical model: In full-precision inference mode, the jitter-blurred original image sequence acquired by S1 is processed... Input a pre-trained image stabilization and enhancement fusion model to obtain stable features that reflect the consistency of the image's geometric structure. and enhanced features The model uses PWC-Net as the optical flow estimation backbone and U-Net as the enhancement branch. Feature fusion is performed through differentiable warp operations and a channel attention fusion module to stabilize the features. With enhanced features The system performs fusion processing to generate fused feature representations, which are then input into the decoding and reconstruction module. To obtain a clear image The specific steps of step S2 are as follows: S2.1 Physical Model Construction: Modeling inter-frame motion as a dense optical flow field V t ,in , representing a pixel The displacement vector from frame t to the reference frame; S2.2 Differentiable Motion Compensation and Stabilized Image Generation: Based on the Dense Optical Flow Field V t Using a differentiable bilinear sampling operator For the current frame image Pixel-level remapping is performed to obtain a stable image that has been initially freed from the effects of jitter. Since the bilinear sampling operator is a differentiable operator, the model can adjust the motion estimation results in reverse according to the error information of the output results during the training phase, thereby realizing the joint optimization of motion modeling and subsequent image processing. S2.3 Stabilizing and Enhancing Feature Extraction: The model includes a stabilizing feature extraction module and an enhancing feature extraction module, stabilizing the image. The input is fed into the stable feature extraction module, which extracts stable features that reflect the consistency of the image's geometric structure. Meanwhile, the original input image The input is fed into the enhanced feature extraction module for feature encoding and reconstruction processing, in order to complete noise suppression, contrast adjustment, and detail information recovery within the feature domain, thereby obtaining enhanced features. ; S2.4 Multi-scale Feature Fusion and Image Reconstruction: The multi-scale feature fusion module performs image reconstruction on the stable features... With enhanced features The system performs fusion processing to generate fused feature representations, which are then input into the decoding and reconstruction module. To obtain a clear image : ; During the training phase, by minimizing the sharp image With corresponding clear truth image The model is optimized using a loss function that includes at least pixel-level L1 loss and perceptual loss to improve the overall performance of the reconstructed image in terms of structural consistency and subjective visual quality. S3: Laser line subpixel extraction and seam area localization, resulting in a clear image obtained in step S2. The laser line sub-pixel extraction and seam area positioning operations are performed, and the specific steps are as follows: Sub-pixel extraction of S3.1 laser lines: First, based on the brightness threshold and gradient magnitude constraints, the clear image is... A coarse screening of laser stripes is performed to construct a candidate pixel set. Within this candidate pixel set, the first and second partial derivatives of the grayscale value of any pixel (x, y) are calculated, and the corresponding Hessian matrix H is constructed. ; Perform eigenvalue decomposition on the Hessian matrix and extract the eigenvector corresponding to the smallest eigenvalue. The normal direction of the laser stripe at that pixel is used to characterize the local geometry of the laser stripe. Based on the constraint relationship between the first and second derivatives of the grayscale function, the sub-pixel offset of the laser center in the neighborhood of this pixel is obtained by solving the following system of linear equations: ; This yields the sub-pixel coordinates of the laser center. Traverse all candidate pixels to obtain a set of laser center points with sub-pixel precision. ; S3.2 Seam ROI Delineation Based on Target Detection: To limit the spatial range of laser line analysis and avoid interference from non-seam areas, the lightweight target detection network YOLOv5 is used. Position the tank runner joint in the middle and output the bounding box. Using the bounding box as the center, and extending outward from it with a preset safety boundary, the final region of interest R is constructed; S4: Intelligent Misalignment Quantization and Judgment through Multi-Feature Fusion: To achieve robust quantization and intelligent judgment of misalignment at the tank joint, based on the laser center point set C obtained in step S3 and the region of interest R of the joint, the following multi-feature fusion analysis is performed: S4.1 Dynamic Attitude Compensation and Physical Reference Alignment: The edge computing server extracts the attitude data of the inertial measurement unit synchronized with the current image frame in real time. Based on the attitude data, a rotation matrix is ​​constructed, and dynamic coordinate transformation is performed on the image feature points and laser point set C to eliminate the geometric projection bias caused by swinging and yaw during the operation of the cage. At the same time, combined with the preset camera intrinsic parameters, the matrix and the installation position calibration matrix are used to establish the mapping relationship between the pixel coordinate system and the actual physical coordinate system, so as to realize the unified dimension conversion of pixel displacement to actual physical length. S4.2 Multimodal Feature Construction of Seam Region: Under the unified coordinate system after pose compensation, extract the image patch corresponding to the region of interest of the seam. and a subset of laser center points located within that region. And construct multimodal features for misalignment analysis; S4.2.1 Laser geometric distortion feature extraction: A subset of laser center points at the seam. Sort by horizontal coordinate, assuming the seam is located at... At this point, the laser center point subset Divided into left-side point sets and the right-side point set Two groups were used, and the lines were fitted using the least squares method for each group. and , yes , yes ,in and Through The slope and intercept obtained from point set estimation and Through The slope and intercept obtained from point set estimation; Based on the above fitting results, the transverse step of the laser line at the joint was calculated. and the bend in the laser line at the seam , , The above two features are used to characterize the geometric discontinuity of the laser line at the joint; S4.2.2 Visual Texture Misalignment Feature Extraction: In the image patch after pose compensation The above method uses deep learning feature descriptors to extract feature points on both sides of the seam and performs matching to obtain a set of matching point pairs. Based on the vertical displacement of the matching point, the visual misalignment estimate is calculated. , This feature is used to reflect the degree of relative misalignment between the left and right sides of the seam; S4.3 Adaptive Weighted Feature Fusion and Decision: Integrating Laser Geometric Features and visual misalignment features As input, the model is a misalignment intelligent judgment model, which learns a nonlinear mapping function. Adaptive weighted fusion is performed based on the quality confidence of each feature to output the fusion misalignment in physical space. , Unit: millimeters; Among them, parameters During training, the system automatically learns weight allocation strategies for each feature under different imaging conditions: when the laser line is clear, it assigns... Higher weighting; when the seam texture contrast is high, it is increased. The weight, ultimately, will Compared with a preset threshold, the preset threshold includes a warning threshold. and alarm threshold Alarm threshold Set to 2.0mm: (1) If < This is considered normal. (2) If ≤ < It was determined to be a warning; (3) If ≥ If an alarm is detected, the system will report an alarm message containing location, image, and quantitative data. When the tank passage is detected to be in an alarm state, the system will upload the alarm message to the ground monitoring center in real time via the wireless communication module.

[0019] S5: Model training and system deployment. Using labeled datasets collected in the experimental platform and real wellbore covering different vibration, lighting and 0-5mm misalignment levels, two deep learning models, the image stabilization and enhancement fusion model and the intelligent judgment model for cage misalignment, are trained. During deployment, the trained models are integrated into the edge computing device at the cage end to realize real-time data acquisition, processing, analysis and alarm.

[0020] This invention discloses a laser and vision-based system and method for detecting misalignment in coal mine vertical shaft cages. This method enables sub-millimeter precision, high reliability, and real-time online monitoring of cage health status under harsh real-world working conditions in coal mine vertical shafts. It employs a technical approach that combines active laser projection to establish an absolute geometric benchmark, high-speed vision to capture dynamic scenes, and deep learning models for intelligent decoupling and decision-making. This effectively isolates cage vibration interference, enhances imaging quality in joint areas, and integrates multimodal features for precise judgment. Ultimately, it achieves stable and accurate identification of 2mm-level cage misalignment under strong vibration environments, successfully realizing intelligent, high-precision, and real-time online monitoring of the health status of coal mine vertical shaft cages.

[0021] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Any other modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention, as long as they do not depart from the spirit and scope of the technical solutions of the present invention, should be covered within the scope of the claims of the present invention.

Claims

1. A laser and vision-based system for detecting misalignment in coal mine vertical shaft guideways, characterized in that, The system includes a line-structured laser emitter mounted on the cage, a high-speed industrial camera, an adaptive lighting unit, an inertial measurement unit, and an edge computing server. The laser emitter projects bright, fine, straight-line light stripes onto the surface of the cage. The high-speed industrial camera acquires image sequences containing both the natural texture of the cage and the laser stripes superimposed on it, at a frame rate higher than the cage's dominant vibration frequency. ; The adaptive lighting unit adjusts the brightness based on ambient light feedback to ensure uniform illumination in the target area; the inertial measurement unit is used to acquire the cage's attitude information. The edge computing server receives image streams from high-speed industrial cameras and cage attitude information acquired by inertial measurement units (IMUs). It then uses the IMU data to perform spatial consistency correction on visual geometric features and outputs the cage track fusion misalignment amount after decoupling from cage vibration interference. .

2. A method for detecting misalignment of coal mine vertical shaft guideways based on laser and vision, characterized in that, The method of using the laser and vision-based coal mine shaft guideway misalignment detection system as described in claim 1 to detect misalignment in coal mine shaft guideways specifically includes the following steps: S1: Multi-source collaborative sensing data acquisition: A laser projects a beam of bright, fine, straight light stripes onto the surface of the cage. A high-speed camera acquires an image sequence containing both the natural texture of the cage and the laser stripes superimposed on it, at a frame rate higher than the dominant vibration frequency of the cage. The supplementary lighting unit adjusts its brightness based on ambient light feedback to ensure uniform illumination of the target area; S2: Image stabilization and enhancement preprocessing based on a differentiable physical model: preprocessing the jitter-blurred original image sequence acquired in S1. Input a pre-trained image stabilization and enhancement fusion model to obtain stable features that reflect the consistency of the image's geometric structure. and enhanced features Regarding the aforementioned stable features With enhanced features The system performs fusion processing to generate fused feature representations, which are then input into the decoding and reconstruction module. To obtain a clear image : ； S3: Laser line subpixel extraction and seam area positioning; S4: Intelligent misalignment quantification and determination based on multi-feature fusion, outputting the cage track fused misalignment amount decoupled from cage vibration interference. and will The system is compared with a preset threshold to determine whether the tank passage is in a normal, warning, or alarm state. S5: Model training and system deployment. The two deep learning models mentioned above are trained using labeled datasets collected in the experimental platform and real wellbore, covering different levels of vibration, illumination and misalignment.

3. The method for detecting misalignment of coal mine vertical shaft guideways based on laser and vision as described in claim 2, characterized in that, The specific steps of step S2 are as follows: S2.1 Physical Model Construction: Modeling inter-frame motion as a dense optical flow field V t ,in , representing a pixel The displacement vector from frame t to the reference frame; S2.2 Differentiable Motion Compensation and Stabilized Image Generation: Based on the Dense Optical Flow Field V t Using a differentiable bilinear sampling operator For the current frame image Pixel-level remapping is performed to obtain a stable image that has been initially freed from the effects of jitter. Since the bilinear sampling operator is a differentiable operator, the model can adjust the motion estimation results in reverse according to the error information of the output results during the training phase, thereby realizing the joint optimization of motion modeling and subsequent image processing. S2.3 Stabilizing and Enhancing Feature Extraction: The model includes a stabilizing feature extraction module and an enhancing feature extraction module, stabilizing the image. The input is fed into the stable feature extraction module, which extracts stable features that reflect the consistency of the image's geometric structure. Meanwhile, the original input image The input is fed into the enhanced feature extraction module for feature encoding and reconstruction processing, in order to complete noise suppression, contrast adjustment, and detail information recovery within the feature domain, thereby obtaining enhanced features. ; S2.4 Multi-scale Feature Fusion and Image Reconstruction: The multi-scale feature fusion module performs image reconstruction on the stable features... With enhanced features The system performs fusion processing to generate fused feature representations, which are then input into the decoding and reconstruction module. To obtain a clear image : ; During the training phase, by minimizing the sharp image With corresponding clear truth image The model is optimized using a loss function that includes at least pixel-level L1 loss and perceptual loss to improve the overall performance of the reconstructed image in terms of structural consistency and subjective visual quality.

4. The method for detecting misalignment of coal mine vertical shaft guideways based on laser and vision as described in claim 2, characterized in that, Step S3 is the clear image obtained in step S2. The laser line sub-pixel extraction and seam area positioning operations are performed, and the specific steps are as follows: Sub-pixel extraction of S3.1 laser lines: First, based on the brightness threshold and gradient magnitude constraints, the clear image is... A coarse screening of laser stripes is performed to construct a candidate pixel set. Within this candidate pixel set, the first and second partial derivatives of the grayscale value of any pixel (x, y) are calculated, and the corresponding Hessian matrix H is constructed. ； Perform eigenvalue decomposition on the Hessian matrix and extract the eigenvector corresponding to the smallest eigenvalue. The normal direction of the laser stripe at that pixel is used to characterize the local geometry of the laser stripe. Based on the constraint relationship between the first and second derivatives of the grayscale function, the sub-pixel offset of the laser center in the neighborhood of this pixel is obtained by solving the following system of linear equations: ； This yields the sub-pixel coordinates of the laser center. Traverse all candidate pixels to obtain a set of laser center points with sub-pixel precision. ; S3.2 Seam ROI Delineation Based on Target Detection: To limit the spatial range of laser line analysis and avoid interference from non-seam areas, the lightweight target detection network YOLOv5 is used. Position the tank runner joint in the middle and output the bounding box. Using the bounding box as the center, and extending outward from it with a preset safety boundary, the final region of interest R is constructed.

5. The method for detecting misalignment of coal mine vertical shaft guideways based on laser and vision as described in claim 2, characterized in that, In step S4, to achieve robust quantification and intelligent determination of misalignment at the tank joint, based on the laser center point set C and the region of interest R of the joint obtained in step S3, the following multi-feature fusion analysis is performed: S4.1 Dynamic Attitude Compensation and Physical Reference Alignment: The edge computing server extracts the attitude data of the inertial measurement unit synchronized with the current image frame in real time. Based on the attitude data, a rotation matrix is ​​constructed, and dynamic coordinate transformation is performed on the image feature points and laser point set C to eliminate the geometric projection bias caused by swinging and yaw during the operation of the cage. At the same time, combined with the preset camera intrinsic parameters, the matrix and the installation position calibration matrix are used to establish the mapping relationship between the pixel coordinate system and the actual physical coordinate system, so as to realize the unified dimension conversion of pixel displacement to actual physical length. S4.2 Multimodal Feature Construction of Seam Region: Under the unified coordinate system after pose compensation, extract the image patch corresponding to the region of interest of the seam. and a subset of laser center points located within that region. And construct multimodal features for misalignment analysis; S4.2.1 Laser geometric distortion feature extraction: A subset of laser center points at the seam. Sort by horizontal coordinate, assuming the seam is located at... At this point, the laser center point subset Divided into left-side point sets and the right-side point set Two groups were used, and the lines were fitted using the least squares method for each group. and , yes , yes ,in and Through The slope and intercept obtained from point set estimation and Through The slope and intercept obtained from point set estimation; Based on the above fitting results, the transverse step of the laser line at the joint was calculated. and the bend in the laser line at the seam , , The above two features are used to characterize the geometric discontinuity of the laser line at the joint; S4.2.2 Visual Texture Misalignment Feature Extraction: In the image patch after pose compensation The above method uses deep learning feature descriptors to extract feature points on both sides of the seam and performs matching to obtain a set of matching point pairs. Based on the vertical displacement of the matching point, the visual misalignment estimate is calculated. , This feature is used to reflect the degree of relative misalignment between the left and right sides of the seam; S4.3 Adaptive Weighted Feature Fusion and Decision: Integrating Laser Geometric Features and visual misalignment features As input, the model is a misalignment intelligent judgment model, which learns a nonlinear mapping function. Adaptive weighted fusion is performed based on the quality confidence of each feature to output the fusion misalignment in physical space. , Unit: millimeters; Among them, parameters During training, the system automatically learns weight allocation strategies for each feature under different imaging conditions: when the laser line is clear, it assigns... Higher weighting; when the seam texture contrast is high, it is increased. The weight, ultimately, will Compared with a preset threshold, the preset threshold includes a warning threshold. and alarm threshold : (1) If < This is considered normal. (2) If ≤ < It was determined to be a warning; (3) If ≥ The system is identified as an alarm and an alarm message containing location, image, and quantification data is reported.

6. The method for detecting misalignment of coal mine vertical shaft guideways based on laser and vision according to claim 2, characterized in that, In step S5, the two deep learning models are trained using labeled datasets collected in the experimental platform and the real wellbore, covering different levels of vibration, illumination and misalignment. During deployment, the trained models are integrated into the edge computing device at the cage end to realize real-time data acquisition, processing, analysis and alarm.

7. The method for detecting misalignment of coal mine vertical shaft guideways based on laser and vision as described in claim 2, characterized in that, In step S1, the frame rate of the high-speed camera is no less than 500 fps.

8. The method for detecting misalignment of coal mine vertical shaft guideways based on laser and vision as described in claim 2, characterized in that, In step S4.3, the alarm threshold Set to 2.0mm.

Images