Method and system for monitoring working face of thin coal seam based on computer vision

Abstract

The invention discloses a thin coal seam working face monitoring method and system based on computer vision, and relates to the field of thin coal seam working face clamping plate monitoring. According to the method, continuous tracking and dynamic feature extraction of the three-dimensional form of the splint are realized by combining multi-source visual data fusion with a time sequence form analysis modeling method. Specifically, a structured light depth camera, an RGB camera and a thermal infrared imager are utilized to obtain morphological characteristics of a clamping plate in different dimensions, and a three-dimensional model with spatial consistency is constructed through fusion; and a morphological risk function is constructed in combination with curvature change, top height difference and time continuity characteristics, so that quantitative, dynamic and real-time evaluation of the clamping plate disengagement risk is realized, and the system can judge potential risks in advance in the early deformation stage of the clamping plate.

Description

Technical Field

[0001] This invention relates to the field of monitoring the interlayer of thin coal seam working faces, specifically a monitoring method and system for thin coal seam working faces based on computer vision. Background Technology

[0002] Existing monitoring methods for thin coal seam working faces primarily rely on manual inspections and single-sensor data acquisition. In mines, equipment such as infrared rangefinders, 2D cameras, or laser scanners are typically used to periodically inspect the working face environment, collecting surface geometric features of the faceplates and cable troughs to determine the presence of risks such as displacement, bending, or detachment. Some systems also incorporate inertial sensors (such as IMUs) or accelerometers for dynamic monitoring to improve measurement frequency and response speed. However, these methods generally depend on local geometric points or low-dimensional image information, failing to provide a continuous description of the overall spatial morphology of the faceplates and exhibiting insufficient ability to capture structural deformation responses in complex coal seam environments.

[0003] The shortcomings of existing technologies are as follows: First, the data sources of single sensors suffer from low information dimensionality, strong noise interference, and weak temporal correlation, making it difficult to accurately reflect the three-dimensional morphological changes of the sandwich panel under dynamic coal seam stress environment; Second, existing monitoring methods mostly remain at the stage of static threshold judgment or single-frame image recognition, and cannot predict the trend of sandwich panel morphological changes, resulting in a lag in the assessment of desquat risk; Third, existing methods lack sufficient correlation analysis between geometric changes and morphological displacement at the top of the sandwich panel, and cannot accurately assess the potential morphological risk status, thus limiting the early warning capability for the operational safety of thin coal seam working faces. Summary of the Invention

[0004] This invention proposes a monitoring method and system for thin coal seam working faces based on computer vision. It aims to build a monitoring technology system for thin coal seam working faces with adaptive analysis capabilities. Through visual fusion modeling and dynamic risk function calculation, it realizes intelligent identification of changes in the shape of the seam slab and accurate determination of the desinking state, providing a more reliable, real-time and automated monitoring means for safe coal mine production.

[0005] One method for monitoring thin coal seam working faces based on computer vision includes the following steps: S1. Collect morphological data of cable clamps using an industrial sensor array, and construct a three-dimensional fusion model using the morphological data; Specifically, an industrial sensor array installed on the working surface synchronously collects information from the cable clamps. A structured light depth camera first acquires depth point cloud data of the clamp surface, and calculates the spatial coordinates of each pixel using the principle of light projection texture deformation to form a dense point cloud frame. An RGB camera synchronously acquires the two-dimensional texture information of the corresponding area, providing color constraints for subsequent point cloud registration and texture mapping. An infrared thermal imager identifies edge temperature gradient changes based on the surface reflection matrix to determine the clamp boundaries and folded areas. Multi-source data are aligned using a spatiotemporal synchronous calibration matrix, and the depth map and color map are fused using an extrinsic calibration matrix. Based on this, voxel filtering is used to remove noise points, and a multi-frame point cloud stitching is achieved using an ICP (Iterative Closest Point) based registration algorithm, resulting in a continuous, dense, and measurable three-dimensional fusion model that accurately reflects the geometry of the clamp.

[0006] S2. Extract the center curve of the 3D fusion model using the deep extremum tracking method, calculate the average curvature of the clamp in the frame based on the discrete point set of the extracted center curve, output parameter pairs, and construct a curvature sequence from the parameter pairs at consecutive time moments. Specifically, the system extracts the central structure of the 3D fusion model using the depth extreme value tracking method. First, the system performs layered scanning of the point cloud data in the fusion model along the trajectory direction. Within each section, representative extreme points are determined based on the depth distribution. By connecting these extreme points layer by layer to form a continuous centerline, the main axis structure of the clamp's shape is established. This main axis curve reflects the overall stress shape characteristics and bending trend of the clamp. Then, the average curvature is calculated based on the set of discrete points on the center curve. The degree of change in curvature represents the concentration and directional deviation of the clamp's deformation. The average curvature of each frame is recorded in chronological order to form a curvature time series, thereby reflecting the dynamic change process of the clamp's shape. Under continuous operation, the system can observe whether the clamp exhibits periodic bending, slow deformation, or abrupt anomalies based on this sequence. The extraction of the center curve and the calculation of curvature together complete the quantitative expression of geometric change characteristics from 3D morphological data.

[0007] S3. Construct the curvature sequence of consecutive time moments into a morphological change time series, predict the curvature state at the next time moment through a sliding window model, and output the predicted morphological deviation value; Specifically, the established curvature time series is used as the main characterization quantity of morphological changes. A sliding window model is used to capture the local dynamic trend of the clamp morphological changes. The sliding window contains curvature data for several consecutive moments. Short-term fluctuation patterns and trend offset information are extracted through time continuity analysis, thereby predicting the curvature state at the next moment. This process is achieved by continuously sliding the time window, enabling the system to perceive the speed and direction of morphological changes in real time. When there is a deviation between the actual measured curvature and the predicted curvature, the deviation is regarded as an early signal of morphological anomaly. The system regards the deviation value as the predicted morphological deviation degree, reflecting the gap between the current state of the clamp and the normal evolutionary trajectory. This prediction mechanism uses time correlation to dynamically model the state of the clamp, so that the system can judge potential risks by the upward trend of the deviation before anomalies occur.

[0008] S4. Based on the geometric height information of the 3D fusion model, calculate the height difference at the top of the clamping plate, and construct a morphological risk function by weighting and combining the predicted morphological deviation value with the calculated height difference at the top of the clamping plate with the relative change rate. Specifically, the geometric height difference at the top of the clamping plate is calculated using a fusion model to reflect its vertical non-uniformity. This height difference is then combined with the predicted morphological deviation to form a multi-dimensional index describing the overall morphological stability of the clamping plate. During the calculation process, the system not only considers the instantaneous state of the current frame but also weights the continuity of adjacent frames, thereby introducing a time accumulation effect to reflect the inertia and abrupt changes in morphological characteristics. The risk function integrates three factors: local bending, vertical deformation, and continuous time change of the clamping plate. This ensures that the obtained morphological risk value can accurately reflect the stress state and deformation trend of the clamping plate on the working surface. When the clamping plate undergoes continuous bending or local morphological abrupt changes, the risk function value will rise rapidly, thus providing the system with a quantifiable morphological risk signal. This process essentially transforms complex geometric morphological features into a computable numerical risk model.

[0009] S5. Input new data into the morphological risk function, calculate the morphological risk value at the current moment and the continuous transformation of the risk value between two consecutive frames, and output the probability value normalized to [0,1] through the Sigmoid function; Specifically, the morphological risk function is transformed into a probabilistic index that can be used for state determination. When new monitoring data is input into the system, the system will input the morphological risk function in real time to calculate the risk value at the current moment. At the same time, it will perform differential analysis with the previous frame data to obtain the continuous rate of change. If the system is running for the first time, the initial baseline value is used as the risk quantity at the previous moment to prevent the model calculation from being interrupted. Subsequently, the system maps the obtained risk value to the standardized probability space. The result is constrained to the range of zero to one by a nonlinear mapping function, thereby reflecting the instability of the current clamp state in the form of continuous probability. The higher the risk value, the closer the output probability is to one, which means that the morphology is closer to the critical state of delamination or deformation. When the risk value is low, the probability approaches zero, indicating that the clamp is in a stable operating range. At the same time, the system monitors the probability difference between consecutive frames. If the probability rises continuously or the fluctuation expands abnormally, it means that the clamp morphology is evolving towards a high-risk area. In this way, the system transforms the physical morphological change process into a continuous probability signal.

[0010] S6. Determine the desinking status based on the calculated range probability value and the set desinking status threshold; Specifically, the system compares the calculated probability results with a preset desquat threshold to determine the operating status of the clamping plate. When the probability value reaches or exceeds the threshold, the system determines that the clamping plate has entered a potential desquat state and triggers an early warning mechanism to send a warning signal to the monitoring terminal. When the probability value is below the threshold, it is considered to be in normal operation. If the probability value remains in a high range for several consecutive time periods, the system further determines that there is a continuous desquat trend and issues a secondary alarm. This process is not only based on risk assessment of a single frame, but also incorporates continuous changes over time to prevent false alarms caused by transient noise. In addition, the system can assess the risk development speed based on the probability growth rate. When the growth rate exceeds a set threshold, it determines that there is a tendency for sudden desquat. This judgment mechanism enables the system to achieve accurate desquat identification and trend warning in dynamic environments. Through this combination of probability and threshold, the system realizes a closed-loop process from data perception, risk calculation to state decision-making.

[0011] Furthermore, a computer vision-based monitoring system for thin coal seam working faces is proposed. This system is implemented based on the computer vision-based monitoring method for thin coal seam working faces described in any of the above embodiments, and includes: The model building module is used to collect morphological data of cable clamps through industrial sensor arrays and build a three-dimensional fusion model based on the morphological data. The curvature calculation module is used to extract the center curve of the 3D fusion model through the depth extreme value tracking method, calculate the average curvature of the clamp in the frame based on the discrete point set of the extracted center curve, output parameter pairs, and construct the parameter pairs of continuous time moments into a curvature sequence. The deviation prediction module constructs a morphological change time series from the curvature sequence at continuous time points, predicts the curvature state at the next time point through a sliding window model, and outputs the predicted morphological deviation value. The morphological risk function construction module is used to calculate the height difference at the top of the clamping plate based on the geometric height information of the 3D fusion model, and construct the morphological risk function by weighting and combining the predicted morphological deviation value with the calculated height difference at the top of the clamping plate with the relative change rate. The data calculation module is used to input new data into the morphological risk function, calculate the morphological risk value at the current moment and the continuous transformation of the risk value between two frames, and output the probability value normalized to [0,1] through the Sigmoid function; The threshold determination module is used to determine the desinking status based on the calculated range probability value and the set desinking status threshold.

[0012] Furthermore, the industrial sensor array includes at least: Structured light depth camera is used to acquire depth point cloud data of the surface of the clamping plate and form point cloud frames; An RGB camera is used to capture two-dimensional texture images of clamps and cable trays; Infrared thermal imagers are used to acquire the infrared reflection matrix of the splice surface to identify splice boundaries and folded areas.

[0013] The beneficial effects of the invention are: This invention achieves continuous tracking and dynamic feature extraction of the three-dimensional morphology of a clamping plate by combining multi-source visual data fusion with temporal morphological analysis and modeling methods. Specifically, this invention utilizes a structured light depth camera, an RGB camera, and an infrared thermal imager to acquire the morphological features of the clamping plate in different dimensions, and constructs a spatially consistent three-dimensional model through fusion. Then, by combining curvature changes, top height differences, and temporal continuity features, a morphological risk function is constructed to achieve quantitative, dynamic, and real-time assessment of the clamping plate delamination risk, enabling the system to identify potential risks in the early stages of clamping plate deformation. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the UAV target perception method based on beam reconstruction and environmental perception adaptation according to an embodiment of the present invention. Detailed Implementation

[0015] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, and not all of them. The components of the embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0017] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention. It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0018] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or machine that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or machine. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or machine that includes said element.

[0019] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0020] Example 1 Among them, such as Figure 1 A computer vision-based monitoring method for thin coal seam working faces includes the following steps: S1. Collect morphological data of cable clamps using an industrial sensor array, and construct a three-dimensional fusion model using the morphological data; Specifically, the principle and process of step S1 are as follows: The spatial morphology of the cable clamp is synchronously acquired using an industrial sensor group installed in the support area of ​​the thin coal seam working face. This sensor group achieves collaborative data acquisition from a depth camera, an RGB camera, and an infrared thermal imager through a multi-channel synchronous triggering system. The depth camera, based on the structured light projection principle, calculates the depth information of points on the clamp surface by measuring fringe deformation; the RGB camera records the two-dimensional texture features of the clamp surface to assist in spatial registration of the depth data; the infrared thermal imager identifies local thermal distribution differences on the clamp surface caused by stress concentration or structural deformation, thereby indirectly reflecting areas of abnormal morphology. Based on the sensor intrinsic and extrinsic parameter calibration results, the system fuses the three types of data in a unified coordinate system and generates multi-source point cloud data through feature alignment and geometric constraint methods. Then, a temporal stitching algorithm is used to establish spatial correspondence between consecutive frames, and after removing redundant noise points, a dense three-dimensional fusion model is generated.

[0021] S2. Extract the center curve of the 3D fusion model using the deep extremum tracking method, calculate the average curvature of the clamp in the frame based on the discrete point set of the extracted center curve, output parameter pairs, and construct a curvature sequence from the parameter pairs at consecutive time moments. Specifically, the principle and process of step S2 are as follows: In the obtained 3D fusion model, the geometric extraction of the center structure of the clamp is achieved through the depth extremum tracking method. Specifically, the system first divides the fusion model into several cross-sectional slices along the length direction. In each slice, extreme value analysis is performed on the depth distribution of the point cloud to identify the depth extremum point that best represents the geometric center of the clamp. Subsequently, the system establishes a correspondence between extreme points between adjacent slices and forms a continuous center curve through interpolation and neighborhood fitting. This curve serves as the geometric skeleton of the clamp's shape, reflecting its overall bending trend along the longitudinal direction. Next, the system calculates the spatial curvature distribution of discrete points on this curve and obtains the average curvature of each frame to quantify the overall bending degree of the clamp at that moment. By arranging the average curvature of consecutive frames in chronological order, the system forms a curvature sequence, establishing a dynamic trajectory of shape change over time. The implementation principle of this process is to use the spatial continuity of the depth gradient in the point cloud to track the geometric evolution of the clamp, enabling the system to obtain continuous temporal information of structural deformation without relying on any mechanical sensors.

[0022] S3. Construct the curvature sequence of consecutive time moments into a morphological change time series, predict the curvature state at the next time moment through a sliding window model, and output the predicted morphological deviation value; Specifically, the implementation principle of step S3 is as follows: The curvature sequence at consecutive time points is input into the morphological change time series model, and a sliding window method is used to analyze the dynamic characteristics within a short time period. The sliding window covers a fixed number of curvature data points in each calculation to capture the changing trend of the clamp within a specific time interval. The system uses the local linear characteristics of the curvature within the window to estimate the predicted curvature at the next time point and compares it with the actual curvature to calculate the morphological deviation value. The magnitude of the deviation value directly reflects the degree of difference between the clamp's morphology and its historical evolution trajectory, thereby revealing whether the morphological change deviates from the normal trend. The implementation principle of this process is to estimate the inertial change of the clamp's morphology through time series prediction, enabling the system to identify abnormal growth trends before significant abrupt changes in morphology occur. When the deviation value continues to increase, the system determines that the clamp's morphology may have a potential risk of folding or deformation.

[0023] S4. Based on the geometric height information of the 3D fusion model, calculate the height difference at the top of the clamping plate, and construct a morphological risk function by weighting and combining the predicted morphological deviation value with the calculated height difference at the top of the clamping plate with the relative change rate. Specifically, the implementation principle of step S4 is as follows: Based on the geometric structure of the 3D fusion model, the system extracts the point cloud of the top boundary region of the clamping plate and calculates its vertical height difference. This height difference reflects the flatness of the upper surface morphology and the uniformity of stress distribution on the clamping plate, and is a key parameter for judging the stability of the clamping plate structure. The system weights and combines the predicted morphological deviation value with the calculated height difference at the top of the clamping plate to form a multi-dimensional morphological risk function. The risk function not only describes the current geometric state, but also introduces the rate of change of time as a correction term to capture the dynamic characteristics of morphological changes. When the height difference at the top of the clamping plate increases rapidly or the predicted deviation continues to accumulate, the risk function value will rise significantly, indicating that the structure has a tendency to fold, warp, or concentrate stress. The design logic of this risk model is to fuse geometric spatial features with predicted trend information, enabling the system to infer the potential risk level through morphological changes themselves without relying on external load monitoring.

[0024] S5. Input new data into the morphological risk function, calculate the morphological risk value at the current moment and the continuous transformation of the risk value between two frames, and output the probability value normalized to [0,1] through the Sigmoid function; Specifically, the implementation principle of step S5 is as follows: After receiving a new frame of morphological data, it is input into the constructed morphological risk function to calculate the risk value at the current moment, and then the difference between the risk value at the previous moment is calculated to obtain the continuous rate of change. When the system runs for the first time, an initial calibration value is used as a reference to ensure the temporal continuity of the risk function. Subsequently, the system uses a nonlinear mapping mechanism to transform the risk value into a standardized probability value, distributing it between 0 and 1, forming a quantifiable morphological stability index. The higher the risk value, the closer the clamp is to an unstable state, and the closer the corresponding output probability is to 1; conversely, it is closer to 0. This normalization process eliminates the difference in risk dimensions by introducing a nonlinear mapping function, making data from different time periods and under different environmental conditions comparable. At the same time, the system monitors the changing trend of the probability between consecutive frames in real time. When the probability increase rate exceeds a set threshold, the system will mark it as a potentially dangerous state.

[0025] S6. Determine the desinking status based on the calculated range probability value and the set desinking status threshold; Specifically, the implementation principle of step S6 is as follows: the probability value output in the previous stage is compared with a preset desquatization threshold to determine whether the clamping plate has entered an abnormal state. When the probability value is greater than the threshold, the system determines that the clamping plate is in the desquatization risk range, immediately triggers an alarm and recording mechanism, and generates an event identifier with a timestamp and stores it in the database; if the probability value is less than the threshold, the clamping plate is considered to be in a stable state, and the system continues to monitor the data at the next moment. In addition, the system also sets a dynamic threshold adjustment mechanism to automatically correct the threshold range according to historical operating data and environmental conditions to adapt to the complex geological environment of different working faces. When the probability value exceeds the threshold at multiple consecutive moments and the rate of increase continues to increase, the system further determines that there is a continuous desquatization trend, thereby taking early measures to stop the machine for inspection or structural reinforcement.

[0026] Furthermore, step S1 specifically includes the following sub-steps: S101. Using industrial sensor groups installed on both sides and above the coal mining machine body, the morphological data of the cable clamps are collected in real time with time step t as the index. S102. Spatial alignment and noise filtering are performed on the collected morphological data to obtain a dense point cloud; S103. Based on the multimodal fusion algorithm, the collected morphological data of the cable clamps are combined with dense point clouds to construct a three-dimensional fusion model.

[0027] Specifically, the morphological data includes: point cloud data at time t, color image, and infrared reflectance intensity matrix.

[0028] Furthermore, the dense point cloud is generated through voxel downsampling and normal estimation. The 3D fusion model includes point cloud frames acquired by a structured light depth camera to reflect the 3D spatial geometry of the cable clamp, 2D texture image frames acquired by an RGB camera to enhance the surface features of the clamp and distinguish it from the background area, and infrared thermal image frames acquired by an infrared thermal imager to identify folded areas and abnormal stress areas of the clamp. These three types of data are registered and weighted in a unified coordinate system using a fusion function to generate a dense 3D fusion model. Specifically, the multi-frame depth point cloud data acquired by the structured light depth camera at different angles and temporal positions are registered in a coordinate system using camera extrinsic parameters (rotation matrix and translation vector), mapping each frame of 2D depth point cloud to a global coordinate system to form a complete 3D point cloud set, thereby reconstructing the overall 3D geometry of the clamp. Additionally, the infrared thermal image frames acquired by the infrared thermal imager are essentially a 2D distribution matrix of the clamp surface temperature, reflecting the thermal characteristics of different locations on the surface through the infrared radiation intensity value corresponding to each pixel. When the clamping plate is subjected to force, bending, or local folding, its contact pressure and frictional stress distribution change, resulting in abnormal local temperature gradients, which manifest as discontinuous areas of high or low brightness in infrared images. By performing spatial gradient analysis and temporal series comparison analysis on infrared thermal image data frames, the temperature change rate and local gradient direction of each pixel can be calculated. Edges of abrupt temperature gradient changes can be extracted and matched with the corresponding coordinates of the depth point cloud, enabling the identification of folded areas and abnormally stressed areas on the clamping plate surface. The construction principle of the above 3D fusion model is as follows: The structured light depth camera outputs a 2D point cloud frame, but each pixel corresponds to a depth value, thus obtaining 3D coordinates (x, y, z). The effective range of this depth information is limited (affected by the incident angle and the reflecting surface), but it can provide a geometric skeleton for the clamping plate surface in the main viewing direction. Secondly, the RGB camera image provides high-resolution texture information of the clamping plate surface. Through the extrinsic parameter calibration matrix, each pixel in the RGB image is mapped to the depth camera coordinate system and registered with the corresponding depth point cloud. After registration, RGB edge, texture gradient, and lighting information are used to repair point cloud voids caused by occlusion or reflection from the depth camera, making the point cloud model denser and more complete. Furthermore, infrared thermal imager data frames provide the temperature distribution field on the surface of the clamping plate. The infrared camera is also spatially calibrated, and its two-dimensional thermal image matrix is ​​projected onto the depth coordinate system through an extrinsic parameter matrix, forming a thermal distribution field in three-dimensional space corresponding to the point cloud position. Although the thermal image data itself does not have Z-axis information, by registering it with known depth coordinates, temperature or thermal stress attributes can be assigned to each point cloud, achieving two-dimensional modeling of both geometry and thermal field.

[0029] Furthermore, the industrial sensor array includes at least: Structured light depth camera is used to acquire depth point cloud data of the surface of the clamping plate and form point cloud frames; An RGB camera is used to capture two-dimensional texture images of clamps and cable trays; Infrared thermal imagers are used to acquire the infrared reflection matrix of the splice surface to identify splice boundaries and folded areas.

[0030] Furthermore, step S2 specifically includes the following sub-steps: S201. Perform cross-sectional scanning in the 3D fusion model, and extract the center curve along the central region of the cable clamp using the depth extreme value tracing method; S202. Calculate the local curvature for discrete points of the center curve, and calculate the average curvature. S203. Construct parameter pairs using time as an index, and construct a curvature sequence from parameter pairs at consecutive time points.

[0031] Furthermore, step S201 specifically includes the following sub-steps: S2011. Based on the output 3D fusion model, obtain the coordinates of each point cloud in the 3D fusion model; S2012. By analyzing each cross section ( Find the local maximum point along the depth direction and continuously trace the extreme points along the direction to form a continuous set of center curve points; The depth direction is the z-axis direction, and the specific process for finding local maxima is as follows: ; The The extreme depth of the i-th point in the vertical direction is represented by the following. Indicating the z-axis direction, the The coordinates of sampling point i along the x, y, and z axes are represented. This represents a dense point cloud, where i represents the index of a sampling point in the point cloud data. This indicates that the location with the highest density is found along the z-axis.

[0032] Specifically, after point cloud registration, the 3D fusion model forms the complete shape of the cable clamp in the entire space. PCA analysis of the entire point cloud yields the principal axis vector of the clamp, representing its extension trend in 3D space. Using this direction as the central axis, the initial central orientation of the clamp is constructed. Further, using the principal axis as a reference, the spatial envelope boundary of the clamp is determined by the projection range of the outermost edge points of the model. Based on this, starting from the clamp head along the principal axis, several orthogonal tangent planes are generated at fixed step sizes (exemplarily 5 mm or 10 mm steps). Each tangent plane is perpendicular to the principal axis. The intersection of these planes with the point cloud of the 3D fusion model forms local spatial sections. These sections are actually the intersection areas of the point cloud and the planes, presenting the 2D contour projection of the clamp on these sections. Furthermore, within each section, the upper and lower surface points of the clamp can be identified through the depth coordinate distribution. Since the clamp is typically thinner along the Z-direction, the depth difference between the upper and lower surfaces on the Z-axis directly reflects local morphological changes. At this point, by searching for the extreme point in the Z-direction along the depth axis within the cross-section, the spatial location of the center of the cable clamp thickness in the cross-section can be found—this point is the center point corresponding to that cross-section. Furthermore, the center points of each of the cross-sections along the extension direction of the cable clamp are continuously extracted and arranged sequentially, forming a continuous spatial curve, namely the center curve of the cable clamp. This curve is used to reflect the morphological changes of the clamp, such as bending, arching, or collapse.

[0033] Furthermore, the continuous set of center curve points is specifically represented as follows: ; The This represents the set of center curve points at the current time t. This represents the total number of points on the center curve.

[0034] Furthermore, in step S202, the specific process for calculating the average curvature is as follows: S2021. Calculate the local curvature of each discrete point in the center curve. Specifically, the finite difference formula is used to calculate the local curvature of discrete uniform arc length points. It should be noted that when calculating the local curvature of the center curve of the clamp, the center curve extracted by the depth extreme value tracking method needs to be parameterized for arc length. The principle of this process is to eliminate the numerical error caused by the uneven discrete distribution of the point cloud. The specific process is as follows: After obtaining the original discrete point set of the center curve from the 3D fusion model, the Euclidean distance between adjacent points is calculated and accumulated to obtain the cumulative arc length distribution of the entire curve. Within this cumulative arc length interval, several arc length nodes are re-divided at fixed intervals. The spatial coordinates of these equal arc length positions are calculated in the original curve point set by cubic spline interpolation or local polynomial fitting, generating a set of points uniformly distributed along the arc length of the curve. Since these newly generated points are evenly spaced in the arc length direction, they can ensure that the discrete approximation of the derivative and the second derivative has good numerical stability and smoothness during calculation. Therefore, they are called discrete uniform arc length points. Subsequently, based on these discrete uniform arc length points, the first and second derivatives are approximated using the difference formula to reflect the local rate of change of direction at each point, and the corresponding local curvature value is then calculated.

[0035] Preferably, the calculation can also be performed on a two-dimensional projection. The principle is as follows: project the central region of the clamping plate from three-dimensional space onto a two-dimensional plane (e.g., project it along the thickness direction onto the main plane of the clamping plate). The central curve can then be viewed as a continuous two-dimensional curve on the plane. To quantitatively describe the curvature of the curve, it is necessary to calculate the curvature. The essence of curvature is to describe the rate of change of the tangent direction of the curve at a certain point—if the curve is straight at that point, the tangent direction does not change, and the curvature is zero; the more severe the curvature, the faster the tangent direction changes, and the larger the curvature value. Based on the above description, for the implementation scheme in this embodiment, the curve consists of continuous coordinate points, each point having a horizontal position (x) and a vertical position (y) on the two-dimensional plane. By calculating the rate of change at each point (i.e., the derivative, which can be understood as the direction of the curve), and then calculating the rate of change of direction (i.e., the second derivative, which can be understood as the acceleration of the curve's curvature), the magnitude of the curvature at each point can be obtained, that is: ; S2022. Calculate the average curvature using the calculated local curvature: ; The The curvature value at the i-th point is represented by the following: express The first derivative, the express The second derivative, the express The first derivative, the express The second derivative, the This represents the curvature state at the current time t.

[0036] Furthermore, in step S203, the output parameter pair includes the current frame time and the curvature state at the current time, which is the average curvature, i.e.: .

[0037] Furthermore, step S3 specifically includes the following sub-steps: S301. Input the curvature sequence into the sliding window prediction model, and output the curvature state at the next moment through the sliding window prediction model; S302. Calculate the predicted shape deviation value based on the curvature state at the current moment and the curvature state at the next moment.

[0038] Furthermore, in step S301, the sliding window prediction model is represented as: ; Among them, the Indicates the index of the current time, the This indicates the predicted curvature state at the next moment. Represents the sliding window prediction function, the Indicates the width of the sliding window, the This represents the curvature state at the current time t, i.e., the mean curvature. This represents the curvature state at time t-w+1.

[0039] Furthermore, in step S302, the predicted morphological deviation value is calculated by combining the predicted curvature state at the next moment with the curvature state at the current moment, that is: The This represents the amount of curvature change, i.e., the predicted shape deviation value.

[0040] Among them, when When this happens, it is determined that the shape of the clamping plate is about to fold. This represents the threshold for the change in curvature.

[0041] Furthermore, step S4 specifically includes the following sub-steps: S401. Obtain the point cloud coordinate set from the 3D fusion model; S402. Extract the approximate horizontal plane region using the vertical direction normal vector to obtain the set of points on the upper surface of the clamping plate: S403. From the set of points on the upper surface of the clamping plate, select the point with the largest height to form the set of points on the top edge, and calculate the height difference at the top of the clamping plate; S404. Construct a morphological risk function by comprehensively predicting the morphological deviation value, the height difference at the top of the clamping plate, and the curvature change: ; Among them, the The morphological risk function represents the current time t. This represents the curvature change weighting coefficient, used to control the weight of the impact of curvature change on the overall risk. This indicates the weighting of the contribution of the height difference between the clamping plates to the overall risk, and is used to control the weighting of the contribution of the height difference between the clamping plates to the overall risk. This represents the height difference weighting coefficient, used for the curvature continuity weighting coefficient. The amount of curvature change, the The height change rate, i.e., the height difference at the top of the clamp, is calculated as the relative ratio of height changes between two consecutive frames. This indicates the maximum height difference at the top of the clamping plates. The curvature state at the current time t is represented by the following. This indicates the curvature state of the previous frame t-1.

[0042] Specifically, in step S402, an engineering approximation assumption is adopted: under normal installation conditions, the upper surface of the clamping plate is usually parallel to the equipment base or installation platform, and the overall tilt angle is small. Therefore, it can be regarded as a locally approximately horizontal planar region. However, strictly speaking, this assumption only holds under certain conditions. After manufacturing, installation, or long-term stress, the clamping plate may experience slight warping, tilting, or bending deformation, causing the normal vector of the upper surface to deviate from the vertical direction. In this case, if a fixed threshold (such as 0.9) is still used to filter points with small angles between the normal vector and the vertical direction, only an approximately horizontal region can be obtained, that is, an approximately horizontal plane region, rather than an absolute upper surface.

[0043] Furthermore, the set of top edge points is as follows: The Represents the set of top edge points, the This represents the maximum height value of the current frame, i.e., the maximum coordinate value in the z-axis direction. This represents the height neighborhood threshold, used to control the range of neighborhoods selected closest to the highest point. The coordinates of the j-th point on the upper surface of the clamping plate are represented by the x-axis, y-axis, and z-axis values. This represents the set of points on the upper surface of the clamping plate; the height difference at the top of the clamping plate is calculated using the minimum and maximum points in the set of points on the top edge, i.e.: The The minimum point in the set of top edge points is represented by the following: This represents the largest point in the set of top edge points.

[0044] Furthermore, step S5 specifically includes the following sub-steps: S501. Based on the constructed morphological risk function, and by inputting new monitoring data into the morphological risk function, calculate the morphological risk value at the current moment; S502. Based on the risk value of the current frame and the risk value of the previous frame, calculate the continuous change amount of the risk values ​​of the two frames, where the continuous change rate is 0 when the current frame is the initial frame. S503. Calculate the normalized probability using the Sigmoid function based on the morphological risk value and the continuous transformation rate.

[0045] Specifically, the calculation of the normalized probability using the Sigmoid function is expressed as follows: ; The Represents the normalized probability value, the This represents the risk weighting coefficient, used to control the influence weight of the risk value at the current time t. This represents the weighting coefficient, used to control the influence of the continuous transformation rate on the probability output.

[0046] Furthermore, the specific process for calculating the continuous transformation amount of the risk value between two frames is expressed as follows: ; Among them, the This represents the continuous change in risk values ​​between two frames. This represents the risk value at the current time t. This represents the risk value at the previous time t-1.

[0047] Furthermore, the specific process of step S6 is as follows: based on the normalized probability value... By setting a threshold for the desinking state Compare and determine the detachment status: ; when When =1, it indicates that based on the current frame's morphological risk value and continuous change rate, the cable clamp has a significant risk of detachment from the groove or is about to detach, and an alarm or shutdown should be triggered; when =0 indicates that the cable clamp is on the normal track.

[0048] Example 2 Furthermore, as a preferred embodiment of the above-described Example 1, an exemplary implementation process is proposed for steps S401-S402: Extracting surface normals from a 3D fusion model Calculate its vertical component Filter out the set of points on the upper surface of the clamping plate: , wherein The set of points on the upper surface of the clamp plate, the The index represents the point cloud index of the point set of the surface region. Represents a three-dimensional fusion model, the The normal vector threshold, typically set to 0.9, is used to filter non-horizontal areas. Specifically, the above implementation aims to identify points on the upper surface of the clamping plate, thereby enabling the analysis of its morphological characteristics and height distribution. The 3D fusion model is constructed from multi-source point cloud data collected by a depth camera, an RGB camera, and an infrared thermal imager. Each point in the model possesses spatial coordinate information and texture or thermal information. The process principle is as follows: local neighborhood analysis is performed on the point cloud surface. By calculating the spatial distribution of each point and its neighboring points, the direction of the point's normal vector is estimated. The normal vector represents the local directional characteristics of the surface at that point, with the vertical component pointing vertically to the working surface. By observing the magnitude of the normal vector in the vertical direction, it can be determined whether the surface of that point is close to a horizontal state. A threshold is set to filter the vertical component of the normal vector. The threshold is typically set to a high value, such as 0.9, indicating that only points with a height consistent with the horizontal plane, i.e., points with a vertical component close to 1, are retained. This effectively eliminates points on the sides of the clamping plate, folded areas, or stray point clouds, because the normal vector directions of these points deviate significantly from the horizontal plane and do not conform to the upper surface characteristics. The process involves filtering to obtain a set of points on the upper surface of the clamping plate, which reflects the plate's geometry. The core principle behind this process lies in using the directionality of the normal vector to distinguish different surface regions, and using the vertical component filtering to ensure that only the upper surface of the clamping plate is analyzed.

[0049] Example 3 Furthermore, as a preferred embodiment of the above-described embodiment one, a computer vision-based thin coal seam working face monitoring system is proposed. This system is implemented based on the computer vision-based thin coal seam working face monitoring method described in any one of the above embodiments, and includes: The model building module is used to collect morphological data of cable clamps through industrial sensor arrays and build a three-dimensional fusion model based on the morphological data. The curvature calculation module is used to extract the center curve of the 3D fusion model through the depth extreme value tracking method, calculate the average curvature of the clamp in the frame based on the discrete point set of the extracted center curve, output parameter pairs, and construct the parameter pairs of continuous time moments into a curvature sequence. The deviation prediction module constructs a morphological change time series from the curvature sequence at continuous time points, predicts the curvature state at the next time point through a sliding window model, and outputs the predicted morphological deviation value. The morphological risk function construction module is used to calculate the height difference at the top of the clamping plate based on the geometric height information of the 3D fusion model, and construct the morphological risk function by weighting and combining the predicted morphological deviation value with the calculated height difference at the top of the clamping plate with the relative change rate. The data calculation module is used to input new data into the morphological risk function, calculate the morphological risk value at the current moment and the continuous transformation rate of two consecutive frames, and output the probability value normalized to [0,1] through the Sigmoid function. The threshold determination module is used to determine the desinking status based on the calculated range probability value and the set desinking status threshold.

[0050] Furthermore, the industrial sensor array includes at least: Structured light depth camera is used to acquire depth point cloud data of the surface of the clamping plate and form point cloud frames; An RGB camera is used to capture two-dimensional texture images of clamps and cable trays; Infrared thermal imagers are used to acquire the infrared reflection matrix of the splice surface to identify splice boundaries and folded areas.

[0051] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A computer vision-based monitoring method for thin coal seam working faces, characterized in that, Includes the following steps: S1. Collect morphological data of cable clamps using an industrial sensor array, and construct a three-dimensional fusion model using the morphological data; S2. Extract the center curve of the 3D fusion model using the deep extremum tracking method, calculate the average curvature of the clamp in the frame based on the discrete point set of the extracted center curve, output parameter pairs, and construct a curvature sequence from the parameter pairs at consecutive time moments. S3. Construct the curvature sequence of consecutive time moments into a morphological change time series, predict the curvature state at the next time moment through a sliding window model, and output the predicted morphological deviation value; S4. Based on the geometric height information of the 3D fusion model, calculate the height difference at the top of the clamping plate, and construct a morphological risk function by weighting and combining the predicted morphological deviation value with the calculated height difference at the top of the clamping plate with the relative change rate. S5. Input new data into the morphological risk function, calculate the morphological risk value at the current moment and the continuous transformation of the risk value between two consecutive frames, and output the probability value normalized to [0,1] through the Sigmoid function; S6. Determine the desinking status based on the calculated range probability value and the set desinking status threshold.

2. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S1 specifically includes the following sub-steps: S101. Using industrial sensor groups installed on both sides and above the coal mining machine body, the morphological data of the cable clamps are collected in real time with time step t as the index. S102. Spatial alignment and noise filtering are performed on the collected morphological data to obtain a dense point cloud; S103. Based on the multimodal fusion algorithm, the collected morphological data of the cable clamps are combined with dense point clouds to construct a three-dimensional fusion model.

3. The method for monitoring thin coal seam working faces based on computer vision as described in claim 2, characterized in that, The industrial sensor group includes at least: Structured light depth camera is used to acquire depth point cloud data of the surface of the clamping plate and form point cloud frames; An RGB camera is used to capture two-dimensional texture images of clamps and cable trays; Infrared thermal imagers are used to acquire the infrared reflection matrix of the splice surface to identify the splice boundaries and folded areas.

4. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S2 specifically includes the following sub-steps: S201. Perform cross-sectional scanning in the 3D fusion model, and extract the center curve along the central region of the cable clamp using the depth extreme value tracing method; S202. Calculate the local curvature for discrete points of the center curve, and calculate the average curvature. S203. Construct parameter pairs using time as an index, and construct a curvature sequence from parameter pairs at consecutive time points.

5. The computer vision-based monitoring method for thin coal seam working faces as described in claim 4, characterized in that, Step S201 specifically includes the following sub-steps: S2011. Based on the output 3D fusion model, obtain the coordinates of each point cloud in the 3D fusion model; S2012. By finding the local maximum point along the depth direction for each cross section and continuously tracing the extreme points along the direction, a continuous set of center curve points is formed.

6. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S3 specifically includes the following sub-steps: S301. Input the curvature sequence into the sliding window prediction model, and output the curvature state at the next moment through the sliding window prediction model; S302. Calculate the predicted morphological deviation value based on the curvature state at the current moment and the curvature state at the next moment.

7. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, In step S301, the sliding window prediction model is represented as follows: ； Among them, the Indicates the index of the current time, the This indicates the predicted curvature state at the next moment. Represents the sliding window prediction function, the Indicates the width of the sliding window, the The curvature state at the current time t is represented by the following. This represents the curvature state at time t-w+1.

8. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S4 specifically includes the following sub-steps: S401. Obtain the point cloud coordinate set from the 3D fusion model; S402. Extract the approximate horizontal plane region using the vertical direction normal vector to obtain the set of points on the upper surface of the clamping plate: S403. From the set of points on the upper surface of the clamping plate, select the point with the largest height to form the set of points on the top edge, and calculate the height difference at the top of the clamping plate; S404. Construct a morphological risk function by comprehensively predicting the morphological deviation value, the height difference at the top of the clamping plate, and the curvature change: ； Among them, the The morphological risk function represents the current time t. This represents the curvature change weighting coefficient, used to control the weight of the impact of curvature change on the overall risk. This indicates the weighting of the contribution of the height difference between the clamping plates to the overall risk, and is used to control the weighting of the contribution of the height difference between the clamping plates to the overall risk. This represents the height difference weighting coefficient, used for the curvature continuity weighting coefficient. The amount of curvature change, the The height change rate is calculated as the relative ratio of height changes between two consecutive frames. Indicates the height difference at the top of the clamping plate, the The curvature state at the current time t is represented by the following. This indicates the curvature state of the previous frame t-1.

9. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S5 specifically includes the following sub-steps: S501. Based on the constructed morphological risk function, and by inputting new monitoring data into the morphological risk function, calculate the morphological risk value at the current moment; S502. Based on the risk value of the current frame and the risk value of the previous frame, calculate the continuous change amount of the risk values ​​of the two frames, where the continuous change rate is 0 when the current frame is the initial frame. S503. Based on the continuous transformation between the morphological risk value and the risk value of the two frames, calculate the normalized probability using the Sigmoid function.

10. The computer vision-based monitoring method for thin coal seam working faces as described in claim 9, characterized in that, The specific process for calculating the continuous transformation of risk values ​​between two frames is as follows: ； Among them, the This represents the continuous change in risk values ​​between two frames. This represents the risk value at the current time t. This represents the risk value at the previous time t-1.

11. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, The specific process of step S6 is as follows: based on the normalized probability value By setting a threshold for the desinking state Compare and determine the detachment status: ； when When =1, it indicates that based on the current frame's morphological risk value and continuous change rate, the cable clamp has a significant risk of detachment from the groove or is about to detach, and an alarm or shutdown should be triggered; when =0 indicates that the cable clamp is on the normal track.

12. A computer vision-based monitoring system for thin coal seam working faces, the system being implemented based on the computer vision-based monitoring method for thin coal seam working faces according to any one of claims 1-11, characterized in that, include: The model building module is used to collect morphological data of cable clamps through industrial sensor arrays and build a three-dimensional fusion model based on the morphological data. The curvature calculation module is used to extract the center curve of the 3D fusion model through the depth extreme value tracking method, calculate the average curvature of the clamp in the frame based on the discrete point set of the extracted center curve, output parameter pairs, and construct the parameter pairs of continuous time moments into a curvature sequence. The deviation prediction module constructs a morphological change time series from the curvature sequence at continuous time points, predicts the curvature state at the next time point through a sliding window model, and outputs the predicted morphological deviation value. The morphological risk function construction module is used to calculate the height difference at the top of the clamping plate based on the geometric height information of the 3D fusion model, and construct the morphological risk function by weighting and combining the predicted morphological deviation value with the calculated height difference at the top of the clamping plate with the relative change rate. The data calculation module is used to input new data into the morphological risk function, calculate the morphological risk value at the current moment and the continuous transformation of the risk value between two frames, and output the probability value normalized to [0,1] through the Sigmoid function; The threshold determination module is used to determine the desinking status based on the calculated range probability value and the set desinking status threshold.

13. The computer vision-based thin coal seam working face monitoring system as described in claim 12, characterized in that, The industrial sensor group includes at least: Structured light depth camera is used to acquire depth point cloud data of the surface of the clamping plate and form point cloud frames; An RGB camera is used to capture two-dimensional texture images of clamps and cable trays; Infrared thermal imagers are used to acquire the infrared reflection matrix of the splice surface to identify the splice boundaries and folded areas.

Images