A computer vision-based thin coal seam working face monitoring method and system

Abstract

The application discloses a kind of thin coal seam working face monitoring method and system based on computer vision, it is related to thin coal seam working face splint monitoring field.The application realizes continuous tracking and dynamic feature extraction to splint three-dimensional form by combining multi-source vision data fusion and time sequence form analysis modeling method.Specifically, the application uses structured light depth camera, RGB camera and infrared thermal imager to obtain the form features of splint in different dimensions, and constructs a three-dimensional model with spatial consistency by fusion;Then, combined with the curvature change, the height difference of the top and the time continuity feature, a form risk function is constructed to realize the quantitative, dynamic and real-time evaluation of splint off-slot risk, so that the system can determine the potential risk in the early deformation stage of splint.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of thin coal seam working face clamping plate monitoring, in particular to a thin coal seam working face monitoring method and system based on computer vision. BACKGROUND

[0002] The existing thin coal seam working face monitoring mainly relies on manual inspection and single sensor data acquisition. In the mine site, devices such as infrared range finder, two-dimensional camera or laser scanner are usually used to periodically detect the working face environment, and the surface geometric features of the clamping plate and cable groove are collected to judge whether there is displacement, bending or out-of-groove risk. Some systems also combine inertial sensors (such as IMU) or accelerometers for dynamic monitoring to improve measurement frequency and response speed. However, such methods generally rely on local geometric points or low-dimensional image information, and cannot continuously describe the overall spatial form of the clamping plate, and the structure deformation response capturing ability under complex coal seam environment is insufficient.

[0003] The defects of the prior art are: firstly, the data source of single sensor has the problems of low information dimension, strong noise interference and weak time sequence correlation, which makes it difficult to accurately reflect the three-dimensional shape change of the clamping plate under the dynamic coal seam stress environment; secondly, the existing monitoring method is mostly limited to static threshold judgment or single frame image recognition stage, and cannot realize the prediction of the clamping plate shape change trend, resulting in lag of out-of-groove risk judgment; thirdly, the existing method lacks correlation analysis of the geometric change and shape deviation of the clamping plate top, and cannot accurately evaluate the potential shape risk state, thereby limiting the early warning ability of the thin coal seam working face operation safety. SUMMARY

[0004] The present application proposes a thin coal seam working face monitoring method and system based on computer vision, aiming to build a thin coal seam working face monitoring technology system with adaptive analysis capability, realize intelligent identification of clamping plate shape change and accurate determination of out-of-groove state through visual fusion modeling and dynamic risk function calculation, and provide a more reliable, real-time and automated monitoring means for coal mine safety production.

[0005] Among them, a thin coal seam working face monitoring method based on computer vision includes the following steps:

[0006] S1. Collect the shape data of the cable clamping plate through the industrial sensor group, and construct a three-dimensional fusion model through the shape data;

[0007] Specifically, an industrial sensor array installed on the working surface synchronously collects information from the cable clamps. A structured light depth camera first acquires depth point cloud data of the clamp surface, and calculates the spatial coordinates of each pixel using the principle of light projection texture deformation to form a dense point cloud frame. An RGB camera synchronously acquires the two-dimensional texture information of the corresponding area, providing color constraints for subsequent point cloud registration and texture mapping. An infrared thermal imager identifies edge temperature gradient changes based on the surface reflection matrix to determine the clamp boundaries and folded areas. Multi-source data are aligned using a spatiotemporal synchronous calibration matrix, and the depth map and color map are fused using an extrinsic calibration matrix. Based on this, voxel filtering is used to remove noise points, and a multi-frame point cloud stitching is achieved using an ICP (Iterative Closest Point) based registration algorithm, resulting in a continuous, dense, and measurable three-dimensional fusion model that accurately reflects the geometry of the clamp.

[0008] S2. Extract the center curve of the 3D fusion model using the deep extremum tracking method, calculate the average curvature of the clamp in the frame based on the discrete point set of the extracted center curve, output parameter pairs, and construct a curvature sequence from the parameter pairs at consecutive time moments.

[0009] Specifically, the system extracts the central structure of the 3D fusion model using the depth extreme value tracking method. First, the system performs layered scanning of the point cloud data in the fusion model along the trajectory direction. Within each section, representative extreme points are determined based on the depth distribution. By connecting these extreme points layer by layer to form a continuous centerline, the main axis structure of the clamp's shape is established. This main axis curve reflects the overall stress shape characteristics and bending trend of the clamp. Then, the average curvature is calculated based on the set of discrete points on the center curve. The degree of change in curvature represents the concentration and directional deviation of the clamp's deformation. The average curvature of each frame is recorded in chronological order to form a curvature time series, thereby reflecting the dynamic change process of the clamp's shape. Under continuous operation, the system can observe whether the clamp exhibits periodic bending, slow deformation, or abrupt anomalies based on this sequence. The extraction of the center curve and the calculation of curvature together complete the quantitative expression of geometric change characteristics from 3D morphological data.

[0010] S3. Construct the curvature sequence of consecutive time moments into a morphological change time series, predict the curvature state at the next time moment through a sliding window model, and output the predicted morphological deviation value;

[0011] Specifically, the established curvature time series is taken as the main representation of the shape change, the local dynamic trend of the splint shape change is captured by using a sliding window model, the sliding window contains curvature data at consecutive time points, short-term fluctuation patterns and trend deviation information are extracted through time continuity analysis, and the curvature state at the next time point is predicted. This process realizes continuous calculation through continuous sliding of the time window, so that the system can perceive the speed and direction of shape change in real time. When the actual measured curvature deviates from the predicted curvature, the deviation is regarded as an early signal of shape anomaly. The system regards the deviation value as the predicted shape deviation degree, reflecting the gap between the current state of the splint and the normal evolution trajectory. This prediction mechanism dynamically models the splint state using time correlation, so that the system can judge potential risks through the rising trend of the deviation before the anomaly occurs.

[0012] S4. According to the geometric height information of the three-dimensional fusion model, the height difference of the top of the splint is calculated, and according to the predicted shape deviation value and the calculated height difference of the top of the splint, a shape risk function is constructed by combining the relative change rate through weighting;

[0013] Specifically, the geometric height difference of the top of the splint is calculated by the fusion model to reflect the vertical non-uniformity, and then the height difference is combined with the predicted shape deviation degree to form a multi-dimensional index describing the overall shape stability of the splint. In the calculation process, the system not only considers the instantaneous state of the current frame, but also weightedly evaluates the continuity of adjacent frames, thereby introducing time accumulation effect to reflect the inertia and mutation characteristics of shape change. The risk function comprehensively considers the local bending, vertical deformation and time continuous change of the splint as a whole, so that the obtained shape risk value can truly reflect the stress state and deformation trend of the splint in the working surface. When the splint is continuously bent or locally mutated, the risk function value will quickly rise, thereby providing the system with a quantifiable shape risk signal. In essence, this process converts complex geometric shape characteristics into an operable risk numerical model.

[0014] S5. Input new data to the shape risk function to calculate the shape risk value at the current time and the continuous transformation quantity of the risk values of two consecutive frames, and output the probability value normalized to [0, 1] through the Sigmoid function;

[0015] Specifically, the morphological risk function is converted into a probability index that can be used for state determination. When new monitoring data is input into the system, the system inputs it into the morphological risk function to calculate the risk value at the current time, and performs differential analysis with the previous frame of data to obtain the continuous change rate. If the system is running for the first time, the initial baseline value is used as the risk value at the previous time to prevent the model from being interrupted during calculation. Subsequently, the system maps the obtained risk value to a standardized probability space, and through a nonlinear mapping function, the result is constrained in the range of zero to one, so as to reflect the instability degree of the current clamp plate state in the form of continuous probability. The higher the risk value, the closer the output probability is to one, which represents that the morphology is close to the critical state of slotted or deformed. When the risk value is low, the probability tends to zero, indicating that the clamp plate is in a stable running interval. At the same time, the system monitors the probability difference of consecutive frames. If the probability continuously rises or the abnormal fluctuation is enlarged, it means that the clamp plate morphology is evolving towards a high-risk area. In this way, the system converts the physical morphology change process into a continuous probability signal.

[0016] S6. Determine the slotted state according to the calculated range probability value in combination with the set slotted state threshold value;

[0017] Specifically, the calculated probability result is compared with the preset slotted threshold value to determine the running state of the clamp plate. When the probability value reaches or exceeds the threshold value, the system determines that the clamp plate has entered a potential slotted state, and triggers a warning mechanism to send a warning signal to the monitoring end. When the probability value is below the threshold value, it is considered to be in a normal running state. If the probability value remains in the high interval for a plurality of consecutive time periods, the system further determines a persistent slotted trend and performs a secondary alarm. This process is not only based on the risk determination of a single frame, but also takes into account the continuous change in the time dimension to prevent false positives caused by temporary noise. In addition, the system can also evaluate the risk development speed according to the probability growth rate. When the growth rate exceeds the set threshold value, it is determined that the morphology has a sudden slotted tendency. Such a determination mechanism enables the system to accurately identify and trend warn the slotted state in a dynamic environment. Through the combination of probability and threshold value, the system realizes a closed-loop process from data perception, risk calculation to state decision.

[0018] Further, a thin coal seam working face monitoring system based on computer vision is proposed, which is realized based on the computer vision-based thin coal seam working face monitoring method in any of the above embodiments, comprising:

[0019] A model construction module is configured to collect morphological data of the cable clamp plate through an industrial sensor group, and construct a three-dimensional fusion model through the morphological data.

[0020] a curvature calculation module, configured to extract a center curve of the three-dimensional fusion model by a depth extremum tracking method, calculate an average bending rate of the clamp in the frame based on a discrete point set of the extracted center curve, output a parameter pair, and construct the parameter pairs of continuous time points into a curvature sequence;

[0021] a deviation prediction module, configured to construct the curvature sequence of continuous time points into a morphological change time sequence, predict a curvature state at a next time point by a sliding window model, and output a predicted morphological deviation value;

[0022] a morphological risk function construction module, configured to calculate a clamp top height difference according to geometric height information of the three-dimensional fusion model, combine the predicted morphological deviation value with the calculated clamp top height difference, and construct a morphological risk function by combining a relative change rate through weighting;

[0023] a data calculation module, configured to input new data into the morphological risk function, calculate a morphological risk value at a current time point and a continuous transformation quantity of two frame risk values, and output a probability value normalized to [0, 1] through a Sigmoid function;

[0024] a threshold determination module, configured to determine a slotted state according to the calculated range probability value and a set slotted state threshold.

[0025] Further, the industrial sensor group at least comprises:

[0026] a structured light depth camera, configured to collect depth point cloud data of a clamp surface to form a point cloud frame;

[0027] an RGB camera, configured to collect a two-dimensional texture image of the clamp and the cable slot;

[0028] an infrared thermal imager, configured to collect an infrared reflection matrix of the clamp surface to identify a clamp boundary and a folding area.

[0029] The present application has the following beneficial effects:

[0030] The present application realizes continuous tracking and dynamic feature extraction of the three-dimensional morphology of the clamp by combining multi-source visual data fusion and time sequence morphological analysis modeling methods. Specifically, the present application uses a structured light depth camera, an RGB camera and an infrared thermal imager to obtain morphological features of the clamp in different dimensions, and constructs a three-dimensional model with spatial consistency by fusion; then, a morphological risk function is constructed by combining curvature change, top height difference and time continuity features, realizing quantitative, dynamic and real-time evaluation of the clamp slotted risk, so that the system can determine potential risks in the early deformation stage of the clamp. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The present application is based on a method schematic diagram of a UAV target perception method adaptive to beam reconstruction and environment perception. DETAILED DESCRIPTION

[0032] The technical solutions of the present application will be described in further detail below with reference to the accompanying drawings, but the scope of protection of the present application is not limited to the following description.

[0033] For the purpose of the present application, the technical solutions and advantages are more clearly and obviously understood, the present application will be further described in detail in conjunction with the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application, that is, the described examples are only a part of the examples of the present application, but not all the examples. The components of the embodiments of the present application generally described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0034] Therefore, the detailed description of the embodiments of the present application provided in the accompanying drawings below is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of the present application. It should be noted that the relationship terms such as "first" and "second" and the like are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations.

[0035] Moreover, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or mechanical equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or mechanical equipment. Without more limitations, the element defined by the sentence "comprises a" does not exclude the presence of another identical element in the process, method, article or mechanical equipment comprising the element.

[0036] The features and properties of the present application will be further described in detail below in conjunction with the examples.

[0037] Example 1

[0038] Among them, such as Figure 1 A thin coal seam working face monitoring method based on computer vision, comprising the following steps:

[0039] S1. Collecting the shape data of the cable clamp plate through the industrial sensor group, and constructing a three-dimensional fusion model through the shape data;

[0040] Specifically, the specific principle flow of step S1 is: the spatial form of the cable clamp plate is synchronously collected by an industrial sensor group installed in the support area of the thin coal seam working face. The sensor group realizes the collaborative collection of data of the depth camera, RGB camera and infrared thermal imager through a multi-channel synchronous triggering system. The depth camera calculates the depth information of the points on the surface of the clamp plate based on the principle of structured light projection by measuring the deformation of the stripes; the RGB camera is responsible for recording the two-dimensional texture features of the surface of the clamp plate to assist the spatial registration of the depth data; the infrared thermal imager is used to identify the local thermal distribution difference of the surface of the clamp plate caused by stress concentration or structural deformation, thereby indirectly reflecting the abnormal area of the form. According to the internal and external parameter calibration results of the sensor, the system fuses the three types of data in a unified coordinate system, and generates multi-source point cloud data through feature alignment and geometric constraint methods. Then, the spatial correspondence between consecutive frames is established by using a time sequence splicing algorithm, and a dense three-dimensional fusion model is generated after removing redundant noise points.

[0041] S2. Extract the center curve of the three-dimensional fusion model by the depth extreme value tracking method, calculate the average bending rate of the clamp plate in the frame based on the discrete point set of the extracted center curve, output the parameter pair, and construct the curvature sequence of the continuous time.

[0042] Specifically, the specific principle flow of step S2 is: in the obtained three-dimensional fusion model, the geometric extraction of the center structure of the clamp plate is realized by the depth extreme value tracking method. Specifically, the system first divides the fusion model into a plurality of cross-sectional slices along the length direction, and performs extreme value analysis on the depth distribution of the point cloud in each slice to identify the depth extreme value point that best represents the geometric center of the clamp plate. Then, the system establishes a corresponding relationship between the extreme value points in adjacent slices, and forms a continuous center curve through interpolation and neighborhood fitting. This curve, as the geometric skeleton of the clamp plate form, can reflect the overall bending trend of the clamp plate along the longitudinal direction. Then, the system calculates the spatial curvature distribution of the discrete points on the curve, and obtains the average curvature of each frame to quantify the overall bending degree of the clamp plate at that time. By arranging the average curvatures of the consecutive frames in time sequence, the system forms a curvature sequence and establishes a dynamic trajectory of the form change over time. The implementation principle of this process is to track the geometric evolution of the clamp plate by using the spatial continuity of the depth gradient in the point cloud, so that the system can obtain continuous time sequence information of the structural deformation without relying on any mechanical sensor.

[0043] S3. Construct the curvature sequence of the continuous time into a form change time sequence, predict the curvature state at the next time through a sliding window model, and output the predicted form deviation value.

[0044] Specifically, the specific implementation principle flow of step S3 is: input the curvature sequence at continuous time into the morphological change time sequence model, and analyze the dynamic characteristics in a short period by using the sliding window method. The sliding window covers a fixed number of curvature data at each calculation to capture the change trend of the splint in a specific time interval. The system estimates the predicted curvature at the next time using the local linear characteristics of the curvature in the window, and compares it with the actual curvature to calculate the morphological deviation value. The size of the deviation value directly reflects the difference between the splint morphology and the historical evolution trajectory, thereby revealing whether the morphological change deviates from the normal trend. The implementation principle of this process is to estimate the inertial change of the splint morphology through time series prediction, so that the system can identify abnormal growth trends before the morphology has a significant mutation. When the deviation value continues to increase, the system determines that there is a potential folding or deformation risk of the splint morphology.

[0045] S4. According to the geometric height information of the three-dimensional fusion model, calculate the height difference of the top of the splint, and according to the predicted morphological deviation value and the calculated height difference of the top of the splint, construct a morphological risk function by weighting and combining the relative change rate;

[0046] Specifically, the specific implementation principle flow of step S4 is: according to the geometric structure of the three-dimensional fusion model, the system extracts the point cloud of the top boundary area of the splint and calculates the vertical height difference. This height difference reflects the flatness of the upper surface of the splint and the uniformity of the stress distribution, which is a key parameter for judging the stability of the splint structure. The system combines the predicted morphological deviation value and the calculated height difference of the top of the splint by weighting to form a multi-dimensional morphological risk function. The risk function not only describes the current geometric state, but also introduces the time change rate as a correction term to capture the dynamic characteristics of the morphological change. When the height difference of the top of the splint increases rapidly or the predicted deviation accumulates continuously, the risk function value will increase significantly, indicating that the structure has a folding, warping or stress concentration trend. The design logic of this risk model is to fuse the geometric spatial characteristics and the predicted trend information, so that the system can infer the potential risk level through the morphological change itself without relying on external load monitoring.

[0047] S5. Input new data into the morphological risk function, calculate the morphological risk value at the current time and the continuous transformation amount of the two frames of risk values, and output the probability value normalized to [0, 1] through the Sigmoid function;

[0048] Specifically, the specific implementation principle flow of step S5 is: after receiving a new frame of shape data, input it into the constructed shape risk function, calculate the risk value at the current time, and difference it with the risk value at the last time to obtain the continuous change rate. When the system is first run, the initial calibration value is used as a reference to ensure the time continuity of the risk function. Subsequently, the system uses a nonlinear mapping mechanism to convert the risk value into a standardized probability value, so that it is distributed between 0 and 1, forming a quantifiable shape stability index. The higher the risk value, the closer the clamp plate is to the instability state, and the corresponding output probability is also closer to 1, and vice versa. The normalization process eliminates the dimensional difference of the risk value by introducing a nonlinear mapping function, making the data at different time periods and under different environmental conditions comparable. At the same time, the system monitors the change trend of the probability between consecutive frames in real time, and when the probability rises at a speed exceeding the set threshold, the system will mark it as a potential dangerous state.

[0049] S6. Determine the stripping state according to the calculated range probability value in combination with the set stripping state threshold;

[0050] Specifically, the specific implementation principle flow of step S6 is: compare the probability value output in the last stage with the preset stripping threshold to determine whether the clamp plate enters an abnormal state. When the probability value is greater than the threshold, the system determines that the shape of the clamp plate is in the stripping risk interval, immediately triggers the alarm and recording mechanism, and generates an event identifier with a time stamp and stores it in the database; if the probability value is less than the threshold, it is considered that the clamp plate is in a stable state, and the system continues to monitor the data at the next time. In addition, the system also sets a dynamic threshold adjustment mechanism to automatically correct the threshold range according to historical operation data and environmental conditions to adapt to the complex geological environment of different working faces. When the probability value exceeds the threshold at multiple consecutive times and the rising rate continues to increase, the system further determines that there is a persistent stripping trend, so that measures such as shutdown inspection or structure reinforcement are taken in advance.

[0051] Further, the step S1 specifically includes the following sub-steps:

[0052] S101. Collect the shape data of the cable clamp plate in real time by the industrial sensor group installed on both sides and above the body of the coal mining machine, with time step t as the index;

[0053] S102. Spatially align and filter the collected shape data to obtain a dense point cloud;

[0054] S103. Construct a three-dimensional fusion model according to a multi-modal fusion algorithm, combining the collected shape data of the cable clamp plate with the dense point cloud.

[0055] Specifically, the shape data includes: point cloud data at time t, color image, and infrared reflection intensity matrix.

[0056] Further, the dense point cloud is generated by voxel down-sampling and normal estimation; the three-dimensional fusion model includes a point cloud frame acquired by a structured light depth camera, used to reflect the three-dimensional spatial geometry of the cable clamp plate, a two-dimensional texture image frame acquired by an RGB camera, used to enhance the surface features of the clamp plate and distinguish the clamp plate from the background area, and an infrared thermal image data frame acquired by an infrared thermal imager, used to identify the folded area and the abnormal stress area of the clamp plate, and the three types of data are registered and weighted fused in a unified coordinate system to generate a dense three-dimensional fusion model. Wherein, the structured light depth camera collects multiple frames of depth point cloud data at different angles and time sequence positions, and performs coordinate system unification and spatial registration through camera external parameters (rotation matrix and translation vector), maps each frame of two-dimensional depth point cloud to the global coordinate system to form a complete three-dimensional point cloud set, and thus reconstructs the overall three-dimensional geometry of the clamp plate. In addition, the infrared thermal image data frame acquired by the infrared thermal imager is essentially a two-dimensional distribution matrix of the surface temperature of the clamp plate, which reflects the thermal features of different positions on the surface through the infrared radiation intensity value of each pixel point. When the clamp plate is stressed, bent or locally folded, the contact pressure and friction stress distribution changes, thereby causing an abnormal local temperature gradient, which is manifested as a highlight or low-light discontinuous area in the infrared image. Through spatial gradient analysis and time sequence comparison analysis of the infrared thermal image data frame, the temperature change rate and local gradient direction of each pixel point are calculated, the temperature gradient mutation edge can be extracted, and the edge is matched with the corresponding coordinates of the depth point cloud to identify the folded area and the abnormal stress area on the surface of the clamp plate. The construction principle flow of the above three-dimensional fusion model is as follows: the structured light depth camera outputs a two-dimensional point cloud frame, but each pixel point corresponds to a depth value, that is, a three-dimensional coordinate (x, y, z) can be obtained. The effective range of this depth information is limited (affected by the incident angle and the reflecting surface), but it can provide the geometric skeleton of the clamp plate surface in the main viewing direction. Secondly, the RGB camera image provides high-resolution texture information of the clamp plate surface. Through the external parameter calibration matrix, each pixel point in the RGB image is mapped to the depth camera coordinate system and registered with the corresponding depth point cloud. After registration, the edges, texture gradients and lighting information of the RGB are used to repair the hollow areas of the point cloud caused by occlusion or reflection of the depth camera, so that the point cloud model is more dense and complete. Further, the infrared thermal imager data frame provides the temperature distribution field of the clamp plate surface. The infrared camera is also spatially calibrated, and the two-dimensional thermal image matrix is projected to the depth coordinate system through the external parameter matrix to form a thermal distribution field corresponding to the position of the point cloud in the three-dimensional space. Although the thermal image data itself has no Z-axis information, by registration with the known depth coordinates, each point cloud can be assigned a temperature or thermal stress attribute, realizing geometric + thermal field dual-dimension modeling.

[0057] Further, the industrial sensor group at least includes:

[0058] Structured light depth camera is used to acquire depth point cloud data of the surface of the clamping plate and form point cloud frames;

[0059] An RGB camera is used to capture two-dimensional texture images of clamps and cable trays;

[0060] Infrared thermal imagers are used to acquire the infrared reflection matrix of the splice surface to identify splice boundaries and folded areas.

[0061] Furthermore, step S2 specifically includes the following sub-steps:

[0062] S201. Perform cross-sectional scanning in the 3D fusion model, and extract the center curve along the central region of the cable clamp using the depth extreme value tracing method;

[0063] S202. Calculate the local curvature for discrete points of the center curve, and calculate the average curvature.

[0064] S203. Construct parameter pairs using time as an index, and construct a curvature sequence from parameter pairs at consecutive time points.

[0065] Furthermore, step S201 specifically includes the following sub-steps:

[0066] S2011. Based on the output 3D fusion model, obtain the coordinates of each point cloud in the 3D fusion model;

[0067] S2012. By analyzing each cross section ( Find the local maximum point along the depth direction and continuously trace the extreme points along the direction to form a continuous set of center curve points;

[0068] The depth direction is the z-axis direction, and the specific process for finding local maxima is as follows:

[0069] ;

[0070] The The extreme depth of the i-th point in the vertical direction is represented by the following. Indicating the z-axis direction, the The coordinates of sampling point i along the x, y, and z axes are represented. This represents a dense point cloud, where i represents the index of a sampling point in the point cloud data. This indicates that the location with the highest density is being searched along the z-axis.

[0071] Specifically, the three-dimensional fusion model forms the complete form of the cable clamp plate in the whole space after point cloud registration. The main shaft direction vector of the clamp plate is obtained by performing PCA analysis on the whole point cloud, and the vector represents the extension trend of the clamp plate in the three-dimensional space. The preliminary center line of the clamp plate is constructed with the direction as the center axis. Further, the space envelope boundary of the clamp plate is determined by the projection range of the outermost edge points of the model with the main shaft direction as the reference. On this basis, a plurality of tangent planes orthogonal to the main shaft are generated along the main shaft direction from the head of the clamp plate at a fixed step (for example, every 5 mm or 10 mm step). Each tangent plane is perpendicular to the main shaft, and the intersection of the plane and the point cloud of the three-dimensional fusion model forms a local space section. These sections are actually the intersection area of the point cloud and the plane, and present the two-dimensional contour projection of the clamp plate on the section. Further, in each section, the upper surface and lower surface points of the clamp plate are identified by the depth coordinate distribution. Since the thickness of the clamp plate is usually small along the Z direction, the depth difference of the upper and lower surfaces on the Z axis directly reflects the local shape change. At this time, the spatial position of the thickness center of the clamp plate in the section can be found by searching for the extreme value point of the Z direction along the depth axis direction in the section, that is, the point is the center point corresponding to the section. Further, in all sections along the extension direction of the clamp plate, the respective center points are continuously extracted and arranged in order, forming a continuous space curve, that is, the center curve of the cable clamp plate. The curve is used to reflect the shape change of the clamp plate, such as bending, arching or collapsing, etc.

[0072] Further, the continuous center curve point set is specifically represented as:

[0073] ;

[0074] The represents the center curve point set at the current time t, and the represents the total number of center curve points.

[0075] Further, in step S202, the specific process of calculating the average bending rate is:

[0076] S2021. Calculate the local curvature of each discrete point in the center curve, specifically, the local curvature is calculated by using the difference formula on the discrete uniform arc length points, it should be noted that when calculating the local curvature of the center curve extracted by the depth extremum tracking method, the center curve needs to be arc length parameterized, the principle of this process is to eliminate the numerical error caused by the uneven distribution of point cloud. The specific process is: after obtaining the original discrete point set of the center curve from the three-dimensional fusion model, the Euclidean distance between adjacent points is calculated and accumulated to obtain the cumulative arc length distribution of the entire curve, a number of arc length nodes are re-divided in the cumulative arc length interval at a fixed interval, and the spatial coordinate points at these arc length positions are calculated in the original curve point set through cubic spline interpolation or local polynomial fitting to generate a sequence of points uniformly distributed along the curve arc length. Since these newly generated points are uniformly spaced in the arc length direction, the discrete approximation of the first and second derivatives has good numerical stability and smoothness in calculation, so they are called discrete uniform arc length points. Subsequently, based on these discrete uniform arc length points, the first and second derivatives are approximated using the difference formula to reflect the local direction change rate of each point, and then the corresponding local curvature value is calculated.

[0077] As preferred, the calculation can also be performed on a two-dimensional projection, the principle process of which is: project the center region of the clamp from three-dimensional space to a two-dimensional plane (such as projecting along the thickness direction to the main plane of the clamp), and the center curve on the plane can be regarded as a continuous two-dimensional curve. In order to quantitatively describe the bending degree of the curve, the curvature needs to be calculated, and the essence of the curvature is to describe the change rate of the tangent direction of the curve at a point - if the curve is straight at this point, the tangent direction does not change, and the curvature is zero; if the bending is more severe, the tangent direction changes faster, and the curvature value is larger. Through the above description, for the implementation scheme in the embodiment, the curve is composed of continuous coordinate points, each point has a horizontal position (x) and a vertical position (y) on the two-dimensional plane. By calculating the change rate (i.e. derivative, which can be understood as the direction of the curve) of each point, and then calculating the direction change rate (i.e. second derivative, which can be understood as the acceleration of the curve bending), the curvature at each point can be obtained, i.e.

[0078] ;

[0079] S2022. Calculate the average curvature by the calculated local curvature:

[0080] ;

[0081] The represents the curvature value of the i-th point, the represents the first derivative of the , and the represents second derivative of the curvature of the current time t, the curvature of the current time t being represented by first derivative of the curvature of the current time t, the curvature of the current time t being represented by second derivative of the curvature of the current time t, the curvature of the current time t being represented by second derivative of the curvature of the current time t, the curvature of the current time t being represented by

[0082] Further, in the step S203, the output parameter pair includes the current frame time and the curvature state of the current time, and the curvature state is the average curvature, i.e. .

[0083] Further, the step S3 specifically includes the following sub-steps:

[0084] S301. Input the curvature sequence into the sliding window prediction model, and output the next time curvature state through the sliding window prediction model;

[0085] S302. Calculate the predicted shape deviation value according to the current time curvature state and the next time curvature state.

[0086] Further, in the step S301, the sliding window prediction model is represented as:

[0087] ;

[0088] wherein, the indicates the current time index, the indicates the predicted next time curvature state, the indicates the sliding window prediction function, the indicates the sliding window width, the indicates the curvature state of the current time t, i.e. the average curvature, indicates the curvature state at t-w+1 time.

[0089] Further, in the step S302, the predicted shape deviation value is calculated by combining the predicted next time curvature state with the curvature state of the current time, i.e. ; the indicates the curvature change amount, i.e. the predicted shape deviation value.

[0090] wherein, when , it is determined that the clamping plate shape will change fold. indicates the curvature change amount threshold.

[0091] Further, the step S4 specifically includes the following sub-steps:

[0092] S401. Obtain a point cloud coordinate set from the three-dimensional fusion model; ​​

[0093] S402. Extract the approximate horizontal plane region using the vertical direction normal vector to obtain the set of points on the upper surface of the clamping plate:

[0094] S403. From the set of points on the upper surface of the clamping plate, select the point with the largest height to form the set of points on the top edge, and calculate the height difference at the top of the clamping plate;

[0095] S404. Construct a morphological risk function by comprehensively predicting the morphological deviation value, the height difference at the top of the clamping plate, and the curvature change:

[0096] ;

[0097] Among them, the The morphological risk function represents the current time t. This represents the curvature change weighting coefficient, used to control the weight of the impact of curvature change on the overall risk. This indicates the weighting used to control the contribution of the height difference between the clamping plates to the overall risk. The curvature continuity weighting coefficient is represented by the following. The amount of curvature change, the The height change rate, i.e., the height difference at the top of the clamp, is calculated as the relative ratio of height changes between two consecutive frames. This indicates the maximum height difference at the top of the clamping plates. The curvature state at the current time t is represented by the following. This indicates the curvature state of the previous frame t-1.

[0098] Specifically, in step S402, an engineering approximation assumption is adopted: under normal installation conditions, the upper surface of the clamping plate is usually parallel to the equipment base or installation platform, and the overall tilt angle is small. Therefore, it can be regarded as a locally approximately horizontal planar region. However, strictly speaking, this assumption only holds under certain conditions. After manufacturing, installation, or long-term stress, the clamping plate may experience slight warping, tilting, or bending deformation, causing the normal vector of the upper surface to deviate from the vertical direction. In this case, if a fixed threshold (such as 0.9) is still used to filter points with small angles between the normal vector and the vertical direction, only an approximately horizontal region can be obtained, that is, an approximately horizontal plane region, rather than an absolute upper surface.

[0099] Furthermore, the set of top edge points is as follows: The Represents the set of top edge points, the This represents the maximum height value of the current frame, i.e., the maximum coordinate value in the z-axis direction. This represents the height neighborhood threshold, used to control the range of neighborhoods selected closest to the highest point. represents the coordinate value of the x-axis, y-axis and z-axis of the jth point on the upper surface of the clamp plate, and the represents the point set on the upper surface of the clamp plate; the height difference of the top of the clamp plate is calculated by the minimum point and the maximum point in the top edge point set, that is: , the represents the minimum point in the top edge point set, and the represents the maximum point in the top edge point set.

[0100] Further, the step S5 specifically includes the following sub-steps:

[0101] S501. According to the constructed morphological risk function, and by inputting new monitoring data into the morphological risk function, the morphological risk value at the current time is calculated;

[0102] S502. According to the current frame risk value and the previous frame risk value, the continuous transformation amount of the two frames of risk values is calculated, wherein when the current frame is the initial frame, the continuous change rate is 0;

[0103] S503. According to the morphological risk value and the continuous transformation rate, the normalized probability is calculated by the Sigmoid function.

[0104] Specifically, the calculation of the normalized probability by the Sigmoid function is specifically represented as:

[0105] ;

[0106] The represents the normalized probability value, and the represents the risk weight coefficient, which is used to control the influence weight of the risk value at the current time t, and the represents the change weight coefficient, which is used to control the influence weight of the continuous transformation rate on the probability output.

[0107] Further, the specific process of calculating the continuous transformation amount of the two frames of risk values is represented as:

[0108] ;

[0109] Among them, the represents the continuous transformation amount of the two frames of risk values, the represents the risk value at the current time t, and the represents the risk value at the previous time t-1.

[0110] Further, the specific process of the step S6 is: according to the normalized probability value , by comparing with the set stripping state threshold , the stripping state is determined:

[0111] ;

[0112] When =1, it means that according to the shape risk value and the continuous transformation rate of the current frame, the cable clamp plate has a significant risk of being out of the groove or about to be out of the groove, and an alarm or shutdown measure should be triggered; when =0, it means that the cable clamp plate is in a normal track.

[0113] Example Two

[0114] Further, as a preferred embodiment of the above-mentioned example one, for steps S401-S402, an exemplary implementation process is proposed:

[0115] Extracting surface normal vector from three-dimensional fusion model , calculating its vertical component . Screening out the upper surface point set of the clamp plate: , wherein represents the upper surface point set of the clamp plate, the represents the point cloud index in the surface area point set, the represents the three-dimensional fusion model, and the represents the normal vector threshold, which is usually 0.9, used to filter non-horizontal areas. Specifically, the above-mentioned embodiment aims to identify the upper surface points of the clamp plate, so as to analyze its shape characteristics and height distribution. The three-dimensional fusion model is constructed by multi-source point cloud data collected by a depth camera, an RGB camera and an infrared thermal imager. Each point in the model has spatial coordinate information and texture or thermal information. The principle of the process is as follows: local neighborhood analysis is performed on the point cloud surface, and the spatial distribution of each point and its neighborhood points is calculated to estimate the direction of the normal vector of the point. The normal vector represents the local directional characteristics of the surface at the point, and the vertical component points to the vertical direction of the working surface. By observing the size of the normal vector in the vertical direction, it can be judged whether the surface of the point is close to the horizontal state; by setting a threshold to screen the vertical component of the normal vector, the threshold is usually a high value, such as 0.9, which means only the points consistent with the horizontal direction in height, i.e. the points with a vertical component close to 1, are retained. The side surfaces, folded areas or stray point clouds of the clamp plate can be effectively eliminated, because the normal vector directions of these points deviate greatly from the horizontal plane and do not meet the upper surface characteristics. The upper surface point set of the clamp plate is obtained by screening, and these points are used to reflect the geometric shape of the clamp plate. The core of the implementation principle of the above-mentioned process lies in distinguishing different surface areas by using the directionality of the normal vector, and ensuring that only the upper surface of the clamp plate is analyzed by screening the vertical component.

[0116] Example Three

[0117] Further, as a preferred embodiment of the above-mentioned embodiment one, a thin coal seam working face monitoring system based on computer vision is proposed, which is implemented based on the computer vision-based thin coal seam working face monitoring method in any of the above-mentioned embodiments, comprising:

[0118] A model construction module is configured to collect the shape data of the cable clamp plate through the industrial sensor group, and construct a three-dimensional fusion model through the shape data.

[0119] A curvature calculation module is configured to extract the center curve of the three-dimensional fusion model through the depth extreme value tracking method, calculate the average bending rate of the clamp plate in the frame based on the discrete point set of the extracted center curve, output the parameter pair, and construct the parameter pair of the continuous time into a curvature sequence.

[0120] A deviation prediction module is configured to construct the curvature sequence of the continuous time into a shape change time sequence, predict the curvature state at the next time through a sliding window model, and output a predicted shape deviation value.

[0121] A shape risk function construction module is configured to calculate the height difference of the top of the clamp plate according to the geometric height information of the three-dimensional fusion model, and construct a shape risk function through the relative change rate combined with the calculated height difference of the top of the clamp plate and the predicted shape deviation value.

[0122] A data calculation module is configured to input new data into the shape risk function, calculate the shape risk value at the current time and the continuous transformation rate of the continuous two frames, and output a probability value normalized to [0, 1] through a Sigmoid function.

[0123] A threshold determination module is configured to determine the off-slot state according to the calculated range probability value combined with the set off-slot state threshold.

[0124] Further, the industrial sensor group at least comprises:

[0125] A structured light depth camera is configured to collect depth point cloud data of the surface of the clamp plate, forming a point cloud frame.

[0126] An RGB camera is configured to collect two-dimensional texture images of the clamp plate and the cable slot.

[0127] An infrared thermal imager is configured to collect infrared reflection matrices of the surface of the clamp plate, and identify the clamp plate boundary and the folding area.

[0128] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and operation described. Accordingly, all such variations are intended to be included within the scope of the present application as defined in the claims below and their equivalents.

Claims

1. A computer vision-based monitoring method for thin coal seam working faces, characterized in that, Includes the following steps: S1. Collect morphological data of cable clamps using an industrial sensor array, and construct a three-dimensional fusion model using the morphological data; S2. Extract the center curve of the 3D fusion model using the deep extremum tracking method, calculate the average curvature of the clamp in the current frame based on the discrete point set of the extracted center curve, output parameter pairs, and construct a curvature sequence from the parameter pairs at consecutive time moments. S3. Construct the curvature sequence of consecutive time moments into a morphological change time series, predict the curvature state at the next time moment through a sliding window model, and output the predicted morphological deviation value; S4. Based on the geometric height information of the 3D fusion model, calculate the height difference at the top of the clamping plate, and construct a morphological risk function by weighting and combining the predicted morphological deviation value with the calculated height difference at the top of the clamping plate with the relative change rate. S5. Input new data into the morphological risk function, calculate the morphological risk value at the current moment and the continuous transformation of the risk value between two consecutive frames, and output the probability value normalized to [0,1] through the Sigmoid function; S6. Determine the desinking status based on the calculated range probability value and the set desinking status threshold; The morphological data includes: point cloud data at time t, color image, and infrared reflectance intensity matrix; The three-dimensional fusion model includes: Point cloud frames acquired by a structured light depth camera are used to reflect the three-dimensional spatial geometry of the cable clamp. Two-dimensional texture image frames acquired via an RGB camera are used to enhance the surface features of the clamping plate and distinguish the clamping plate from the background area; and Infrared thermal image data frames acquired by an infrared thermal imager are used to identify folded areas and abnormally stressed areas of the clamping plate. The point cloud frame, the two-dimensional texture image frame, and the infrared thermal image data frame are registered and weighted fused in a unified coordinate system through a fusion function. The morphological risk function is expressed as: ； Among them, the The morphological risk function represents the current time t. This represents the curvature change weighting coefficient, used to control the weight of the impact of curvature change on the overall risk. This indicates the weight of the contribution of the height difference between the control plates to the overall risk. The curvature continuity weighting coefficient is represented by the following. The amount of curvature change, the The height change rate, i.e., the height difference at the top of the clamp, is calculated as the relative ratio of height changes between two consecutive frames. This indicates the maximum height difference at the top of the clamping plates. The curvature state at the current time t is represented by the following. This indicates the curvature state of the previous frame t-1; Specifically, this is expressed through the Sigmoid function as follows: ； The Represents the normalized probability value, the This represents the risk weighting coefficient, used to control the influence weight of the risk value at the current time t. This represents the change weighting coefficient, used to control the influence of the continuous transformation rate on the probability output. This represents the continuous change in risk values ​​between two frames.

2. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S1 specifically includes the following sub-steps: S101. Using industrial sensor groups installed on both sides and above the coal mining machine body, the morphological data of the cable clamps are collected in real time with time step t as the index. S102. Spatial alignment and noise filtering are performed on the collected morphological data to obtain a dense point cloud; S103. Based on the multimodal fusion algorithm, the collected morphological data of the cable clamps are combined with dense point clouds to construct a three-dimensional fusion model.

3. The method for monitoring thin coal seam working faces based on computer vision as described in claim 2, characterized in that, The industrial sensor group includes at least: Structured light depth camera is used to acquire depth point cloud data of the surface of the clamping plate and form point cloud frames; An RGB camera is used to capture two-dimensional texture images of clamps and cable trays; Infrared thermal imagers are used to acquire the infrared reflection matrix of the splice surface to identify splice boundaries and folded areas.

4. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S2 specifically includes the following sub-steps: S201. Perform cross-sectional scanning in the 3D fusion model, and extract the center curve along the central region of the cable clamp using the depth extreme value tracing method; S202. Calculate the local curvature for discrete points of the center curve, and calculate the average curvature. S203. Construct parameter pairs using time as an index, and construct a curvature sequence from parameter pairs at consecutive time points.

5. The method for monitoring thin coal seam working faces based on computer vision as described in claim 4, characterized in that, Step S201 specifically includes the following sub-steps: S2011. Based on the output 3D fusion model, obtain the coordinates of each point cloud in the 3D fusion model; S2012. By finding the local maximum point along the depth direction for each cross section and continuously tracing the extreme points along the direction, a continuous set of center curve points is formed.

6. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S3 specifically includes the following sub-steps: S301. Input the curvature sequence into the sliding window prediction model, and output the curvature state at the next moment through the sliding window prediction model; S302. Calculate the predicted shape deviation value based on the curvature state at the current moment and the curvature state at the next moment.

7. The method for monitoring thin coal seam working faces based on computer vision as described in claim 6, characterized in that, In step S301, the sliding window prediction model is represented as follows: ； Among them, the Indicates the index of the current time, the This indicates the predicted curvature state at the next moment. Represents the sliding window prediction function, the Indicates the width of the sliding window, the The curvature state at the current time t is represented by the following. This represents the curvature state at time t-w+1.

8. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S4 specifically includes the following sub-steps: S401. Obtain the point cloud coordinate set from the 3D fusion model; S402. Extract the approximate horizontal plane region using the vertical direction normal vector to obtain the set of points on the upper surface of the clamping plate: S403. From the set of points on the upper surface of the clamping plate, select the point with the largest height to form the set of points on the top edge, and calculate the height difference at the top of the clamping plate; S404. Construct a morphological risk function by comprehensively predicting the morphological deviation value, the height difference at the top of the clamping plate, and the curvature change.

9. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, Step S5 specifically includes the following sub-steps: S501. Based on the constructed morphological risk function, and by inputting new monitoring data into the morphological risk function, calculate the morphological risk value at the current moment; S502. Based on the risk value of the current frame and the risk value of the previous frame, calculate the continuous change amount of the risk values ​​of the two frames, where the continuous change rate is 0 when the current frame is the initial frame. S503. Based on the continuous transformation between the morphological risk value and the risk value of the two frames, calculate the normalized probability using the Sigmoid function.

10. The computer vision-based monitoring method for thin coal seam working faces as described in claim 9, characterized in that, The specific process for calculating the continuous transformation of risk values ​​between two frames is as follows: ； Among them, the This represents the continuous change in risk values ​​between two frames. This represents the risk value at the current time t. This represents the risk value at the previous time t-1.

11. The method for monitoring thin coal seam working faces based on computer vision as described in claim 1, characterized in that, The specific process of step S6 is as follows: based on the normalized probability value By setting a threshold for the desinking state Compare and determine the detachment status: ； when When =1, it indicates that based on the current frame's morphological risk value and continuous change rate, the cable clamp has a significant risk of detachment from the groove or is about to detach, and an alarm or shutdown should be triggered; when =0 indicates that the cable clamp is on the normal track.

12. A computer vision-based monitoring system for thin coal seam working faces, the system being implemented based on the computer vision-based monitoring method for thin coal seam working faces according to any one of claims 1-11, characterized in that, include: The model building module is used to collect morphological data of cable clamps through industrial sensor arrays and build a three-dimensional fusion model based on the morphological data. The curvature calculation module is used to extract the center curve of the 3D fusion model through the depth extreme value tracking method, calculate the average curvature of the clamp in the frame based on the discrete point set of the extracted center curve, output parameter pairs, and construct the parameter pairs of continuous time moments into a curvature sequence. The deviation prediction module constructs a morphological change time series from the curvature sequence at continuous time points, predicts the curvature state at the next time point through a sliding window model, and outputs the predicted morphological deviation value. The morphological risk function construction module is used to calculate the height difference at the top of the clamping plate based on the geometric height information of the 3D fusion model, and construct the morphological risk function by weighting and combining the predicted morphological deviation value with the calculated height difference at the top of the clamping plate with the relative change rate. The data calculation module is used to input new data into the morphological risk function, calculate the morphological risk value at the current moment and the continuous transformation of the risk value between two frames, and output the probability value normalized to [0,1] through the Sigmoid function; The threshold determination module is used to determine the desinking status based on the calculated range probability value and the set desinking status threshold.

13. The computer vision-based thin coal seam working face monitoring system as described in claim 12, characterized in that, The industrial sensor group includes at least: Structured light depth camera is used to acquire depth point cloud data of the surface of the clamping plate and form point cloud frames; An RGB camera is used to capture two-dimensional texture images of clamps and cable trays; Infrared thermal imagers are used to acquire the infrared reflection matrix of the splice surface to identify splice boundaries and folded areas.

Images