A hydraulic support motion monitoring system and method based on two-level progressive sensing
By using a hydraulic support motion monitoring system based on two-level progressive sensing, combined with sensor initial screening and visual verification, the problem of distinguishing between sensor failure and mechanical inadequacy has been solved, reducing resource consumption and improving fault location accuracy and processing efficiency.
Patent Information
- Application Number
- CN202611566066.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-10
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies cannot effectively distinguish between sensor malfunctions and mechanical malfunctions, leading to resource waste and inaccurate judgments. Furthermore, the continuous operation of visual monitoring solutions results in high resource consumption.
A hydraulic support motion monitoring system based on two-level progressive sensing is adopted. By combining sensor initial screening and visual verification, a logic is established to trigger visual verification when the initial screening is abnormal. The system uses sensor groups and pan-tilt cameras to monitor motion, thereby achieving accurate differentiation between sensor failures and mechanical motion.
It reduces the continuous operating load and computing power consumption of the vision system, provides a clear direction for fault location, significantly shortens the fault handling time, and improves the accuracy and efficiency of the monitoring system.
Smart Images

Figure CN123109784A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of automated control and condition monitoring technology for hydraulic supports in fully mechanized coal mining faces, and in particular to a hydraulic support motion monitoring system and method based on two-level progressive sensing. Background Technology
[0002] Hydraulic supports are the core support equipment in fully mechanized mining faces. The accurate execution of their actions, such as lowering, pulling, raising, and pushing, directly affects the safety and efficiency of automated production at the working face. Currently, the monitoring of the hydraulic support's operational status mainly employs the following two technical approaches:
[0003] I. Multi-sensor-based state monitoring scheme. This scheme uses stroke sensors, column pressure sensors, and tilt sensors installed on the hydraulic support to collect and analyze displacement, pressure, and attitude parameters during the support's operation in real time. For example, patent CN105737791A discloses a method for detecting the posture of a hydraulic support in a large-angle fully mechanized mining face, using a dual-axis tilt sensor to detect the true tilt angle and tilt azimuth of the base, and combining distance and pressure sensors to achieve support posture detection; patent CN116202574A proposes a method for evaluating the working state of a hydraulic support, collecting data through pressure, tilt, and displacement sensors, using a BP neural network for first-level fusion, and then combining it with a fuzzy comprehensive evaluation model to determine the working state of the support; other literature proposes a hydraulic support monitoring system based on multi-sensor information fusion technology, which fuses data such as tilt angle, displacement, and pressure; research published in "Industrial and Mining Automation" in 2023 points out that hydraulic support pressure and posture are key state data that can be directly monitored, and their fusion analysis is the basis for intelligent control of the support. Furthermore, the intelligent control method for automatic hydraulic support relocation in coal mines divides the relocation process into three stages: column lowering, relocation, and column raising, with pressure sensors, stroke sensors, and angle sensors respectively sensing the motion status. A common characteristic of these solutions is their complete reliance on sensor data for judgment. When a sensor malfunctions or drifts, the system cannot distinguish whether the problem lies with the sensor or with a genuine mechanical malfunction, easily leading to misjudgments.
[0004] II. Vision-Based Support Status Monitoring Scheme. With the development of machine vision technology, some solutions have begun to incorporate pan-tilt cameras to assist in monitoring the movement of hydraulic supports. For example, patent CN103244164B discloses a hydraulic support assembly with a video control system, which uses a video camera and a control pan-tilt unit to track the moving hydraulic supports. The "Automatic Vision Detection Method for Hydraulic Support Loss Status at Working Faces" published in *Industrial and Mining Automation* in 2024 uses AI video analysis to detect lost supports. Another patent proposes a scheme using pan-tilt cameras to collect image information for auxiliary monitoring of hydraulic support movement. A common feature of these solutions is that the vision system operates continuously, performing uninterrupted video acquisition and AI analysis on all supports, resulting in significant consumption of computing resources and network bandwidth.
[0005] In recent years, solutions have emerged that simply use sensors and vision technologies side-by-side. For example, some research has proposed a remote intelligent monitoring method for the posture of hydraulic supports based on depth vision and sensor fusion; another patent discloses a method for detecting the movement of hydraulic supports, which simultaneously collects audio and image data and constructs audio and video recognition algorithm models for mutual verification. However, in the above solutions, the sensors and vision systems always work simultaneously, and the vision system does not selectively activate based on the initial judgment results of the sensors. It fails to form a progressive verification logic of "sensor initial screening → anomaly triggering → visual verification," and still suffers from resource waste and coarse judgment.
[0006] In summary, existing technologies have not yet proposed a system architecture that can perform preliminary screening using sensors and trigger the corresponding gimbal camera for fixed-point visual verification only when the preliminary screening is abnormal, in order to distinguish between the two fault modes of "sensor failure" and "actuator malfunction" and reduce the continuous operating load of the vision system. Summary of the Invention
[0007] To address the issues of existing sensor monitoring schemes failing to distinguish between sensor malfunctions and mechanical malfunctions, existing visual monitoring schemes consuming excessive resources due to continuous operation, and the resource waste and inaccurate judgment issues inherent in parallel sensor and vision monitoring schemes, this application proposes a hydraulic support motion monitoring system and method based on two-level progressive sensing.
[0008] The technical solution adopted in this application is: a hydraulic support action monitoring system based on two-level progressive sensing, including a sensing layer, an edge computing layer and an output layer, wherein the sensing layer includes a sensor group and a pan-tilt camera installed on each hydraulic support;
[0009] The edge computing layer includes a first-level verification module, an anomaly management module, a second-level verification module, and a comprehensive diagnostic module. The first-level verification module verifies the position of the completed hydraulic support action based on real-time data from the sensor array and preset multi-dimensional judgment conditions. The anomaly management module generates anomaly information based on the first-level verification module, determines the optimal allocation scheme between the anomaly and the pan-tilt camera, and drives the pan-tilt camera to align with the target hydraulic support exhibiting anomalies. The second-level verification module identifies the target hydraulic support, calculates its piston rod extension length, generates the actual position information of the target hydraulic support action, and outputs visual analysis results. The comprehensive diagnostic module cross-compares the visual analysis results output by the second-level verification module with the sensor data output by the first-level verification module to obtain the final diagnostic conclusion.
[0010] The output layer is used to output diagnostic conclusions.
[0011] Furthermore, the perception layer includes:
[0012] The stroke sensor of the pushing cylinder installed in each hydraulic support pushing jack is used to collect the displacement of the piston rod of the pushing cylinder in real time.
[0013] The column pressure sensor installed in the lower cavity of each hydraulic support column is used to collect the column support pressure in real time.
[0014] Tilt sensors installed on the top beam, shield beam, and base of each hydraulic support are used to collect pitch and roll angles in real time.
[0015] A pan-tilt camera is installed at intervals below the top beam of the hydraulic support. The pan-tilt camera is in a constantly awake state and includes a pan-tilt mechanism, a camera unit, and a tilt measurement unit. The tilt measurement unit is used to collect the pitch angle of the camera unit in real time.
[0016] Furthermore, the first-level verification module is connected to the push cylinder stroke sensor, column pressure sensor, and tilt sensor to receive real-time data from each sensor and to verify the position of the completed hydraulic support action based on preset multi-dimensional judgment conditions corresponding to the four actions of lowering the column, pulling the support, raising the column, and pushing the slide.
[0017] Furthermore, the multidimensional determination conditions include travel threshold conditions, pressure threshold conditions, and attitude angle change range conditions.
[0018] Furthermore, the second-level verification module is connected to the PTZ camera resource scheduling module and the PTZ camera signal. After receiving the PTZ camera resource scheduling information, it initiates visual reasoning analysis of the real-time video stream of the camera unit, identifies key points on the target hydraulic support, its column, and the piston rod of the push cylinder, and calculates the piston rod extension length based on the key point results to generate the actual positioning information of the target hydraulic support.
[0019] Furthermore, the abnormal event management module includes an abnormal event generation module, an abnormal event queue management module, and a PTZ camera resource scheduling module. The abnormal event generation module generates abnormal event information when the sensor data of a hydraulic support fails to meet the multi-dimensional judgment conditions N times consecutively. The abnormal event information carries at least the frame number of the target hydraulic support for subsequent visual verification. The abnormal event queue management module arranges received abnormal events into the abnormal event queue according to the abnormality priority level and the order of reception time. The PTZ camera resource scheduling module is connected to the abnormal event generation module and the PTZ camera signal. It receives abnormal event information, obtains the optimal allocation scheme between the abnormal event and the PTZ camera, calculates the target horizontal yaw angle, target vertical pitch angle, and target focal length of the PTZ camera based on the frame number of the target hydraulic support, and drives the PTZ camera to rotate to the target angle and focal length so that the camera unit is aligned with the target hydraulic support.
[0020] Furthermore, the final diagnostic conclusions in the comprehensive diagnostic module include:
[0021] If the visual analysis results show that the mechanical action has been completed but the sensor data is abnormal, it is determined to be a sensor malfunction.
[0022] If the visual analysis results show that the mechanical action is not in place, but the sensor data matches the visual analysis results, then it is determined that the actuator is faulty;
[0023] If the visual analysis results show that the mechanical action is not in place and the deviation direction of the sensor data is inconsistent with the multidimensional judgment conditions, it is judged as a compound fault.
[0024] A hydraulic support motion monitoring method based on two-level progressive sensing, using the aforementioned system, includes the following steps:
[0025] Step 1: Real-time acquisition of sensor data;
[0026] Step 2: First-level verification: Initial screening using sensor fusion;
[0027] Step 3: Abnormal event generation and PTZ camera resource scheduling;
[0028] Step 4: Automatic positioning of the pan-tilt camera;
[0029] Step 5: Second-level verification: Visual review and detailed diagnosis;
[0030] Step 6: Comprehensive diagnosis and conclusion output.
[0031] Furthermore, step 4 specifically includes:
[0032] Step 4.1: Calculate the height of the pan-tilt camera above the ground;
[0033] Based on the principle that the hinge points of the middle trough of the scraper conveyor are arranged at equal intervals along the working surface, and according to the known distance between adjacent hinge points of the middle trough of the scraper conveyor in the image, the height of the pan-tilt camera above the ground is calculated using the multi-point least squares estimation method.
[0034] Step 4.2: The gimbal camera automatically aligns with the target hydraulic support. Using the target hydraulic support number and the current hydraulic support number of the gimbal camera, the rotation angle of the gimbal mechanism is calculated to align the camera unit with the target hydraulic support.
[0035] Furthermore, step 5 specifically includes:
[0036] Step 5.1: Depth calculation and coordinate transformation;
[0037] Measure the actual outer diameter of the piston rod beforehand. In the image, the top and bottom positions of the piston rod are located by the target detection algorithm, and the pixel width at these two sections is measured. According to the pinhole imaging model, the depth of the top and bottom of the piston rod is obtained, and then the three-dimensional vector of the piston rod endpoint is calculated.
[0038] Step 5.2: Identification of column frame number and hydraulic cylinder frame number;
[0039] Step 5.3: Calculate the extension distance of the column piston rod, and generate the actual positioning information of the target hydraulic support based on the calculated data.
[0040] The advantages of this application over the prior art are as follows:
[0041] This application establishes a first-level verification module (sensor fusion initial screening) and a second-level verification module (visual review and detailed diagnosis), and establishes a causal relationship of "initial screening failure → generation of abnormal events → triggering visual review," so that visual inference analysis is only initiated as needed when sensor data is abnormal. Compared with existing technologies that allow visual inference to run continuously all the time, this application's solution significantly reduces the continuous operating load and computing power consumption of the vision system.
[0042] Existing pure sensor solutions cannot determine whether abnormal sensor data indicates a sensor malfunction or a failure of the actuator to perform its intended action. This application addresses this by using a comprehensive diagnostic module to cross-reference visual analysis results with initial sensor screening data: if the visual system indicates the mechanical action is in place but the sensor data is abnormal, the sensor is identified as faulty; if the visual system indicates the mechanical action is not in place but the sensor data matches, the actuator is identified as faulty. This ability to differentiate between these factors provides maintenance personnel with a clear direction for fault location, avoiding blind troubleshooting and significantly reducing troubleshooting time.
[0043] In existing technologies, the installation height of gimbal cameras is typically fixed after manual measurement, which cannot adapt to dynamic changes such as bracket lifting and the undulation of the base plate. This application utilizes the geometric constraint that the spacing between the hinge points in the middle trough of the scraper conveyor is a known constant, and calculates the ground clearance of the gimbal camera in real time using the pixel coordinates of multiple hinge points in a monocular image and camera intrinsic parameters. This method requires no additional sensors and relies solely on the structural characteristics of the scraper conveyor itself to complete dynamic calibration, providing an accurate reference height for subsequent gimbal camera angle calculations and visual measurements.
[0044] This application utilizes the prior information that the true outer diameter of the piston rod is a known constant. By measuring the pixel width at two cross-sections at the top and bottom of the piston rod in the image, the depth at both ends is inferred based on a pinhole imaging model, thereby reconstructing the three-dimensional coordinates of both ends and calculating the extension length of the piston rod. This method eliminates the need for any manual markings on the piston rod, resulting in low deployment costs, simple maintenance, and can be accomplished with a single-lens pan-tilt camera, without the need for complex calibration and stereo matching required by binocular vision. Attached Figure Description
[0045] The following description, in conjunction with the accompanying drawings, further illustrates this application:
[0046] Figure 1 This is a system architecture diagram for this application;
[0047] Figure 2 This is a flowchart of the method used in this application. Detailed Implementation
[0048] like Figure 1 and Figure 2 As shown, this application provides a hydraulic support motion monitoring system based on two-level progressive sensing, including a sensing layer, an edge computing layer, and an output layer.
[0049] The perception layer includes:
[0050] The stroke sensor of the pushing cylinder installed in each hydraulic support pushing jack is used to collect the displacement of the piston rod of the pushing cylinder in real time.
[0051] The column pressure sensor installed in the lower cavity of each hydraulic support column is used to collect the column support pressure in real time.
[0052] Tilt sensors installed on the top beam, shield beam, and base of each hydraulic support are used to collect pitch and roll angles in real time.
[0053] A pan-tilt camera is installed at intervals below the top beam of the hydraulic support. This camera is always-on and includes a pan-tilt mechanism, a camera unit, and a tilt measurement unit. The tilt measurement unit is used to acquire the pitch angle of the camera unit in real time. The interval or position of the pan-tilt camera is determined based on the field of view of the camera unit; one pan-tilt camera can be installed every two hydraulic supports or every four hydraulic supports.
[0054] The edge computing layer includes:
[0055] The first-level verification module is connected to the push cylinder stroke sensor, column pressure sensor, and tilt sensor. It is used to receive real-time data from each sensor and verify the position of the completed hydraulic support action according to the preset multi-dimensional judgment conditions corresponding to the four actions of column lowering, support pulling, column raising, and push sliding. The multi-dimensional judgment conditions include stroke threshold conditions, pressure threshold conditions, and attitude angle change range conditions.
[0056] The abnormal event management module is connected to the first-level verification module and the second-level verification module. It is used to generate abnormal event information based on the first-level verification module, obtain the optimal allocation scheme between the abnormal event and the PTZ camera based on the abnormal event information, and drive the PTZ camera to aim at the target hydraulic support with abnormality.
[0057] The second-level verification module is connected to the PTZ camera resource scheduling module and the PTZ camera signal. After receiving the PTZ camera resource scheduling information, it starts the camera unit to perform real-time visual reasoning analysis of the video stream, identifies key points on the target hydraulic support and the column and piston rod of the push cylinder, calculates the piston rod extension length based on the key point results, and generates the actual positioning information of the target hydraulic support.
[0058] The comprehensive diagnostic module, connected to the first-level verification module and the second-level verification module, is used to cross-compare the visual analysis results output by the second-level verification module with the sensor data output by the first-level verification module. If the visual analysis results show that the mechanical action has been completed but the sensor data is abnormal, it is determined to be a sensor fault; if the visual analysis results show that the mechanical action has not been completed but the sensor data matches it, it is determined to be an actuator fault; if the visual analysis results show that the mechanical action has not been completed and the deviation direction of the sensor data is inconsistent with the multi-dimensional judgment conditions, it is determined to be a compound fault.
[0059] The output layer is connected to the comprehensive diagnostic module and is used to output diagnostic conclusions.
[0060] The abnormal event management module includes an abnormal event generation module, an abnormal event queue management module, and a PTZ camera resource scheduling module.
[0061] The abnormal event generation module is connected to the first-level verification module. It is used to generate abnormal event information when the sensor data of a certain hydraulic support fails to meet the multi-dimensional judgment conditions for M consecutive times. The abnormal event information carries at least the frame number and abnormal priority level of the target hydraulic support for subsequent visual verification.
[0062] The exception event queue management module, connected to the exception event generation module, is used to arrange received exception events into the exception event queue according to exception priority and reception time.
[0063] The gimbal camera resource scheduling module is connected to the abnormal event generation module and the gimbal camera signal. It is used to receive abnormal event information, obtain the optimal allocation scheme between the abnormal event and the gimbal camera, calculate the target horizontal yaw angle, target vertical pitch angle and target focal length of the gimbal camera according to the frame number of the target hydraulic support, and drive the gimbal camera to rotate to the target angle and focal length so that the camera unit is aligned with the target hydraulic support.
[0064] Based on the above system, this application also proposes a hydraulic support motion monitoring method based on two-level progressive sensing, including the following steps:
[0065] Step 1: Real-time acquisition of sensor data;
[0066] The detection data of each sensor is collected in real time by using a push cylinder stroke sensor installed on the hydraulic support, a column pressure sensor, and tilt sensors installed on the top beam, shield beam, and base respectively.
[0067] Step 2: First-level verification: Initial screening using sensor fusion;
[0068] Based on the type of action currently being performed by the hydraulic support, multi-dimensional judgment conditions corresponding to the action type are retrieved from a variety of preset action judgment conditions. The action types include column lowering action, support pulling action, column raising action, and push sliding action.
[0069] Multidimensional determination conditions include:
[0070] Stroke determination criteria: Whether the displacement collected by the push cylinder stroke sensor reaches the preset target stroke allowable deviation range;
[0071] Pressure determination criteria: After the column is raised, whether the support pressure collected by the column pressure sensor reaches the preset pressure threshold; after the column is lowered, whether the support pressure collected by the column pressure sensor drops below the preset pressure relief threshold (e.g., below 5% of the rated working pressure).
[0072] Attitude determination conditions: A model of the correspondence between the support height and the shield beam inclination angle is established in advance based on regression analysis of downhole measured data. During the lifting and lowering of the support column, the theoretical change in the shield beam inclination angle is predicted based on the target lifting / lowering amount. After the lifting and lowering of the support column is completed, it is determined whether the actual change in the shield beam inclination angle reaches the allowable deviation range of the theoretical change in the shield beam inclination angle.
[0073] When the displacement, support pressure, and tilt angle change all simultaneously meet their respective judgment conditions, the hydraulic support is determined to be in position.
[0074] Step 3: Abnormal event generation and PTZ camera resource scheduling;
[0075] When the sensor data of a hydraulic support fails to meet the multi-dimensional judgment condition in step 2 for N consecutive times, an abnormal event information is generated. The abnormal event information carries at least the support number of the hydraulic support. and abnormal priority level Let the current set of abnormal events be... A combination of pan-tilt-zoom (PTZ) cameras can be used. For each abnormal event Calculate the priority level using the following formula. :
[0076] ;
[0077] In the formula, The number of consecutive failures. This represents the maximum number of consecutive failures. The degree to which sensor data deviates from a threshold. If the sensor data deviates from the upper limit of the threshold, Historical failure rate The weights of each item.
[0078] In a specific embodiment of determining the stroke of the push cylinder, for the push-pull action, Subtract the distance to the end of the push from the maximum distance. For the maximum travel distance, in the context of a break-up action, This represents the absolute value of the deviation between the preset tensioning amount and the actual tensioning amount. This represents the maximum travel distance.
[0079] In a specific embodiment of column pressure determination, for the column lifting action, Max is (rated initial support force - lifting column end pressure, 0). For the rated initial support force, the column lowering action, To reduce the pressure at the end of the column, This is the rated initial support force.
[0080] The abnormal event queue management module will arrange the received abnormal events into the abnormal event queue according to the abnormality priority level and the time of receipt.
[0081] Based on the premise that cameras are installed at intervals on all hydraulic supports in the longwall mining face and one camera can monitor multiple adjacent hydraulic supports simultaneously, the pan-tilt camera resource scheduling module calculates the sum of priority levels of abnormal hydraulic supports belonging to the same pan-tilt camera's field of view under constraints of pan-tilt camera field of view coverage, pan-tilt camera capacity, and inference computing power. With the goal of maximizing the sum of the above priority levels, it obtains the optimal allocation scheme between abnormal events and pan-tilt cameras and drives the allocated pan-tilt cameras to rotate to the target angle.
[0082] Example: A total of 100 hydraulic supports are installed. A pan-tilt camera is installed for every five hydraulic supports, for a total of 20 cameras (i.e., one pan-tilt camera is installed below the top beam of each of the 3rd, 8th, 13th, etc.). Each pan-tilt camera can capture the movement of its corresponding five hydraulic supports. Ten hydraulic supports experienced abnormal events: supports 5-6, supports 18-23, supports 49, supports 50, and supports 70. Therefore, supports 18-20 are attributed to the camera installed by the 4th pan-tilt camera, and supports 21-23... are attributed to the camera installed by the 4th pan-tilt camera. All hydraulic supports are assigned to the 5th PTZ camera, and hydraulic supports 49-50 are assigned to the 10th PTZ camera. The sum of the priority levels of abnormal events for hydraulic supports 18-20, 21-23, and 49-50 are calculated respectively. Then, these sums are compared with the priority levels of abnormal events for hydraulic supports 5, 6, and 70, and the maximum value is selected as the target.
[0083] Step 4: Automatic positioning of the pan-tilt camera, including:
[0084] Step 4.1: Calculate the height of the pan-tilt camera above the ground;
[0085] The hinge points (or connecting pin center points) of the central trough of the scraper conveyor are arranged at equal intervals along the working surface, with the distance between adjacent hinge points being... These are known constants. The pan-tilt camera is installed below the top beam and captures the movement of the lifting column. Quantitative visual calculations often require transforming the captured object from the camera coordinate system to the world coordinate system. This transformation process requires knowing the height of the camera (image unit). The hinge point spacing of the central trough of the scraper conveyor can be used as a reference point to calculate the pan-tilt camera's height above the ground.
[0086] Let the height of the pan-tilt camera above the ground be... (To be determined). In this embodiment, the height of the gimbal camera above the ground is estimated using the multi-point least squares method:
[0087] When detected in the image At one hinge point ( The corresponding camera view direction vector in the world coordinate system is , This can be obtained by combining the pixel coordinates of the image with the camera angle. Let the length of the line-of-sight vector at the hinge point be... , ,but:
[0088] ;
[0089] ;
[0090] , They are respectively , The z-component in the world coordinate system.
[0091] In reality, the central trough of a scraper conveyor lies flat on the ground, coplanar with the x and y directions. Therefore, the first... The and the first Estimated distance between hinge points for: , for Components in the xy plane, for The components in the xy plane, , After substituting the actual value, we get:
[0092] ;
[0093] make , , can be obtained ;
[0094] In the formula, For the first The and the first Estimated physical distance between hinge points The modulus of the result of the difference operation between two vectors.
[0095] The objective is to minimize the weighted sum of squared residuals for all pairs of points:
[0096] ;
[0097] In the formula, These are the weighting coefficients; For the first The and the first The actual physical distance between each hinge point = The central trough of the scraper conveyor is straight, and the distance between adjacent hinge points is fixed. Therefore, it is easy to know the distance between the first hinge point and the second hinge point. 1 times The distance between the first and third hinge points is 2 times .
[0098] The objective function is about The convex quadratic function has a unique global minimum, and its analytical solution is:
[0099] ;
[0100] In the formula, for The optimal solution.
[0101] Step 4.2: The pan-tilt camera automatically aligns with the target hydraulic support;
[0102] When an anomaly in the initial sensor screening triggers a visual verification, the system obtains the target hydraulic support number. The current hydraulic support number of the pan-tilt camera is [number missing]. It is necessary to calculate the rotation angle of the gimbal mechanism so that the camera unit is aligned with the target hydraulic support.
[0103] Let the longitudinal distance from the pan-tilt camera to the target reference plane be... The target reference plane is a plane passing through the target observation point and parallel to the fully mechanized mining face, and the center distance between adjacent supports is... Then the horizontal yaw angle :
[0104] ;
[0105] Let the height of the target observation point (such as the middle of the push rod or column) be... (Given constants, preset according to different observation locations), the height difference is: ;
[0106] The horizontal distance is: ;
[0107] The vertical pitch angle is: ;
[0108] The gimbal camera aims at the target hydraulic support by adjusting the horizontal yaw angle and the vertical pitch angle.
[0109] Step 5: Second-level verification: Visual review and detailed diagnosis;
[0110] Step 5.1: Depth calculation and coordinate transformation;
[0111] Measure the actual outer diameter of the piston rod beforehand. In the image, the top (far from the cylinder end) and the tail (near the cylinder end) of the piston rod are located using an object detection algorithm, and the pixel width at these two cross-sections is measured. and .
[0112] Based on the pinhole imaging model, the depth from the gimbal camera to the top of the piston rod... Depth of the pan-tilt camera from the bottom of the piston rod ;in The pixel focal length is located along the x-axis. This represents the pixel width at the top cross-section of the piston rod. This represents the pixel width at the cross-section of the piston rod's tail end.
[0113] The pixel coordinates of the piston rod end point in the image are: The corresponding 3D vector in the camera coordinate system is:
[0114] ;
[0115] In the formula, Let be the three-dimensional vector of the piston rod endpoint in the camera coordinate system. These are the horizontal and vertical pixel focal lengths (in pixels). These are the pixel coordinates of the midpoint in the image.
[0116] The three-dimensional vector of the piston rod end point in the world coordinate system is ,in The transformation vector is used to transform any point from the camera coordinate system to the world coordinate system.
[0117] Step 5.2: Identification of column frame number and hydraulic cylinder frame number, the calculation formula is as follows:
[0118] ;
[0119] In the formula, For the identified hydraulic support number, This is the current hydraulic support number of the pan-tilt camera. The x-axis coordinate of the piston rod end point in the world coordinate system. This represents the center-to-center distance between adjacent supports.
[0120] In actual monitoring, the column and the push cylinder in this step may not belong to the same hydraulic support. For example, if the column data of frame 2 is abnormal and the push cylinder data of frame 3 is abnormal, it is necessary to match which column belongs to frame 2 and which push cylinder belongs to frame 3 on the image.
[0121] Step 5.3: Calculate the extension distance of the column piston rod;
[0122] Piston rod extension length for:
[0123] ;
[0124] In the formula, Let be the three-dimensional coordinate vector of the piston rod extension end. The three-dimensional coordinate vector of the piston rod extension tail end. To take the modulus of the result of the first and last vector difference operation, , , The three-dimensional coordinates of the piston rod extension end are given. , , The three-dimensional coordinates of the piston rod extension end are given.
[0125] Step 6: Comprehensive diagnosis and conclusion output;
[0126] Cross-compare the visual analysis results from step 5 with the sensor data from step 2:
[0127] If the visual analysis results show that the mechanical action has been completed but the sensor data is abnormal, it is determined to be a sensor malfunction.
[0128] If the visual analysis results show that the mechanical action is not in place, but the sensor data matches the visual analysis results, then it is determined that the actuator is faulty;
[0129] If the visual analysis results show that the mechanical action is not in place and the deviation direction of the sensor data is inconsistent with the multidimensional judgment conditions, it is judged as a compound fault.
[0130] Output the above diagnostic conclusion.
[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A hydraulic support motion monitoring system based on two-level progressive sensing, characterized in that: It includes a perception layer, an edge computing layer, and an output layer. The perception layer includes a sensor array and a gimbal camera mounted on each hydraulic support. The edge computing layer includes a first-level verification module, an anomaly management module, a second-level verification module, and a comprehensive diagnostic module. The first-level verification module verifies the position of the completed hydraulic support action based on real-time data from the sensor array and preset multi-dimensional judgment conditions. The anomaly management module generates anomaly information based on the first-level verification module, determines the optimal allocation scheme between the anomaly and the pan-tilt camera, and drives the pan-tilt camera to align with the target hydraulic support exhibiting anomalies. The second-level verification module identifies the target hydraulic support, calculates its piston rod extension length, generates the actual position information of the target hydraulic support action, and outputs visual analysis results. The comprehensive diagnostic module cross-compares the visual analysis results output by the second-level verification module with the sensor data output by the first-level verification module to obtain the final diagnostic conclusion. The output layer is used to output diagnostic conclusions.
2. The hydraulic support motion monitoring system based on two-level progressive sensing according to claim 1, characterized in that: The perception layer includes: The stroke sensor of the pushing cylinder installed in each hydraulic support pushing jack is used to collect the displacement of the piston rod of the pushing cylinder in real time. The column pressure sensor installed in the lower cavity of each hydraulic support column is used to collect the column support pressure in real time. Tilt sensors installed on the top beam, shield beam, and base of each hydraulic support are used to collect pitch and roll angles in real time. A pan-tilt camera is installed at intervals below the top beam of the hydraulic support. The pan-tilt camera is in a constantly awake state and includes a pan-tilt mechanism, a camera unit, and a tilt measurement unit. The tilt measurement unit is used to collect the pitch angle of the camera unit in real time.
3. The hydraulic support motion monitoring system based on two-level progressive sensing according to claim 2, characterized in that: The first-level verification module is connected to the push cylinder stroke sensor, column pressure sensor, and tilt sensor. It is used to receive real-time data from each sensor and verify the position of the completed hydraulic support action based on preset multi-dimensional judgment conditions corresponding to the four actions of lowering the column, pulling the support, raising the column, and pushing the slide.
4. The hydraulic support motion monitoring system based on two-level progressive sensing according to claim 3, characterized in that: The multidimensional determination conditions include travel threshold conditions, pressure threshold conditions, and attitude angle change range conditions.
5. The hydraulic support motion monitoring system based on two-level progressive sensing according to claim 1, characterized in that: The second-level verification module is connected to the PTZ camera resource scheduling module and the PTZ camera signal. After receiving the PTZ camera resource scheduling information, it initiates visual reasoning analysis of the real-time video stream of the camera unit, identifies key points on the target hydraulic support, the column of the target hydraulic support, and the piston rod of the push cylinder, and calculates the extension length of the piston rod based on the key point results to generate the actual positioning information of the target hydraulic support.
6. The hydraulic support motion monitoring system based on two-level progressive sensing according to claim 1, characterized in that: The abnormal event management module includes an abnormal event generation module, an abnormal event queue management module, and a PTZ camera resource scheduling module. The abnormal event generation module generates abnormal event information when the sensor data of a hydraulic support fails to meet the multidimensional judgment conditions N times consecutively. The abnormal event information carries at least the support number of the target hydraulic support for subsequent visual verification. The abnormal event queue management module arranges received abnormal events into the abnormal event queue according to the abnormality priority level and the order of receipt time. The gimbal camera resource scheduling module is connected to the abnormal event generation module and the gimbal camera signal. It is used to receive abnormal event information, obtain the optimal allocation scheme between the abnormal event and the gimbal camera, calculate the target horizontal yaw angle, target vertical pitch angle and target focal length of the gimbal camera according to the frame number of the target hydraulic support, and drive the gimbal camera to rotate to the target angle and focal length so that the camera unit is aligned with the target hydraulic support.
7. The hydraulic support motion monitoring system based on two-level progressive sensing according to claim 1, characterized in that: The final diagnostic conclusions in the comprehensive diagnostic module include: If the visual analysis results show that the mechanical action has been completed but the sensor data is abnormal, it is determined to be a sensor malfunction. If the visual analysis results show that the mechanical action is not in place, but the sensor data matches the visual analysis results, then it is determined that the actuator is faulty; If the visual analysis results show that the mechanical action is not in place and the deviation direction of the sensor data is inconsistent with the multidimensional judgment conditions, it is judged as a compound fault.
8. A method for monitoring the movement of a hydraulic support based on two-level progressive sensing, characterized in that: Using the system as described in any one of claims 1-7, the method includes the following steps: Step 1: Real-time acquisition of sensor data; Step 2: First-level verification: Initial screening using sensor fusion; Step 3: Abnormal event generation and PTZ camera resource scheduling; Step 4: Automatic positioning of the pan-tilt camera; Step 5: Second-level verification: Visual review and detailed diagnosis; Step 6: Comprehensive diagnosis and conclusion output.
9. A hydraulic support motion monitoring method based on two-level progressive sensing according to claim 8, characterized in that: Step 4 specifically includes: Step 4.1: Calculate the height of the pan-tilt camera above the ground; Based on the principle that the hinge points of the middle trough of the scraper conveyor are arranged at equal intervals along the working surface, and according to the known distance between adjacent hinge points of the middle trough of the scraper conveyor in the image, the height of the pan-tilt camera above the ground is calculated using the multi-point least squares estimation method. Step 4.2: The gimbal camera automatically aligns with the target hydraulic support. Using the target hydraulic support number and the current hydraulic support number of the gimbal camera, the rotation angle of the gimbal mechanism is calculated to align the camera unit with the target hydraulic support.
10. A hydraulic support motion monitoring method based on two-level progressive sensing according to claim 8, characterized in that: Step 5 specifically includes: Step 5.1: Depth calculation and coordinate transformation; The true outer diameter of the piston rod is measured in advance. The top and bottom positions of the piston rod are located in the image using a target detection algorithm. The pixel width at these two cross-sections is measured. Based on the pinhole imaging model, the depth of the top and bottom of the piston rod is obtained. Then, the three-dimensional vector of the piston rod endpoint is calculated. Step 5.2: Identification of column frame number and hydraulic cylinder frame number; Step 5.3: Calculate the extension distance of the column piston rod, and generate the actual positioning information of the target hydraulic support based on the calculated data.
Citation Information
Patent Citations
A hydraulic support assembly with a video control system
CN103244164B
Position and orientation detection method of large-inclination-angle fully-mechanized coal mining face hydraulic support
CN105737791A
