Method and device for measuring telescopic angle of multi-stage face guard of hydraulic support

Abstract

The invention relates to the technical field of position and posture state detection of underground coal mine fully-mechanized coal mining equipment, in particular to a method and device for measuring the telescopic angle of a hydraulic support multi-stage face guard based on machine vision. According to the measurement method based on machine vision, real-time and high-precision measurement of the telescopic angle of the multistage face guard of the hydraulic support is achieved, and the problems that a traditional contact type measurement method is prone to interference and difficult to install and maintain, and an existing vision measurement method cannot accurately reflect the linkage overall posture of the multistage face guard are effectively solved. The explosion-proof multi-view industrial camera and the key point detection and dynamic compensation technology are adopted, so that the stability and reliability of measurement are improved, the measurement result is displayed in real time through a visual interface, accurate data support is provided for multi-stage coordinated linkage control of the hydraulic support, and the working efficiency of the hydraulic support is improved. The safe operation level and the intelligent degree of underground coal mine fully-mechanized coal mining equipment are remarkably improved, and important practical significance and wide application prospects are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of pose state detection of fully-mechanized mining equipment underground in coal mines, and in particular to a method and device for measuring the telescopic angle of a multi-stage rib protection plate of a hydraulic support based on machine vision. Background Art

[0002] In recent years, with the development of coal mine intelligentization, the depth and thickness of coal mining have been continuously increasing, and the number of large mining height working faces has also increased accordingly. Hydraulic supports with two-stage or three-stage rib protection plates are mostly used in large mining height working faces, and the telescopic multi-stage rib protection plates effectively fit the coal wall to ensure the support effect. Therefore, achieving accurate monitoring of the telescopic angles of each stage of the rib protection plate is of great practical significance for the multi-stage coordinated linkage control of the hydraulic support, dynamically adapting to the changes of the coal seam, and ensuring the safe and intelligent production of coal mines.

[0003] Traditional measurement methods mostly rely on contact type inclination sensors, but they are easily affected by dust and vibration interference, and the installation and maintenance workload is large. In addition, due to the multi-stage linkage of the multi-stage rib protection plate, this single-point measurement method cannot accurately reflect the overall posture of the rib protection plate. To solve the above problems, the patent with the application number 202010551198.8 discloses an interference recognition and monitoring system for the rib protection plate of a hydraulic support and the shearer drum. By using video image recognition technology to detect the contour edge lines of the rib protection plate and the shearer drum, and then determining the interference warning level between the rib protection plate and the drum according to the threshold range of the distance between the two; the patent with the application number 201910722429.4 discloses a monitoring method for the rib protection plate based on video image recognition. By using the target detection algorithm to identify the rib protection plate in the video image, it is judged whether the rib protection plate in the video image frame is in place for rib protection; the patent with the application number 202111596613.2 discloses a method for measuring the telescopic angle of the rib protection plate based on video image sequences. According to the mapping relationship between the motion trajectory of key points and the telescopic angle of the rib protection plate, the machine vision measurement of the telescopic angle of a single-stage rib protection plate is realized, but the measurement of the angles of multi-stage rib protection plates cannot be realized.

[0004] In summary, the prior art has realized the monitoring of whether the rib protection plate is in place for rib protection or the relative position state between the rib protection plate and the shearer drum, and has realized the detection of the telescopic angle of a single-stage rib protection plate based on machine vision, but currently, the coordinated measurement of the telescopic angles of each stage of the multi-stage rib protection plate cannot be realized. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem in the prior art that the telescopic angles of each section of the multi-stage rib protection plate cannot be measured in real time and with high precision.

[0006] To solve the above technical problem, the present invention provides a method for measuring the telescopic angle of a multi-stage rib protection plate of a hydraulic support, including:

[0007] Collect video frame images of the target hydraulic support using a camera;

[0008] Based on the video frame images of the target hydraulic support, perform object detection on the key points of the target hydraulic support. The key points include the points at both ends inside the top beam of the target hydraulic support, the hinge points between the guard plates at all levels, and the end points on both sides of the end of the last-level guard plate;

[0009] Obtain the three-dimensional spatial coordinates of the key points;

[0010] Based on the three-dimensional spatial coordinates, calculate the normal vectors of the top beam plane and the normal vectors of the guard plates at all levels;

[0011] Based on the normal vectors, calculate the angles between every two adjacent planes respectively to obtain the telescopic angles of each-level guard plate of the target hydraulic support.

[0012] Preferably, the collecting of the video frame images of the target hydraulic support includes:

[0013] Collect video frame images of the target hydraulic support using explosion-proof industrial multi-cameras calibrated by the Zhang Zhengyou calibration method pre-installed at the positions of the top beams of multiple hydraulic supports. Among them, one camera is installed every preset number of hydraulic supports.

[0014] Preferably, reflective marking points are pre-set on the key points. Among them, each reflective marking point has a corresponding number.

[0015] Preferably, after collecting the video frame images of the target hydraulic support, it further includes:

[0016] Preprocess the video frame images of the target hydraulic support using the adaptive histogram equalization method.

[0017] Preferably, the performing of object detection on the key points of the target hydraulic support based on the video frame images of the target hydraulic support includes:

[0018] Based on the video frame images of the target hydraulic support, use the YOLO-Mini model to detect the key points of the target hydraulic support.

[0019] Preferably, the obtaining of the three-dimensional spatial coordinates of the key points includes:

[0020] Determine the origin of the coordinate system and establish a three-dimensional rectangular coordinate system;

[0021] Use the perspective projection model to map the two-dimensional image coordinates of the key points into the three-dimensional coordinate system according to the internal and external parameters of the camera to obtain the three-dimensional spatial coordinates of the key points.

[0022] Preferably, calculating the normal vectors of the top beam plane and the plane of each stage of rib protection plate based on the three-dimensional space coordinates includes:

[0023] Performing plane fitting on the top beam plane of the hydraulic support and the plane of each stage of rib protection plate respectively based on the three-dimensional space coordinates to obtain the plane equations of each plane;

[0024] Obtaining the normal vectors of each plane based on the plane equations of each plane.

[0025] Preferably, after obtaining the telescopic angle of each stage of rib protection plate of the target hydraulic support, it further includes:

[0026] Obtaining multiple groups of telescopic angle data of multiple stages of rib protection plates of the target hydraulic support according to multiple frame video images of the target hydraulic support;

[0027] Using a Kalman filter to fuse and correct the multiple groups of telescopic angle data of multiple stages of rib protection plates of the target hydraulic support to obtain multiple groups of telescopic angle data of multiple stages of rib protection plates of the target hydraulic support after dynamic compensation.

[0028] Preferably, after obtaining the multiple groups of telescopic angle data of multiple stages of rib protection plates of the target hydraulic support after dynamic compensation, it further includes:

[0029] Feeding back the multiple groups of telescopic angle data of multiple stages of rib protection plates of the target hydraulic support after dynamic compensation to the visualization interface according to the time stamp.

[0030] The present invention also provides a device for measuring the telescopic angle of multiple stages of rib protection plates of a hydraulic support, including:

[0031] An image acquisition module for acquiring video frame images of the target hydraulic support by using a camera;

[0032] A key point detection module for performing target detection on the key points of the target hydraulic support based on the video frame images of the target hydraulic support, where the key points include the points at both ends inside the top beam of the target hydraulic support, the hinge points between each stage of rib protection plates, and the end points on both sides of the end of the last stage of rib protection plate;

[0033] A three-dimensional coordinate acquisition module for acquiring the three-dimensional space coordinates of the key points;

[0034] A plane normal vector acquisition module for calculating the normal vectors of the top beam plane and the plane of each stage of rib protection plate based on the three-dimensional space coordinates;

[0035] A telescopic angle acquisition module for calculating the included angle between every two adjacent planes respectively based on the normal vectors to obtain the telescopic angle of each stage of rib protection plate of the target hydraulic support.

[0036] The above technical solutions of the present invention have the following advantages compared with the prior art:

[0037] The method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support according to the present invention realizes real-time and high-precision measurement of the telescopic angle of the multi-stage rib protection plate of the hydraulic support through a machine vision-based measurement method, effectively solving the problems of traditional contact measurement methods being susceptible to interference, difficult to install and maintain, and the existing vision measurement methods being unable to accurately reflect the overall posture of multi-stage linkage. The explosion-proof multi-camera industrial camera, key point detection and dynamic compensation technology adopted not only improve the stability and reliability of the measurement, but also display the measurement results in real time through a visualization interface, providing accurate data support for the multi-stage coordinated linkage control of the hydraulic support, significantly improving the safe operation level and intelligent degree of fully-mechanized mining equipment in coal mines, and having important practical significance and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to make the content of the present invention easier to be clearly understood, the following further details the present invention according to specific embodiments of the present invention in conjunction with the drawings, where:

[0039] Figure 1 is the implementation flowchart of a method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support provided by the present invention;

[0040] Figure 2 is a schematic diagram of the camera installation position provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of key point settings provided by an embodiment of the present invention;

[0042] Description of reference numerals: 2-1: support top beam; 2-2: first-stage rib protection plate; 2-3: second-stage rib protection plate; 2-4: third-stage rib protection plate; 2-5: industrial camera. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] The core of the present invention is to provide a method and device for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support, effectively realizing real-time and high-precision measurement of the telescopic angle of the multi-stage rib protection plate of the hydraulic support.

[0044] In order to enable those skilled in the art to better understand the solution of the present invention, the following further details the present invention in conjunction with the drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] Please refer to Figure 1 , Figure 1 , which is the implementation flowchart of a method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support provided by the present invention; the specific operation steps are as follows:

[0046] S101: Collect video frame images of the target hydraulic support using a camera;

[0047] S102: Based on the video frame images of the target hydraulic support, perform target detection on the key points of the target hydraulic support. The key points include the points at both ends inside the top beam of the target hydraulic support, the hinge points between the guard plates at each level, and the end points on both sides of the end of the last-level guard plate;

[0048] S103: Obtain the three-dimensional spatial coordinates of the key points;

[0049] S104: Calculate the normal vectors of the top beam plane and the normal vectors of the guard plates at each level based on the three-dimensional spatial coordinates;

[0050] S105: Calculate the angle between every two adjacent planes based on the normal vectors to obtain the telescopic angle of each level of the guard plate of the target hydraulic support.

[0051] Based on the above embodiments, this embodiment details step S101:

[0052] In one embodiment, as Figure 2 shown, every few hydraulic supports (for example, it can be 3), install a camera at the position of the support top beam. The top beam position is selected because this position is relatively stable and can better cover the full telescopic range of multiple levels of guard plates.

[0053] Adjust the field of view angle of the camera to ensure that its field of view can cover the full telescopic range of multiple levels of guard plates. This means that in any telescopic state of the guard plate, the camera can completely capture the image of the guard plate. At the same time, ensure that there is a certain field of view overlap between adjacent cameras. The overlapping part of the field of view can be used for subsequent image stitching or data fusion to improve the integrity and accuracy of the measurement. For example, when the field of view of a single camera cannot completely cover a certain area of the guard plate, the overlapping part of adjacent cameras can provide supplementary information.

[0054] In one embodiment, in order to adapt to special environments such as underground coal mines, use an industrial multi-camera with explosion-proof design to ensure the safe operation of the equipment. The camera should be installed firmly and reliably to avoid the camera position shifting due to factors such as vibration, which may affect the measurement accuracy. Special brackets or fixing devices can be used to install the camera on the top beam of the hydraulic support and perform appropriate protective treatments such as dust prevention and moisture protection.

[0055] In one embodiment, use the Zhang Zhengyou calibration method to calibrate the explosion-proof industrial multi-camera (this is a method of camera calibration using a planar template). By taking images of the calibration board at different positions and angles, obtain the internal parameters of the camera (such as focal length, principal point coordinates, etc.) and external parameters (such as the attitude of the camera relative to the calibration board):

[0056] Prepare the calibration board: The calibration board is usually a flat template with a checkerboard or dots, and its geometric dimensions and the positions of the feature points are known.

[0057] Take calibration images: Take images of the calibration board from multiple different angles and positions, ensuring that the calibration board occupies different positions and orientations in the images to obtain sufficient data for calibration.

[0058] Data processing: Input the captured calibration images into the calibration algorithm, and calculate the internal and external parameters of the camera through computation. These parameters are crucial for subsequent conversion of image coordinates to three-dimensional space coordinates.

[0059] During the calibration process, ensure that the feature points of the calibration board are clearly visible, avoiding occlusion and blurring to improve the calibration accuracy.

[0060] When taking calibration images, parameters such as the aperture and focal length of the camera should be kept consistent with those during actual measurement to ensure the applicability of the calibration results.

[0061] In one embodiment, an explosion-proof industrial multi-camera calibrated by the Zhang Zhengyou calibration method installed at the positions of the top beams of multiple hydraulic supports in advance is used to collect video frame images of the target hydraulic support. Among them, a camera is installed every preset number of hydraulic supports. The multi-camera can provide image information from multiple perspectives, which helps to capture the motion state of the rib protection plate more comprehensively and reduce the influence of factors such as occlusion:

[0062] According to the actual environment and measurement requirements, reasonably set parameters such as the resolution and frame rate of the camera. A higher resolution can provide clearer image details, which helps to improve the accuracy of key point detection; an appropriate frame rate can ensure real-time performance and timely capture the dynamic changes of the rib protection plate.

[0063] Considering that there may be insufficient light in environments such as underground coal mines, parameters such as the sensitivity and exposure time of the camera also need to be appropriately adjusted to ensure that the quality of the collected images meets the requirements.

[0064] During the operation of the hydraulic support, the camera continuously collects video frame images of the movement of the rib protection plate. The collected image data will serve as the basis for subsequent processing and analysis.

[0065] Based on the above embodiments, this embodiment details step S102:

[0066] In one embodiment, the points at both ends inside the top beam of the hydraulic support (such as a pair of symmetric points at both ends inside the inner side of the support top beam) are set as key points, and these two points can be used as reference points to determine the position and attitude of the top beam. The hinge points between the guard plates at all levels are set as key points. These hinge points are the key positions for the movement of the guard plates. By detecting the position changes of these points, the telescopic angle of the guard plates can be accurately calculated. The end points on both sides of the end of the last-level guard plate are set as key points to determine the end position of the last-level guard plate.

[0067] In one embodiment, such as Figure 3 , the points at both ends inside the top beam of the hydraulic support (A, A ′ ), the hinge points between the guard plates at all levels (the hinge points O, O ′ between the first level and the top beam, the hinge points B, B ′ between the first level and the second level, the hinge points C, C ′ between the second level and the third level), and the end points at the ends of the third-level guard plate (D, D ′ ) are set as key points.

[0068] In one embodiment, reflective marking points are set at the key points. The reflective marking points should have good reflective performance so that they can be clearly detected in the image. The reflective material and color should form a distinct contrast with the surrounding environment to reduce the interference of factors such as ambient light. The shape and size of the marking points should be appropriate, being able to be accurately detected and not affecting the normal movement of the guard plates. For example, small reflective patches in the shape of a circle or a square can be used.

[0069] In one embodiment, each reflective marking point is numbered. The numbers should be unique and regular, facilitating quick identification and reference in the subsequent image processing and calculation processes. Record the position information of each marking point, including its actual position coordinates on the hydraulic support (such as the coordinates relative to the top beam) and the approximate position range in the image, which helps improve the efficiency and accuracy of key point detection.

[0070] Based on the above embodiments, the adaptive histogram equalization method is used to preprocess the video frame image of the target hydraulic support:

[0071] The CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm is a local histogram equalization method. It divides the image into multiple small regions (called "tiles"), performs histogram equalization within each region, and limits the range of contrast enhancement to avoid over-amplifying noise. In environments such as underground coal mines, due to factors such as uneven lighting and dust, the image may have low-contrast regions. The CLAHE algorithm can enhance the contrast of these regions, making details such as key points more clearly visible, thereby improving the accuracy and reliability of subsequent key point detection. The parameters of the CLAHE algorithm include the size of the tile, contrast limit, etc. The size of the tile should be selected according to the size and detail distribution of the image. Generally, smaller tiles can better adapt to local contrast changes, but the computational cost is relatively large; the contrast limit parameter is used to control the degree of contrast enhancement and avoid image distortion caused by over-enhancement.

[0072] In addition to the CLAHE algorithm, other denoising methods such as median filtering and Gaussian filtering can be adopted according to the specific situation of the image to further improve the image quality. If there are no special requirements for color information in subsequent processing, the image can be converted to a grayscale image to reduce the amount of data and improve the processing speed.

[0073] In one embodiment, based on the target hydraulic support video frame image, the YOLO-Mini model is used to detect the key points of the target hydraulic support:

[0074] The YOLO (You Only Look Once) series of models is an efficient real-time object detection algorithm, characterized by fast speed and relatively high accuracy. YOLO-Mini may be a lightweight improved version of the YOLO model, more suitable for running in resource-constrained industrial environments, and can quickly and accurately detect key points in the image.

[0075] In order to enable the YOLO-Mini model to accurately detect reflective marking points, a large amount of image data marked with reflective marking points needs to be prepared as training samples. These images should cover different scenarios, lighting conditions, and rib shield postures to improve the generalization ability of the model.

[0076] The preprocessed image is input into the YOLO-Mini model, and the model will output the position information of each reflective marking point in the image, usually including the center coordinates, confidence, etc. of the marking point. The detection results are screened and optimized to remove misdetected points with low confidence and retain the accurate key point position information. For example, a confidence threshold can be set, and only detection results with a confidence higher than this threshold are considered valid key points.

[0077] The detection accuracy is an important indicator for key point detection, directly affecting the accuracy of subsequent angle calculation. By optimizing the structure of the YOLO-Mini model, the selection of training data, and the annotation quality, etc., the detection accuracy can be improved. The detection efficiency is also crucial, especially in real-time measurement scenarios. The fast detection ability of the YOLO-Mini model can ensure that the system can obtain the key point position information in real time, meeting the dynamic measurement requirements during the operation of the hydraulic support.

[0078] In one embodiment, the specific steps for key point detection are as follows:

[0079] Assume the input image is I, and the output of the YOLO-Mini model is the set of detected key points {P i}, where each key point P i includes its image coordinates (x i , y i ) and confidence level c i . The detection process can be expressed as {P i} = YOLO-Mini(I); where each key point P i can be expressed as P i = {(x i , y i ), c i};

[0080] Since there may be false detections during the detection process, it is necessary to perform confidence level screening on the detection results. Set a confidence level threshold c th . Only when the confidence level c i of the key point is greater than this threshold, is the key point considered valid. The screening process can be expressed as:

[0081]

[0082] where P i′ represents the key points after screening.

[0083] Perform post-processing on the key points after screening, such as non-maximum suppression (NMS), to remove the key points detected repeatedly and retain the optimal detection results. Specifically:

[0084] According to the confidence level from high to low, for each key point after screening in turn, calculate its overlap degree with other key points, and remove the key points with lower confidence levels whose overlap degree with it is higher than the preset overlap degree threshold. The commonly used overlap degree calculation method is the intersection over union (IoU). The calculation formula of IoU is as follows:

[0085]

[0086] Among them, Area(D i ∩D j ) represents the intersection area of the bounding boxes of the detection results D i and D j , and Area(D i ∪D j ) represents the union area of the bounding boxes of the detection results D i and D j .

[0087] Based on the above embodiments, this embodiment elaborates on step S103 in detail:

[0088] Determine the origin of the coordinate system and establish a three-dimensional space rectangular coordinate system: Select a suitable reference point (such as a certain fixed point on the top beam of the hydraulic support) as the origin of the coordinate system and establish a three-dimensional space rectangular coordinate system. This coordinate system is used to describe the position and attitude of the hydraulic support and its rib protection plate in space.

[0089] Using the perspective projection model, map the two-dimensional image coordinates of the key points to the three-dimensional space coordinate system according to the internal and external parameters of the camera to obtain the three-dimensional space coordinates of the key points: Use the perspective projection model to convert the image coordinates into three-dimensional space coordinates. The perspective projection model takes into account the internal parameters of the camera (such as focal length, principal point coordinates, etc.) and external parameters (such as the attitude of the camera relative to the coordinate system). Through this model, the two-dimensional image coordinates of the key points detected in the image can be mapped to the three-dimensional space coordinate system; the specific conversion process involves a series of mathematical calculations, including matrix operations, etc.

[0090] In one embodiment, through coordinate conversion, the three-dimensional space coordinates of points O / A / B / C / D and O’ / A’ / B’ / C’ / D can be obtained, and these coordinates are the basis for subsequent angle calculations.

[0091] Based on the above embodiments, this embodiment elaborates on step S104 in detail:

[0092] Based on the three-dimensional space coordinates, perform plane fitting on the top beam plane of the hydraulic support and the rib protection plate planes at all levels to obtain the plane equations of each plane;

[0093] Based on the plane equations of each plane, obtain the normal vectors of each plane.

[0094] In one embodiment, the method of plane fitting can adopt the least squares method, SVD decomposition method, etc., and by calculation, make the distance between the fitting plane and the key points the smallest. Specifically:

[0095] The specific steps of the least squares method are as follows: For the top beam plane or any face guard plane, select three or more key points P1(x1, y1, z1), P2(x2, y2, z2), …, P m , and fit a plane by the least squares method. The plane equation can be expressed as: ax + by + cz + d = 0, and the normal vector n of the plane can be obtained from the coefficients of the plane equation as n = (a, b, c).

[0096] The specific steps of the SVD decomposition method are as follows: For the top beam plane or any face guard plane, select three or more key points P1(x1, y1, z1), P2(x2, y2, z2), …, P m , and calculate the centroid of the point set Subtract the centroid from each point to obtain the matrix A: Perform singular value decomposition on A, A = UΣV T , and the normal vector n is the last column of V (corresponding to the smallest singular value).

[0097] In one embodiment, for the top beam plane, the first-level face guard plane, the second-level face guard plane, and the third-level face guard plane, plane fitting is respectively performed according to the key point coordinates thereon (i.e., m = 4 in the above formula: P1(x1, y1, z1), P2(x2, y2, z2), …, P4). For the top beam plane, the reflective marking points at both ends inside the top beam and the hinge points with the first-level face guard plane can be selected as fitting data. For each face guard plane, the hinge points with adjacent planes can be selected as fitting data. For the top beam plane, its normal vector is n top ; for the first-level face guard plane, its normal vector is n1; for the second-level face guard plane, its normal vector is n2; for the third-level face guard plane, its normal vector is n3.

[0098] In one embodiment, the normal vectors of each plane are calculated according to the least squares method or the SVD decomposition method. The normal vector of a plane is a vector perpendicular to the plane, which plays a key role in subsequent angle calculations and is used to represent the direction of each plane.

[0099] Based on the above embodiments, this embodiment details step S105:

[0100] Calculate the angle between adjacent face guard planes using their normal vectors. Let the normal vectors of two planes be n i and n i+1 , and the angle θ between them can be calculated by the following formula:

[0101]

[0102] where, n i ·ni+1 Denote the normal vector as n i and n i+1 The dot product of, |n i | and |n i+1 | respectively represent the modulus of the normal vector n i and n i+1 .

[0103] For every two adjacent planes, calculate the angle between them respectively. For example, calculate the angle between the top beam plane and the first-stage rib protection plate plane, the angle between the first-stage rib protection plate plane and the second-stage rib protection plate plane, the angle between the second-stage rib protection plate plane and the third-stage rib protection plate plane, etc. By calculating these angles in sequence, the telescopic angles of each stage of rib protection plates can be obtained. These angles reflect the attitude changes of the rib protection plates in space and are of great significance for the multi-stage coordinated linkage control of the hydraulic support. Specifically:

[0104] For the angle between the top beam plane and the first-stage rib protection plate plane:

[0105]

[0106] For the angle between the first-stage rib protection plate plane and the second-stage rib protection plate plane:

[0107]

[0108] For the angle between the second-stage rib protection plate plane and the third-stage rib protection plate plane:

[0109]

[0110] The Kalman filter is an efficient self-recursive filter that can estimate the state of the system from a series of noisy measurement data. Its basic principle is to establish a state space model of the system and continuously correct the state estimate value of the system through two steps of prediction and update, thereby suppressing the influence of noise. In this application, the state of the system can be represented as the true values of the telescopic angles of each stage of rib protection plates, and the measurement data is the angle values calculated through image processing. Due to the possible presence of noise (such as camera jitter, environmental interference, etc.) in the image acquisition and processing process, resulting in certain errors in the measurement data, the Kalman filter can fuse and correct these data.

[0111] Based on the above embodiments, according to the multi-frame video images of the target hydraulic support, obtain multiple groups of telescopic angle data of the multi-stage rib protection plates of the target hydraulic support, and use the Kalman filter to fuse and correct the multiple groups of telescopic angle data of the multi-stage rib protection plates of the target hydraulic support to obtain multiple groups of telescopic angle data of the multi-stage rib protection plates of the target hydraulic support after dynamic compensation.

[0112] In this embodiment, the telescopic angle data of the canopy board calculated from multiple frames of images is input into a Kalman filter. The filter performs prediction and update operations on each frame of data according to the dynamic model of the system (such as the motion law of the canopy board) and the measurement model (such as the angle calculation model). By introducing the Kalman filter, the jitter noise caused by factors such as camera jitter and environmental noise can be effectively suppressed, making the measurement results more stable and smooth. This is very important for the real-time monitoring and control of the operation state of the hydraulic support, and can improve the reliability and accuracy of the system.

[0113] The performance of the Kalman filter is affected by its parameters (such as process noise covariance, observation noise covariance, etc.). According to the noise characteristics and system dynamic characteristics in actual applications, these parameters need to be reasonably adjusted to achieve the best filtering effect.

[0114] Based on the above embodiments, the present invention also designs an intuitive visualization interface for real-time displaying the measured values of the telescopic angles of each level of the canopy board. The visualization interface can display data in a graphical manner, such as drawing the dynamic model of the canopy board and updating its telescopic angle in real time. Other relevant information, such as measurement timestamps, camera status, alarm information, etc., can also be displayed on the interface so that the operator can comprehensively understand the operation situation of the system.

[0115] The measured values of the telescopic angles of the canopy board after dynamic compensation are displayed in real time in the visualization interface according to the timestamps. The update frequency of the data should match the operating speed of the hydraulic support and the measurement requirements to ensure that the operator can obtain the latest angle information in a timely manner. Multiple forms such as tables and curve graphs can be used to display the data, which is convenient for the operator to analyze and monitor. For example, by drawing a curve graph of the angle changing with time, the motion trend and abnormal conditions of the canopy board can be intuitively observed.

[0116] The measured results of the telescopic angles of the canopy board displayed in real time can provide data support for the multi-level coordinated linkage control of the hydraulic support. According to these angle information, the control system can timely adjust the telescopic actions of the canopy board to adapt to the changes in the coal seam, ensuring the support effect of the hydraulic support and the safe production of the coal mine. At the same time, these measurement data can also be used for equipment fault diagnosis and maintenance. By analyzing the historical data, potential equipment problems can be found, and maintenance and repairs can be carried out in advance to reduce the probability of equipment failures.

[0117] The embodiment of the present invention also provides a device for measuring the telescopic angles of multiple levels of the canopy board of a hydraulic support; the specific device may include:

[0118] An image acquisition module for using a camera to acquire video frame images of the target hydraulic support;

[0119] The key point detection module is used to perform target detection on the key points of the target hydraulic support based on the target hydraulic support video frame image. The key points include the points at both ends inside the top beam of the target hydraulic support, the hinge points between the guard plates at all levels, and the end points on both sides of the end of the last-level guard plate.

[0120] The three-dimensional coordinate acquisition module is used to acquire the three-dimensional space coordinates of the key points.

[0121] The plane normal vector acquisition module is used to calculate the normal vector of the top beam plane and the normal vectors of the guard plate planes at all levels based on the three-dimensional space coordinates.

[0122] The telescopic angle acquisition module is used to calculate the angle between every two adjacent planes respectively based on the normal vectors, and obtain the telescopic angles of the guard plates at all levels of the target hydraulic support.

[0123] The telescopic angle measuring device for the multi-level guard plates of the hydraulic support in this embodiment is used to implement the foregoing telescopic angle measuring method for the multi-level guard plates of the hydraulic support. Therefore, the specific implementation manners in the telescopic angle measuring device for the multi-level guard plates of the hydraulic support can be seen in the embodiment part of the foregoing telescopic angle measuring method for the multi-level guard plates of the hydraulic support. For example, the image acquisition module, the key point detection module, the three-dimensional coordinate acquisition module, the plane normal vector acquisition module, and the telescopic angle acquisition module are respectively used to implement steps S101, S102, S103, S104, and S105 in the foregoing telescopic angle measuring method for the multi-level guard plates of the hydraulic support. Therefore, the specific implementation manners can refer to the descriptions of the corresponding individual part embodiments and will not be elaborated herein.

[0124] A specific embodiment of the present invention further provides a telescopic angle measuring device for the multi-level guard plates of a hydraulic support, including: a memory for storing a computer program; a processor for implementing the steps of the foregoing telescopic angle measuring method for the multi-level guard plates of a hydraulic support when executing the computer program.

[0125] A specific embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the foregoing telescopic angle measuring method for the multi-level guard plates of a hydraulic support when executed by a processor.

[0126] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a means for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 means for implementing the functions specified in one or more blocks.

[0128] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 means for implementing the functions specified in one or more blocks.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operating steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 means for implementing the functions specified in one or more blocks.

[0130] Obviously, the above embodiments are merely examples for clear illustration and are not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to exhaustively list all implementation manners here. And the obvious changes or modifications derived therefrom are still within the protection scope of the present invention.

Claims

1. A method for measuring the telescopic angle of a multi-stage rib protection plate of a hydraulic support, characterized in that, Including: Collecting video frame images of the target hydraulic support using a camera; Based on the video frame images of the target hydraulic support, performing object detection on the key points of the target hydraulic support, where the key points include the points at both ends inside the top beam of the target hydraulic support, the hinge points between the various gangue protection plates, and the end points on both sides of the end of the last-level gangue protection plate; Obtaining the three-dimensional spatial coordinates of the key points; Calculating the normal vectors of the top beam plane and the planes of the various gangue protection plates based on the three-dimensional spatial coordinates; Calculating the angles between every two adjacent planes based on the normal vectors to obtain the telescopic angles of each level of the gangue protection plate of the target hydraulic support.

2. The method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support according to claim 1, characterized in that, The collecting of the video frame images of the target hydraulic support includes: Collecting video frame images of the target hydraulic support using explosion-proof industrial multi-cameras calibrated by the Zhang Zhengyou calibration method and pre-installed at the positions of the top beams of multiple hydraulic supports, where one camera is installed every preset number of hydraulic supports.

3. The method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support according to claim 2, wherein, Reflective marking points are pre-set on the key points, and each reflective marking point has a corresponding number.

4. The method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support according to claim 1 or 3, characterized in that, After collecting the video frame images of the target hydraulic support, it further includes: Preprocessing the video frame images of the target hydraulic support using the adaptive histogram equalization method.

5. The method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support according to claim 1 or 3, characterized in that, The performing of object detection on the key points of the target hydraulic support based on the video frame images of the target hydraulic support includes: Based on the video frame images of the target hydraulic support, using the YOLO-Mini model to detect the key points of the target hydraulic support.

6. The method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support according to claim 1 or 3, characterized in that, The obtaining of the three-dimensional spatial coordinates of the key points includes: Determining the origin of the coordinate system and establishing a three-dimensional rectangular coordinate system; Using the perspective projection model to map the two-dimensional image coordinates of the key points into the three-dimensional coordinate system according to the internal and external parameters of the camera to obtain the three-dimensional spatial coordinates of the key points.

7. The method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support according to claim 6, characterized in that, The calculating of the normal vectors of the top beam plane and the planes of the various gangue protection plates based on the three-dimensional spatial coordinates includes: Performing plane fitting on the top beam plane of the hydraulic support and the planes of the various gangue protection plates respectively based on the three-dimensional spatial coordinates to obtain the plane equations of each plane; Obtaining the normal vectors of each plane based on the plane equations of each plane.

8. The method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support according to claim 1, characterized in that, After obtaining the telescopic angles of each level of the gangue protection plate of the target hydraulic support, it further includes: According to the multi-frame video images of the target hydraulic support, obtaining multiple sets of telescopic angle data of the multi-level gangue protection plates of the target hydraulic support; Using the Kalman filter to fuse and correct the multiple sets of telescopic angle data of the multi-level gangue protection plates of the target hydraulic support to obtain multiple sets of telescopic angle data of the multi-level gangue protection plates of the target hydraulic support after dynamic compensation.

9. The method for measuring the telescopic angle of the multi-stage rib protection plate of the hydraulic support according to claim 8, characterized in that, After obtaining the multiple sets of telescopic angle data of the multi-level gangue protection plates of the target hydraulic support after dynamic compensation, it further includes: According to the time stamp, feeding back the multiple sets of telescopic angle data of the multi-level gangue protection plates of the target hydraulic support after dynamic compensation to the visualization interface.

10. A telescopic angle measuring device for a multi-stage rib protection plate of a hydraulic support, characterized in that, Including: An image acquisition module for collecting video frame images of the target hydraulic support using a camera; A key point detection module for performing object detection on the key points of the target hydraulic support based on the video frame images of the target hydraulic support, where the key points include the points at both ends inside the top beam of the target hydraulic support, the hinge points between the various gangue protection plates, and the end points on both sides of the end of the last-level gangue protection plate; A three-dimensional coordinate acquisition module for acquiring the three-dimensional spatial coordinates of the key points; A plane normal vector acquisition module for calculating the normal vectors of the top beam plane and the plane of each stage of the rib protection plate based on the three-dimensional spatial coordinates; A telescopic angle acquisition module for calculating the included angle between every two adjacent planes respectively based on the normal vectors to obtain the telescopic angle of each stage of the rib protection plate of the target hydraulic support.

Images