Hydraulic support face guard state monitoring and early warning method and device based on three-dimensional modeling

Abstract

The invention relates to a hydraulic support face guard state monitoring and early warning method and device based on three-dimensional modeling, and the method comprises the steps: carrying out the multi-conversion mapping among a world coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system based on an original working condition image of a fully mechanized coal mining face in combination with a face guard three-dimensional model and folding / unfolding safety line information; generating a working condition image fused with the three-dimensional projection; a deep learning instance segmentation algorithm is adopted to extract the contour of the face guard, and a to-be-detected area is dynamically divided; the folding state of the face guard in the advancing direction of the coal mining machine and the unfolding state of the rear face guard are monitored by comparing the overlapping area of the contour of the face guard and the projection area of the safety line; and frame numbers and early warning information are automatically marked for the abnormal state face guard plates, and a visual early warning image is generated. According to the invention, through a double verification mechanism of three-dimensional space projection and two-dimensional visual identification, full-period state monitoring of the face guard is realized, hardware dependence is reduced, three-dimensional space characterization is enriched, and high monitoring accuracy is maintained in an underground complex environment.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, in particular to the field of artificial intelligence technology, and more particularly to a method and device for monitoring and warning the state of a hydraulic support guard plate based on three-dimensional modeling. Background Art

[0002] In fully mechanized mining faces, interference between the shearer and the hydraulic support guards is a significant factor impacting safe production. The hydraulic support guards support the coal wall, preventing spalling and roof collapse. As the shearer advances along the face, the hydraulic support guards fold promptly; after the shearer passes, the hydraulic support guards deploy promptly. This is crucial for improving the intelligent capabilities of coal mines and ensuring production safety.

[0003] Related technologies monitor the status of support plates using contact sensor-based monitoring methods, image recognition-based support plate status monitoring methods, or virtual reality-based collision detection methods. Contact sensor-based monitoring methods directly measure the position and angle of the support plates by installing devices such as angle sensors and displacement sensors on them. However, these methods are susceptible to factors such as coal dust, humidity, and vibration in the complex underground coal mine environment, resulting in high sensor failure rates and high maintenance costs. Furthermore, measurement results are easily affected by factors such as machine body tilt, making accuracy difficult to guarantee. Image recognition-based support plate status monitoring methods utilize fog and dust image sharpening algorithms and machine vision measurement methods to monitor the retraction angle of hydraulic support support plates. While these methods avoid the shortcomings of contact sensors, image quality is difficult to guarantee in complex lighting and coal dust environments. Furthermore, target detection is based solely on two-dimensional images, lacking accurate representation of three-dimensional spatial relationships. Virtual reality-based collision detection methods combine virtual rays and bounding boxes during remote control of fully mechanized mining face equipment, enabling collision detection and early warning monitoring between fully mechanized mining face equipment. Although this method can simulate collision detection in a virtual environment, it requires a large amount of sensor data support and is difficult to match with the actual working surface environment in real time.

[0004] In summary, the relevant technologies only focus on real-time collision detection, and do not cover the tracking of the guard plate reset status after the coal mining machine passes, and cannot form closed-loop management; they are highly dependent on hardware equipment, resulting in extremely high costs; they cannot accurately identify and locate the guard plate in three-dimensional space, and are difficult to adapt to complex environments, resulting in large calculation errors and low monitoring accuracy. Summary of the Invention

[0005] One object of the present invention is to provide a hydraulic support guard plate status monitoring and early warning method based on three-dimensional modeling, which combines three-dimensional modeling and computer vision technology to achieve full-cycle status monitoring of the hydraulic support guard plate, form a closed-loop management, reduce dependence on hardware equipment, and thus reduce costs; it can adapt to complex environments, accurately locate and identify the guard plate in three-dimensional space, reduce calculation errors, and improve monitoring accuracy. Another object of the present invention is to provide a hydraulic support guard plate status monitoring and early warning device based on three-dimensional modeling. Another object of the present invention is to provide a computer-readable medium. Another object of the present invention is to provide a computer device.

[0006] In order to achieve the above objectives, the present invention discloses a method for monitoring and warning the status of a hydraulic support guard plate based on three-dimensional modeling, comprising:

[0007] Obtain the original working condition image of the fully mechanized mining face, the constructed 3D model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the shearer movement information;

[0008] Through the image acquisition device information and the original working condition image, the three-dimensional model information and safety line information of the hydraulic support guard plate are converted and mapped into multiple coordinate systems to generate a fused working condition image;

[0009] Through deep learning algorithms, the original working condition image is segmented and the guard plate contour information is extracted;

[0010] Based on the shearer motion information, the risk monitoring of the guard plate status is carried out according to the fusion of working condition images and guard plate contour information to generate risk monitoring results;

[0011] If the risk monitoring results show that the guard plate status is abnormal, a guard plate status warning image is generated based on the fusion of the working condition image and the guard plate contour information.

[0012] Preferably, the image acquisition device information includes posture information and device internal parameters, and the three-dimensional model information of the hydraulic support guard plate includes a three-dimensional space point set of the guard plate;

[0013] Through the image acquisition device information and the original working condition image, the three-dimensional model information and safety line information of the hydraulic support guard plate are converted and mapped into multiple coordinate systems to generate a fused working condition image, including:

[0014] Convert the posture information into quaternion to generate a rotation matrix;

[0015] By using the rotation matrix, the three-dimensional space point set and safety line information of the guard plate are converted from the world coordinate system to the camera coordinate system, thereby generating the three-dimensional space point set and safety line information of the guard plate in the camera coordinate system;

[0016] Through perspective projection, the three-dimensional space point set and safety line information of the guard plate in the camera coordinate system are converted into the two-dimensional position information and safety line information of the guard plate in the image coordinate system;

[0017] Through the internal parameters of the device, the two-dimensional position information of the guard plate and the safety line information in the image coordinate system are converted into the three-dimensional spatial point set and safety line information of the guard plate in the pixel coordinate system;

[0018] The two-dimensional position information of the side guard plate and the safety line information in the pixel coordinate system are mapped to the original working condition image for visualization processing to generate a fused working condition image.

[0019] Preferably, the original working condition image is segmented by an instance through a deep learning algorithm to extract the guard plate contour information, including:

[0020] Performing image preprocessing on the original working condition image to generate a preprocessed original working condition image;

[0021] Through deep learning algorithms, the guard plate positions are identified on the pre-processed original working condition images to generate the initial contour information of each guard plate;

[0022] The initial contour information is post-processed to generate the guard plate contour information.

[0023] Preferably, the shearer movement information includes the shearer position and movement direction, the fused working condition image includes safety line information, and the safety line information includes a folding safety line and an unfolding safety line;

[0024] Based on the shearer motion information, the guard plate status is monitored based on the fusion of working condition images and guard plate contour information, generating risk monitoring results, including:

[0025] According to the folding safety line and the unfolding safety line, the folding auxiliary judgment area and the unfolding auxiliary judgment area are respectively determined in the fused working condition image;

[0026] According to the preset support range, the shearer position and movement direction, the guard plate area to be inspected is divided in the fused working condition image;

[0027] According to the side guard plate contour information, the folding auxiliary judgment area and the unfolding auxiliary judgment area, the state of the side guard plate in the side guard plate area to be inspected is monitored and a risk monitoring result is generated.

[0028] Preferably, the guard plate area to be detected includes a forward direction area and a reverse direction area;

[0029] Based on the side guard plate contour information, the folding auxiliary judgment area, and the unfolding auxiliary judgment area, the side guard plate status within the side guard plate area to be inspected is monitored and risk monitoring results are generated, including:

[0030] Determine whether the overlapping area between the side guard plate contour information in the forward direction area and the folding auxiliary judgment area is greater than a preset folding threshold;

[0031] If so, generate a risk monitoring result indicating that the guard plate is in an abnormal state;

[0032] If not, generate a risk monitoring result that the guard plate is in a normal state;

[0033] Determine whether the overlapping area between the side guard plate contour information in the reverse direction area and the deployment auxiliary judgment area is greater than a preset deployment threshold;

[0034] If so, generate a risk monitoring result indicating that the guard plate is in a normal state;

[0035] If not, a risk monitoring result indicating that the guard plate is in an abnormal state is generated.

[0036] Preferably, the fused working condition image includes the guard plate frame number of each guard plate;

[0037] The method also includes:

[0038] The guard plate contour information is matched with the guard plate two-dimensional position information in the fusion working condition image, and the guard plate frame number corresponding to each guard plate contour information is determined and marked.

[0039] Preferably, generating a guard plate status warning image based on the fusion working condition image and guard plate contour information includes:

[0040] Determine the center point of the guard plate according to the guard plate contour information corresponding to the abnormal state guard plate;

[0041] Generate guard board status warning information based on the guard board frame number corresponding to the abnormal guard board and the preset warning text;

[0042] According to the preset warning annotation format, the guard plate status warning information is added to the guard plate center point corresponding to the guard plate with abnormal status in the fusion working condition image to generate the guard plate status warning image.

[0043] Preferably, the modeling parameters of the three-dimensional model information of the hydraulic support guard plate are determined by performing parameter adaptive iterative optimization based on a deep learning algorithm.

[0044] The present invention also discloses a hydraulic support guard plate state monitoring and early warning device based on three-dimensional modeling, comprising:

[0045] An acquisition unit is used to acquire the original working condition image of the fully mechanized mining face, the constructed three-dimensional model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine movement information;

[0046] A coordinate system conversion unit is used to perform multiple coordinate system conversion and mapping on the three-dimensional model information and safety line information of the hydraulic support guard plate through the image acquisition device information and the original working condition image, so as to generate a fused working condition image;

[0047] A deep learning unit is used to perform instance segmentation on the original working condition image and extract the guard plate contour information through a deep learning algorithm;

[0048] The risk monitoring unit is used to perform risk monitoring on the guard plate status based on the shearer motion information, the fusion working condition image and the guard plate contour information, and generate risk monitoring results;

[0049] The status warning unit is used to generate a guard plate status warning image based on the fusion of the working condition image and the guard plate contour information if the risk monitoring result shows that the guard plate status is abnormal.

[0050] The present invention also discloses a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0051] The present invention also discloses a computer device, including a memory and a processor, wherein the memory is used to store information including program instructions, the processor is used to control the execution of program instructions, and the processor implements the above method when executing the program.

[0052] The present invention also discloses a computer program product, comprising a computer program / instruction, which implements the above method when the computer program / instruction is executed by a processor.

[0053] The present invention obtains the original working condition image of the comprehensive mining working face, the constructed three-dimensional model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine motion information; through the image acquisition equipment information and the original working condition image, the three-dimensional model information and the safety line information of the hydraulic support guard plate are converted and mapped into multiple coordinate systems to generate a fused working condition image; through the deep learning algorithm, the original working condition image is instance segmented to extract the guard plate contour information; based on the coal mining machine motion information, the guard plate status is risk monitored according to the fused working condition image and the guard plate contour information to generate a risk monitoring result; if the risk monitoring result shows that the guard plate status is abnormal, a guard plate status warning image is generated according to the fused working condition image and the guard plate contour information, which combines three-dimensional modeling and computer vision technology to realize full-cycle status monitoring of the hydraulic support guard plate, form a closed-loop management, reduce dependence on hardware equipment, and thus reduce costs; it can adapt to complex environments, accurately locate and identify guard plates in three-dimensional space, reduce calculation errors, and improve monitoring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A schematic diagram of the structure of a hydraulic support guard plate status monitoring and early warning system based on three-dimensional modeling provided by an embodiment of the present invention;

[0056] Figure 2 A flowchart of a method for monitoring and warning the status of a hydraulic support guard plate based on three-dimensional modeling provided by an embodiment of the present invention;

[0057] Figure 3 A flowchart of another method for monitoring and warning the status of a hydraulic support guard plate based on three-dimensional modeling provided in an embodiment of the present invention;

[0058] Figure 4 A flowchart of quaternion posture projection provided by an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of the structure of a hydraulic support guard plate state monitoring and early warning device based on three-dimensional modeling provided by an embodiment of the present invention;

[0060] Figure 6 A schematic structural diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0062] It should be noted that the method and device for monitoring and warning the status of a hydraulic support guard plate based on three-dimensional modeling disclosed in this application can be used in the field of artificial intelligence technology, and can also be used in any field other than the field of artificial intelligence technology. The application field of the method and device for monitoring and warning the status of a hydraulic support guard plate based on three-dimensional modeling disclosed in this application is not limited.

[0063] In order to facilitate the understanding of the technical solution provided by this application, the relevant contents of the technical solution of this application are first explained below. With the continuous deepening of the intelligent construction of coal mines, the intelligent technology of comprehensive mining working faces is also developing rapidly, and realizing less-managed or even unmanned operations has become an industry development trend. When the coal mining machine advances along the working face, if the guard plate of the hydraulic support fails to be folded correctly and in time, it may cause the coal mining machine to collide with the guard plate, causing equipment damage or even safety accidents. At the same time, after the coal mining machine passes, the correct operation of the guard plate is also related to the safety and stability of the working face. If the guard plates are not deployed in time after the coal mining machine passes, it will bring a series of serious hazards: First, the guard plates are an important part of the support roof management system. Their main function is to prevent coal wall collapse and control roof sinking. Undeployed guard plates will cause gaps to form between the goaf and the working face, causing the roof to lose effective support and increase the risk of roof collapse; second, failure to deploy the guard plates will cause the coal wall to lose lateral support, especially in the working face of soft coal seams, which can easily lead to accidents such as coal wall spalling and roof collapse; third, failure to deploy the guard plates correctly will affect the effective management of the goaf, which may lead to uneven pressure distribution in the goaf and trigger dynamic phenomena on the working face, such as impact ground pressure and other safety accidents; fourth, during the advancement of the working face, if the guard plates are not deployed in time, it will affect the normal movement of the next cycle support, resulting in obstruction of the working face advancement and affecting production efficiency; finally, failure to deploy the guard plates correctly will affect the ventilation effect of the working face, which may lead to the accumulation of harmful gases such as gas and coal dust, increasing safety risks.

[0064] Therefore, to realize the whole process monitoring of the folding status of the side guard, it is necessary not only to pay attention to whether the front side guard is folded in time when the coal mining machine passes through, but also to monitor whether the rear side guard is unfolded in time when the coal mining machine passes through, so as to ensure the safe and efficient operation of the fully mechanized mining working face.

[0065] This invention uses 3D modeling and computer vision to monitor and warn the status of side guards. It aims to solve the following key technical problems in existing side guard status monitoring and collision warning technologies, thereby significantly improving the safety and intelligence level of fully mechanized mining working faces:

[0066] 1. Lack of 3D spatial representation: Existing 2D detection methods cannot accurately reflect the spatial position of the side guard, resulting in inaccurate collision risk assessments. This method significantly increases the misjudgment rate when the side guard tilt angle changes. This invention overcomes the limitations of 2D detection methods by establishing a precise 3D spatial model to accurately position and represent the side guard's position in 3D space.

[0067] 2. Dynamic parameter drift: When camera installation parameters change due to harsh environments such as downhole frame movement and vibration, the lack of an effective adaptive calibration mechanism significantly reduces system stability and reliability. This invention dynamically adjusts system parameters by leveraging the differences between deep learning recognition results and geometric projection results, improving the system's environmental adaptability and detection accuracy.

[0068] 3. Insufficient full-cycle monitoring: Existing technologies primarily focus on collision warnings before a shearer passes through, but fail to effectively track the guard plate's status during its reset phase, preventing closed-loop management. This invention accurately determines the guard plate's folded state by comparing its contour with the projected safety line, and issues precise warnings for unfolded guard plates. Furthermore, a differentiated monitoring strategy is designed to distinguish between different scenarios before and after a shearer passes, improving the relevance and effectiveness of warnings and achieving full-cycle monitoring.

[0069] 4. Hardware Dependence and High Cost: Traditional solutions rely on additional sensors such as millimeter-wave radar and lidar, which not only increases system deployment and maintenance costs but also reduces the system's adaptability in complex underground environments. This invention utilizes an efficient coordinate system conversion mechanism to accurately project the 3D guard plate model and safety line onto the image plane, improving detection accuracy. By combining deep learning with geometric projection methods, the guard plate's position in the image is precisely identified and its corresponding frame number is determined, achieving precise positioning.

[0070] Figure 1 A structural diagram of a hydraulic support guard plate status monitoring and early warning system based on three-dimensional modeling provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the system includes: a data acquisition layer 100, a core processing layer 200 and an early warning control layer 300. The data acquisition layer 100 is in communication connection with the core processing layer 200, and the core processing layer 200 is in communication connection with the early warning control layer 300.

[0071] The data acquisition layer 100 is used to capture raw working condition images and the device's PTZ (Pan-Tilt-Zoom) parameters using an image acquisition device. Alternatively, the image acquisition device can be an industrial camera; the raw working condition images have a resolution of 1280×960 and a frame rate of 30 FPS.

[0072] The core processing layer 200 is used to perform 3D modeling of the side guard and define its geometric parameters (width 1400mm, height 1340mm); realize the mapping from 3D coordinates to the image plane; extract the side guard outline through the lightweight SOLOv2 model (inference speed 34ms / frame) to achieve instance segmentation; and calibrate the 3D modeling parameters based on the parameter adaptive optimization algorithm to achieve dynamic optimization.

[0073] The early warning control layer 300 is responsible for executing early warning strategies. Specifically, this includes identifying the guard plate frame numbers of abnormal conditions, labeling them with warning information, and generating a warning image of the guard plate status. It is also used to reset the monitoring mechanism. In practical applications at fully mechanized coal mining faces, guard plates must not only be correctly folded when a shearer approaches, but also promptly reset (deploy) after the shearer passes to ensure safe production at the face.

[0074] In the technical solution provided by the embodiment of the present invention, the original working condition image of the comprehensive mining working face, the constructed three-dimensional model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine motion information are obtained; the three-dimensional model information and the safety line information of the hydraulic support guard plate are converted and mapped into multiple coordinate systems through the image acquisition equipment information and the original working condition image to generate a fused working condition image; the original working condition image is instance segmented through a deep learning algorithm to extract the guard plate contour information; based on the coal mining machine motion information, the guard plate status is risk monitored according to the fused working condition image and the guard plate contour information to generate a risk monitoring result; if the risk monitoring result shows that the guard plate status is abnormal, a guard plate status warning image is generated according to the fused working condition image and the guard plate contour information, which combines three-dimensional modeling and computer vision technology to realize full-cycle status monitoring of the hydraulic support guard plate, form a closed-loop management, reduce dependence on hardware equipment, and thus reduce costs; it can adapt to complex environments, accurately locate and identify the guard plate in three-dimensional space, reduce calculation errors, and improve monitoring accuracy.

[0075] It is worth mentioning that Figure 1 The hydraulic support guard plate condition monitoring and early warning system based on 3D modeling is also applicable to Figure 2 or Figure 3 The method of identifying new words in the field will not be described in detail here.

[0076] The following describes the implementation of the method for monitoring and warning the status of a hydraulic support guard plate based on 3D modeling, provided in an embodiment of the present invention, using a 3D modeling-based hydraulic support guard plate status monitoring and warning device as an example. It is understood that the method for monitoring and warning the status of a hydraulic support guard plate based on 3D modeling, provided in an embodiment of the present invention, may include, but is not limited to, the 3D modeling-based hydraulic support guard plate status monitoring and warning device.

[0077] Figure 2 A flow chart of a method for monitoring and warning the status of a hydraulic support guard plate based on three-dimensional modeling provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0078] Step 101: Acquire the original working condition image of the fully mechanized mining face, the constructed three-dimensional model information of the hydraulic support side guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine movement information.

[0079] In this embodiment of the present invention, the original working condition image is an unprocessed working condition image captured in real time by an image acquisition device. The three-dimensional model of the hydraulic support guard plate is constructed in three-dimensional space within a world coordinate system with the image acquisition device as the origin. Safety lines are defined in this three-dimensional space to obtain safety line information, including folding safety lines and unfolding safety lines. Image acquisition device information includes posture information and device internal parameters. The three-dimensional model information of the hydraulic support guard plate includes a set of three-dimensional spatial points of the guard plate and the guard plate frame number of each guard plate.

[0080] As an optional solution, the image acquisition device is an industrial camera.

[0081] Step 102: Perform multiple coordinate system conversion and mapping on the three-dimensional model information and safety line information of the hydraulic support guard plate using the image acquisition device information and the original working condition image to generate a fused working condition image.

[0082] In this embodiment of the present invention, quaternions are used to represent the posture information of the image acquisition device to implement coordinate system transformation and 3D model projection. Multiple coordinate system transformations include the transformation from the world coordinate system to the camera coordinate system, the transformation from the camera coordinate system to the image coordinate system, and the transformation from the image coordinate system to the pixel coordinate system.

[0083] Step 103: Use a deep learning algorithm to perform instance segmentation on the original working condition image and extract the guard plate contour information.

[0084] In an embodiment of the present invention, the deep learning algorithm is an instance segmentation method, for example, the SOLOv2 algorithm.

[0085] Specifically, a deep learning algorithm is used to identify the position of the side guard in the original working condition image and extract the contour information of the side guard.

[0086] Step 104: Based on the shearer motion information, the guard plate status is risk monitored according to the fusion working condition image and the guard plate contour information to generate a risk monitoring result.

[0087] In an embodiment of the present invention, the shearer motion information includes the shearer position and motion direction, the fused working condition image includes safety line information, and the safety line information includes a folding safety line and an unfolding safety line.

[0088] In this embodiment of the present invention, the frame number of each side guard is determined based on the correspondence between its position in the fused working condition image and the projection of the 3D model. The folding state of the side guard is determined by comparing the spatial relationship between the side guard's outline and the folding and unfolding safety lines. The risk monitoring results are then determined by comparing the side guard's actual folding state with the preset folding state.

[0089] Step 105: If the risk monitoring result shows that the guard plate is in an abnormal state, a guard plate state warning image is generated based on the fusion of the working condition image and the guard plate contour information.

[0090] In the embodiment of the present invention, an accurate warning is issued for a side guard plate in an abnormal state, and the corresponding frame number and warning information are marked in the fusion working condition image to generate a side guard plate status warning image.

[0091] In the technical solution provided by the embodiment of the present invention, the original working condition image of the comprehensive mining working face, the constructed three-dimensional model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine motion information are obtained; the three-dimensional model information and the safety line information of the hydraulic support guard plate are converted and mapped into multiple coordinate systems through the image acquisition equipment information and the original working condition image to generate a fused working condition image; the original working condition image is instance segmented through a deep learning algorithm to extract the guard plate contour information; based on the coal mining machine motion information, the guard plate status is risk monitored according to the fused working condition image and the guard plate contour information to generate a risk monitoring result; if the risk monitoring result shows that the guard plate status is abnormal, a guard plate status warning image is generated according to the fused working condition image and the guard plate contour information, which combines three-dimensional modeling and computer vision technology to realize full-cycle status monitoring of the hydraulic support guard plate, form a closed-loop management, reduce dependence on hardware equipment, and thus reduce costs; it can adapt to complex environments, accurately locate and identify the guard plate in three-dimensional space, reduce calculation errors, and improve monitoring accuracy.

[0092] Figure 3 A flowchart of another method for monitoring and warning the status of a hydraulic support guard plate based on three-dimensional modeling is provided in an embodiment of the present invention, as shown in FIG. Figure 3 As shown, the method includes:

[0093] Step 201: Acquire the original working condition image of the fully mechanized mining face, the constructed three-dimensional model information of the hydraulic support side guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine movement information.

[0094] In the embodiment of the present invention, each step is performed by a hydraulic support guard plate state monitoring and early warning device based on three-dimensional modeling.

[0095] In this embodiment of the present invention, the image acquisition device is an industrial camera. Image acquisition device information includes, but is not limited to, posture information and device internal parameters. The three-dimensional model information of the hydraulic support guard plate includes, but is not limited to, the guard plate's three-dimensional spatial point set, the guard plate's initial position and posture, and each guard plate's unique frame number. The guard plate's unique frame number uniquely identifies the guard plate. Coal mining machine motion information includes, but is not limited to, the shearer's position and direction of motion. Safety line information includes folding and unfolding safety lines.

[0096] In the embodiment of the present invention, a world coordinate system is established with the image acquisition device as the origin, and the directions of the X-axis, Y-axis, and Z-axis are defined. Specifically, the X-axis is horizontal to the right, the Y-axis is vertically upward, and the Z-axis points in the direction of the coal mining machine.

[0097] In the embodiment of the present invention, a three-dimensional geometric model of the side guard is constructed based on actual measurement data, the initial position and posture of the side guard in the world coordinate system are defined, and a unique frame number is assigned to each side guard.

[0098] In an embodiment of the present invention, the folding safety line is defined in three-dimensional space, representing the maximum height to which the support guard plate needs to be folded when the coal mining machine passes; the unfolding safety line is defined in three-dimensional space, representing the minimum height to which the support guard plate needs to be unfolded after the coal mining machine passes.

[0099] It is worth noting that the positions and shapes of the folding safety line and the unfolding safety line are determined according to the size of the coal mining machine and the actual conditions of the working surface, and the embodiments of the present invention do not limit this.

[0100] As another alternative, in addition to using the geometric model to construct a 3D model of the hydraulic support guard plate, a depth camera can also be used for 3D reconstruction. The specific implementation is as follows:

[0101] 1. Depth map acquisition: Use a depth camera to obtain a depth map of the work surface; combine it with the RGB image to generate point cloud data with depth information.

[0102] 2. 3D model reconstruction: Based on the point cloud data, reconstruct the 3D model of the guardrail; use algorithms such as Random Sample Consensus Algorithm (RANSAC) to fit the plane and boundaries of the guardrail.

[0103] 3. Safety line information definition: Define the safety line information on the reconstructed 3D model; determine the position and shape of the safety line information based on the size of the coal mining machine and the actual situation of the working face.

[0104] Step 202: Perform multiple coordinate system conversion and mapping on the three-dimensional model information and safety line information of the hydraulic support guard plate using the image acquisition device information and the original working condition image to generate a fused working condition image.

[0105] Figure 4 A flowchart of a quaternion posture projection provided by an embodiment of the present invention is as follows: Figure 4 As shown, step 202 specifically includes:

[0106] Step 2021: Perform quaternion conversion on the posture information to generate a rotation matrix.

[0107] In the embodiment of the present invention, the posture information includes a pitch angle (Tilt) and a horizontal rotation angle (Pan). Specifically, the pitch angle and the rotation angle are converted into quaternions to derive a rotation matrix.

[0108] In the embodiment of the present invention, the quaternion representation method avoids the universal lock problem in the Euler angle representation and improves the accuracy of posture expression.

[0109] Step 2022: Convert the three-dimensional space point set and safety line information of the side guard plate from the world coordinate system to the camera coordinate system through the rotation matrix, and generate the three-dimensional space point set and safety line information of the side guard plate in the camera coordinate system.

[0110] Specifically, the three-dimensional space point set (points_4d), folding safety line (points_plate_fold) and unfolding safety line (points_plate_unfold) of the side guard are transformed from the world coordinate system to the camera coordinate system through the rotation matrix, and the three-dimensional space point set, folding safety line and unfolding safety line of the side guard are generated in the camera coordinate system.

[0111] Step 2023: Convert the three-dimensional space point set and safety line information of the side guard plate in the camera coordinate system into the three-dimensional space point set and safety line information of the side guard plate in the image coordinate system through perspective projection.

[0112] Specifically, through perspective projection, the three-dimensional spatial point set of the guard plate, the folded safety line and the unfolded safety line in the camera coordinate system are projected onto the two-dimensional image plane to generate the two-dimensional position information of the guard plate, the folded safety line and the unfolded safety line in the image coordinate system.

[0113] Step 2024: Using the device internal parameters, convert the two-dimensional position information of the side guard plate and the safety line information in the image coordinate system into the three-dimensional spatial point set of the side guard plate and the safety line information in the pixel coordinate system.

[0114] In this embodiment of the present invention, the Zhang Zhengyou calibration method is used to determine the device's internal parameters, including but not limited to focal length and principal point coordinates. Specifically, the focal length and principal point coordinates are used to convert the two-dimensional position information of the side guard, the folding safety line, and the unfolding safety line in the image coordinate system into the two-dimensional position information of the side guard, the folding safety line, and the unfolding safety line in the pixel coordinate system.

[0115] Step 2025: Map the two-dimensional position information of the side guard plate and the safety line information in the pixel coordinate system to the original working condition image for visualization processing to generate a fused working condition image.

[0116] It is worth noting that the three-dimensional spatial point set of the side guard plate, the two-dimensional position information of the side guard plate, the folding safety line and the unfolding safety line are all position information in each coordinate system.

[0117] In an embodiment of the present invention, the two-dimensional position information and safety line information of the guard plate in the pixel coordinate system are mapped by the pre-solved distortion coefficient to generate a distorted effect image of the real camera imaging; the guard plate contour where the two-dimensional position information of the guard plate is located is converted into a polygon in the image coordinate system; the guard plate contour, folded safety line and unfolded safety line are mapped and fused to the original working condition image, and the polygon filling (cv2.fillPoly) function is called to fill the internal area of ​​the polygon corresponding to the guard plate contour to achieve the guard plate visualization effect; the image fusion (cv2.addWeighted) function is called to fuse the polygon corresponding to the filled guard plate contour, folded safety line and unfolded safety line with the original working condition image to generate a fused working condition image. Since the fused working condition image is generated based on the three-dimensional model information of the hydraulic support guard plate, the three-dimensional model information of the hydraulic support guard plate defines the unique frame number of each guard plate, so the fused working condition image includes the guard plate frame number of each guard plate.

[0118] In this embodiment of the present invention, the fused working condition image provides a basis for subsequent judgment on whether the side guard is folded or unfolded. Through precise quaternion posture expression and camera distortion model, accurate mapping between three-dimensional space and two-dimensional image is achieved, providing a reliable spatial reference for subsequent side guard status judgment and collision warning.

[0119] The present invention achieves precise positioning and posture expression of the side guard plate in three-dimensional space by establishing an accurate three-dimensional space model and combining it with quaternion posture expression, thereby improving detection accuracy.

[0120] As an alternative, in addition to using quaternions to represent the camera's attitude, you can also use Euler angles. Euler angles are more intuitive and easier to understand, but they may encounter gimbal lock issues at certain angles. The specific implementation is as follows:

[0121] 1. Euler angle representation: Use pitch, yaw, and roll angles to represent the camera's posture; construct a rotation matrix to achieve coordinate system transformation.

[0122] 2. Projection transformation: Use the perspective projection matrix to project the 3D points onto the 2D image plane; consider the intrinsic and extrinsic parameters of the camera to achieve accurate projection transformation.

[0123] Step 203: perform image preprocessing on the original working condition image to generate a preprocessed original working condition image.

[0124] In the embodiment of the present invention, image preprocessing includes but is not limited to normalization, resizing and other preprocessing operations.

[0125] In the embodiment of the present invention, image preprocessing is performed on the original working condition image to improve image quality, reduce the influence of factors such as coal dust and light, and reduce the influence of environmental factors, thereby improving accuracy.

[0126] Step 204: Using a deep learning algorithm, identify the position of the side guard plates on the pre-processed original working condition image to generate initial contour information of each side guard plate.

[0127] In this embodiment of the present invention, the deep learning algorithm includes the SOLOv2 algorithm. Specifically, the preprocessed raw working condition image is input into the SOLOv2 algorithm for instance segmentation, identifying the position of the side guards in the image and outputting a pixel-level mask for each side guard, thereby achieving accurate contour extraction and obtaining the initial contour information of each side guard.

[0128] As another alternative, in addition to using deep learning instance segmentation methods, traditional image processing methods can also be used to identify the side guards. The specific implementation is as follows:

[0129] 1. Image preprocessing: Preprocess the input image, including grayscale, filtering, edge enhancement and other operations; enhance image quality and reduce the influence of factors such as coal dust and light.

[0130] 2. Edge detection: Use Canny, Sobel and other algorithms to perform edge detection and extract the contour features of the guard plate.

[0131] 3. Shape analysis: Use methods such as Hough transform to detect straight lines and rectangles; identify the position and outline of the guard plate based on its geometric characteristics.

[0132] As another alternative, in addition to using the deep learning instance segmentation method, feature point matching can also be used to identify the side guard. The specific implementation is as follows:

[0133] 1. Feature point extraction: Use algorithms such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and ORB (Oriented FAST and Rotated BRIEF) to extract feature points from the image; calculate descriptors of the feature points for subsequent matching.

[0134] 2. Feature point matching: Match the feature points in the current image with the pre-stored feature points of the guard plate template; use algorithms such as RANSAC to remove false matches.

[0135] 3. Position determination: Based on the matching results, determine the position and outline of the guard plate in the image; use homography transformation to deal with the impact of perspective changes.

[0136] Step 205: Post-process the initial contour information to generate the side guard plate contour information.

[0137] In this embodiment of the present invention, post-processing includes, but is not limited to, removing noise and outliers to generate the guard plate outline information. Post-processing this initial outline information can improve image quality, thereby increasing the accuracy of the guard plate outline and providing a foundation for subsequent guard plate frame number recognition.

[0138] Step 206: Match the side guard plate contour information with the side guard plate two-dimensional position information in the fused working condition image, and determine and mark the side guard plate frame number corresponding to each side guard plate contour information.

[0139] In this embodiment of the present invention, the x-coordinates of each pixel in the side guard profile information are averaged to obtain a mean x-coordinate value; a sorting order is determined based on the horizontal rotation angle of the camera; and according to the sorting order, each side guard is sorted based on the mean x-coordinate value to obtain an ordered side guard sequence. Specifically, if the horizontal rotation angle is less than a preset angle threshold, indicating that the camera is deflected to the left, the side guards are sorted from largest to smallest based on the mean x-coordinate value to obtain an ordered side guard sequence. If the horizontal rotation angle is greater than or equal to the preset angle threshold, indicating that the camera is facing the media wall or deflected to the right, the side guards are sorted from smallest to largest based on the mean x-coordinate value to obtain an ordered side guard sequence.

[0140] It is worth noting that the angle threshold can be set according to actual conditions, and the embodiment of the present invention does not limit this.

[0141] In an embodiment of the present invention, the boundary of each guard plate is calculated, including the minimum x-coordinate and the maximum x-coordinate. According to the ordered guard plate sequence, the minimum x-coordinate and the maximum x-coordinate of each guard plate in the guard plate contour information are matched with the minimum x-coordinate and the maximum x-coordinate of each guard plate in the guard plate two-dimensional position information. The guard plate frame number corresponding to each guard plate contour information is determined and marked.

[0142] This invention utilizes a boundary-matching-based identification method to accurately identify each guard plate identification number, resolving the existing difficulty in identification. By taking into account the camera's tilt and pan angles, the system can adapt to different viewing angles, improving its adaptability.

[0143] Furthermore, in actual production scenarios, there is a deviation between the calculated side guard plate position and the actual side guard plate position. When the deviation is relatively large or the recognition accuracy is not high, duplicate side guard plate frame numbers will appear. The frame number value can be increased or decreased to ensure the uniqueness of the frame number assignment for each side guard plate, thereby realizing abnormal handling of the frame number assignment and improving the judgment accuracy.

[0144] As another alternative, in addition to using the boundary matching method, you can also directly use the deep learning method to identify the frame number. The specific implementation is as follows:

[0145] 1. Dataset construction: Collect images of guard plates with different frame numbers; annotate the images and mark the frame number corresponding to each guard plate.

[0146] 2. Model training: Use deep learning models such as convolutional neural networks (CNN) and long short-term memory networks (LSTM) for training; optimize model parameters to improve the accuracy of license plate recognition.

[0147] 3. Frame number recognition: Use the trained model to directly identify the frame numbers of the guard plates in the image and output the frame number information corresponding to each guard plate.

[0148] As another alternative, in addition to using the boundary matching method, you can also perform frame number recognition based on the position of the guard plate in space. The specific implementation is as follows:

[0149] 1. Spatial position calculation: Based on the position of the guard plate in the image and the camera parameters, calculate the position of the guard plate in three-dimensional space; establish a mapping relationship between the spatial position and the frame number.

[0150] 2. Frame number allocation: Determine the corresponding frame number based on the position of the guard plate in space; consider the actual layout of the work surface and handle special situations.

[0151] Step 207 : According to the folding safety line and the unfolding safety line, a folding auxiliary judgment area and an unfolding auxiliary judgment area are respectively determined in the fused working condition image.

[0152] In the embodiment of the present invention, a folding auxiliary judgment area and an unfolding auxiliary judgment area are defined in the fused working condition image according to the folding safety line and the unfolding safety line projected by the three-dimensional model.

[0153] Specifically, the area enclosed by the folding safety line and the lower bottom edge of the fused working condition image is determined as the folding auxiliary judgment area, and the folding auxiliary judgment area indicates the area where the guard plate should not be located after being folded; the area enclosed by the unfolding safety line and the lower bottom edge of the fused working condition image is determined as the unfolding auxiliary judgment area, and the unfolding auxiliary judgment area indicates the area where the lower bottom edge of the guard plate should be located after the guard plate is unfolded.

[0154] Step 208: According to the preset support range, the position and movement direction of the coal mining machine, the guard plate area to be detected is divided in the fused working condition image.

[0155] In an embodiment of the present invention, the support range is a range of a specified number of hydraulic support side guard plates from the coal shearer position along the direction of movement of the coal shearer, and a range of a specified number of hydraulic support side guard plates from the coal shearer position along the opposite direction of the coal shearer movement direction.

[0156] Specifically, the fused working condition image is divided according to the range of the bracket, and the divided area is the guard plate area to be detected.

[0157] It is worth noting that the support range can be set according to actual needs and is not limited in this embodiment of the present invention. As an optional solution, the support range is the range of seven hydraulic support panels from the coal shearer position in the direction of coal shearer movement, and the range of seven hydraulic support panels from the coal shearer position in the opposite direction of coal shearer movement.

[0158] Step 209: Based on the side guard plate contour information, the folding auxiliary judgment area, and the unfolding auxiliary judgment area, risk monitoring is performed on the status of the side guard plate within the side guard plate area to be detected, and a risk monitoring result is generated. If the risk monitoring result shows that the side guard plate is abnormal, step 210 is executed; if the risk monitoring result shows that the side guard plate is normal, the process ends.

[0159] In the embodiment of the present invention, the guard plate area to be detected includes a forward direction area and a reverse direction area, and the boundary between the forward direction area and the reverse direction area is the position of the coal mining machine.

[0160] In the embodiment of the present invention, step 209 specifically includes:

[0161] Step 2091: Determine whether the overlapping area between the side guard plate contour information in the forward direction area and the folding auxiliary judgment area is greater than a preset folding threshold. If so, execute step 2092; if not, execute step 2093.

[0162] In the embodiment of the present invention, by S 1overlap =S plate ∩S fold_linr , calculate the overlapping area between the guard plate contour information in the motion direction area and the folding auxiliary judgment area. 1overlap is the overlapping area, S plate The guard plate area indicated by the guard plate outline information, S fold_line This is the folding auxiliary judgment area.

[0163] In the embodiment of the present invention, the preset folding threshold can be set according to actual needs, and the embodiment of the present invention does not limit this. As an optional solution, the overlapping area can be the number of pixels, and the folding threshold can be 200 pixels.

[0164] In an embodiment of the present invention, if the overlapping area is greater than the folding threshold, it indicates that there is still overlap between the side guard plate and the folding auxiliary judgment area after folding, and step 2092 is continued; if the overlapping area is less than or equal to the folding threshold, it indicates that there is no overlap between the side guard plate and the folding auxiliary judgment area after folding, and step 2093 is continued.

[0165] Step 2092: Generate a risk monitoring result of abnormal guard plate status, and proceed to step 210.

[0166] In the embodiment of the present invention, if the side guard plate still overlaps with the auxiliary folding judgment area after being folded, it indicates that the side guard plate is not safely folded and is in an abnormal state, and step 210 is continued.

[0167] Step 2093: Generate a risk monitoring result indicating that the guard plate is in a normal state, and the process ends.

[0168] In the embodiment of the present invention, if there is no overlap between the folded side guard and the auxiliary folding judgment area, it indicates that the side guard has been safely folded and is in normal condition, and the status of the next side guard is determined; if there is no next side guard, the process ends.

[0169] Step 2094: Determine whether the overlapping area between the side guard plate contour information in the reverse direction area and the deployment auxiliary determination area is greater than a preset deployment threshold. If so, execute step 2095; if not, execute step 2096.

[0170] In the embodiment of the present invention, by S 2overlap =S plate ∩S unfold_line , calculate the overlapping area between the guard plate contour information in the opposite direction of the movement direction and the expansion auxiliary judgment area. 2overlap is the overlapping area, S plate The guard plate area indicated by the guard plate outline information, S unfold_line To expand the auxiliary judgment area.

[0171] In the embodiment of the present invention, the preset expansion threshold can be set according to actual needs, and the embodiment of the present invention does not limit this. As an optional solution, the expansion threshold is 10 pixels.

[0172] In this embodiment of the present invention, if the overlapping area is greater than the expansion threshold, it indicates that there is overlap between the side guard plate after expansion and the expansion auxiliary judgment area, and step 2095 is continued to be executed; if the overlapping area is less than or equal to the expansion threshold, it indicates that there is no overlap between the side guard plate after expansion and the expansion auxiliary judgment area, and step 2096 is continued to be executed.

[0173] As another alternative, in addition to using the overlap analysis method, the folding state of the side guard can also be determined based on angle measurement. The specific implementation is as follows:

[0174] 1. Angle extraction: Extract the main axis direction of the guard plate; calculate the angle between the guard plate and the horizontal or vertical plane.

[0175] 2. Status judgment: Set an angle threshold. If the angle is less than the threshold, it is judged to be in a folded state; if the angle is greater than the threshold, it is judged to be in an unfolded state.

[0176] As another alternative, in addition to using the overlap analysis method, we can also directly use deep learning methods to determine the folding state of the side guard. The specific implementation is as follows:

[0177] 1. Dataset construction: Collect images of side guards in different folding states; annotate the images and mark the folding state of each side guard.

[0178] 2. Model training: Use deep learning models such as CNN for training; optimize model parameters to improve state judgment accuracy.

[0179] 3. Status judgment: Use the trained model to directly judge the folding status of the side guards and output the folding status information of each side guard.

[0180] Step 2095: Generate a risk monitoring result indicating that the guard plate is in a normal state, and the process ends.

[0181] In the embodiment of the present invention, after the side guard plate is deployed, there is an overlap with the deployment auxiliary judgment area, indicating that the side guard plate has been safely deployed and is in a normal state, and the process ends.

[0182] Step 2096: Generate a risk monitoring result of abnormal guard plate status, and continue to step 210.

[0183] In the embodiment of the present invention, if there is no overlap between the side guard plate after deployment and the deployment auxiliary judgment area, it indicates that the side guard plate is not deployed safely, the state is abnormal, and the process ends.

[0184] The present invention automatically monitors the folding and unfolding states of the side guards in real time during the movement of the coal mining machine, promptly discovers side guards in abnormal states, and accurately identifies side guard area, folding auxiliary judgment area and unfolding auxiliary judgment area through computer vision technology, thereby improving monitoring accuracy, real-timeness and automation. Without manual intervention, the safety of operations is effectively guaranteed, forming a complete side guard plate status monitoring system, which improves the safety and intelligence level of the fully mechanized mining working face. By optimizing algorithms and models, the system can realize real-time monitoring and early warning to meet the needs of intelligent production in coal mines.

[0185] The present invention improves the safety of coal mining machine operations by accurately detecting the folding / unfolding status of the side guard plate, timely discovering potential collision risks, and comparing the spatial relationship between the side guard plate contour and the projected safety line to accurately judge the folding status of the side guard plate, and issuing precise early warnings for unfolded side guard plates, thereby improving the accuracy and effectiveness of the early warnings.

[0186] Step 210: Determine the center point of the side guard plate according to the side guard plate contour information corresponding to the side guard plate in the abnormal state.

[0187] As an optional solution, the center point of the guard plate can be determined by calculating the average x-coordinate and the average y-coordinate of each point in the guard plate outline information. in, is the average value of the x-coordinate, is the average value of the y coordinate.

[0188] In the embodiment of the present invention, by calculating the center point of the side guard plate contour, it is ensured that the display position of the subsequent side guard plate status warning information matches the corresponding side guard plate.

[0189] Step 211: Generate guard plate status warning information according to the guard plate frame number corresponding to the abnormal guard plate and the preset warning text.

[0190] Specifically, for a guard plate in an abnormal state, a guard plate status warning message including the guard plate frame number and warning text is generated. For example, if the guard plate frame number is "#41" and the warning text is "Warn!!", the guard plate status warning message will be "#41Warn!!".

[0191] Step 212: According to a preset warning annotation format, guard plate status warning information is added to the guard plate center point corresponding to the guard plate with abnormal status in the fused working condition image to generate a guard plate status warning image.

[0192] In the embodiment of the present invention, the warning label format can be set according to actual needs and is not limited in this embodiment of the present invention. As an optional solution, red is used in the fused working condition image to highlight the guardrails with abnormal status according to their contour information. The monitoring results are intuitively displayed in the image, allowing personnel to quickly locate the position of the abnormal guardrail for timely inspection.

[0193] Furthermore, green can be used in the fusion working condition image to highlight the guard plates in normal condition according to the guard plate contour information, and the monitoring results can be intuitively displayed in the image, so that the staff can check the position of the normal guard plates.

[0194] Specifically, guard plate status warning information is added to the guard plate center point corresponding to the guard plate with abnormal status in the fusion working condition image, and the guard plate with abnormal status is highlighted to generate a guard plate status warning image.

[0195] The present invention achieves an intuitive visualization effect by marking the unfolded side guard in the image and adding warning information including the frame number, which makes it easier for operators to quickly locate the problem.

[0196] Furthermore, the guard plate status warning image is saved as a result file, and the guard plate status warning information is output to the system for real-time monitoring.

[0197] This invention provides face operators with intuitive warnings of the side guard plate status through clear visual indicators and text messages, effectively improving the safety and intelligence of fully mechanized mining faces. The system dynamically adjusts its warning strategy based on the shearer's operating status, ensuring that the side guard plates are promptly folded during the shearer's advance and deployed after the shearer passes, achieving full-cycle monitoring.

[0198] Furthermore, the modeling parameters of the three-dimensional model information of the hydraulic support guard plate are determined by adaptive iterative optimization of parameters based on a deep learning algorithm. The modeling parameters can be adaptively optimized at preset time intervals, thereby improving the accuracy of the modeling. Specifically, before establishing a three-dimensional model, it is necessary to determine the actual values ​​of some key points in space, such as the vertical height of the camera from the ground, the camera installation position, the scraper conveyor position, the width of the support, the spacing between the supports, and other key modeling parameters. In each fully mechanized mining working face, parameters such as the width of the support, the spacing between the supports, and the width of the scraper conveyor are all fixed values ​​and can be directly obtained. However, parameters such as the vertical height of the camera from the ground, the angle between the camera and the horizontal plane, the horizontal distance from the camera to the coal wall, and the horizontal distance from the scraper conveyor to the coal wall are not fixed values. They change with the change of the moving frame, or errors may occur due to the different positions where workers install the camera.

[0199] The present invention first constructs a model with initialized modeling parameters, and then uses an adaptive method to solve the modeling parameters that need to be solved. Specifically, the trained SOLOv2 instance segmentation method is used to identify the image; the position of the scraper conveyor is calculated using a method based on three-dimensional modeling; the difference between the scraper conveyor and the support guard plate identified by the SOLOv2 instance segmentation algorithm and the one calculated based on the three-dimensional modeling principle is calculated as the loss, with white representing the area where the calculation is correct and black representing the area where the calculation is wrong; the constrained minimization method of the multivariate scalar function is used to solve the modeling parameters such as the vertical height of the camera from the ground, the angle between the camera and the horizontal plane, the horizontal distance from the camera to the coal wall, and the horizontal distance from the scraper conveyor to the coal wall that minimize the loss, so as to achieve adaptive optimization of the modeling parameters.

[0200] Through the above steps, this method integrates deep learning, computer vision and 3D modeling technologies, has high adaptability and accuracy, and provides an innovative solution for improving the level of mine safety monitoring.

[0201] It is worth noting that the acquisition, storage, use, and processing of data in the technical solutions of this application are in compliance with the relevant provisions of laws and regulations. The user information in the embodiments of this application is obtained through legal and compliant channels, and the acquisition, storage, use, and processing of user information are authorized and agreed by the customer.

[0202] It is worth noting that the information collected in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good customs, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0203] It is worth noting that the technical solution provided in this application provides users with corresponding operation entrances for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered.

[0204] In the technical solution of the hydraulic support guard plate status monitoring and early warning method based on three-dimensional modeling provided by the embodiment of the present invention, the original working condition image of the fully mechanized mining face, the constructed three-dimensional model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine motion information are obtained; the three-dimensional model information and the safety line information of the hydraulic support guard plate are converted and mapped into multiple coordinate systems through the image acquisition equipment information and the original working condition image to generate a fused working condition image; the original working condition image is instance segmented through a deep learning algorithm to extract the guard plate contour information; based on the coal mining machine motion information, the guard plate status is risk monitored according to the fused working condition image and the guard plate contour information to generate a risk monitoring result; if the risk monitoring result shows that the guard plate status is abnormal, a guard plate status early warning image is generated according to the fused working condition image and the guard plate contour information. By combining three-dimensional modeling and computer vision technology, full-cycle status monitoring of the hydraulic support guard plate is realized, closed-loop management is formed, and dependence on hardware equipment is reduced, thereby reducing costs; the method can adapt to complex environments, accurately locate and identify the guard plate in three-dimensional space, reduce calculation errors, and improve monitoring accuracy.

[0205] Figure 5 A schematic diagram of a hydraulic support guard plate state monitoring and early warning device based on three-dimensional modeling provided by an embodiment of the present invention, wherein the device is used to execute the above-mentioned hydraulic support guard plate state monitoring and early warning method based on three-dimensional modeling, such as Figure 5 As shown, the device includes: an acquisition unit 11, a coordinate system conversion unit 12, a deep learning unit 13, a risk monitoring unit 14 and a status warning unit 15.

[0206] The acquisition unit 11 is used to acquire the original working condition image of the fully mechanized mining face, the constructed three-dimensional model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine movement information.

[0207] The coordinate system conversion unit 12 is used to perform multiple coordinate system conversion and mapping on the three-dimensional model information and safety line information of the hydraulic support guard plate through the image acquisition device information and the original working condition image, and generate a fused working condition image.

[0208] The deep learning unit 13 is used to perform instance segmentation on the original working condition image through a deep learning algorithm to extract the guard plate contour information.

[0209] The risk monitoring unit 14 is used to perform risk monitoring on the guard plate status based on the coal mining machine movement information and the fusion working condition image and the guard plate contour information to generate a risk monitoring result.

[0210] The status warning unit 15 is used to generate a guard plate status warning image based on the fusion of the working condition image and the guard plate contour information if the risk monitoring result shows that the guard plate status is abnormal.

[0211] In an embodiment of the present invention, the image acquisition device information includes posture information and device internal parameters, and the three-dimensional model information of the hydraulic support guard plate includes a three-dimensional space point set of the guard plate; the coordinate system conversion unit 12 includes: a quaternion conversion module 121, a camera coordinate system conversion module 122, an image coordinate system conversion module 123, a pixel coordinate system conversion module 124 and a visualization processing module 125.

[0212] The quaternion conversion module 121 is used to perform quaternion conversion on the posture information to generate a rotation matrix.

[0213] The camera coordinate system conversion module 122 is used to convert the guard plate 3D space point set and safety line information from the world coordinate system to the camera coordinate system through the rotation matrix, and generate the guard plate 3D space point set and safety line information in the camera coordinate system.

[0214] The image coordinate system conversion module 123 is used to convert the three-dimensional space point set of the side guard plate and the safety line information in the camera coordinate system into the two-dimensional position information of the side guard plate and the safety line information in the image coordinate system through perspective projection.

[0215] The pixel coordinate system conversion module 124 is used to convert the two-dimensional position information of the side guard plate and the safety line information in the image coordinate system into the two-dimensional position information of the side guard plate and the safety line information in the pixel coordinate system through the device internal parameters.

[0216] The visualization processing module 125 is used to map the two-dimensional position information of the side guard plate and the safety line information in the pixel coordinate system to the original working condition image for visualization processing to generate a fused working condition image.

[0217] In the embodiment of the present invention, the deep learning unit 13 includes: an image pre-processing module 131 , a position recognition module 132 and a post-processing module 133 .

[0218] The image preprocessing module 131 is used to perform image preprocessing on the original working condition image to generate a preprocessed original working condition image.

[0219] The position recognition module 132 is used to identify the position of the side guard plates on the pre-processed original working condition image through a deep learning algorithm, and generate the initial contour information of each side guard plate.

[0220] The post-processing module 133 is used to perform post-processing on the initial contour information to generate the side guard plate contour information.

[0221] In an embodiment of the present invention, the coal mining machine motion information includes the position and motion direction of the coal mining machine, the fused working condition image includes safety line information, and the safety line information includes a folding safety line and an unfolding safety line; the risk monitoring unit 14 includes: an auxiliary area determination module 141, a division module 142 and a status monitoring module 143.

[0222] The auxiliary region determination module 141 is used to determine a folding auxiliary judgment region and an unfolding auxiliary judgment region in the fused working condition image according to the folding safety line and the unfolding safety line.

[0223] The division module 142 is used to divide the guard plate area to be detected in the fused working condition image according to the preset support range, the position and the movement direction of the coal mining machine.

[0224] The state monitoring module 143 is used to perform risk monitoring on the state of the side guard plate in the side guard plate area to be inspected based on the side guard plate contour information, the folding auxiliary judgment area and the unfolding auxiliary judgment area, and generate a risk monitoring result.

[0225] In an embodiment of the present invention, the guard plate area to be detected includes a forward direction area and a reverse direction area; the status monitoring module 143 is specifically used to determine whether the overlapping area between the guard plate contour information in the forward direction area and the folding auxiliary judgment area is greater than a preset folding threshold; if so, generate a risk monitoring result of abnormal guard plate status; if not, generate a risk monitoring result of normal guard plate status; determine whether the overlapping area between the guard plate contour information in the reverse direction area and the expansion auxiliary judgment area is greater than a preset expansion threshold; if so, generate a risk monitoring result of normal guard plate status; if not, generate a risk monitoring result of abnormal guard plate status.

[0226] In the embodiment of the present invention, the fused working condition image includes the side guard frame number of each side guard; the device further includes: a frame number recognition unit 16.

[0227] The frame number recognition unit 16 is used to match the side guard plate contour information with the side guard plate two-dimensional position information in the fused working condition image, and determine and mark the side guard plate frame number corresponding to each side guard plate contour information.

[0228] In the embodiment of the present invention, the status warning unit 15 includes: a center point determination module 151 , a warning information generation module 152 , and a warning image generation module 153 .

[0229] The center point determination module 151 is used to determine the center point of the side guard plate according to the side guard plate contour information corresponding to the side guard plate in an abnormal state.

[0230] The warning information generating module 152 is used to generate guard plate status warning information according to the guard plate frame number corresponding to the guard plate in abnormal status and the preset warning text.

[0231] The warning image generation module 153 is used to add guard plate status warning information to the guard plate center point corresponding to the guard plate with abnormal status in the fusion working condition image according to the preset warning annotation format, and generate a guard plate status warning image.

[0232] In the scheme of the embodiment of the present invention, the original working condition image of the comprehensive mining working face, the constructed three-dimensional model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine motion information are obtained; the three-dimensional model information and the safety line information of the hydraulic support guard plate are converted and mapped into multiple coordinate systems through the image acquisition equipment information and the original working condition image to generate a fused working condition image; the original working condition image is instance segmented through a deep learning algorithm to extract the guard plate contour information; based on the coal mining machine motion information, the guard plate status is risk monitored according to the fused working condition image and the guard plate contour information to generate a risk monitoring result; if the risk monitoring result shows that the guard plate status is abnormal, a guard plate status warning image is generated according to the fused working condition image and the guard plate contour information, which combines three-dimensional modeling and computer vision technology to realize full-cycle status monitoring of the hydraulic support guard plate, form a closed-loop management, reduce dependence on hardware equipment, and thus reduce costs; it can adapt to complex environments, accurately locate and identify the guard plate in three-dimensional space, reduce calculation errors, and improve monitoring accuracy.

[0233] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer device. Specifically, the computer device may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0234] An embodiment of the present invention provides a computer device, including a memory and a processor, the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, the steps of the embodiment of the above-mentioned hydraulic support guard plate status monitoring and early warning method based on three-dimensional modeling are implemented. For a specific description, please refer to the embodiment of the above-mentioned hydraulic support guard plate status monitoring and early warning method based on three-dimensional modeling.

[0235] Reference below Figure 6 , which shows a structural diagram of a computer device 600 suitable for implementing an embodiment of the present application.

[0236] like Figure 6 As shown, the computer device 600 includes a central processing unit (CPU) 601, which can perform various appropriate tasks and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the computer device 600 are also stored in the RAM 603. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0237] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including devices such as a hard disk; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that computer programs read therefrom can be installed in the storage section 608 as needed.

[0238] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program including program code for executing the methods illustrated in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication portion 609 and / or installed from removable media 611.

[0239] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0240] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0241] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0242] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0243] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0244] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0245] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0246] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0247] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0248] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0249] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0250] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for monitoring and warning the condition of a hydraulic support guard plate based on three-dimensional modeling, characterized in that: The method comprises: Obtain the original working condition image of the fully mechanized mining face, the constructed 3D model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the shearer movement information; Performing multiple coordinate system conversion and mapping on the three-dimensional model information and safety line information of the hydraulic support guard plate through the image acquisition device information and the original working condition image to generate a fused working condition image; By using a deep learning algorithm, instance segmentation is performed on the original working condition image to extract the guard plate contour information; Based on the shearer motion information, risk monitoring of the guard plate state is performed according to the fused working condition image and the guard plate contour information to generate a risk monitoring result; If the risk monitoring result shows that the guard plate is in an abnormal state, a guard plate state warning image is generated based on the fused working condition image and the guard plate contour information.

2. The method for monitoring and warning the condition of the hydraulic support guard plate based on three-dimensional modeling according to claim 1 is characterized in that: The image acquisition device information includes posture information and device internal parameters, and the three-dimensional model information of the hydraulic support guard plate includes a three-dimensional space point set of the guard plate; The method of performing multiple coordinate system conversion and mapping on the three-dimensional model information and safety line information of the hydraulic support guard plate by using the image acquisition device information and the original working condition image to generate a fused working condition image includes: Performing quaternion conversion on the posture information to generate a rotation matrix; The three-dimensional space point set and safety line information of the side guard plate are converted from the world coordinate system to the camera coordinate system by using the rotation matrix, thereby generating the three-dimensional space point set and safety line information of the side guard plate in the camera coordinate system; Through perspective projection, the three-dimensional space point set and safety line information of the guard plate in the camera coordinate system are converted into the two-dimensional position information and safety line information of the guard plate in the image coordinate system; Through the internal parameters of the device, the two-dimensional position information of the guard plate and the safety line information in the image coordinate system are converted into the three-dimensional spatial point set and safety line information of the guard plate in the pixel coordinate system; The two-dimensional position information of the side guard plate and the safety line information in the pixel coordinate system are mapped to the original working condition image for visualization processing to generate a fused working condition image.

3. The method for monitoring and warning the condition of the hydraulic support guard plate based on three-dimensional modeling according to claim 1 is characterized in that: The method of performing instance segmentation on the original working condition image and extracting the guard plate contour information by using a deep learning algorithm includes: Performing image preprocessing on the original working condition image to generate a preprocessed original working condition image; Through deep learning algorithms, the guard plate positions are identified on the pre-processed original working condition images to generate the initial contour information of each guard plate; The initial contour information is post-processed to generate the side guard plate contour information.

4. The method for monitoring and warning the condition of the hydraulic support guard plate based on three-dimensional modeling according to claim 1 is characterized in that: The shearer motion information includes the shearer position and motion direction, and the fused working condition image includes safety line information, which includes a folding safety line and an unfolding safety line; The step of performing risk monitoring on the state of the side guard plate based on the shearer motion information and the fused working condition image and the side guard plate contour information to generate a risk monitoring result includes: According to the folding safety line and the unfolding safety line, respectively determining a folding auxiliary judgment area and an unfolding auxiliary judgment area in the fused working condition image; According to the preset support range, the shearer position and movement direction are used to divide the guard plate area to be detected in the fused working condition image; According to the side guard plate contour information, the folding auxiliary judgment area and the unfolding auxiliary judgment area, risk monitoring is performed on the state of the side guard plate in the side guard plate area to be detected to generate a risk monitoring result.

5. The method for monitoring and warning the condition of the hydraulic support guard plate based on three-dimensional modeling according to claim 4 is characterized in that: The guard plate area to be detected includes a forward direction area and a reverse direction area; The step of performing risk monitoring on the state of the side guard plate within the side guard plate area to be detected based on the side guard plate contour information, the folding auxiliary judgment area, and the unfolding auxiliary judgment area to generate a risk monitoring result includes: Determining whether an overlapping area between the side guard plate contour information in the forward direction area and the folding auxiliary determination area is greater than a preset folding threshold; If so, generate a risk monitoring result indicating that the guard plate is in an abnormal state; If not, generate a risk monitoring result that the guard plate is in a normal state; Determining whether an overlapping area between the side guard plate contour information in the reverse direction area and the deployment auxiliary judgment area is greater than a preset deployment threshold; If so, generate a risk monitoring result indicating that the guard plate is in a normal state; If not, a risk monitoring result indicating that the guard plate is in an abnormal state is generated.

6. The method for monitoring and warning the condition of the hydraulic support guard plate based on three-dimensional modeling according to claim 1 is characterized in that: The fused working condition image includes the guard plate frame number of each guard plate; The method further comprises: The side guard plate contour information is matched with the side guard plate two-dimensional position information in the fused working condition image, and the side guard plate frame number corresponding to each side guard plate contour information is determined and marked.

7. The method for monitoring and warning the condition of the hydraulic support guard plate based on three-dimensional modeling according to claim 6 is characterized in that: Generating a guard plate status warning image according to the fused working condition image and the guard plate contour information includes: Determine the center point of the guard plate according to the guard plate contour information corresponding to the abnormal state guard plate; Generate guard board status warning information based on the guard board frame number corresponding to the abnormal guard board and the preset warning text; According to a preset warning annotation format, the guard plate status warning information is added to the guard plate center point corresponding to the guard plate with an abnormal status in the fused working condition image to generate a guard plate status warning image.

8. The method for monitoring and warning the condition of the hydraulic support guard plate based on three-dimensional modeling according to claim 1 is characterized in that: The modeling parameters of the three-dimensional model information of the hydraulic support guard plate are determined by performing parameter adaptive iterative optimization based on a deep learning algorithm.

9. A hydraulic support guard plate state monitoring and early warning device based on three-dimensional modeling, characterized in that: The device comprises: An acquisition unit is used to acquire the original working condition image of the fully mechanized mining face, the constructed three-dimensional model information of the hydraulic support guard plate, the defined safety line information, the image acquisition equipment information and the coal mining machine movement information; A coordinate system conversion unit is used to perform multiple coordinate system conversion and mapping on the three-dimensional model information and safety line information of the hydraulic support guard plate through the image acquisition device information and the original working condition image, so as to generate a fused working condition image; A deep learning unit, configured to perform instance segmentation on the original working condition image and extract the guard plate contour information using a deep learning algorithm; a risk monitoring unit, configured to perform risk monitoring on the state of the side guard plate based on the shearer motion information, the fused working condition image and the side guard plate contour information, and generate a risk monitoring result; A status warning unit is used to generate a guard plate status warning image based on the fused working condition image and guard plate contour information if the risk monitoring result shows that the guard plate status is abnormal.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for monitoring and warning the condition of the hydraulic support guard plate based on three-dimensional modeling as described in any one of claims 1 to 8 is implemented.

11. A computer device comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, characterized in that: When the program instructions are loaded and executed by the processor, the hydraulic support guard plate status monitoring and early warning method based on three-dimensional modeling as described in any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the method for monitoring and warning the condition of the hydraulic support guard plate based on three-dimensional modeling as described in any one of claims 1 to 8 is implemented.

Images