Method and device for determining dangerous areas of fully mechanized mining face based on three-dimensional modeling

Abstract

The present invention provides a method and device for determining a dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling, the method comprising: obtaining the intrinsic parameters and distortion parameters of a camera in the fully mechanized mining face; obtaining the three-dimensional modeling parameters of the fully mechanized mining face; establishing a world coordinate system with the center of the camera as the origin; constructing a three-dimensional model of the fully mechanized mining face based on the three-dimensional modeling parameters; obtaining the external parameters of the camera, and converting the world coordinate system into the camera coordinate system based on the external parameters; converting the camera coordinate system into the image coordinate system using the intrinsic parameters of the camera; converting the image coordinate system into the pixel coordinate system, and performing distortion correction based on the distortion parameters; mapping the fully mechanized mining face constructed based on the three-dimensional modeling to the camera screen after distortion correction; and determining the pixel coordinates of the dangerous area in the mapped camera screen. The present invention can not only accurately identify the dangerous area, but also has a small amount of calculation. It also adopts an automatic parameter calculation method, which can automatically adapt to various new fully mechanized mining face environments.

Description

Technical Field

[0001] The present invention relates to the technical field of coal mining machine dangerous area identification, and in particular to a method and device for determining a dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling. Background Art

[0002] The fully mechanized coal mining face is the core area of ​​coal mining, and its safe production is of vital importance. With the advancement of intelligent mine construction, the use of computer vision technology to realize automatic identification and early warning of dangerous areas in the working face has become an important research direction. At present, the identification of dangerous areas in coal mines is not mature enough, or it cannot be done in real time, dynamically, and intelligently. The amount of calculation is too large and it is difficult to adapt to new and untrained working face environments.

[0003] For example, in order to solve the problem of potential safety hazards caused by people entering dangerous areas when the shearer is cutting coal, the existing technology has designed a dangerous area alarm method for shearers when cutting coal. The dangerous area is determined by the manual preset area by the staff and the moving direction of the shearer. When the shearer is cutting coal, once it is confirmed that there are people working in the dangerous area, an alarm is sent to prevent the workers in the head area from being harmed.

[0004] However, this technology relies on the method of presetting areas to identify dangerous areas. When using this method, it is not possible to identify dangerous areas in real time, dynamically, and intelligently. For example, for different scenes, the dangerous areas need to be reset, and the setting of dangerous areas depends on human subjective judgment; and when the camera rotates or moves, the original dangerous areas will become invalid.

[0005] For example, in order to identify the dangerous areas of tower cranes and ensure the safety of operators, the yoloV5S model was used, and a series of improvements were made to the yoloV5S model: 1) the attention module was used; 2) a small target detection layer was added; 3) the bounding box loss function was changed. The improved yoloV5S algorithm was used to detect and locate the hook in the dynamic scene of tower crane lifting and transportation, and the location and height of the hook were combined to predict the falling point range, delineate the dangerous area, improve the safety factor under the tower crane, and reduce the accident rate.

[0006] However, this method has the following defects:

[0007] 1. Based on the recognition result of yoloV5S, the recognition result is a rectangular area, which cannot divide the dangerous area well.

[0008] 2. The identification results need to be calculated to obtain the final required dangerous area, and new errors are inevitably introduced during the calculation.

[0009] 3. The model has limited generalization ability and is difficult to adapt to new, untrained work surface environments.

[0010] 4. Lack of explainability, unable to provide specific basis for determining dangerous areas.

[0011] 5. The computation is large, the hardware requirements are high, and the real-time performance is poor. Summary of the invention

[0012] In view of this, the present invention provides a method and device for determining a dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling to solve at least one of the problems mentioned above.

[0013] In order to achieve the above object, the present invention adopts the following scheme:

[0014] According to a first aspect of the present invention, a method for determining a dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling is provided, the method comprising:

[0015] Obtain the intrinsic parameters and distortion parameters of the camera in the fully mechanized mining face;

[0016] Acquiring three-dimensional modeling parameters of the fully mechanized mining working face;

[0017] Constructing a three-dimensional model of the fully mechanized mining working face based on the three-dimensional modeling parameters;

[0018] Establishing a world coordinate system with the camera center as the origin;

[0019] Acquire the external parameters of the camera, and convert the world coordinate system into a camera coordinate system based on the external parameters;

[0020] Converting the camera coordinate system into an image coordinate system using the intrinsic parameters of the camera;

[0021] Converting the image coordinate system into a pixel coordinate system, and performing distortion correction based on the distortion parameters;

[0022] Map the fully mechanized mining face constructed based on 3D modeling to the camera image after distortion correction;

[0023] The pixel coordinates of the dangerous area are determined in the mapped camera image.

[0024] As an embodiment of the present invention, the intrinsic parameters of the camera in the above method are an intrinsic parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the distortion parameters include radial distortion coefficients and tangential distortion coefficients.

[0025] As an embodiment of the present invention, the above method for obtaining the intrinsic parameters and distortion parameters of the camera in the comprehensive mining working face includes: using Zhang Zhengyou calibration method to calibrate the intrinsic parameters and distortion parameters of the camera to obtain the intrinsic parameters and distortion parameters of the camera in the comprehensive mining working face.

[0026] As an embodiment of the present invention, the three-dimensional modeling parameters in the above method include fixed parameters and variable parameters, the fixed parameters include: support width, support spacing and scraper conveyor width; the variable parameters include: the vertical height of the camera from the ground, the angle between the camera and the horizontal plane, the horizontal distance from the camera to the coal wall and the horizontal distance from the scraper conveyor to the coal wall.

[0027] As an embodiment of the present invention, the above method also includes: adaptively solving the modeling parameters of the three-dimensional model of the comprehensive mining working face during coal mining machine operation based on the DeepLabV3+ semantic segmentation algorithm, and updating the three-dimensional model according to the solved modeling parameters.

[0028] As an embodiment of the present invention, the method adaptively solves the modeling parameters of the three-dimensional model of the fully mechanized working face in the coal mining machine operation based on the DeepLabV3+ semantic segmentation algorithm, and updates the three-dimensional model according to the solved modeling parameters, including:

[0029] Use the trained DeepLabV3+ semantic segmentation algorithm to identify the image;

[0030] The position of the scraper conveyor is calculated using a 3D modeling-based approach;

[0031] Calculate the difference between the positions of the scraper conveyor and the support guard plate identified by the DeepLabV3+ semantic segmentation algorithm and the positions of the scraper conveyor and the support guard plate calculated based on the three-dimensional modeling principle as the value of the loss function; use the constrained minimization method of multivariate scalar function to solve the variable parameters that minimize the loss function;

[0032] The three-dimensional model is updated using the fixed parameters and the solved variable parameters.

[0033] According to a second aspect of the present invention, a device for determining a dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling is provided, the device comprising:

[0034] A camera parameter acquisition unit is used to obtain the internal parameters and distortion parameters of the camera in the fully mechanized mining working face;

[0035] A modeling parameter acquisition unit, used to acquire the three-dimensional modeling parameters of the fully mechanized mining working face;

[0036] A three-dimensional modeling unit, used for constructing a three-dimensional model of the fully mechanized mining working face based on the three-dimensional modeling parameters;

[0037] A world coordinate establishing unit, used for establishing a world coordinate system with the camera center as the origin;

[0038] A camera coordinate conversion unit, used for acquiring the external parameters of the camera, and converting the world coordinate system into a camera coordinate system based on the external parameters;

[0039] An image coordinate conversion unit, used to convert the camera coordinate system into an image coordinate system using the intrinsic parameters of the camera;

[0040] A pixel coordinate conversion unit, used for converting the image coordinate system into a pixel coordinate system, and performing distortion correction based on the distortion parameters;

[0041] A picture mapping unit is used to map the fully mechanized mining face constructed based on three-dimensional modeling to a camera picture after distortion correction;

[0042] The danger area determination unit is used to determine the pixel coordinates of the danger area in the mapped camera image.

[0043] As an embodiment of the present invention, the internal parameters of the above-mentioned camera are an internal parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the distortion parameters include radial distortion coefficients and tangential distortion coefficients.

[0044] As an embodiment of the present invention, the above-mentioned camera parameter acquisition unit is specifically used to: use Zhang Zhengyou calibration method to calibrate the intrinsic parameters and distortion parameters of the camera to obtain the intrinsic parameters and distortion parameters of the camera in the comprehensive mining working face.

[0045] As an embodiment of the present invention, the above-mentioned three-dimensional modeling parameters include fixed parameters and variable parameters. The fixed parameters include: bracket width, bracket spacing and scraper conveyor width; the variable parameters include: the vertical height of the camera from the ground, the angle between the camera and the horizontal plane, the horizontal distance from the camera to the coal wall and the horizontal distance from the scraper conveyor to the coal wall.

[0046] As an embodiment of the present invention, the above-mentioned device also includes: a model updating unit, which is used to adaptively solve the modeling parameters of the three-dimensional model of the comprehensive mining working face during coal mining machine operation based on the DeepLabV3+ semantic segmentation algorithm, and update the three-dimensional model according to the solved modeling parameters.

[0047] As an embodiment of the present invention, the model updating unit includes:

[0048] Image recognition module, used to identify images using the trained DeepLabV3+ semantic segmentation algorithm;

[0049] A position calculation module, used for calculating the position of the scraper conveyor using a method based on three-dimensional modeling;

[0050] The difference calculation module is used to calculate the difference between the scraper conveyor and the support guard plate identified by the DeepLabV3+ semantic segmentation algorithm and the scraper conveyor and the support guard plate calculated based on the three-dimensional modeling principle, as the value of the loss function; the solution module is used to use the constrained minimization method of the multivariate scalar function to solve the variable parameters that minimize the loss function;

[0051] An updating module is used to update the three-dimensional model using the fixed parameters and the solved variable parameters.

[0052] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0053] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0054] According to a fifth aspect of the present invention, there is provided a computer program product, comprising a computer program / instructions, which implement the steps of the above method when executed by a processor.

[0055] It can be seen from the above technical solution that the method and device for determining the dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling provided by the present invention have the following beneficial effects:

[0056] 1. High-precision identification of dangerous areas: Through precise geometric modeling and coordinate system conversion, the present invention can accurately locate dangerous areas in the fully mechanized mining working face, greatly improve identification accuracy, and reduce the probability of false alarms and missed alarms.

[0057] 2. Strong dynamic adaptability: The three-dimensional model-based method enables the system to adapt to changes in the camera position in real time, overcoming the problem that the screen annotation method fails when the camera moves.

[0058] 3. High computational efficiency: Compared with the method that fully relies on deep learning, the geometric model of the present invention has a small amount of computation, can quickly calculate the dangerous area, meet the needs of real-time monitoring, and reduce the hardware requirements. It takes 9.6ms to calculate once on an Intel(R) Xeon(R) Gold 5218R CPU.

[0059] 4. Adaptive parameter optimization: By combining deep learning recognition results and initialization parameters, an adaptive parameter optimization algorithm is designed, which enables the system to automatically adjust parameters in the new fully mechanized mining face environment without human intervention.

[0060] 5. Strong generalization ability: The method based on the three-dimensional model has good generalization ability and can be easily applied to different fully mechanized mining faces, overcoming the problem of decreased recognition accuracy of pure deep learning methods in new environments.

[0061] 6. Reduce labor costs: The method of the present invention can automatically adapt to different working surface environments, without the need for frequent manual parameter adjustments and a large amount of data labeling work, significantly reducing the labor costs of system deployment and maintenance.

[0062] 7. Improve interpretability: Compared with the black-box deep learning method, the method based on the three-dimensional geometric model of the present invention has better interpretability, facilitates understanding and debugging of the system, and improves the reliability and maintainability of the system.

[0063] 8. Improved safety: Through rapid and accurate identification of dangerous areas, the present invention can promptly discover potential safety hazards and provide timely warnings to workers, thereby significantly improving the safety production level of the fully mechanized mining face.

[0064] 9. Easy to promote: The method of the present invention does not rely on specific hardware equipment, has wide applicability, and is easy to promote and apply in different coal mines and fully mechanized mining faces, which is beneficial to improving the safety production level of the entire industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0066] Figure 1 It is a flow chart of a method for determining a dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling provided in an embodiment of the present application;

[0067] Figure 2 is a schematic diagram of a process of performing distortion correction based on distortion parameters provided in an embodiment of the present application;

[0068] Figure 3 is a schematic diagram of a process for adaptively updating modeling parameters provided in an embodiment of the present application;

[0069] Figure 4 It is a structural schematic diagram of a device for determining a dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling provided in an embodiment of the present application;

[0070] Figure 5 It is a schematic block diagram of the system structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0072] At present, the identification of dangerous areas in coal mines is not mature enough, or it is not possible to identify dangerous areas in real time, dynamically and intelligently, or the amount of calculation is too large, making it difficult to adapt to new and untrained working face environments. Therefore, the purpose of this application is to provide a method and device for identifying dangerous areas in a fully mechanized working face based on three-dimensional modeling, which can not only accurately identify dangerous areas, but also has a small amount of calculation, and it takes 9.6ms to calculate once on an Intel(R) Xeon(R) Gold 5218R CPU. In addition, it can automatically adapt to various new working face environments.

[0073] like Figure 1 The figure is a flow chart of a method for determining a dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling provided in an embodiment of the present application. The method comprises the following steps:

[0074] Step S101: Obtain the intrinsic parameters and distortion parameters of the camera in the fully mechanized mining face.

[0075] Preferably, the intrinsic parameters of the camera in this step are the intrinsic parameter matrix K, which includes the focal length of the camera in the x direction (fx), the focal length of the camera in the y direction (fy), the coordinates of the image principal point in the x-axis direction (cx), and the coordinates of the image principal point in the y-axis direction (cy); and the distortion parameters include radial distortion coefficients (k1, k2, k3) and tangential distortion coefficients (p1, p2).

[0076] Further preferably, the Zhang Zhengyou calibration method can be used to calibrate the intrinsic parameters and distortion parameters of the camera to obtain the intrinsic parameters and distortion parameters of the camera in the fully mechanized mining working face. The following are the specific steps:

[0077] Step 1: Prepare the calibration plate: In a laboratory environment, make a calibration plate and ensure that the calibration plate is flat and wrinkle-free.

[0078] Step 2, image acquisition: a) Photograph the calibration plate at different positions and angles, usually 15-20 images are required; b) Ensure that the calibration plate is fully visible in each image and covers most of the image area; c) The acquired images should cover the calibration plate at different depths and tilt angles.

[0079] Step 3: Corner detection: a) Preprocess each image, such as grayscale, contrast enhancement, etc.; b) Use a corner detection algorithm (such as Harris corner detector) to accurately locate the intersection of the chessboard; c) Perform sub-pixel optimization on the detection results to improve the accuracy of the corner position.

[0080] Step 4: Initial parameter estimation: a) Assume an ideal pinhole camera model and ignore distortion; b) Use the detected corner coordinates to establish a mapping relationship from the world coordinate system to the image coordinate system; c) Use the direct linear transformation (DLT) method to estimate the initial internal and external parameters of the camera.

[0081] Step 5, parameter optimization: a) Introduce radial distortion and tangential distortion models; b) Construct an objective function to minimize the reprojection error (the difference between the actual detected corner position and the projected position estimated based on the current parameters); c) Use nonlinear optimization methods such as the Levenberg-Marquardt algorithm to iteratively optimize the camera internal and external parameters and distortion parameters.

[0082] Step 6, error evaluation: a) calculate the reprojection error of each corner point; b) calculate the overall root mean square (RMS) reprojection error; c) if the error exceeds a predetermined threshold (usually less than 0.5 pixels), return to step 5 to continue optimization or re-acquire the image.

[0083] Step 7, parameter extraction: a) Extract the camera intrinsic parameter matrix K from the optimization results, including focal length (fx, fy) and principal point coordinates (cx, cy); b) Extract the distortion coefficients, including radial distortion coefficients (k1, k2, k3) and tangential distortion coefficients (p1, p2); c) Extract fx, fy, cx, cy from the intrinsic parameter matrix K.

[0084] Step 8: Result verification: a) Use the obtained parameters to perform distortion correction on the new image and check the correction effect; b) If the correction effect is not good, it may be necessary to redo the calibration process or adjust the calibration board design.

[0085] Step S102: Acquire the three-dimensional modeling parameters of the fully mechanized mining working face.

[0086] Preferably, the three-dimensional modeling parameters in this step include fixed parameters and variable parameters, wherein fixed parameters are parameters that will not change during the operation, and variable parameters are parameters that will frequently change during the operation. Fixed parameters include, but are not limited to, support width, support spacing, and scraper conveyor width; variable parameters include, but are not limited to, the vertical height of the camera from the ground, the angle between the camera and the horizontal plane, the horizontal distance from the camera to the coal wall, and the horizontal distance from the scraper conveyor to the coal wall.

[0087] These parameters will first read the above three-dimensional modeling parameters from the system configuration file. If they cannot be read, the above modeling parameters will be initialized.

[0088] Step S103: constructing a three-dimensional model of the fully mechanized mining face based on the three-dimensional modeling parameters.

[0089] Based on the modeling parameters obtained in step S102, the present invention constructs an accurate three-dimensional model of the fully mechanized mining face. The specific construction process of the three-dimensional model is as follows:

[0090] 1. Single bracket modeling: Construct the edge line of a single bracket directly opposite the camera.

[0091] 2. Ground-coal wall intersection line modeling: Construct the intersection line between the ground plane directly opposite the camera and the coal wall plane.

[0092] 3. Scraper conveyor modeling: Construct two edge lines of the scraper conveyor directly opposite the camera.

[0093] 4. Modeling of the bracket base: construct the outline of the bracket base.

[0094] Each of the above edges is precisely defined by 5 key points to capture the geometric features of the stent. This multi-point definition method allows for a more accurate description of the shape of the stent, including possible curved surfaces or irregular edges.

[0095] 5. Model expansion: According to the pre-set support quantity and support spacing parameters, the single support model is copied and adjusted to form a complete support group. Similarly, the ground-coal wall intersection, scraper conveyor, and support base sections are expanded into a continuous system model.

[0096] The modeling results of this step can clearly display the key structural elements of the fully mechanized mining face through lines, such as the outline and edge line of the support, the ground-coal wall intersection line, the edge line of the scraper conveyor, the outline line of the support base, etc. This high-precision three-dimensional digital model provides a solid foundation for the subsequent positioning and visualization of dangerous areas, greatly improving the accuracy and reliability of the system.

[0097] Step S104: establishing a world coordinate system with the camera center as the origin.

[0098] Specifically, this step may include the following three sub-steps:

[0099] 1) Determine the origin: Define the camera optical center as the origin of the world coordinate system (0,0,0).

[0100] 2) Define the coordinate axes:

[0101] X-axis: parallel to the ground, pointing in the direction of advancement of the working surface.

[0102] Y axis: parallel to the ground and perpendicular to the X axis.

[0103] Z axis: perpendicular to the ground, with upward as the positive direction.

[0104] 3) Quantifying key points: Convert all key points defined in step S103 into three-dimensional coordinates (X, Y, Z) in the world coordinate system.

[0105] Step S105: Acquire the external parameters of the camera, and convert the world coordinate system into a camera coordinate system based on the external parameters.

[0106] Specifically, this step may include the following two sub-steps:

[0107] 1) Obtain camera external parameters: Obtain the horizontal rotation angle (Pan) and pitch angle (Tilt) of the camera in the current frame, and calculate the rotation matrix R based on the camera's Pan and Tilt.

[0108] 2) Coordinate conversion: Use the formula P_camera = R*P_world to convert the point in the world coordinate system to the camera coordinate system, where P_world and P_camera represent the points in the world coordinate system and the camera coordinate system respectively.

[0109] Step S106: converting the camera coordinate system into an image coordinate system using the intrinsic parameters of the camera.

[0110] Specifically, this step may include the following two sub-steps:

[0111] 1) Apply the projection matrix: Use the camera intrinsic parameter matrix K to project the points in the camera coordinate system to the image plane.

[0112] 2) Coordinate transformation: Use the formula p_image = K*P_camera, where p_image is a point in the image coordinate system.

[0113] Step S107: converting the image coordinate system into a pixel coordinate system, and performing distortion correction based on the distortion parameters.

[0114] Specifically, in this step, converting the image coordinate system into a pixel coordinate system may include the following two sub-steps:

[0115] 1) Scaling transformation: Convert points in the image coordinate system to pixel coordinates according to the image resolution and sensor size.

[0116] 2) Origin translation: Move the coordinate origin from the center of the image to the upper left corner of the image.

[0117] And as Figure 2 As shown, the distortion correction based on the distortion parameters in this step may include the following sub-steps:

[0118] Step S1071: Obtain pixel coordinates in the distorted image, that is, the coordinates of pixel points in the uncorrected image.

[0119] Step S1072: Convert the coordinates of the pixel points in the uncorrected image into normalized coordinates, that is, convert the coordinates in the pixel coordinate system into a coordinate system with the center of the image as the origin and the unit as pixel.

[0120] Step S1073: Calculate the radial distortion increment using the radial distortion coefficients k1, k2, k3.

[0121] Step S1074: Calculate the tangential distortion increment using the tangential distortion coefficients p1 and p2.

[0122] Step S1075: Calculate the corrected normalized coordinates based on the radial distortion increment and the tangential distortion increment.

[0123] Step S1076: convert the corrected normalized coordinates back to pixel coordinates to obtain corrected pixel coordinates.

[0124] Through the above steps, a distortion-corrected image can be obtained, thereby improving the accuracy of subsequent dangerous area identification.

[0125] Step S108: Map the fully mechanized mining face constructed based on the three-dimensional modeling to the camera image after distortion correction.

[0126] The three-dimensional model constructed in step S103 is mapped to the camera picture after the distortion correction in step S107, so as to achieve alignment between the three-dimensional model and the actual picture.

[0127] Step S109: Determine the pixel coordinates of the dangerous area in the mapped camera image.

[0128] In the mapped camera image, the pixel coordinates of the dangerous area are determined according to predefined rules, so as to achieve accurate positioning of the dangerous area. For example, the boundary line of a specific support, the intersection line of the ground and the coal wall, etc. can be defined as the boundary of the dangerous area.

[0129] Because before building a three-dimensional model, it is necessary to determine the actual values ​​of some key points in space, that is, the three-dimensional modeling parameters in the above step S102. In each comprehensive mining working face, parameters such as the width of the bracket, the spacing between the brackets, the width of the scraper conveyor, etc. are all fixed values ​​and can be directly obtained. However, parameters such as the vertical height of the camera from the ground, the angle between the camera and the horizontal plane, the horizontal distance from the camera to the coal wall, and the horizontal distance from the scraper conveyor to the coal wall are not fixed values. They change with the change of the moving frame, or errors are caused by the different positions where the workers install the camera. Therefore, in order to make the final recognition result more accurate, the above method of the present application also includes the following steps:

[0130] Based on the DeepLabV3+ semantic segmentation algorithm, the modeling parameters of the three-dimensional model of the fully mechanized mining working face in the coal mining machine operation are adaptively solved, and the three-dimensional model is updated according to the solved modeling parameters.

[0131] Specifically, if Figure 3 As shown, the above-mentioned DeepLabV3+ semantic segmentation algorithm is used to adaptively solve the modeling parameters of the three-dimensional model of the fully mechanized working face in the coal mining machine, and updating the three-dimensional model according to the solved modeling parameters can further include the following steps:

[0132] Step S301: Use the trained DeepLabV3+ semantic segmentation algorithm to identify the image.

[0133] Step S302: Calculate the position of the scraper conveyor using a method based on three-dimensional modeling.

[0134] Step S303: Calculate the difference between the scraper conveyor and support guard plate identified by the DeepLabV3+ semantic segmentation algorithm and the positions of the scraper conveyor and support guard plate calculated based on the three-dimensional modeling principle, and use the difference as the value of the loss function; Step S304: Use the constrained minimization method of multivariate scalar function to solve the variable parameters that minimize the loss function.

[0135] Step S305: Update the three-dimensional model using the fixed parameters and the solved variable parameters.

[0136] By comparing and iteratively optimizing the DeepLabV3 semantic segmentation results with the 3D model calculation results, the model parameters can be effectively adjusted to make the 3D model more consistent with the real fully mechanized mining face scene, thereby improving the accuracy of the model.

[0137] The above method of the present application is further described below through a specific embodiment:

[0138] The fully mechanized mining face is an important area in coal mining. There are some dangerous areas that need special attention. The following are the main dangerous areas of the fully mechanized mining face:

[0139] Coal mining machine operation area: When large-scale coal mining machinery and equipment are in operation, there is a risk of mechanical injury in the surrounding area;

[0140] Coal wall area: The coal wall refers to the unmined side of the coal seam, which may suddenly collapse or fall;

[0141] Conveyor area: When belt conveyors and other equipment are in operation, there is a risk of entrapment and crushing;

[0142] Support moving area: When the hydraulic support moves, there is a risk of squeezing around it.

[0143] The three-dimensional model system established according to the above method can calculate the working face area when the camera is rotated to any angle, and flexibly define the dangerous area according to the current situation. The following takes the coal mining machine operation area as an example to explain in detail. The processing of other dangerous areas is similar to this example and will not be repeated here.

[0144] When large-scale coal mining machinery and equipment are in operation, there is a risk of mechanical injury in the surrounding area. Therefore, it is necessary to calculate the dangerous area formed by the current camera perspective, that is, the boundary line of a certain bracket. For example, according to business needs, the boundary of bracket No. 72 can be set as the boundary of the dangerous area. When one camera is installed on every 6 brackets, the most suitable camera is the camera of bracket No. 68 for observation. The position of the boundary line of bracket No. 72 in the current camera is calculated using the above-mentioned 3D modeling-based method and marked with a red line. The intersection of the ground plane and the coal wall plane and the contour line of the bracket base are also marked. The area surrounded by this line is the dangerous area.

[0145] From the above, it can be seen that the method for determining the dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling provided by the present invention has the following beneficial effects:

[0146] 1. High-precision identification of dangerous areas: Through precise geometric modeling and coordinate system conversion, the present invention can accurately locate dangerous areas in the fully mechanized mining working face, greatly improve identification accuracy, and reduce the probability of false alarms and missed alarms.

[0147] 2. Strong dynamic adaptability: The three-dimensional model-based method enables the system to adapt to changes in the camera position in real time, overcoming the problem that the screen annotation method fails when the camera moves.

[0148] 3. High computational efficiency: Compared with the method that fully relies on deep learning, the geometric model of the present invention has a small amount of calculation, can quickly calculate the dangerous area, meet the needs of real-time monitoring, and reduce the requirements for hardware.

[0149] 4. Adaptive parameter optimization: By combining deep learning recognition results and initialization parameters, an adaptive parameter optimization algorithm is designed, which enables the system to automatically adjust parameters in the new fully mechanized mining face environment without human intervention.

[0150] 5. Strong generalization ability: The method based on the three-dimensional model has good generalization ability and can be easily applied to different fully mechanized mining faces, overcoming the problem of decreased recognition accuracy of pure deep learning methods in new environments.

[0151] 6. Reduce labor costs: The method of the present invention can automatically adapt to different working surface environments, without the need for frequent manual parameter adjustments and a large amount of data labeling work, significantly reducing the labor costs of system deployment and maintenance.

[0152] 7. Improve interpretability: Compared with the black-box deep learning method, the method based on the three-dimensional geometric model of the present invention has better interpretability, facilitates understanding and debugging of the system, and improves the reliability and maintainability of the system.

[0153] 8. Improved safety: Through rapid and accurate identification of dangerous areas, the present invention can promptly discover potential safety hazards and provide timely warnings to workers, thereby significantly improving the safety production level of the fully mechanized mining face.

[0154] 9. Easy to promote: The method of the present invention does not rely on specific hardware equipment, has wide applicability, and is easy to promote and apply in different coal mines and fully mechanized mining faces, which is beneficial to improving the safety production level of the entire industry.

[0155] like Figure 4 The structure diagram of a device for determining a dangerous area of ​​a fully mechanized working face based on three-dimensional modeling provided by an embodiment of the present application is shown, and the device includes: a camera parameter acquisition unit 410, a modeling parameter acquisition unit 420, a three-dimensional modeling unit 430, a world coordinate establishment unit 440, a camera coordinate conversion unit 450, an image coordinate conversion unit 460, a pixel coordinate conversion unit 470, a screen mapping unit 480 and a dangerous area determination unit 490, which are sequentially connected. Among them:

[0156] The camera parameter acquisition unit 410 is used to obtain the intrinsic parameters and distortion parameters of the camera in the fully mechanized mining working face.

[0157] The modeling parameter acquisition unit 420 is used to acquire the three-dimensional modeling parameters of the fully mechanized mining working face.

[0158] The three-dimensional modeling unit 430 is used to construct a three-dimensional model of the fully mechanized mining working face based on the three-dimensional modeling parameters.

[0159] The world coordinate system establishing unit 440 is used to establish a world coordinate system with the camera center as the origin.

[0160] The camera coordinate conversion unit 450 is used to obtain the external parameters of the camera and convert the world coordinate system into the camera coordinate system based on the external parameters.

[0161] The image coordinate conversion unit 460 is used to convert the camera coordinate system into an image coordinate system using the intrinsic parameters of the camera.

[0162] The pixel coordinate conversion unit 470 is used to convert the image coordinate system into a pixel coordinate system and perform distortion correction based on the distortion parameters.

[0163] The picture mapping unit 480 is used to map the fully mechanized mining face constructed based on three-dimensional modeling to the camera picture after distortion correction.

[0164] The dangerous area determination unit 490 is used to determine the pixel coordinates of the dangerous area in the mapped camera image.

[0165] Preferably, the internal parameters of the above-mentioned camera are an internal parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the distortion parameters include radial distortion coefficients and tangential distortion coefficients.

[0166] Preferably, the above-mentioned camera parameter acquisition unit is specifically used to: use Zhang Zhengyou calibration method to calibrate the intrinsic parameters and distortion parameters of the camera to obtain the intrinsic parameters and distortion parameters of the camera in the comprehensive mining working face.

[0167] Preferably, the above-mentioned three-dimensional modeling parameters include fixed parameters and variable parameters, the fixed parameters include: bracket width, bracket spacing and scraper conveyor width; the variable parameters include: the vertical height of the camera from the ground, the angle between the camera and the horizontal plane, the horizontal distance from the camera to the coal wall and the horizontal distance from the scraper conveyor to the coal wall.

[0168] Preferably, the device also includes: a model updating unit, which is used to adaptively solve the modeling parameters of the three-dimensional model of the comprehensive mining working face during coal mining machine operation based on the DeepLabV3+ semantic segmentation algorithm, and update the three-dimensional model according to the solved modeling parameters.

[0169] Preferably, the above-mentioned model updating unit includes:

[0170] Image recognition module, used to identify images using the trained DeepLabV3+ semantic segmentation algorithm;

[0171] A position calculation module, used for calculating the position of the scraper conveyor using a method based on three-dimensional modeling;

[0172] The difference calculation module is used to calculate the difference between the scraper conveyor and the support guard plate identified by the DeepLabV3+ semantic segmentation algorithm and the scraper conveyor and the support guard plate calculated based on the three-dimensional modeling principle, and use the difference as the value of the loss function; the solution module is used to use the constrained minimization method of the multivariate scalar function to solve the variable parameters that minimize the loss function;

[0173] An updating module is used to update the three-dimensional model using the fixed parameters and the solved variable parameters.

[0174] For a detailed description of the above-mentioned units and modules, please refer to the corresponding description in the aforementioned method embodiment, which will not be further elaborated here.

[0175] From the above, it can be seen that the device for determining the dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling provided by the present invention has the following beneficial effects:

[0176] 1. High-precision identification of dangerous areas: Through precise geometric modeling and coordinate system conversion, the present invention can accurately locate dangerous areas in the fully mechanized mining working face, greatly improve identification accuracy, and reduce the probability of false alarms and missed alarms.

[0177] 2. Strong dynamic adaptability: The three-dimensional model-based method enables the system to adapt to changes in the camera position in real time, overcoming the problem that the screen annotation method fails when the camera moves.

[0178] 3. High computational efficiency: Compared with the method that fully relies on deep learning, the geometric model of the present invention has a small amount of calculation, can quickly calculate the dangerous area, meet the needs of real-time monitoring, and reduce the requirements for hardware.

[0179] 4. Adaptive parameter optimization: By combining deep learning recognition results and initialization parameters, an adaptive parameter optimization algorithm is designed, which enables the system to automatically adjust parameters in the new fully mechanized mining face environment without human intervention.

[0180] 5. Strong generalization ability: The method based on the three-dimensional model has good generalization ability and can be easily applied to different fully mechanized mining faces, overcoming the problem of decreased recognition accuracy of pure deep learning methods in new environments.

[0181] 6. Reduce labor costs: The method of the present invention can automatically adapt to different working surface environments, without the need for frequent manual parameter adjustments and a large amount of data labeling work, significantly reducing the labor costs of system deployment and maintenance.

[0182] 7. Improve interpretability: Compared with the black-box deep learning method, the method based on the three-dimensional geometric model of the present invention has better interpretability, facilitates understanding and debugging of the system, and improves the reliability and maintainability of the system.

[0183] 8. Improved safety: Through rapid and accurate identification of dangerous areas, the present invention can promptly discover potential safety hazards and provide timely warnings to workers, thereby significantly improving the safety production level of the fully mechanized mining face.

[0184] 9. Easy to promote: The method of the present invention does not rely on specific hardware equipment, has wide applicability, and is easy to promote and apply in different coal mines and fully mechanized mining faces, which is beneficial to improving the safety production level of the entire industry.

[0185] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above method is implemented when the processor executes the program.

[0186] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program for executing the above method.

[0187] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the method described in the claim when the computer program / instruction is executed by a processor.

[0188] like Figure 5 As shown, the electronic device 600 may further include: a communication module 110, an input unit 120, an audio processor 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily have to include Figure 5 In addition, the electronic device 600 may also include Figure 5 For components not shown, reference may be made to the prior art.

[0189] like Figure 5 As shown, the central processor 100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The central processor 100 receives inputs and controls the operations of various components of the electronic device 600 .

[0190] The memory 140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The information related to the hydraulic support control method may be stored, and a program for executing the related information may also be stored. The central processing unit 100 may execute the program stored in the memory 140 to implement information storage or processing.

[0191] The input unit 120 provides input to the CPU 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used to display display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0192] The memory 140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 140 may also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 600 through the central processor 100.

[0193] The memory 140 may also include a data storage unit 143 for storing data, such as digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0194] The communication module 110 is a transmitter / receiver that transmits and receives signals via the antenna 111. The communication module 110 (transmitter / receiver) is coupled to the central processor 100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0195] Based on different communication technologies, multiple communication modules 110 may be provided in the same electronic device, such as low-power wireless communication modules such as Radio Frequency Identification (RFID), Ultra-WideBand (UWB), ZigBee, and Bluetooth. The communication module 110 (transmitter / receiver) is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide an audio output via the speaker 131 and receive an audio input from the microphone 132, thereby realizing a common telecommunication function. The audio processor 130 may include any suitable buffer, decoder, amplifier, and the like. In addition, the audio processor 130 is also coupled to the central processor 100, so that recording can be performed on the local machine through the microphone 132, and the sound stored on the local machine can be played through the speaker 131.

[0196] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0197] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0198] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0199] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0200] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for determining dangerous areas of fully mechanized mining working faces based on three-dimensional modeling, characterized in that: The method comprises: Obtain the intrinsic parameters and distortion parameters of the camera in the fully mechanized mining face; Acquiring three-dimensional modeling parameters of the fully mechanized mining working face; Constructing a three-dimensional model of the fully mechanized mining working face based on the three-dimensional modeling parameters; Establishing a world coordinate system with the camera center as the origin; Acquire the external parameters of the camera, and convert the world coordinate system into a camera coordinate system based on the external parameters; Converting the camera coordinate system into an image coordinate system using the intrinsic parameters of the camera; Converting the image coordinate system into a pixel coordinate system, and performing distortion correction based on the distortion parameters; Map the fully mechanized mining face constructed based on 3D modeling to the camera image after distortion correction; The pixel coordinates of the dangerous area are determined in the mapped camera image.

2. The method for determining the dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling according to claim 1 is characterized in that: The intrinsic parameters of the camera are an intrinsic parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the distortion parameters include radial distortion coefficients and tangential distortion coefficients.

3. The method for determining the dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling according to claim 2 is characterized in that: The obtaining of the intrinsic parameters and distortion parameters of the camera in the fully mechanized mining working face includes: using the Zhang Zhengyou calibration method to calibrate the intrinsic parameters and distortion parameters of the camera to obtain the intrinsic parameters and distortion parameters of the camera in the fully mechanized mining working face.

4. The method for determining the dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling according to claim 1 is characterized in that: The three-dimensional modeling parameters include fixed parameters and variable parameters. The fixed parameters include: support width, support spacing and scraper conveyor width; the variable parameters include: the vertical height of the camera from the ground, the angle between the camera and the horizontal plane, the horizontal distance from the camera to the coal wall and the horizontal distance from the scraper conveyor to the coal wall.

5. The method for determining the dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling according to claim 4 is characterized in that: The method also includes: adaptively solving modeling parameters of the three-dimensional model of the fully mechanized mining face during coal mining operation based on the DeepLabV3+ semantic segmentation algorithm, and updating the three-dimensional model according to the solved modeling parameters.

6. The method for determining the dangerous area of ​​a fully mechanized mining face based on three-dimensional modeling according to claim 5 is characterized in that: The method of adaptively solving the modeling parameters of the three-dimensional model of the fully mechanized working face in the coal mining machine operation based on the DeepLabV3+ semantic segmentation algorithm, and updating the three-dimensional model according to the solved modeling parameters includes: Use the trained DeepLabV3+ semantic segmentation algorithm to identify the image; The position of the scraper conveyor is calculated using a 3D modeling-based approach; Calculate the difference between the positions of the scraper conveyor and the support guard plate identified by the DeepLabV3+ semantic segmentation algorithm and the scraper conveyor and the support guard plate calculated based on the three-dimensional modeling principle, and use the difference as the value of the loss function; use the constrained minimization method of multivariate scalar function to solve the variable parameters that minimize the loss function; The three-dimensional model is updated using the fixed parameters and the solved variable parameters.

7. A device for determining dangerous areas of fully mechanized mining working faces based on three-dimensional modeling, characterized in that: The device comprises: A camera parameter acquisition unit is used to obtain the internal parameters and distortion parameters of the camera in the fully mechanized mining working face; A modeling parameter acquisition unit, used to acquire the three-dimensional modeling parameters of the fully mechanized mining working face; A three-dimensional modeling unit, used for constructing a three-dimensional model of the fully mechanized mining working face based on the three-dimensional modeling parameters; A world coordinate establishing unit, used for establishing a world coordinate system with the camera center as the origin; A camera coordinate conversion unit, used for acquiring the external parameters of the camera, and converting the world coordinate system into a camera coordinate system based on the external parameters; An image coordinate conversion unit, used to convert the camera coordinate system into an image coordinate system using the intrinsic parameters of the camera; A pixel coordinate conversion unit, used for converting the image coordinate system into a pixel coordinate system, and performing distortion correction based on the distortion parameters; A picture mapping unit is used to map the fully mechanized mining face constructed based on three-dimensional modeling to a camera picture after distortion correction; The danger area determination unit is used to determine the pixel coordinates of the danger area in the mapped camera image.

8. The device for determining dangerous areas of fully mechanized mining working faces based on three-dimensional modeling according to claim 7, characterized in that: The intrinsic parameters of the camera are an intrinsic parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the distortion parameters include radial distortion coefficients and tangential distortion coefficients.

9. The device for determining dangerous areas of fully mechanized mining working faces based on three-dimensional modeling according to claim 8, characterized in that: The camera parameter acquisition unit is specifically used to: use Zhang Zhengyou calibration method to calibrate the intrinsic parameters and distortion parameters of the camera to obtain the intrinsic parameters and distortion parameters of the camera in the comprehensive mining working face.

10. The device for determining dangerous areas of fully mechanized mining working faces based on three-dimensional modeling according to claim 7, characterized in that: The three-dimensional modeling parameters include fixed parameters and variable parameters. The fixed parameters include: support width, support spacing and scraper conveyor width; the variable parameters include: the vertical height of the camera from the ground, the angle between the camera and the horizontal plane, the horizontal distance from the camera to the coal wall and the horizontal distance from the scraper conveyor to the coal wall.

11. The device for determining dangerous areas of fully mechanized mining working faces based on three-dimensional modeling according to claim 10, characterized in that: The device also includes: a model updating unit, which is used to adaptively solve the modeling parameters of the three-dimensional model of the fully mechanized working face in the coal mining machine operation based on the DeepLabV3+ semantic segmentation algorithm, and update the three-dimensional model according to the solved modeling parameters.

12. The device for determining dangerous areas of fully mechanized mining working faces based on three-dimensional modeling according to claim 11, characterized in that: The model updating unit comprises: Image recognition module, used to identify images using the trained DeepLabV3+ semantic segmentation algorithm; A position calculation module, used for calculating the position of the scraper conveyor using a method based on three-dimensional modeling; A difference calculation module is used to calculate the difference between the scraper conveyor and the support guard plate identified by the DeepLabV3+ semantic segmentation algorithm and the positions of the scraper conveyor and the support guard plate calculated based on the three-dimensional modeling principle, and use the difference as the value of the loss function; A solution module, used for solving the variable parameters that minimize the loss function using a constrained minimization method of a multivariate scalar function; An updating module is used to update the three-dimensional model using the fixed parameters and the solved variable parameters.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images