Method and device for determining the position of support plate and support number of support

Abstract

The present invention provides a method and device for determining the position and support number of a support guard plate, the method comprising: using a camera installed on a comprehensive mining face to obtain an image of the support guard plate; obtaining the camera's internal parameters, external parameters and distortion parameters; based on the camera's internal parameters, external parameters and distortion parameters, converting the three-dimensional world coordinates of the support guard plate into a camera coordinate system, an image coordinate system, and a pixel coordinate system in sequence; using a deep learning model to identify the pixel coordinates of the support guard plate in the image; determining the actual position of the support guard plate in the image based on the pixel coordinates; based on the actual position, determining the frame number of the support guard plate through a position-frame number mapping table. The present invention can accurately convert the position of the support guard plate into a position in a pixel coordinate system, and infer its frame number based on the position-frame number mapping table, greatly improving the accuracy of frame number judgment. In addition, the present application adopts an automatic parameter seeking method, which can automatically adapt to the dynamic changes of the comprehensive mining face.

Description

Technical Field

[0001] The invention relates to the technical field of identification of a fully mechanized coal mining face, and in particular to a method and a device for determining a support guard plate position and a support number. Background Art

[0002] The fully mechanized mining face is a key operating area in coal mining, where the stability and control of the support guard plate is directly related to the safety and efficiency of coal mining. The determination of the support guard plate frame number is an important part of the monitoring and management of the fully mechanized mining face. Traditional frame number recognition relies on manual or simple visual detection methods, which have many limitations in complex mine environments. For example, factors such as insufficient light, high dust content, and frequent support movement in the fully mechanized mining face will have a negative impact on the recognition results.

[0003] At present, the visual recognition technology used in some fully mechanized mining faces is usually based on a single image processing method. The recognition effect is easily affected by lighting, angle and occlusion, resulting in low frame number recognition accuracy and difficulty in meeting actual needs. In addition, in most cases, the camera can only capture a partial image of the support guard plate, making it difficult to directly recognize the accurate frame number, which poses a challenge to the automation and intelligence of fully mechanized mining work.

[0004] In order to solve the above problems, there is an urgent need for a system and method that can accurately and automatically identify the frame number of the support guard plate in a complex environment. Summary of the invention

[0005] In view of this, the present invention provides a method and device for determining the position of a support guard plate and the support number of a support to solve at least one of the above-mentioned problems.

[0006] In order to achieve the above object, the present invention adopts the following scheme:

[0007] According to a first aspect of the present invention, a method for determining a support plate position and a support number of a support is provided, the method comprising:

[0008] The image of the support guard plate is obtained by using a camera installed on the fully mechanized mining face;

[0009] Obtaining intrinsic parameters, extrinsic parameters and distortion parameters of the camera;

[0010] Based on the intrinsic parameters, extrinsic parameters and distortion parameters of the camera, the three-dimensional world coordinates of the support guard plate are converted into a camera coordinate system, an image coordinate system and a pixel coordinate system in sequence;

[0011] Use a deep learning model to identify the pixel coordinates of the support plate in the image;

[0012] Determine the actual position of the support plate of the support in the image based on the pixel coordinates;

[0013] Based on the actual position, the frame number of the support guard plate is determined through a position-frame number mapping table.

[0014] As an embodiment of the present invention, the intrinsic parameters of the camera in the above method are an intrinsic parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the external parameters of the camera include the horizontal rotation angle and the pitch angle; the distortion parameters include the radial distortion coefficient and the tangential distortion coefficient.

[0015] As an embodiment of the present invention, the above method converts the three-dimensional world coordinates of the support guard plate into a camera coordinate system, an image coordinate system, and a pixel coordinate system in sequence based on the intrinsic parameters, extrinsic parameters, and distortion parameters of the camera, including:

[0016] Establishing a world coordinate system with the camera center as the origin;

[0017] Converting the world coordinate system into a camera coordinate system based on the external reference;

[0018] Converting the camera coordinate system into an image coordinate system using the intrinsic parameters;

[0019] The image coordinate system is converted into a pixel coordinate system, and distortion correction is performed based on the distortion parameters.

[0020] As an embodiment of the present invention, the method of using a deep learning model to identify the pixel coordinates of the support plate of the stent in the image includes: using the DeepLabv3+ semantic segmentation algorithm to identify the pixel coordinates of the support plate of the stent in the image.

[0021] As an embodiment of the present invention, the above method also includes: based on the dynamic changes of the comprehensive mining working face, using the DeepLabv3+ semantic segmentation algorithm to dynamically adjust the distance and deflection angle parameters between the camera and the support guard plate.

[0022] As an embodiment of the present invention, the method of dynamically adjusting the distance and deflection angle parameters between the camera and the support guard plate using the DeepLabv3+ semantic segmentation algorithm includes:

[0023] The pixel coordinates of the support plate of the stent in the image identified by the DeepLabv3+ semantic segmentation algorithm are used to form the outline of the support plate of the stent;

[0024] On the outline of the support plate of the bracket, a corner detection algorithm is used to identify and extract key feature points;

[0025] Calculate the three-dimensional coordinates of the key feature point in the world coordinate system using the intrinsic parameters and extrinsic parameters of the camera and the pixel coordinates of the key feature point;

[0026] Calculate the distance between the camera and the support plate of the bracket using the three-dimensional coordinates of the key feature points;

[0027] The deflection angle of the camera relative to the support guard plate is calculated using the external parameters of the camera and the three-dimensional coordinates of the support guard plate.

[0028] According to a second aspect of the present invention, a device for determining a support plate position and a support number of a support is provided, the device comprising:

[0029] An image acquisition unit, used for acquiring an image of the support guard plate by using a camera installed on the fully mechanized mining working face;

[0030] A parameter acquisition unit, used to acquire the intrinsic parameters, extrinsic parameters and distortion parameters of the camera;

[0031] A coordinate conversion unit, used for converting the three-dimensional world coordinates of the support guard plate into a camera coordinate system, an image coordinate system, and a pixel coordinate system in sequence based on the intrinsic parameters, extrinsic parameters, and distortion parameters of the camera;

[0032] A guard plate recognition unit, used to recognize the pixel coordinates of the support guard plate in the image using a deep learning model;

[0033] An actual position determination unit, used for determining the actual position of the support plate of the support in the image based on the pixel coordinates;

[0034] The frame number determining unit is used to determine the frame number of the support guard plate based on the actual position through a position-frame number mapping table.

[0035] As an embodiment of the present invention, the intrinsic parameters of the above-mentioned camera are an intrinsic parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the external parameters of the camera include a horizontal rotation angle and a pitch angle; the distortion parameters include a radial distortion coefficient and a tangential distortion coefficient.

[0036] As an embodiment of the present invention, the coordinate conversion unit includes:

[0037] A coordinate system establishment module, used to establish a world coordinate system with the camera center as the origin;

[0038] A first conversion module, configured to convert the world coordinate system into a camera coordinate system based on the external reference;

[0039] A second conversion module, used to convert the camera coordinate system into an image coordinate system using the intrinsic reference;

[0040] The third conversion module is used to convert the image coordinate system into a pixel coordinate system and perform distortion correction based on the distortion parameters.

[0041] As an embodiment of the present invention, the above-mentioned guard plate identification unit is specifically used to: use the DeepLabv3+ semantic segmentation algorithm to identify the pixel coordinates of the support plate of the stent in the image.

[0042] As an embodiment of the present invention, the above-mentioned device also includes: an adaptive unit, which is used to dynamically adjust the distance and deflection angle parameters between the camera and the support guard plate based on the dynamic changes of the comprehensive mining working face using the DeepLabv3+ semantic segmentation algorithm.

[0043] As an embodiment of the present invention, the above-mentioned adaptive unit includes:

[0044] A contour acquisition module, used to form the contour of the stent guard plate based on the pixel coordinates of the stent guard plate in the image identified by the DeepLabv3+ semantic segmentation algorithm;

[0045] A feature point acquisition module is used to identify and extract key feature points on the outline of the support plate of the bracket using a corner point detection algorithm;

[0046] A three-dimensional coordinate calculation module, used to calculate the three-dimensional coordinates of the key feature point in the world coordinate system by using the intrinsic parameters and extrinsic parameters of the camera and the pixel coordinates of the key feature point;

[0047] A distance acquisition module, used to calculate the distance between the camera and the support plate of the bracket using the three-dimensional coordinates of the key feature points;

[0048] The deflection angle acquisition module is used to calculate the deflection angle of the camera relative to the support guard plate by using the external parameters of the camera and the three-dimensional coordinates of the support guard plate.

[0049] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0050] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0051] According to a fifth aspect of the present invention, there is provided a computer program product, comprising a computer program / instructions, which implement the steps of the above method when executed by a processor.

[0052] It can be seen from the above technical solution that the method and device for determining the position of the support guard plate and the support number provided by the present invention have the following beneficial effects:

[0053] 1. Through precise coordinate system conversion and parameter calculation, the present application can accurately convert the position of the support plate of the bracket into a position in the pixel coordinate system, and identify its frame number based on the position-frame number mapping table, thereby greatly improving the recognition accuracy.

[0054] 2. This application can automatically adapt to the dynamic changes of the fully mechanized mining face, such as changes in lighting conditions, equipment position, etc. No matter how the camera installation position changes, the system can automatically adjust the parameters, that is, automatically calculate the distance between the camera and the support guard plate, the deflection angle of the camera relative to the support guard plate, and realize accurate frame number recognition, so this application can be applied to different fully mechanized mining face environments.

[0055] 3. This application can effectively improve the operating efficiency and safety of the comprehensive mining face through automated and accurate frame number recognition, and reduce the risks caused by manual recognition errors.

[0056] 4. It has been verified that the system delay of this application is less than 10ms (Intel(R) Xeon(R) Gold 5218R CPU), which achieves near real-time frame number calculation and display, can process and feedback the recognition results in real time, and provide timely support and support plate information to operators or automatic control systems, which is conducive to the efficient management of comprehensive mining working faces. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0058] Figure 1 It is a flowchart of a method for determining the position of a support plate and a support number of a support provided in an embodiment of the present application;

[0059] Figure 2 is a schematic diagram of the process of camera coordinate transformation provided by an embodiment of the present application;

[0060] Figure 3 is a schematic diagram of a process of performing distortion correction based on the distortion parameters provided in an embodiment of the present application;

[0061] Figure 4 It is a schematic diagram of a process of dynamically adjusting a camera using the DeepLabv3+ semantic segmentation algorithm provided in an embodiment of the present application;

[0062] Figure 5 It is a structural schematic diagram of a device for determining the position of a support guard plate and a support number of a support provided in an embodiment of the present application;

[0063] Figure 6 is a schematic diagram of the structure of a coordinate conversion unit provided in an embodiment of the present application;

[0064] Figure 7 It is a structural schematic diagram of a device for determining the position of a support guard plate and a support number of a support provided by another embodiment of the present application;

[0065] Figure 8 is a schematic diagram of the structure of the adaptive unit provided in an embodiment of the present application;

[0066] Fig. 9 It is a schematic block diagram of the system structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0068] At present, the visual recognition technology used in some comprehensive mining working faces is usually based on a single image processing method. The recognition effect is easily affected by lighting, angle and occlusion, resulting in low frame number recognition accuracy and difficulty in meeting actual needs. In addition, in most cases, the camera can only capture a partial image of the support guard plate, making it difficult to directly identify the accurate frame number, which poses a challenge to the automation and intelligence of comprehensive mining work. Therefore, the purpose of this application is to provide a method and device for identifying the position and number of a support guard plate, which can not only accurately identify the frame number of the support guard plate, but also automatically adapt to various new working face environments.

[0069] like Figure 1 The figure is a flow chart of a method for determining the position of a support plate and a support number of a support provided by an embodiment of the present application, the method comprising the following steps:

[0070] Step S101: Use a camera installed on the fully mechanized mining face to obtain an image of the support guard plate.

[0071] Multiple cameras are installed on the fully mechanized mining face to capture images of the support guards. The camera positions need to be planned in advance to ensure that the support guards can be captured and that the captured images are clear enough to meet the needs of subsequent image processing and recognition. The frequency of image capture needs to be determined according to actual needs, such as capturing images at regular intervals or continuously in real time. In addition, the image format and resolution also need to be pre-set.

[0072] Step S102: Acquire the intrinsic parameters, extrinsic parameters and distortion parameters of the camera.

[0073] In this embodiment, this step involves camera calibration. Preferably, the camera internal parameters in this step are the internal parameter matrix K, which includes the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the external parameters of the camera include the horizontal rotation angle (Pan) and the pitch angle (Tilt); the distortion parameters include the radial distortion coefficient and the tangential distortion coefficient.

[0074] To obtain these parameters, camera calibration tools and methods are usually required. For example, the Zhang Zhengyou calibration method can be used in this embodiment to calculate the camera parameters by taking a series of calibration plate images of known size and position, and then using the corresponding algorithm. The specific steps for obtaining the camera intrinsic parameters and distortion parameters are as follows:

[0075] Step 1: Prepare the calibration plate: In a laboratory environment, make a calibration plate and ensure that it is flat and wrinkle-free.

[0076] Step 2, image acquisition: a) Photograph the calibration plate at different positions and angles, usually 15-20 images are required; b) Ensure that the calibration plate is fully visible in each image and covers most of the image area; c) The acquired images should cover the calibration plate at different depths and tilt angles.

[0077] Step 3: Corner detection: a) Preprocess each image, such as grayscale, contrast enhancement, etc.; b) Use a corner detection algorithm (such as Harris corner detector) to accurately locate the intersection of the chessboard; c) Perform sub-pixel optimization on the detection results to improve the accuracy of the corner position.

[0078] Step 4: Initial parameter estimation: a) Assume an ideal pinhole camera model and ignore distortion; b) Use the detected corner coordinates to establish a mapping relationship from the world coordinate system to the image coordinate system; c) Use the direct linear transformation (DLT) method to estimate the initial internal and external parameters of the camera.

[0079] Step 5, parameter optimization: a) Introduce radial distortion and tangential distortion models; b) Construct an objective function to minimize the reprojection error (the difference between the actual detected corner position and the projected position estimated based on the current parameters); c) Use nonlinear optimization methods such as the Levenberg-Marquardt algorithm to iteratively optimize the camera internal and external parameters and distortion parameters.

[0080] Step 6, error evaluation: a) calculate the reprojection error of each corner point; b) calculate the overall root mean square (RMS) reprojection error; c) if the error exceeds a predetermined threshold (usually less than 0.5 pixels), return to step 5 to continue optimization or re-acquire the image.

[0081] Step 7, parameter extraction: a) Extract the camera intrinsic parameter matrix K from the optimization results, including focal length (fx, fy) and principal point coordinates (cx, cy); b) Extract the distortion coefficients, including radial distortion coefficients (k1, k2, k3) and tangential distortion coefficients (p1, p2); c) Extract fx, fy, cx, cy from the intrinsic parameter matrix K.

[0082] Step 8: Result verification: a) Use the obtained parameters to perform distortion correction on the new image and check the correction effect; b) If the correction effect is not good, it may be necessary to redo the calibration process or adjust the calibration board design.

[0083] Step S103: based on the intrinsic parameters, extrinsic parameters and distortion parameters of the camera, the three-dimensional world coordinates of the support support plate are converted into a camera coordinate system, an image coordinate system and a pixel coordinate system in sequence.

[0084] The purpose of performing a series of coordinate transformations in this step is to link the three-dimensional position information (world coordinate system) of the support guard plate in the real world with the two-dimensional image information (pixel coordinate system) captured by the camera. If these transformations are not performed, it is impossible to correspond the pixel points in the image with the actual position of the support guard plate. Specifically, in this embodiment, the necessity of these transformations is reflected in the following aspects: First, the support guard plate is a three-dimensional object, and the image captured by the camera is a two-dimensional plane. Therefore, it is necessary to project the three-dimensional world coordinates onto the two-dimensional image plane through projection transformation in order to find the position of the support guard plate in the image, and this projection transformation depends on the intrinsic and extrinsic parameters of the camera. Secondly, the camera has a specific position and posture in three-dimensional space (determined by extrinsic parameters), which will affect the projection position of the support guard plate in the image. If the extrinsic parameters of the camera are not considered, the three-dimensional world coordinates cannot be accurately projected onto the image plane. Finally, there are various distortions in the camera lens, such as radial distortion and tangential distortion. These distortions will make the straight lines in the image look curved and affect the accuracy of the position. Therefore, it is necessary to use distortion parameters to correct the image coordinates to obtain more accurate pixel coordinates.

[0085] The world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system are different coordinate systems. They have different coordinate units and reference points. Coordinate transformation is required to convert between these coordinate systems and establish a unified coordinate framework for subsequent calculations and analysis. Without these coordinate transformations, the deep learning model (DeepLabv3+ semantic segmentation algorithm) can only identify the pixel coordinates of the support guard in the image, but cannot determine its position in the real world, and it is impossible to determine the support number. Therefore, this step is an important step in the final identification of the support guard frame number in this application.

[0086] Preferably, Figure 2 As shown, this step may further include the following sub-steps:

[0087] Step S1031: establishing a world coordinate system with the camera center as the origin.

[0088] Specifically, this step may include the following three sub-steps:

[0089] 1) Determine the origin: Define the camera optical center as the origin of the world coordinate system (0,0,0).

[0090] 2) Define the coordinate axes:

[0091] X-axis: parallel to the ground, pointing in the direction of advancement of the working surface.

[0092] Y axis: parallel to the ground and perpendicular to the X axis.

[0093] Z axis: perpendicular to the ground, with upward as the positive direction.

[0094] 3) Quantifying key points: Convert all key points defined in step S103 into three-dimensional coordinates (X, Y, Z) in the world coordinate system.

[0095] Step S1032: Convert the world coordinate system into a camera coordinate system based on the external parameters.

[0096] Specifically, this step includes the following two sub-steps:

[0097] 1) Calculate the rotation matrix R through the horizontal rotation angle (Pan) and the pitch angle (Tilt).

[0098] 2) Coordinate conversion: Use the formula P_camera = R*P_world to convert the point in the world coordinate system to the camera coordinate system, where P_world and P_camera represent the points in the world coordinate system and the camera coordinate system respectively.

[0099] Step S1033: using the intrinsic parameters to convert the camera coordinate system into an image coordinate system.

[0100] Specifically, this step may include the following two sub-steps:

[0101] 1) Apply the projection matrix: Use the camera intrinsic parameter matrix K to project the points in the camera coordinate system to the image plane.

[0102] 2) Coordinate transformation: Use the formula p_image = K*P_camera, where p_image is a point in the image coordinate system.

[0103] Step S1034: converting the image coordinate system into a pixel coordinate system, and performing distortion correction based on the distortion parameters.

[0104] Specifically, in this step, converting the image coordinate system into a pixel coordinate system may include the following two sub-steps:

[0105] 1) Scaling transformation: Convert points in the image coordinate system to pixel coordinates according to the image resolution and sensor size.

[0106] 2) Origin translation: Move the coordinate origin from the center of the image to the upper left corner of the image.

[0107] And as Figure 3 As shown, the distortion correction based on the distortion parameters in this step may include the following sub-steps:

[0108] Step S10341: Obtain pixel coordinates in the distorted image, that is, the coordinates of pixel points in the uncorrected image.

[0109] Step S10342: Convert the coordinates of the pixel points in the uncorrected image into normalized coordinates, that is, convert the coordinates in the pixel coordinate system into a coordinate system with the center of the image as the origin and the unit as pixel.

[0110] Step S10343: Calculate the radial distortion increment using the radial distortion coefficients k1, k2, k3.

[0111] Step S10344: Calculate the tangential distortion increment using the tangential distortion coefficients p1 and p2.

[0112] Step S10345: Calculate the corrected normalized coordinates based on the radial distortion increment and the tangential distortion increment.

[0113] Step S10346: Convert the corrected normalized coordinates back to pixel coordinates to obtain corrected pixel coordinates.

[0114] Through the above steps, the distortion-corrected image can be obtained.

[0115] Step S104: using a deep learning model to identify the pixel coordinates of the support plate of the support in the image.

[0116] This step uses a deep learning model to process the image, identify the area of ​​the stent guard plate in the image, and determine its position in the pixel coordinate system. The deep learning model needs to be trained in advance, and the training data should contain a large number of stent guard plate images and their corresponding pixel coordinate labels. The output of the model can be the pixel coordinates of the stent guard plate or the mask of the stent guard plate. If it is the mask of the stent guard plate, further processing is required to extract the pixel coordinates.

[0117] Preferably, in this step, the DeepLabv3+ semantic segmentation algorithm can be used to identify the pixel coordinates of the stent guard plate in the image. Using the DeepLabv3+ semantic segmentation algorithm model to identify the pixel coordinates of the stent guard plate in the image can be achieved by the following steps:

[0118] 1. Data preparation and model training:

[0119] First, prepare a dataset containing a certain number of stent guard images. Each image needs to be annotated at the pixel level to mark the pixel area where the stent guard is located. The dataset needs to contain stent guard images under various lighting conditions, angles, and backgrounds to improve the robustness and generalization ability of the model.

[0120] Then select the DeepLabv3+ semantic segmentation algorithm model and train it using the prepared dataset. During the training process, you need to select a suitable loss function (such as the cross entropy loss function), optimizer (such as the Adam optimizer), and hyperparameters. During the training process, you need to monitor the performance of the model, such as precision, recall, and F1 value, to determine the best training parameters and model.

[0121] 2. Image preprocessing:

[0122] First, the input image is resized to the size required by the DeepLabv3+ model. Then the image pixel values ​​are normalized to a specific range (e.g., between 0 and 1) to improve the training efficiency and stability of the model.

[0123] 3. Model Reasoning:

[0124] Input the preprocessed image into the trained DeepLabv3+ model. The DeepLabv3+ model will output a pixel-level segmentation result of the same size as the input image. Each pixel will be assigned a category label, such as "bracket guard" or "background".

[0125] 4. Post-processing:

[0126] According to the output of the DeepLabv3+ model, the mask of the support guard is extracted. In the mask, the pixel value of the support guard area is 1, and the pixel value of other areas is 0. Then the pixel coordinates of the support guard are extracted from the mask, which can be achieved by finding the pixel points with a value of 1 in the mask. Here, the center point coordinates or bounding box coordinates of the support guard can also be further calculated.

[0127] 5. Result verification and optimization:

[0128] Evaluate the results of model recognition, calculate indicators such as precision, recall, and F1 value, and evaluate the performance of the model. According to the evaluation results, optimize the model, such as adjusting model parameters, adding training data, or improving data annotation. It should be noted that the output results of the DeepLabv3+ model may contain some noise or errors, and post-processing is required to remove noise and improve accuracy.

[0129] Through the above steps, the DeepLabv3+ model can be used to identify the pixel coordinates of the support plate in the image.

[0130] Step S105: determining the actual position of the support plate of the support in the image based on the pixel coordinates.

[0131] This step is to convert the pixel coordinates obtained in step S104 into the actual position of the support plate in the image. This requires combining the coordinate transformation performed in step S103 to reversely convert the pixel coordinates back to the three-dimensional coordinates in the world coordinate system or the camera coordinate system, so as to determine the actual position of the support plate.

[0132] Step S106: Based on the actual position, determine the frame number of the support guard plate through a position-frame number mapping table.

[0133] This step is to determine the rack number of the rack according to the actual position of the rack guard plate determined in step S105 using a pre-established position-rack number mapping table. This mapping table needs to be established according to the actual layout of the rack, and the actual position of the rack is associated with the corresponding rack number. By querying this mapping table, the rack number of the rack guard plate can be finally determined.

[0134] The fully mechanized mining face is a dynamically changing environment, and the position and posture of the support guard plate will change with the operation of the coal mining machine. In order to ensure that the parameters in the 3D modeling process are close to the actual values ​​of the fully mechanized mining face, to ensure the accuracy of the modeling and the accuracy of the subsequent calculation process, it is necessary to adjust the parameter values ​​in real time.

[0135] Therefore, in order to adapt to the dynamically changing environment, the present application also provides a step of adaptively adjusting parameters in the above method, that is, the above method also includes: based on the dynamic changes of the comprehensive mining working face, using the DeepLabv3+ semantic segmentation algorithm to dynamically adjust the distance and deflection angle parameters between the camera and the support guard plate.

[0136] Preferably, Figure 4 As shown, the above-mentioned method of dynamically adjusting the distance and deflection angle parameters between the camera and the support guard plate by using the DeepLabv3+ semantic segmentation algorithm may include the following sub-steps:

[0137] Step S401: Based on the pixel coordinates of the support plate in the image identified by the DeepLabv3+ semantic segmentation algorithm, the outline of the support plate is formed.

[0138] In this step, the DeepLabv3+ model outputs the pixel-level segmentation result, which extracts the outline of the support plate. The output result of the DeepLabv3+ semantic segmentation algorithm is usually a mask, in which the pixel value of the support plate area is 1 and the other areas are 0. From this mask, the boundary pixels of the support plate can be extracted to form the outline of the support plate, which requires some image processing techniques, such as edge detection and contour tracking, to remove noise and refine the outline. The final result is a set of pixels representing the outline of the support plate.

[0139] Step S402: using a corner detection algorithm to identify and extract key feature points on the outline of the support plate.

[0140] This step is to use the corner detection algorithm to identify and extract key feature points on the support guard plate contour generated in step S401. Corner points are points with significant features in the image, usually located at the corners of the contour or the intersection of the edges. Commonly used corner detection algorithms include Harris corner detection, FAST corner detection, SIFT feature point detection and SURF feature point detection, and the choice of which algorithm depends on the actual situation and accuracy requirements. The extracted key feature points are usually the four corner points of the support guard plate, or some other representative feature points. These feature points will be used for subsequent three-dimensional coordinate calculations.

[0141] Step S403: Calculate the three-dimensional coordinates of the key feature point in the world coordinate system using the intrinsic parameters and extrinsic parameters of the camera and the pixel coordinates of the key feature point.

[0142] This step is a key step in 3D reconstruction. Using the camera intrinsic parameters and extrinsic parameters obtained in step S102 and the pixel coordinates of the key feature points extracted in step S402, the 3D coordinates of these feature points in the world coordinate system can be calculated. Here, the projection transformation and 3D reconstruction technology in computer vision can be used to convert the 2D image information into 3D space information.

[0143] Step S404: Calculate the distance between the camera and the support plate of the bracket using the three-dimensional coordinates of the key feature points.

[0144] This step uses the three-dimensional coordinates of the key feature points calculated in step S403 to calculate the distance between the camera and the support guard plate. The simplest method is to calculate the Euclidean distance between the origin of the camera coordinate system and the key feature point (such as the center point or a corner point) of the support guard plate. Of course, in this embodiment, the distance between different points between the camera and the support guard plate can also be calculated according to actual needs.

[0145] Step S405: Calculate the deflection angle of the camera relative to the support guard plate by using the external parameters of the camera and the three-dimensional coordinates of the support guard plate.

[0146] This step calculates the deflection angle of the camera relative to the support guard plate, which requires the use of the camera's external parameters and the three-dimensional coordinates of the support guard plate calculated in step S403. The specific calculation method depends on the reference direction of the defined deflection angle. This involves the decomposition of the rotation matrix or other related geometric transformations to calculate the angular relationship between the camera and the support guard plate.

[0147] The output results of these steps (distance and deflection angle) will be used to control the camera and dynamically adjust the distance and angle between the camera and the support guard plate to ensure the best imaging effect. The above process is a closed-loop control system that requires real-time feedback and adjustment to ensure the normal operation of the system.

[0148] It can be seen from the above technical solution that the method for determining the position of the support guard plate and the support number provided by the present invention has the following beneficial effects:

[0149] 1. Through precise coordinate system conversion and parameter calculation, the present application can accurately convert the position of the support plate of the bracket into the position in the pixel coordinate system, and determine its frame number based on the position-frame number mapping table, which greatly improves the calculation accuracy.

[0150] 2. This application can automatically adapt to the dynamic changes of the fully mechanized mining working face, such as changes in lighting conditions, equipment position, etc. Regardless of how the camera installation position changes, the system can automatically adjust the parameters, that is, automatically calculate the distance between the camera and the support guard plate, the deflection angle of the camera relative to the support guard plate, and realize accurate frame number determination, so this application can be applied to different fully mechanized mining working face environments.

[0151] 3. This application can effectively improve the operating efficiency and safety of the comprehensive mining face through automated and accurate frame number calculation, and reduce the risks caused by manual identification errors.

[0152] 4. It has been verified that the system delay of this application is less than 10ms (on Intel(R) Xeon(R) Gold 5218R CPU), achieving near real-time frame number calculation and display, capable of real-time processing and feedback of identification results, providing operators or automatic control systems with instant support and support plate information, and facilitating efficient management of comprehensive mining working faces.

[0153] like Figure 5 The figure shows a schematic diagram of a device for determining the position of a support plate and a support number of a support provided by an embodiment of the present application, the device comprising: an image acquisition unit 510, a parameter acquisition unit 520, a coordinate conversion unit 530, a support plate identification unit 540, an actual position determination unit 550 and a support number determination unit 560, which are sequentially connected. Among them:

[0154] The image acquisition unit 510 is used to acquire the image of the support guard plate using a camera installed on the comprehensive mining working face.

[0155] The parameter acquisition unit 520 is used to acquire the intrinsic parameters, extrinsic parameters and distortion parameters of the camera.

[0156] The coordinate conversion unit 530 is used to convert the three-dimensional world coordinates of the support guard plate into a camera coordinate system, an image coordinate system, and a pixel coordinate system in sequence based on the intrinsic parameters, extrinsic parameters and distortion parameters of the camera.

[0157] The support plate recognition unit 540 is used to recognize the pixel coordinates of the support plate of the stent in the image by using a deep learning model.

[0158] The actual position determination unit 550 is used to determine the actual position of the support plate of the support plate in the image based on the pixel coordinates.

[0159] The frame number determination unit 560 is used to determine the frame number of the support guard plate based on the actual position through a position-frame number mapping table.

[0160] Preferably, the intrinsic parameters of the above-mentioned camera are an intrinsic parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the external parameters of the camera include the horizontal rotation angle and the pitch angle; the distortion parameters include the radial distortion coefficient and the tangential distortion coefficient.

[0161] Preferably, Figure 6 As shown, the coordinate conversion unit 530 includes:

[0162] The coordinate system establishing module 531 is used to establish a world coordinate system with the camera center as the origin.

[0163] The first conversion module 532 is used to convert the world coordinate system into a camera coordinate system based on the external parameters.

[0164] The second conversion module 533 is used to convert the camera coordinate system into an image coordinate system using the intrinsic parameters.

[0165] The third conversion module 534 is used to convert the image coordinate system into a pixel coordinate system and perform distortion correction based on the distortion parameters.

[0166] Preferably, the above-mentioned guard plate identification unit 540 is specifically used to: use the DeepLabv3+ semantic segmentation algorithm to identify the pixel coordinates of the support plate of the stent in the image.

[0167] Preferably, Figure 7 As shown, the above-mentioned device also includes: an adaptive unit 570, which is used to dynamically adjust the distance and deflection angle parameters between the camera and the support guard plate based on the dynamic changes of the comprehensive mining working face and using the recognition results of the DeepLabv3+ semantic segmentation algorithm.

[0168] Preferably, Figure 8 As shown, the above-mentioned adaptive unit 570 includes:

[0169] The contour acquisition module 571 is used to form the contour of the support plate of the support plate based on the pixel coordinates of the support plate of the support plate in the image identified by the DeepLabv3+ semantic segmentation algorithm.

[0170] The feature point acquisition module 572 is used to identify and extract key feature points on the outline of the support guard plate using a corner point detection algorithm.

[0171] The three-dimensional coordinate calculation module 573 is used to calculate the three-dimensional coordinates of the key feature point in the world coordinate system by using the intrinsic parameters and extrinsic parameters of the camera and the pixel coordinates of the key feature point.

[0172] The distance acquisition module 574 is used to calculate the distance between the camera and the support plate using the three-dimensional coordinates of the key feature points.

[0173] The deflection angle acquisition module 575 is used to calculate the deflection angle of the camera relative to the support guard plate by using the external parameters of the camera and the three-dimensional coordinates of the support guard plate.

[0174] For a detailed description of the above-mentioned units and modules, please refer to the corresponding description in the aforementioned method embodiment, which will not be further elaborated here.

[0175] It can be seen from the above technical solution that the device for determining the position of the support guard plate and the support number provided by the present invention has the following beneficial effects:

[0176] 1. Through precise coordinate system conversion and parameter calculation, the present application can accurately convert the position of the support plate of the bracket into the position in the pixel coordinate system, and infer its frame number based on the position-frame number mapping table, thereby greatly improving the accuracy.

[0177] 2. This application can automatically adapt to the dynamic changes of the fully mechanized mining working face, such as changes in lighting conditions, equipment position, etc. Regardless of how the camera installation position changes, the system can automatically adjust the parameters, that is, automatically calculate the distance between the camera and the support guard plate, the deflection angle of the camera relative to the support guard plate, and realize accurate frame number calculation, so this application can be applied to different fully mechanized mining working face environments.

[0178] 3. This application can effectively improve the operating efficiency and safety of the comprehensive mining face through automated and accurate frame number calculation, and reduce the risks caused by manual identification errors.

[0179] 4. It has been verified that the system delay of this application is less than 10ms (on Intel(R) Xeon(R) Gold 5218R CPU), achieving near real-time frame number recognition and display, and being able to process and feedback recognition results in real time, providing timely support and support plate information to operators or automatic control systems, which is conducive to the efficient management of comprehensive mining working faces.

[0180] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above method is implemented when the processor executes the program.

[0181] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program for executing the above method.

[0182] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the method described in the claim when the computer program / instruction is executed by a processor.

[0183] like Fig. 9 As shown, the electronic device 600 may further include: a communication module 110, an input unit 120, an audio processor 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily have to include Fig. 9 In addition, the electronic device 600 may also include Fig. 9 For components not shown, reference may be made to the prior art.

[0184] like Fig. 9 As shown, the central processor 100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The central processor 100 receives inputs and controls the operations of various components of the electronic device 600.

[0185] The memory 140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The information related to the hydraulic support control method may be stored, and a program for executing the related information may also be stored. The central processing unit 100 may execute the program stored in the memory 140 to implement information storage or processing.

[0186] The input unit 120 provides input to the CPU 100. The input unit 120 is, for example, a key or a touch input device. The power supply 170 is used to provide power to the electronic device 600. The display 160 is used to display display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0187] The memory 140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 140 may also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 600 through the central processor 100.

[0188] The memory 140 may also include a data storage unit 143 for storing data, such as digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0189] The communication module 110 is a transmitter / receiver that transmits and receives signals via the antenna 111. The communication module 110 (transmitter / receiver) is coupled to the central processor 100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0190] Based on different communication technologies, multiple communication modules 110 may be provided in the same electronic device, such as low-power wireless communication modules such as Radio Frequency Identification (RFID), Ultra-WideBand (UWB), ZigBee, and Bluetooth. The communication module 110 (transmitter / receiver) is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide an audio output via the speaker 131 and receive an audio input from the microphone 132, thereby realizing a common telecommunication function. The audio processor 130 may include any suitable buffer, decoder, amplifier, and the like. In addition, the audio processor 130 is also coupled to the central processor 100, so that recording can be performed on the local machine through the microphone 132, and the sound stored on the local machine can be played through the speaker 131.

[0191] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0195] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for determining the position of a support plate and a support number of a support, characterized in that: The method comprises: Use the camera installed on the fully mechanized mining face to obtain the image of the support guard plate; Obtaining intrinsic parameters, extrinsic parameters and distortion parameters of the camera; Based on the intrinsic parameters, extrinsic parameters and distortion parameters of the camera, the three-dimensional world coordinates of the support guard plate are converted into a camera coordinate system, an image coordinate system and a pixel coordinate system in sequence; Use a deep learning model to identify the pixel coordinates of the support plate in the image; Determine the actual position of the support plate of the support in the image based on the pixel coordinates; Based on the actual position, the frame number of the support guard plate is determined through a position-frame number mapping table.

2. The method for determining the position of the support plate and the support number of the support as claimed in claim 1, characterized in that: The intrinsic parameters of the camera are an intrinsic parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the extrinsic parameters of the camera include the horizontal rotation angle and the pitch angle; the distortion parameters include the radial distortion coefficient and the tangential distortion coefficient.

3. The method for determining the position of the support plate and the support number of the support as claimed in claim 1, characterized in that: The method of sequentially converting the three-dimensional world coordinates of the support support plate into a camera coordinate system, an image coordinate system, and a pixel coordinate system based on the intrinsic parameters, extrinsic parameters, and distortion parameters of the camera includes: Establishing a world coordinate system with the camera center as the origin; Converting the world coordinate system into a camera coordinate system based on the external reference; Converting the camera coordinate system into an image coordinate system using the intrinsic parameters; The image coordinate system is converted into a pixel coordinate system, and distortion correction is performed based on the distortion parameters.

4. The method for determining the position of the support plate and the support number of the support as claimed in claim 1, characterized in that: The method of using a deep learning model to identify the pixel coordinates of the support plate of the support in the image includes: using a DeepLabv3+ semantic segmentation model to identify the pixel coordinates of the support plate of the support in the image.

5. The method for determining the position of the support plate and the support number of the support as claimed in claim 4, characterized in that: The method also includes: based on the dynamic changes of the fully mechanized mining working face, using the DeepLabv3+ semantic segmentation algorithm to identify the results and dynamically adjust the distance and deflection angle parameters between the camera and the support guard plate.

6. The method for determining the position of the support plate and the support number of the support as claimed in claim 5, characterized in that: The method of dynamically adjusting the distance and deflection angle parameters between the camera and the support guard plate by using the DeepLabv3+ semantic segmentation algorithm includes: Based on the pixel coordinates of the support plate of the stent in the image identified by DeepLabv3+, a contour of the support plate of the stent is formed; On the outline of the support plate of the bracket, a corner detection algorithm is used to identify and extract key feature points; Calculate the three-dimensional coordinates of the key feature point in the world coordinate system using the intrinsic parameters and extrinsic parameters of the camera and the pixel coordinates of the key feature point; Calculate the distance between the camera and the support plate of the bracket using the three-dimensional coordinates of the key feature points; The deflection angle of the camera relative to the support guard plate is calculated using the external parameters of the camera and the three-dimensional coordinates of the support guard plate.

7. A device for determining the position of a support plate and a support number of a support, characterized in that: The device comprises: An image acquisition unit, used for acquiring an image of the support guard plate by using a camera installed on the fully mechanized mining working face; A parameter acquisition unit, used to acquire the intrinsic parameters, extrinsic parameters and distortion parameters of the camera; A coordinate conversion unit, used for converting the three-dimensional world coordinates of the support guard plate into a camera coordinate system, an image coordinate system, and a pixel coordinate system in sequence based on the intrinsic parameters, extrinsic parameters, and distortion parameters of the camera; A guard plate recognition unit, used to recognize the pixel coordinates of the support guard plate in the image using a deep learning model; An actual position determination unit, used for determining the actual position of the support plate of the support in the image based on the pixel coordinates; The frame number determining unit is used to determine the frame number of the support guard plate based on the actual position through a position-frame number mapping table.

8. The device for determining the position of the support plate and the support number of the support as claimed in claim 7, characterized in that: The intrinsic parameters of the camera are an intrinsic parameter matrix, including the focal length of the camera in the x direction, the focal length of the camera in the y direction, the coordinates of the image principal point in the x-axis direction, and the coordinates of the image principal point in the y-axis direction; the extrinsic parameters of the camera include the horizontal rotation angle and the pitch angle; the distortion parameters include the radial distortion coefficient and the tangential distortion coefficient.

9. The device for determining the position of the support plate and the support number of the support as claimed in claim 8, characterized in that: The coordinate conversion unit comprises: A coordinate system establishment module, used to establish a world coordinate system with the camera center as the origin; A first conversion module, configured to convert the world coordinate system into a camera coordinate system based on the external reference; A second conversion module, used to convert the camera coordinate system into an image coordinate system using the intrinsic reference; The third conversion module is used to convert the image coordinate system into a pixel coordinate system and perform distortion correction based on the distortion parameters.

10. The device for determining the position of the support plate and the support number of the support as claimed in claim 7, characterized in that: The support plate recognition unit is specifically used to: use the DeepLabv3+ semantic segmentation algorithm to recognize the pixel coordinates of the support plate of the stent in the image.

11. The device for determining the position of the support plate and the support number of the support as claimed in claim 10, characterized in that: The device also includes: an adaptive unit, which is used to dynamically adjust the distance and deflection angle parameters between the camera and the support guard plate based on the dynamic changes of the comprehensive mining working face using the DeepLabv3+ semantic segmentation algorithm.

12. The device for determining the position of the support plate and the support number of the support as claimed in claim 11, characterized in that: The adaptive unit comprises: A contour acquisition module, used to form the contour of the stent guard plate based on the pixel coordinates of the stent guard plate in the image identified by the DeepLabv3+ semantic segmentation algorithm; A feature point acquisition module is used to identify and extract key feature points on the outline of the support plate of the bracket using a corner point detection algorithm; A three-dimensional coordinate calculation module, used to calculate the three-dimensional coordinates of the key feature point in the world coordinate system by using the intrinsic parameters and extrinsic parameters of the camera and the pixel coordinates of the key feature point; A distance acquisition module, used to calculate the distance between the camera and the support plate of the bracket using the three-dimensional coordinates of the key feature points; The deflection angle acquisition module is used to calculate the deflection angle of the camera relative to the support guard plate by using the external parameters of the camera and the three-dimensional coordinates of the support guard plate.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images