An intelligent method and device for ore positioning and crushing control

Abstract

This invention provides an intelligent ore positioning and crushing control method and apparatus. The method includes: simultaneously capturing images of a mine pass area using two binocular cameras; identifying large ore and a breaker hammer using a deep learning instance segmentation algorithm; extracting matching feature points from the target image areas of the identified large ore and breaker hammer, and reconstructing a 3D point cloud from the matching feature points; calculating the crushing point location of the large ore; calculating the motion information for moving the breaker hammer to the crushing point of the large ore; sending a movement control signal to the crusher's electronic control device based on the motion information to control the crusher to move the breaker hammer to the crushing point of the large ore; and sending a crushing control signal to perform the crushing operation on the large ore. The intelligent ore positioning and crushing control method and apparatus provided by this invention achieves rapid and stable operation, a wide visual detection range, and high image resolution, ensuring precise positioning and efficient secondary crushing of ore in underground mine passes.

Description

Technical Field

[0001] This invention relates to the field of ore crushing technology in mining, and more specifically, to an intelligent ore positioning and crushing control method and device. Background Technology

[0002] Mineral resources, as important non-renewable resources, play a vital role in national economic development. However, many mining companies are currently facing multiple challenges, including difficulty in recruiting workers, high costs, slow profit growth, and insufficient protection of personnel safety.

[0003] Ore crushing is a crucial operation in mining operations, and it is conducted in multiple stages. The first stage of crushing uses blasting, and its output largely depends on the particle size distribution of the ore. The second stage of crushing typically involves using a crusher to break down large pieces of ore that are blocked by metal screens. The metal screens can block large pieces of ore, and the crusher breaks them down into smaller pieces that flow into the subsequent ore transport process, preventing blockages in the transport.

[0004] In existing technologies, secondary crushing operations at underground ore chutes mostly involve workers operating crushers on-site to crush large ores blocked by screens. A very small number of these operations utilize remote control of the crushers. On-site operation of crushers by underground workers presents a harsh working environment, with noise and dust significantly impacting worker health, especially in deep mines where worker safety is compromised. Remote control, a semi-automated approach, allows workers to manually control the crusher from the surface by watching video feeds from cameras at the ore chute. However, this method suffers from significant signal delays and frequent interruptions. Delayed video feedback can occasionally lead to damage to the ore chute screens by the breaker hammers, severely impacting operator comfort and ultimately hindering production efficiency.

[0005] Because each ore piece varies in size and shape, and multiple pieces are often stacked, traditional machine vision methods are insufficient for precise detection and positioning. Furthermore, the large size of the ore pass, sometimes exceeding 10 meters in length and width, makes accurate ore identification and positioning a significant challenge. Ordinary depth cameras often fall short in terms of positioning accuracy and detection range. Using conventional depth cameras and 3D deep learning for ore identification suffers from insufficient accuracy, low image resolution, limited recognition angle and distance, and high computational demands, resulting in long system response times and failing to meet the requirements for real-time crusher control.

[0006] Therefore, to address the above shortcomings, there is a need to provide an intelligent ore positioning and crushing control method and device to achieve rapid and stable operation, wide visual detection range, and high image resolution, ensuring accurate ore positioning and efficient and rapid secondary crushing at underground mine ore passes, and creating a digital mine working scenario for unmanned management of fully mechanized mining faces. Summary of the Invention

[0007] This invention provides an intelligent ore positioning and crushing control method and device, which achieves rapid and stable operation, wide visual detection range, and high image resolution, ensuring accurate ore positioning and efficient and rapid secondary crushing at underground mine ore passes, and creating a digital mine working scenario for unmanned management of fully mechanized mining faces.

[0008] An embodiment of the first aspect of the present invention provides an intelligent ore positioning and crushing control method, comprising the following steps: simultaneously capturing images of a pass area using two binocular cameras; identifying large ore and hydraulic breakers by using a deep learning instance segmentation algorithm; extracting matching feature points from the target image regions of the identified large ore and hydraulic breakers, and reconstructing a three-dimensional point cloud from the matching feature points; calculating the crushing point location of the large ore based on the three-dimensional point cloud reconstruction data of the large ore; calculating the action information of the crusher moving the hydraulic breaker to the crushing point of the large ore based on the three-dimensional point cloud reconstruction data of the hydraulic breaker; sending a movement control signal to the electronic control device of the crusher based on the action information to control the crusher to move the hydraulic breaker to the crushing point of the large ore; and sending a crushing control signal after the hydraulic breaker reaches the crushing point of the large ore to perform crushing operations on the large ore.

[0009] Preferably, the image sensors of the two binocular cameras have a resolution of not less than 5 million pixels, both binocular cameras are installed near the chute area, and the installation distance between the two binocular cameras is not less than 1 meter.

[0010] Preferably, in the deep learning instance segmentation algorithm, an improved SOLOV2 detection model is used to identify large ore and hydraulic breakers. The improvements to the SOLOV2 detection model include at least one of the following: i. Dividing the image input from the camera into multiple sub-regions, determining the regions of interest and regions of non-interest, inputting the regions of interest into the neural network for deep processing, and discarding image information from the regions of non-interest; ii. Using MobileNetV3 as the backbone network; iii. During model training, the training process is divided into three stages based on the learning rate: In the first stage, the learning rate increases continuously with the number of training iterations; in the second stage, the learning rate remains stable without changing with the number of training iterations; in the third stage, the learning rate is inversely proportional to the number of training iterations, decreasing as the number of training iterations increases, with the rate of decrease being initially rapid and then slowing down, until the learning rate approaches 0 when the number of training iterations increases infinitely close to positive infinity; iv. The feature pyramid is set to three layers; v. The loss function for the masked branch uses the BCE Loss function combined with the Sigmoid activation function; where the specific calculation formula for the BCE Loss function is:

[0011]

[0012] Where o represents output, which is the probability of judging a certain item as true calculated after the network passes through the forward propagation, and its value ranges from [0,1]; t represents target, which is the label in the sample that has been marked as true or false, and its value is 0 or 1; i represents the i-th item among the n output items to be judged, and o[i] and t[i] represent the calculated true value and the labeled true value corresponding to the i-th item; the specific calculation formula of the BCE Loss function can also be:

[0013]

[0014] Where weight is a matrix of the same size as o or t; vi. In the non-maximum suppression stage, the SoftNMS function is introduced, and its specific function expression is as follows:

[0015]

[0016] Where iou is the intersection-to-union ratio, which is the ratio of the area of ​​the intersection to the area of ​​the union of the predicted ranges of two closely spaced objects; M is a predicted range to be compared; b i Let s be the i-th prediction range to be compared near M, and D be the set of prediction ranges that have been determined to be retained. i The confidence score for the original prediction range is S, and the adjusted confidence score is S. i σ is an adjustment range parameter that can be set manually.

[0017] Preferably, the captured images are identified using a deep learning instance segmentation algorithm to identify large ore and hydraulic breakers. Specifically, this involves the following steps: capturing target images and labeling the large ore and hydraulic breakers in the images; selecting whether to set key regions based on the labeling; performing preprocessing and data augmentation based on the selection of key regions; preprocessing including filtering and noise reduction, and data augmentation including rotation, translation, and scaling operations; selecting K-fold cross-validation or dividing the training and validation datasets according to the dataset size; setting hyperparameters such as learning rate and number of training iterations; calling the training method function to train the network model and monitoring the training progress; using the trained model if the training results meet expectations; exporting the model and deploying it in an industrial control computer environment; inputting the images via data stream and performing preprocessing, then inputting the images into the model for inference; and returning the post-processed data output by the model to the branch process that called instance segmentation.

[0018] Preferably, matching feature points are extracted from the identified target image regions of large ore and hydraulic breakers, and 3D point cloud reconstruction is performed on the matching feature points. Specifically, this includes the following steps: applying Gaussian blurring to the image at different scales, then downsampling to construct a Gaussian scale space; generating a Gaussian difference pyramid by subtracting each adjacent layer of the pyramid; detecting extreme points for each pixel; assigning orientations to key points, with each key point corresponding to a reference orientation, and using image gradient methods to determine the stable orientation of the local structure; using histograms to statistically analyze the gradients and orientations of pixels in the neighborhood, selecting the most frequent gradient as the principal orientation of the key point; finally, each key point contains three pieces of information—position, scale, and orientation—to describe the image features; and performing scale-invariant processing on the target within the sub-region of interest. After feature extraction, the optimal matching point pairs are selected using a fast nearest neighbor matching algorithm. Utilizing the principle of binocular stereo vision and combining the projection relationship of the imaging target in 3D space, the 3D camera coordinates of the feature matching points in each frame of the two binocular cameras are calculated. The 3D coordinates of the hydraulic breaker in the coordinate systems of the two binocular cameras are then determined. Hand-eye calibration is performed by adding Q marker points at different locations in the workspace of the crusher. Based on the crusher and camera coordinates corresponding to the Q marker points, the least squares method is used to obtain the spatial transformation matrix of the two binocular cameras in the crusher coordinate system. The 3D point clouds corresponding to the two binocular cameras are then transformed into two sets of corresponding 3D point clouds in the crusher coordinate system using the spatial transformation matrix. Their union is taken as the final 3D point cloud in the crusher coordinate system, where Q≥1.

[0019] Preferably, the location of the crushing point of the large ore is calculated based on the three-dimensional point cloud reconstruction data of the large ore, specifically including the following steps: obtaining the outline information of the large ore based on the three-dimensional point cloud coordinate information of the large ore in the crusher coordinate system, and calculating the centroid of the large ore based on the outline information, with the centroid point p... aimThe calculation is as follows: Let the three-dimensional coordinates of the large ore point cloud in the crusher coordinate system be P = {p1, p2, p3, ... p n},but,

[0020]

[0021] Select the x-axis, y-axis coordinates and p. aim The point with the largest z-axis value within the range where the x-axis and y-axis coordinates are both less than a set threshold is the break point.

[0022] Preferably, in the step of calculating the action information of the crusher moving the breaker to the crushing point of the large ore based on the 3D point cloud reconstruction data of the breaker, the large ore is sorted according to the crushing point coordinates of each large ore, and the point closer to the breaker is selected as the first large ore to be crushed. Based on the 3D coordinate data of the first large ore crushing point, the crushing order of the remaining large ore is sorted according to the principle of minimizing the energy consumption of the crusher. After sorting, the movement path of the breaker is calculated based on the 3D coordinate data of the first large ore and the breaker, and the motion control data of each arm of the breaker is generated according to the movement path.

[0023] Preferably, in the step of sending a movement control signal to the electronic control device of the crusher based on the action information, and controlling the crusher to move the breaker hammer to the crushing point of the large ore, the movement control signal is sent to the electronic control device of the crusher via a CAN bus interface or an RJ45 network interface.

[0024] Preferably, when the breaker hammer reaches the crushing point of the large ore, a crushing control signal is sent to perform the crushing operation on the large ore. When the distance between the breaker hammer and the selected large ore crushing point is calculated to be less than a set threshold, it is determined that the breaker hammer has reached the crushing point. A crushing control signal is then sent to the electronic control device of the crusher to enable the breaker hammer to perform the crushing operation on the large ore until the large ore is crushed into small pieces that automatically fall from the screen of the chute.

[0025] A second aspect of the present invention also provides an intelligent ore positioning and crushing control device, comprising: two binocular cameras configured to simultaneously capture images of a pass area; an identification module configured to identify large ore and a breaker hammer by using a deep learning instance segmentation algorithm; a three-dimensional point cloud reconstruction module configured to extract matching feature points from the identified target image areas of the large ore and the breaker hammer, and to perform three-dimensional point cloud reconstruction on the matching feature points; and a position determination module configured to... The system calculates the location of the crushing point of the large ore based on the 3D point cloud reconstruction data of the breaker hammer; the action determination module is configured to calculate the action information of the crusher to move the breaker hammer to the crushing point of the large ore based on the 3D point cloud reconstruction data of the breaker hammer; the movement control module is configured to send a movement control signal to the electronic control device of the crusher based on the action information, and control the crusher to move the breaker hammer to the crushing point of the large ore; the crushing control module is configured to send a crushing control signal after the breaker hammer reaches the crushing point of the large ore, and to carry out the crushing operation on the large ore.

[0026] A third aspect of the present invention also provides an intelligent ore positioning and crushing control device, which includes a memory and a processor: wherein the memory is used to store executable program code; and the processor is used to read the executable program code stored in the memory to execute the intelligent ore positioning and crushing control method.

[0027] A fourth aspect of the present invention also provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements an intelligent ore positioning and crushing control method.

[0028] The intelligent ore positioning and crushing control method, device, equipment, and computer-readable storage medium provided by this invention achieve rapid and stable operation, wide visual detection range, and high image resolution, ensuring accurate ore positioning and efficient and rapid secondary crushing at underground mine ore passes, and creating a digital mine working scenario for unmanned management of fully mechanized mining faces.

[0029] Additional aspects and advantages of the invention will become apparent in the following description or may be learned by practice of the invention. Attached Figure Description

[0030] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0031] Figure 1 A flowchart illustrating an intelligent ore positioning and crushing control method according to a first embodiment of the present invention is shown.

[0032] Figure 2 A schematic diagram of the information flow of an intelligent ore positioning and crushing control device according to a second embodiment of the present invention is shown.

[0033] Figure 3 A diagram showing the composition of an intelligent ore positioning and crushing control device according to a second embodiment of the present invention is provided.

[0034] Figure 4 A structural diagram of an intelligent ore positioning and crushing control device according to a second embodiment of the present invention is shown.

[0035] Figure 5 A structural diagram of an intelligent ore positioning and crushing control device according to a third embodiment of the present invention is shown.

[0036] Figure 6 A structural diagram of a computer-readable storage medium illustrating an intelligent ore positioning and crushing control method according to a fourth embodiment of the present invention is shown. Detailed Implementation

[0037] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0038] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0039] The following discussion provides several embodiments of the invention. While each embodiment represents a single combination of the invention, different embodiments of the invention can be substituted or combined, and therefore the invention can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes A, B, and C, and another embodiment includes a combination of B and D, then the invention should also be considered to include embodiments containing one or more other all other possible combinations of A, B, C, and D, even if such embodiments are not explicitly described in the following text.

[0040] Figure 1 A flowchart illustrating an intelligent ore positioning and crushing control method according to a first embodiment of the present invention is shown. Figure 1 As shown, the method includes the following steps:

[0041] Step S01: Simultaneously photograph the ore pass area using two binocular cameras;

[0042] Step S02: The captured images are identified using a deep learning instance segmentation algorithm to identify large ore and hydraulic breakers;

[0043] Step S03: Extract matching feature points from the target image regions of the identified large ore and hydraulic breaker, and reconstruct three-dimensional point cloud from the matching feature points;

[0044] Step S04: Calculate the location of the breakage point of the large ore based on the 3D point cloud reconstruction data of the large ore;

[0045] Step S05: Calculate the action information of the crusher moving the breaker to the crushing point of the large ore based on the 3D point cloud reconstruction data of the breaker.

[0046] Step S06: Send a movement control signal to the electric control device of the crusher according to the action information to control the crusher to move the breaker hammer to the crushing point of the large ore;

[0047] Step S07: After the breaker reaches the crushing point of the large ore, a crushing control signal is sent to crush the large ore.

[0048] The above embodiments of the present invention involve installing two high-resolution binocular cameras near the ore pass screen. These cameras capture images of the ore pass area. A deep learning network model is used to segment and identify instances in the captured images, extracting large ore blocks obstructed by the ore pass screen and the crusher's breaker hammer. Matching feature points are extracted from the detected large ore and breaker hammer target image areas, and 3D point cloud reconstruction is performed on the matched feature points. The 3D point cloud coordinates based on the individual coordinate systems of the two binocular cameras are then transformed to a unified crusher coordinate system. Based on the ore point cloud coordinates in the crusher coordinate system, the ore crushing point location is extracted. Appropriate control signals are sent to the crusher's electronic control device to control the crusher to move the breaker hammer to the large ore crushing point for crushing. Using two binocular cameras effectively solves the problem of insufficient ore positioning accuracy of ordinary binocular depth cameras over large areas, and also effectively solves the problem of breaker hammer and ore occlusion. The method of identification followed by positioning effectively solves the problem of insufficient accuracy in ore positioning over large areas. Figure 3 The computational complexity of point cloud extraction for target recognition and localization is a significant issue.

[0049] In the first embodiment of this invention, the improved SOLOv2 detection model is used in the deep learning instance segmentation algorithm to identify large ore and breakers. The improved SOLOv2 instance segmentation model reduces the time consumption for target identification while improving the accuracy of target identification, especially for identifying individual ore pieces in a stacked ore pile. The use of two high-resolution binocular cameras and a method of identification followed by localization enables the entire system to perform stable and smooth intelligent crushing operations in real-time at a rate of no less than 10 frames per second while maintaining high-precision localization. The first embodiment of this invention has the advantages of simple structure, high identification accuracy, high positioning accuracy, anti-occlusion, strong real-time performance, and high crushing efficiency, and can effectively solve the problem of intelligent unmanned operation of secondary crushing of large ore in underground ore ore passes.

[0050] In the first embodiment of this invention, two high-resolution binocular cameras with image sensor pixels of no less than 5 million pixels are used. The two binocular cameras are installed near the ore pass screen, with a spacing of no less than 1 meter between them. Installing the two high-resolution binocular cameras near the ore pass screen effectively solves the problem of insufficient positioning accuracy over a large area, and also effectively solves the problem of ore and breaker hammer obstructing each other and affecting image data analysis. Before installation underground, the two high-resolution binocular cameras are stereo calibrated using a binocular stereo calibration method to obtain their respective intrinsic and extrinsic parameters and distortion coefficients. After installation at the underground ore pass, the two high-resolution binocular cameras are calibrated using a common nine-point calibration method based on the breaker coordinate system, obtaining the transformation matrix of the two binocular cameras based on the breaker.

[0051] In the first embodiment of this invention, an improved SOLOv2 deep learning instance segmentation model is used for real-time identification of ore and hydraulic breakers. Based on the ore pass scenario, with the goals of reducing single-frame latency and ensuring accuracy, improvements were made to SOLOv2 in several aspects, including preprocessing, backbone network, learning rate, FPN (Feature Pyramid), Loss function, and NMS (Non-maximum suppression). Based on the improved SOLOv2 detection model, not only is the accuracy of ore and hydraulic breaker identification at the ore pass improved, especially the accuracy of identifying individual ore pieces in a stacked ore pile, but the time consumption during identification is also reduced, ensuring the real-time processing of high-speed, large-data streams. Generating the ore and hydraulic breaker identification model based on the SOLOv2 model requires training on a sample set acquired by a stereo camera. Considering factors such as illumination, time, ore distribution, local occlusion, and surrounding conditions, at least 10,000 relevant samples were collected for model training. First, image enhancement, image denoising, and similarity transformation were performed on the images. Then, the ore and hydraulic breakers in the images were accurately labeled. Next, the labeled dataset was used to train and verify the accuracy. Finally, the model was deployed to the system for instance segmentation.

[0052] Specifically, regarding preprocessing: In terms of speed, since most mine cameras are fixed in position, capturing objects of similar types with clear distinctions between static and dynamic objects, we adopted the concept of key regions to conserve computing resources. The images input from the cameras were divided into 16 4x4 regions. Only the selected regions of interest were input into the neural network for the most time-consuming depth processing, while image information from regions of no interest was discarded, ultimately improving detection efficiency. Regarding inference accuracy, we added marker points to the hydraulic breaker. A single image, multiple points for verification, and cross-correction improved the accuracy of detecting key target information on the hydraulic breaker in each instance.

[0053] Regarding the backbone network: To improve the inference speed of the network in real-world environments, we replaced the backbone network of the deep learning network with MobileNetV3. MobileNetV3 is a lightweight backbone network commonly used in relatively small devices such as mobile phones and CCTV cameras. These devices generally cannot accommodate high-performance computing devices like desktop computers. The emergence of MobileNetV3 is primarily to adapt to devices with limited computing power as much as possible without significantly sacrificing inference accuracy. Compared to the traditional ResNet family of backbone networks, MobileNetV3 has a much smaller overall parameter scale. This means that such a backbone network is easier to train, has faster inference speeds in deployed application scenarios, and conserves limited computing resources, making it more suitable for applications that require real-time detection in practical engineering environments.

[0054] Regarding the learning rate: During model training, because the scenes we identify are relatively simple and fixed, with few significant scene changes, the training and validation datasets appear very similar. After a large number of training iterations, the model is prone to overfitting, reducing its generalizability across different scenes. To improve this, we modified the learning rate setting, adopting a variable, dynamic learning rate. The entire training process is divided into three stages based on the learning rate. In the first stage, the learning rate increases continuously with the number of training iterations, mainly to quickly approach the convergence region of the loss function from an initial state. In the second stage, the learning rate remains stable without changing with the number of training iterations, in order to stably reach the vicinity of the convergence region. In the third stage, the learning rate has an inverse proportional relationship with the number of epochs (one generation of training), that is, as the number of epochs increases, the learning rate decreases, and the rate of decrease is initially rapid and then slows down. When the number of epochs increases infinitely close to positive infinity, the learning rate approaches 0. The purpose of this is to suppress the tendency for overfitting models to appear more likely as the number of training iterations increases. This is because in mining scenarios, a large number of training iterations can easily lead to model overfitting, and such overfitting models can sometimes perform well on multiple metrics, making it difficult to distinguish between overfitting models and truly excellent models.

[0055] Regarding Feature Pyramid Network (FPN): The main principle is to scale the image to different sizes, then use small-scale feature maps to detect large-scale objects within the same image, and vice versa. The main value of the FPN network lies in its ability to extract features at different scales from an image hierarchically, and then fuse these features across multiple scales to identify objects of the same type. Practice has proven it to be a crucial component of multi-scale object detection network models. The original SOLOv2 used five FPN layers, but there was overlap between these layers, primarily to provide a "double insurance" effect and minimize the missed detection rate of small objects in a single image. Considering the specific needs of mining applications, we improved it to three feature scales, focusing on the accuracy and speed of identifying large objects. Reducing the number of FPN layers helps decrease the total amount of data processed and transmitted by the network model, which is more helpful in improving prediction speed.

[0056] Regarding the loss function: To better adapt to multi-label classification scenarios, the loss function for the `mask` branch uses the BCE Loss function combined with the Sigmoid activation function. The specific calculation formula for the loss function is as follows:

[0057]

[0058] Where: o represents output, which refers to the probability of judging a certain item as true after the network has passed through forward propagation, and its value ranges from [0,1]; t represents target, which refers to the label in the sample that has been marked as true or false, and its value must be 0 or 1. I represents the i-th item among the n output items that need to be judged, and o[i] and t[i] represent the calculated true value and the labeled true value corresponding to the i-th item.

[0059] You can also specify weights, and more generally represent them as follows:

[0060]

[0061] Where: weight is a matrix of the same size as o or t; introducing weight allows for manual adjustment of the loss value of each loss function output according to its importance; more important parts correspond to larger weights, and less important parts correspond to smaller weights; if weight is set to all 1, it is equivalent to the above formula.

[0062] BCE Loss is a proven loss function that is well-suited for classification tasks. It is typically used for binary classification tasks, and can be applied to multi-class classification tasks after simple stacking. It performs well when the number of classes is relatively small.

[0063] Regarding Non-Maximum Suppression (NMS): NMS is mainly used in the post-processing stage after image inference to find the best result from multiple potentially correct inference results. However, often we need to manually set the IoU threshold for NMS, which can easily lead to either an excessively high value causing missed detections or an excessively low value suppressing correct calibrations. To improve the confidence setting issue, SoftNMS is introduced in the NMS stage, and its mathematical expression is as follows:

[0064]

[0065] Where: iou is the intersection-union ratio, which is the ratio of the area of ​​the intersection to the area of ​​the union of the predicted ranges of two closely spaced objects; M is a predicted range to be compared; bi is the i-th predicted range to be compared near M; D is the set of predicted ranges that have been determined to be retained; si is the confidence score of the original predicted range; Si is the confidence score after adjustment; σ is an adjustment range parameter that can be set manually.

[0066] SoftNMS establishes an inverse correlation between IoU and Si by adding a Gaussian function; that is, increasing IoU leads to decreasing Si. This inverse suppression helps to select results with less extreme confidence from the candidate boxes.

[0067] To generate an ore and breaker hammer recognition model based on the SOLOv2 model, training is required using a sample set collected by a stereo camera. Factors such as illumination, time, ore distribution, partial occlusion, and surrounding conditions must be considered, and at least ten thousand labeled sample images should be collected for model training.

[0068] In the first embodiment of the present invention, step S02 further includes the following specific steps:

[0069] Step S021: Capture the target image and label the large ore and hydraulic breakers in the image; capture a clear target image and accurately label the categories such as ore and hydraulic breakers in the image.

[0070] Step S022: Select whether to set key regions based on the annotation situation; perform preprocessing and data augmentation based on the selection of key regions. The preprocessing includes filtering and noise reduction, and the data augmentation includes rotation, translation, and scaling operations. Select K-fold cross-validation or divide the training dataset and validation dataset by a specified ratio based on the dataset size.

[0071] Step S023: Set hyperparameters such as learning rate and number of training iterations, call the training method function to train the network model, and monitor the training progress.

[0072] Step S024: If the training results meet expectations, then use the trained model.

[0073] Step S025: Export the model and deploy it in the industrial control computer environment.

[0074] Step S026: Input the image via data stream and perform preprocessing, then input it into the model for inference, and return the post-processed result data output by the model to the branch process that calls instance segmentation.

[0075] In the first embodiment of the present invention, step S03 further includes the following specific steps:

[0076] Step S031: Apply Gaussian blur to the image at different scales, then downsample to construct a Gaussian scale space. Based on this, subtract each adjacent two layers of the pyramid to generate a Gaussian difference pyramid.

[0077] Step S032: Perform extreme point detection for each pixel; assign directions to key points, with each key point corresponding to a reference direction, and use the image gradient method to find the stable direction of the local structure. Then, perform extreme point detection for each pixel. Assuming that each group of Gaussian pyramids has (S+3) layers, then each group of Gaussian difference pyramids has (S+2) layers. Only within the same group, the gradient values ​​of 26 points from the second layer to the second-to-last layer are compared to find the maximum or minimum value.

[0078] Step S033: Use histograms to count the gradient and direction of pixels in the neighborhood, and select the one with the most gradients as the main direction of the keypoint. Each keypoint contains three pieces of information: position, scale, and direction, describing the image features. Then, assign directions to the keypoints, with each keypoint corresponding to a reference direction. Use the image gradient method to find the stable direction of the local structure. Assuming the grayscale value of the sub-image in the pyramid at (x,y) in the image coordinate system is L(x,y), the formulas for calculating the gradient magnitude m(x,y) and direction θ(x,y) are as follows:

[0079]

[0080] θ(x,y)=tan -1 ((L(x,y+1)-L(x,y-1)) / L(x+1,y)-L(x-1,y))).

[0081] Step S034: After performing scale-invariant feature extraction on the target within the region of interest, the optimal matching point pair is selected using the Fast Nearest Neighbor (Flann) algorithm. Histograms are used to statistically analyze the gradient and orientation of pixels in the neighborhood, and the gradient with the most gradients is taken as the principal orientation of the keypoint. Each keypoint contains three pieces of information: location, scale, and orientation. The best overall description of the image features is obtained by combining these three information. After performing SIFT feature extraction on the target within the region of interest, the optimal matching point pair is selected using the Fast Nearest Neighbor (Flann) algorithm. Then, the accuracy of the matching point pair registration is further optimized using the KNN nearest neighbor classification algorithm.

[0082] Step S035: Using the principle of binocular stereo vision and combining the projection relationship of the imaging target in three-dimensional space, calculate the three-dimensional camera coordinates of the feature matching points in each frame of the two binocular cameras.

[0083] Step S036: Calculate the three-dimensional coordinates of the hydraulic breaker in the coordinate systems of the two binocular cameras.

[0084] Step S037: Hand-eye calibration is performed by adding Q marker points at different locations in the crusher's working space. Based on the crusher and camera coordinates corresponding to the Q marker points, the least squares method is used to obtain the spatial transformation matrices of the two binocular cameras in the crusher coordinate system. The 3D point clouds corresponding to the two binocular cameras are transformed into two sets of corresponding 3D point clouds in the crusher coordinate system using the spatial transformation matrices. The union of these matrices is taken as the final 3D point cloud in the crusher coordinate system, where Q≥1. In a preferred embodiment, hand-eye calibration is performed by adding 9 marker points at different locations in the crusher's working space. Based on the crusher and camera coordinates corresponding to the 9 points, the least squares method is used to obtain the spatial transformation matrices of the two binocular cameras in the crusher coordinate system. The 3D point clouds corresponding to the two binocular cameras are transformed into two sets of corresponding 3D point clouds in the crusher coordinate system using the transformation matrices. The union of these matrices is taken as the final 3D point cloud in the crusher coordinate system. Assuming the target's 3D coordinate points in the two stereo camera coordinate systems are C1, C2, and the corresponding 3D coordinate points in the crusher coordinate system are P1, P2, and the spatial transformation matrices between them are Rt1, Rt2, then the transformation process from camera 3D coordinates to crusher 3D coordinates is as follows:

[0085] P i =Rt i ·C i i = 1, 2;

[0086] The final point cloud data P of the ore in the crusher coordinate system is:

[0087] P = P1∪P2.

[0088] In the first embodiment of the present invention, the improved SOLOv2 instance segmentation algorithm is used to detect ore and breaker targets in each frame of the image. Then, SIFT feature point detection is performed on all ROIs (regions of interest) corresponding to the rocks and breaker in the detected binocular camera left and right images. Next, fast nearest neighbor matching (Flann) is used to initially filter matching point pairs in the left and right images, and then nearest neighbor classification (KNN) is used to further filter the most similar feature points. Since the ore crushing operation area at the ore pass is very large, the three-dimensional point cloud coordinates based on the two-dimensional feature points matched in the two images acquired by the two image sensors in the binocular camera are calculated based on the two-dimensional feature points of each binocular camera. Then, the three-dimensional point cloud coordinates in the coordinate systems of the two binocular cameras are transformed to the three-dimensional point cloud coordinates based on the coordinate system of the breaker using the coordinate transformation matrix of the two binocular cameras relative to the breaker. Finally, the unified target three-dimensional point cloud coordinates based on the breaker coordinate system are obtained by taking the union of the transformed three-dimensional point clouds.

[0089] In the first embodiment of the present invention, the method for acquiring a 3D point cloud using a single binocular camera is as follows:

[0090] Based on the prior calibration information and the image coordinates of the corresponding feature points in the binocular images, it is assumed that the pixel corresponds to the point (u). l ,v l ), (u r ,v r Assume the intrinsic parameter matrices of the stereo cameras are M and M respectively. l M r The extrinsic parameters are respectively Rt l Rt r The pixel coordinates of the left and right images were collected as follows: (u l ,v l ), (u r ,v r If the target's three-dimensional coordinates in the camera coordinate system are (X,Y,Z), then the projection relationship of the spatial imaging process can be expressed by the following formula:

[0091]

[0092]

[0093] From the dual-target calibration, the intrinsic and extrinsic parameter matrices of the camera can be obtained, expanded as follows, where f x ,f y The scaling factors are represented in the x and y directions, u0 and v0 are the image coordinates corresponding to the intersection of the imaging plane and the optical axis, and R in the extrinsic parameter matrix represents a 3*3 rotation matrix and t represents a 1*3 translation matrix.

[0094]

[0095]

[0096] Multiplying the intrinsic and extrinsic parameter matrices yields the projection matrix (4x3) of the target in the binocular camera imaging process, whose elements can be derived from P. i l, P i r,i = 00,01,...,23 indicates that all 12 elements here are constants, representing various influencing factors of spatial mapping relationships on the imaging results. These factors collectively affect the projection result from the world's three-dimensional coordinates to the image's two-dimensional coordinates. Their values ​​can be obtained by multiplying the intrinsic parameter matrix (3*3) and the extrinsic parameter matrix (4*4), and the specific solution is given in the implementation method. Based on the principle of stereo imaging, the three-dimensional coordinates (X,Y,Z) of the feature point pairs can be deduced. The camera imaging calculation formula is as follows:

[0097]

[0098]

[0099] Expanding the above two equations, we get

[0100]

[0101] Finally, the coordinates (X, Y, Z) of the spatial object point can be solved using the least squares method.

[0102] In the first embodiment of the present invention, ore contour information is obtained based on the three-dimensional point cloud coordinate information of the ore in the crusher coordinate system, and the centroid of the ore is calculated based on the contour information. Points with higher elevations near the centroid are selected as ore crushing points. Assume that the three-dimensional coordinates of the ore point cloud in the crusher coordinate system are P = {p1, p2, p3, ..., p...} n}, then the centroid p aim The calculation formula is as follows:

[0103]

[0104] Choose the x, y coordinates and p aim The point with the largest z-coordinate value within a range where both x and y coordinates are less than a manually set threshold is identified as the breakage point. Since the selection of breakage points is crucial in the crushing operation, we use two high-resolution binocular cameras to acquire images, and then use an improved SOLOv2 instance segmentation model to first identify and then locate the ore and the breaker hammer, effectively ensuring the accuracy of breakage point identification and location.

[0105] In the first embodiment of the present invention, when multiple ores exist in the ore pass, the system sorts them according to the coordinates of the crushing points of each ore, prioritizing the point closer to the breaker hammer as the first ore to be crushed. Then, based on the three-dimensional coordinate data of the first ore's crushing point and the principle of minimizing the energy consumption of the crusher, the system sorts the remaining ores in order of crushing. After sorting, the system calculates the movement path of the breaker hammer based on the three-dimensional coordinate data of the first ore and the breaker hammer, and generates motion control data for the operation of each arm of the breaker hammer based on the movement path.

[0106] In a preferred embodiment, step S05 includes the following steps:

[0107] Step S051: Sort the large ore according to the coordinates of the crushing points of each large ore, and select the points closer to the breaker hammer as the first large ore to be crushed. Based on the three-dimensional coordinate data of the crushing points of the first large ore, sort the remaining large ore in order of crushing according to the principle of minimizing the energy consumption of the crusher.

[0108] Step S052: After sorting, calculate the movement path of the breaker based on the three-dimensional coordinate data of the preferred large ore and the breaker, and generate motion control data for the operation of each arm of the breaker based on the movement path.

[0109] Specifically, multiple ore blocks in the 3D ore point cloud are sorted. In a preferred embodiment, the sorting algorithm prioritizes crushing ore blocks closer to the breaker hammer, and then crushes those blocks farther away. Having determined the start and end points of the crusher, this preferred embodiment generates an optimal path for the breaker hammer to perform its crushing action, achieving obstacle avoidance and path simplification, thus improving the crusher's efficiency. This preferred embodiment employs an A*-based 3D optimal path planning algorithm, ensuring the crushing head avoids collisions or taking invalid paths during its movement to the target point. A quadratic B-spline algorithm is used to smooth and optimize the path, overcoming vibrations during the robotic arm's movement. The optimal path is selected using the A* algorithm to crush the target ore. Let the estimation function F(n) of the current node x be defined as:

[0110] F(n) = g(n) + h(n)

[0111] Where g(n) is the actual distance the crushing head travels from the starting point to node x, and h(n) is the minimum distance estimate from the current node x to the crushing point. In this invention, h(n) is expressed as a Euclidean distance:

[0112]

[0113] Among them, (x n ,yn,z n (x) represents the current node of the breaker head's movement position. d ,y d ,z d Let ) represent the breaking point. Starting from the starting point, calculate the F(n) value of each of its child nodes. Select the child node with the smallest F(n) value as the next point in the search, iterating repeatedly until the next node becomes the target point. The obstacle avoidance path searched by the A* algorithm with a uniform step size consists of line segments. During the movement of the robotic arm, vibrations are inevitable. This invention uses a quadratic B-spline curve to smooth the optimal path, enabling the crusher to perform crushing operations stably. Let the control vertices be D1, D2, ..., D... n N i,k If (t) is a basis function of a (K-1) degree B-spline curve, then the mathematical expression of the K-order (K-1) degree B-spline curve is:

[0114]

[0115] The quadratic B-spline curve used in this invention is set such that each segment consists of 3 control points, and its basis matrix is ​​represented as follows:

[0116]

[0117] This preferred embodiment enables the crusher robotic arm to achieve obstacle avoidance and operate on the optimal path when performing crushing operations.

[0118] In the first embodiment of the present invention, the system sends control signals to the crusher's electrical control device via a CAN bus interface or an RJ45 network interface to control the operation of the crusher's hammer. The input interface of the crusher's electrical control device can be either a CAN interface or an RJ45 interface. To ensure compatibility with different crushers, the system's control signal output interface has both types of interfaces, and the appropriate control interface is selected based on the input interface of the crusher's electrical control device.

[0119] In the first embodiment of the present invention, after calculating the distance between the breaker and the selected ore crushing point coordinate data and finding that the distance between them is less than a set threshold, it is determined that the breaker has reached the crushing point. At this time, the system sends a control signal to the crusher's electrical control device to start the breaker crushing, so that the breaker crushes the ore until the large ore is crushed into small pieces and automatically falls from the chute screen.

[0120] Figure 2 A schematic diagram of the information flow of the intelligent ore positioning and crushing control device according to a second embodiment of the present invention is shown. As shown, two high-resolution binocular cameras are connected to a network switch via network cables. The network switch is connected to an industrial control computer via a network cable. The industrial control computer is connected to the electrical control device of the crusher via a CAN bus or an RJ45 network cable. The electrical control device is a unit component of the hydraulic control of the crusher. It can control the flow of hydraulic pressure in the crusher through electrical signals, thereby controlling the rotation of the crusher base, the extension and retraction of each arm, and the hammering of the breaker.

[0121] Figure 3 A diagram showing the composition of an intelligent ore positioning and crushing control device according to a second embodiment of the present invention is provided.

[0122] In this embodiment, the intelligent ore positioning and crushing control device includes: two binocular cameras configured to simultaneously capture images of the ore pass area; an identification module configured to identify the captured images using a deep learning instance segmentation algorithm, identifying large ore and a breaker hammer; a 3D point cloud reconstruction module configured to extract matching feature points from the identified target image areas of the large ore and the breaker hammer, and to perform 3D point cloud reconstruction on the matching feature points; a position determination module configured to calculate the crushing point position of the large ore based on the 3D point cloud reconstruction data of the large ore; an action determination module configured to calculate the action information of the crusher moving the breaker hammer to the crushing point of the large ore based on the 3D point cloud reconstruction data of the breaker hammer; a movement control module configured to send a movement control signal to the electronic control device of the crusher based on the action information, controlling the crusher to move the breaker hammer to the crushing point of the large ore; and a crushing control module configured to send a crushing control signal after the breaker hammer reaches the crushing point of the large ore, performing crushing operations on the large ore.

[0123] The system comprises two high-resolution binocular cameras and their data transmission interfaces, synchronously acquiring each frame of image data in real time. The recognition module includes a deep learning instance segmentation algorithm based on an improved SOLOv2 instance segmentation model, capable of simultaneously identifying and detecting ore and the breaker hammer. The 3D point cloud reconstruction module extracts and matches SIFT feature points from the ore and breaker hammer ROI regions in the data acquired by the two binocular cameras, reconstructs the 3D coordinates of the extracted matching 2D point pairs, and then transforms the 3D point cloud coordinates from the two binocular cameras to a unified 3D coordinate system under the crusher coordinate system. The position determination module calculates and obtains the ore's crushing point based on the ore point cloud coordinates in the crusher coordinate system. The motion determination module sorts the crushing order of multiple ore pieces and plans the optimal motion path from the breaker hammer to the target crushing point. The movement control module and crushing control module include a bus interface and device for sending crushing control signals to the crusher's electronic control unit. All modules are interconnected and operate efficiently, achieving real-time processing of image data at a rate of no less than 10 frames per second and outputting control information to the crusher.

[0124] The intelligent ore positioning and crushing control device described in the above embodiment can achieve the following effects: Unlike existing operating methods, whether underground workers control the crusher on-site or the ground remotely controls the underground ore pass crusher, it saves labor costs. Using the system of this invention, no worker intervention is required during operation; the equipment automatically completes the crushing operation. This not only frees underground operators from dusty and noisy working environments, especially those potentially life-threatening ones, but also avoids the problems of inefficient operation and low work efficiency associated with remote control. Regarding the challenge of real-time, large-space, accurate ore identification and positioning, especially the identification and positioning of individual ores in a stacked ore pile, we cleverly solved this problem by using two high-resolution binocular cameras combined with an improved SOLOv2 instance segmentation model, enabling the system to perform stable and smooth intelligent crushing operations in real-time at a speed of no less than 10 frames per second.

[0125] In one embodiment, two high-resolution binocular cameras are installed near the ore pass screen. Each camera has a high-performance processor, which can be a GPU, FPGA, or DSP processor; preferably, a high-performance FPGA processor with an integrated ARM core from Xilinx is used. In the binocular cameras, the images output from the image sensors are directly fed into the FPGA processor. Image preprocessing is performed in the FPGA, and then, using a combination of ARM processor and FPGA hardware acceleration, a deep learning network model is used within the camera to perform instance segmentation and recognition on the acquired images, extracting large ore blocks blocked by the ore pass screen and the breaker hammer of the crusher. Then, the processor uses the SIFT algorithm to extract two-dimensional feature points based on the target's ROI region. After matching the extracted two-dimensional feature points, the three-dimensional point cloud coordinates of the ore and the breaker hammer are reconstructed. Two binocular cameras transmit their calculated 3D point cloud coordinates to an industrial control computer via a network cable and a switch. The software running on the industrial control computer converts the 3D point cloud coordinates generated by the two cameras, based on their respective camera coordinate systems, to a unified crusher coordinate system. Then, based on the ore point cloud coordinates in the crusher coordinate system, the location of the ore crushing point is calculated. Finally, based on the ore crushing point and the 3D coordinate data of the breaker hammer, the path control information for the crusher to perform the crushing operation is calculated. The industrial control computer sends appropriate control signals to the crusher's electrical control device via a CAN bus interface or an RJ45 network interface, controlling the crusher to move the breaker hammer to the large ore crushing point to perform the crushing operation.

[0126] Figure 4 A structural diagram of an intelligent ore positioning and crushing control device according to a second embodiment of the present invention is shown.

[0127] In this embodiment, the intelligent ore positioning and crushing control device includes: two binocular cameras 1 and 2, configured to simultaneously capture images of the ore pass area; the two binocular cameras 1 and 2 are respectively connected to a switch 3 via network cables; the switch 3 is connected to an industrial control computer 4 via a network cable; and the industrial control computer is connected to the electrical control device of the crusher via a CAN bus or an RJ45 network cable; a recognition module, configured to recognize the captured images using a deep learning instance segmentation algorithm to identify the large ore 7 and the breaker hammer located on the grid screen 8; and a three-dimensional point cloud reconstruction module, configured to extract matching feature points from the identified target image areas of the large ore 7 and the breaker hammer, and to... The system includes: a feature point matching module for 3D point cloud reconstruction; a position determination module configured to calculate the crushing point position of the large ore 7 based on the 3D point cloud reconstruction data; an action determination module configured to calculate the action information for the crusher 6 to move the breaker hammer to the crushing point of the large ore 7 based on the 3D point cloud reconstruction data of the breaker hammer; a movement control module configured to send a movement control signal to the electronic control device 5 of the crusher 6 based on the action information, controlling the crusher 6 to move the breaker hammer to the crushing point of the large ore 7; and a crushing control module configured to send a crushing control signal after the breaker hammer reaches the crushing point of the large ore 7, to perform crushing operations on the large ore 7.

[0128] Figure 5 A structural diagram of an intelligent ore positioning and crushing control device according to a third embodiment of the present invention is shown. (Refer to the following...) Figure 5 It shows a structural schematic diagram of an intelligent ore positioning and crushing control device 300 suitable for implementing embodiments of the present disclosure.

[0129] The electronic devices in this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (e.g., vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0130] like Figure 5As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0131] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0132] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of embodiments of this disclosure.

[0133] Figure 6 A structural diagram of a computer-readable storage medium illustrating an intelligent ore positioning and crushing control method according to a fourth embodiment of the present invention is shown. Figure 6 As shown, the computer-readable storage medium 40 according to an embodiment of the present disclosure stores non-transitory computer-readable instructions 41 thereon. When the non-transitory computer-readable instructions 41 are executed by a processor, all or part of the steps of the intelligent ore positioning and crushing control method of the foregoing embodiments of the present disclosure are performed.

[0134] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0135] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0136] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: construct a base page, the page code of which is used to build the environment required for the operation of the business page and / or implement the same workflow abstracted from similar business scenarios; construct one or more page templates, the page templates being used to provide code templates for implementing business functions in the business scenario; based on the corresponding page templates, through code conversion of the specific functions of each page in the business scenario, generate the final page code for each page in the business scenario; and merge the generated final page code of each page into the page code of the base page to generate the code for the business page.

[0137] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0140] This invention provides an intelligent ore positioning and crushing control method, device, equipment, and computer-readable storage medium. It achieves rapid and stable operation, a wide visual detection range, and high image resolution, ensuring precise ore positioning and efficient secondary crushing in underground mine ore passes, creating a digital mine working scenario for unmanned management of fully mechanized mining faces. This invention uses two high-resolution binocular cameras instead of a single binocular camera. This is because the working range of the crusher in the ore pass area is relatively large, and a single binocular camera, including TOF cameras and LiDAR sensors, cannot meet the required positioning accuracy, especially for the crushing point. Using two high-resolution binocular cameras can basically meet the positioning accuracy requirements within a 5-centimeter range of the crushing point. Furthermore, two binocular cameras can effectively solve the problem of mutual occlusion between multiple ores and the crushing head, which cannot be solved by a single camera. During recognition, an improved SOLOv2 instance segmentation model is used, which can accurately obtain the edge contours of the ore and the crushing head. Unlike other target recognition and detection methods, which typically select the target using a bounding box without obtaining edge contour information, this approach can easily miss parts of other ores when encountering multiple ore stacks, or even identify multiple ores together as a single ore. The biggest advantage of using the SOLOv2 instance segmentation model is its accurate identification of individual ores within an ore stack, facilitating the localization of ore crushing points. For 3D point cloud reconstruction, we first identify the ore and crushing head, then extract feature points from their image regions, and finally reconstruct the 3D point cloud based on the matched feature points. This effectively reduces computational load while ensuring accurate ore and crushing head localization. This method allows the entire software algorithm to run within 60ms, ensuring the system achieves a processing speed of 10 frames per second for image processing and crusher control information output. Since the crusher operates in real-time, the real-time performance of the entire system must also be guaranteed.

[0141] In this invention, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance; the term "multiple" refers to two or more unless otherwise explicitly defined. The terms "install," "connect," "link," and "fix" should be interpreted broadly. For example, "connect" can be a fixed connection, a detachable connection, or an integral connection; "link" can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0142] In the description of this invention, it should be understood that the terms "upper" and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or unit referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0143] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0144] The above are merely some embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An intelligent ore positioning and crushing control method, comprising the following steps: S01: Two binocular cameras are used to simultaneously capture images of the ore pass area. After the two high-resolution binocular cameras are installed at the ore pass in the mine, the two binocular cameras are calibrated based on the crusher coordinate system using the common nine-point calibration method to obtain the transformation matrix of the two binocular cameras based on the crusher. S02: The captured images are identified using a deep learning instance segmentation algorithm, simultaneously identifying large ore and hydraulic breakers; this step specifically includes: S021: Capture the target image and label the large ore and hydraulic breaker in the image; S022: Select whether to set key regions based on the annotation situation, and perform preprocessing and data augmentation based on the selection of key regions; S023: Set the hyperparameters for learning rate and training iterations, call the training method function to train the network model, and monitor the training progress; S024: If the training results meet expectations, then use the trained model; S025: Export the model and complete its deployment in an industrial control computer environment; S026: Input the image via data stream and perform preprocessing, then input it into the model for inference, and return the post-processed result data output by the model to the branch process that calls instance segmentation; S03: Extract matching feature points from the target image regions of the identified large ore and the hydraulic breaker, and reconstruct a three-dimensional point cloud from the matching feature points; this step specifically includes: S031: Apply Gaussian blur to the image at different scales, then downsample to construct a Gaussian scale space. Based on this, subtract each adjacent two layers of the pyramid to generate a Gaussian difference pyramid. S032: Perform extreme point detection for each pixel; S033: Assign directions to key points, with each key point corresponding to a reference direction, and use the image gradient method to find the stable direction of the local structure; S034: Use histograms to count the gradient and orientation of pixels in the neighborhood, and take the one with the most gradients as the main orientation of the key point. Finally, each key point contains three pieces of information: position, scale, and orientation, to describe the image features. After performing scale-invariant feature extraction on the target in the sub-region of interest, the optimal matching point pair is selected by the fast nearest neighbor matching algorithm. S035: Calculate the 3D camera coordinates of the feature matching points in each frame of the two stereo cameras; S036: Determine the three-dimensional coordinates of the hydraulic breaker in the coordinate systems of the two binocular cameras; S037: Hand-eye calibration is performed by adding Q different marker points in the working space of the crusher. Based on the crusher and camera coordinates corresponding to the Q marker points, the least squares method is used to obtain the spatial transformation matrix of the two binocular cameras based on the crusher coordinate system. The three-dimensional point clouds corresponding to the two binocular cameras are transformed into two sets of corresponding three-dimensional point clouds in the crusher coordinate system through the spatial transformation matrix. The union of them is taken as the final three-dimensional point cloud in the crusher coordinate system, where Q≥1. S04: Calculate the location of the breakage point of the large ore based on the three-dimensional point cloud reconstruction data of the large ore. S05: Based on the three-dimensional point cloud reconstruction data of the hydraulic breaker, determine the current three-dimensional spatial position of the hydraulic breaker in real time, and calculate the action information of the crusher to move the hydraulic breaker to the crushing point of the large ore based on the position. S06: Based on the action information, a movement control signal is sent to the electronic control device of the crusher to control the crusher to move the breaker hammer to the crushing point of the large ore; S07: When the breaker reaches the crushing point of the large ore, it sends a crushing control signal to crush the large ore.

2. The intelligent ore positioning and crushing control method according to claim 1, characterized in that, In step S01, the image sensors of the two binocular cameras have a resolution of no less than 5 million pixels. Both binocular cameras are installed near the well pass area, and the installation distance between the two binocular cameras is no less than 1 meter. Before the two high-resolution binocular cameras are installed downhole, their respective binocular camera stereo calibration is completed by the binocular stereo calibration method to obtain the intrinsic and extrinsic parameters and distortion coefficients of the binocular cameras.

3. The intelligent ore positioning and crushing control method according to claim 1, characterized in that, In step S02, the deep learning instance segmentation algorithm uses an improved SOLOv2 detection model to identify large ore and hydraulic breakers; the improvement of the SOLOv2 detection model includes the following steps: i. Divide the image input from the camera into multiple sub-regions, determine the sub-regions of interest and the sub-regions of non-interest, input the sub-regions of interest into a neural network for deep processing, and discard the image information in the sub-regions of non-interest; ii. Use MobileNetV3 as the backbone network; iii. During the model training process, the training process is divided into three stages according to the different learning rates. In the first stage, the learning rate increases continuously with the number of training iterations. In the second stage, the learning rate does not change with the number of training iterations and remains stable. In the third stage, the learning rate has an inverse proportional relationship with the number of training iterations. As the number of training iterations increases, the learning rate decreases, and the rate of decrease is fast at first and then slows down. When the number of training iterations increases infinitely and approaches positive infinity, the learning rate approaches 0. iv. Set the feature pyramid to three layers; v. For the branch in the masked portion, the loss function uses the BCE Loss function combined with the Sigmoid activation function; where, The specific formula for calculating the BCE Loss function is as follows: Where o represents output, which is the probability of judging a certain item as true after the network passes through the forward propagation, and its value ranges from [0,1]; t represents target, which is the label in the sample that has been marked as true or false, and its value is 0 or 1; i represents the i-th item among the n output items that need to be judged, and o[i] and t[i] represent the calculated true value and the labeled true value corresponding to the i-th item; Alternatively, the specific formula for calculating the BCE Loss function is as follows: ； Wherein, weight is a matrix of the same size as o or t; vi. In the non-maximum suppression stage, the SoftNMS function is introduced, and its specific function expression is as follows: ； Where iou is the intersection-to-union ratio, which is the ratio of the area of ​​the intersection to the area of ​​the union of the predicted ranges of two closely spaced objects; M is a predicted range to be compared; b i Let s be the i-th prediction range to be compared near M, and D be the set of prediction ranges that have been determined to be retained. i The confidence score for the original prediction range is S, and the adjusted confidence score is S. i σ is the manually set adjustment range parameter.

4. The intelligent ore positioning and crushing control method according to claim 1, characterized in that, In the step of sending a movement control signal to the electrical control device of the crusher based on the action information, and controlling the crusher to move the breaker hammer to the crushing point of the large ore, the movement control signal is sent to the electrical control device of the crusher through a CAN bus interface or an RJ45 network interface.

5. The intelligent ore positioning and crushing control method according to claim 1, characterized in that, The preprocessing in step S022 includes filtering and noise reduction, and the data augmentation includes rotation, translation, and scaling operations; K-fold cross-validation is selected according to the size of the dataset, or the training dataset and validation dataset are divided into a specified ratio.

6. The intelligent ore positioning and crushing control method according to claim 1, characterized in that, Step S04 includes: Based on the three-dimensional point cloud coordinates of the large ore in the crusher coordinate system, the outline information of the large ore is obtained, and the centroid of the large ore is calculated based on the outline information. The centroid point paim is calculated as follows: Let the three-dimensional point cloud of the large ore be P={p1,p2,p3,…pn} in the crusher coordinate system. Then, ； The point with the largest z-axis value within the range where the x-axis and y-axis coordinates of the Paim point are both less than a set threshold is selected as the break point.

7. The intelligent ore positioning and crushing control method according to claim 1, characterized in that, Step S05 includes: S051: Sort the large ore according to the coordinates of the crushing points of each large ore, and select the points closer to the breaker hammer as the first large ore to be crushed. Based on the three-dimensional coordinate data of the crushing points of the first large ore, sort the crushing order of the remaining large ore according to the principle of minimizing the energy consumption of the crusher. S052: After sorting, calculate the movement path of the breaker based on the three-dimensional coordinate data of the preferred large ore and the breaker, and generate motion control data for the operation of each arm of the breaker based on the movement path.

8. The intelligent ore positioning and crushing control method according to any one of claims 1-7, characterized in that, In the step of sending a crushing control signal after the breaker hammer reaches the crushing point of the large ore to perform the crushing operation, when the distance between the breaker hammer and the selected large ore crushing point is calculated to be less than a set threshold, it is determined that the breaker hammer has reached the crushing point. The electronic control device of the crusher sends a crushing control signal to make the breaker hammer perform the crushing operation on the large ore until the large ore is crushed into small pieces of ore that automatically fall from the grid screen of the chute.

Images