System and method for rock fracturing determination

Abstract

The present invention pertains to a system and method to determine rock fracturing during the breaking process, specifically when using rock breakers in the mining industry. More precisely, the present invention relates to a system and a method that, using sensory information composed of images, point clouds, or both, automatically determines, in real time, whether a rock which a rock breaker is attempting to break is fractured or not.

Description

[0001] METHOD FOR ROCK FRACTURING DETERMINATION

[0002] FIELD OF THE INVENTION

[0003] The present invention relates to a system and a method for determining the fracturing of rocks during the rock-breaking process carried out by rock breakers utilized particularly in the mining industry. More precisely, the present invention relates to a system and a method that, using sensory information composed of images, point clouds or both of them, automatically determine, in real time, whether a rock which a rock breaker is attempting to break was fractured or not.

[0004] BACKGROUND

[0005] In different types of mining operations, rock fragmentation is a necessary process for the correct operation of the subsequent stages of mineral recovery. Commonly, this task is carried out through the use of rock breakers, also known as hydraulic hammers or impact hammers.

[0006] In underground mining, the process usually includes the extraction of the material from different extraction points and the transportation to the plant by means of LHD (Load-Haul-Dump) vehicles, which transport the material to discharge shafts or directly to a primary crusher. In both cases, the material falls on structures called transfer grates or grids, which retain rocks that are larger than the appropriate size for the next stage of the process. In order to pass the retained rocks through the grid, it is necessary to carry out a fragmentation process with rock breakers.

[0007] In the case of open pit mining, rock breakers are used in all primary crusher facilities, where rock breakers are used to reduce the size of the rocks that are not suitable for crushing.

[0008] In the state of the art, the most part of technologies are developed to remotely control rock breakers, i.e. using tele-operation, by means of a remote operating station located in a more comfortable and safer place. However, in the last years, the world trend has been to increase the automation of the mining operations, reducing to the minimum the human intervention, in order to provide more efficient methods and systems for the rock fragmentation process, taking advantage of the safety and better productivity associated with the automation. The determination of rock fracturing represents an essential component for the autonomous operation of rock breakers, because if the rock was actually fractured, then the rock breaker has to finish the fracturing process and go to a next stage in order to determine a new target rock.

[0009] An exemplary technology of the prior art is disclosed in the document by Lampinen, S., Niu L. et al., “Autonomous robotic rock breaking using a real-time 3D visual perception system”, Journal of Field Robotics, May 2021 , pag 980-1006. In this document it is proposed an autonomous breaker system that includes a visual perception system (VPS) capable of detecting multiple irregularly shaped rocks, a robotic control system featuring a decision making mechanism for determining the breaking order when dealing with multiple rocks, and a comprehensive manipulator control system. However, the authors recognize in the same document some shortcomings of the experimental setup proposed, in particular the matter related to the break instances. The authors have commented about “the naive approach for detecting break instances, or more specifically, the lack of such system”. Moreover, the solutions proposed in this document to overcome this shortcoming are based on external force estimations, as well as acceleration measurements.

[0010] In view of the above, it is noted that it is necessary to provide more efficient systems and methods for the rock fragmentation process, increasing the automation in rock breaker operations, particularly by providing a system and method to automatically determine fracturing during the rock-breaking process.

[0011] SUMMARY OF THE INVENTION

[0012] The invention refers to a system and a method for determining the fracturing of rocks during the rock-breaking process carried out by rock breakers. More particularly, the present invention relates to a system and method that, using sensory information composed of images, point clouds or both of them, automatically determine, in real time, whether a rock which a rock breaker is attempting to break was fractured or not.

[0013] The complete understanding of the invention may be obtained using the drawings, description and claims included in this application.

[0014] The proposed invention is based on the concept that the fracturing of a rock being hit by a rock breaker can be determined by observing changes in the rock's appearance using either images or point clouds, or both of them. When the rock is fractured, these changes in the rock's appearance can be observed, in the case of using images, mainly through the contour of the rock or the number of observed rocks, and in the case of using point clouds, these changes can be mainly observed through either its shape or the density of the points, or both of them.

[0015] In a first aspect of the invention, a rock fracture determination system is provided, which is able to implement a method to automatically determine, in real time, whether a rock which a rock breaker is attempting to break was fractured or not. The system is able to work using images provided by at least one monocular camera, point clouds provided by at least one range sensor, or using both images and point clouds.

[0016] The preferred embodiment of the rock fracture determination system comprises the following subsystems:

[0017] - a generation of processing boundaries subsystem, which provides a three-dimensional region that delimits which sensory information should be included in the processing, either point clouds, images or both of them;

[0018] - a visual rock segmentation subsystem, which segments the rocks contained on the image’s processing boundaries, in image coordinates, and generates masks for each segmented rock;

[0019] - a determination of regions of interest around the rock breaker’s end effector subsystem, which generates a three-dimensional region around the end effector, using information of the rock breaker’s configuration at a given time instant, and the direct kinematic model of the rock breaker;

[0020] - a rock characterization using point clouds subsystem, which process the point clouds that are to be used for the rock fracture determination system, at first filtering the points eliminating those points outside the processing delimitation regions and subsequently, using an encoder specially designed, generating a fixed-size representation, called PT-vector, which encapsulates the information contained in the original point cloud;

[0021] - a feature generation subsystem, which produces a set of features to be used for the determination of rock fractures, and that on the preferred embodiment computes two types of features: features based on segmentation masks and features computed using the PT-vector for point clouds; and - a fracture detection subsystem, which analyze the set of features previously generated, and in the preferred embodiment uses at least three statistical classifiers: one classifier for analyzing the features obtained by processing the image obtained by each camera, one classifier for analyzing the features obtained by processing the point cloud, and a third classifier, that makes a decision from the previous classifiers, and generates a Boolean signal that determines if the fracture of a rock took place or not.

[0022] In another aspect of the invention, a rock fracture determination method is provided, which using sensory information composed of images, point clouds or both of them, automatically determines, in real time, whether a rock which a rock breaker is attempting to break was fractured or not.

[0023] In a preferred implementation the method comprises acquiring data based on images provided by at least one monocular camera, and based on point clouds provided by at least one range sensor, or based on both of them.

[0024] In a preferred implementation, the method comprises additionally the steps of generating processing boundaries region represented as a three-dimensional region that delimits which sensory information should be included in the processing, either point clouds, images or both of them; segmenting the rocks contained on the image’s processing boundaries, in image coordinates, and generating masks for each segmented rock; determining a regions of interest around the rock breaker’s end effector subsystem, performing a rock characterization processing the point clouds, filtering and subsequently encoding them to generate a fixed-size representation, which encapsulates the information contained in the original point cloud; producing a set of features to be used for the determination of rock fractures, computing two types of features: features based on segmentation masks and features computed using point clouds; and analyzing the set of features previously generated, using at least three statistical classifiers: one classifier for analyzing the features obtained by processing the image obtained by each camera, one classifier for analyzing the features obtained by processing the point cloud, and a third classifier, that makes a decision from the previous classifiers, and generating a Boolean signal that determines if the fracture of a rock took place or not. DESCRIPTION OF THE DRAWINGS

[0025] FIG. 1 illustrates a general diagram of a preferred embodiment of the rock fracture determination system according to the present invention.

[0026] FIG. 2 illustrates examples of segmentation masks obtained when processing image regions within appropriate processing boundaries.

[0027] FIG. 3 illustrates a general diagram of the classification process realized by the fracture detection subsystem in a preferred embodiment of the rock fracture determination system.

[0028] FIG 4 illustrates the experimental setup implemented for an example embodiment. On the left, it is shown the Bobcat E10 mini-excavator, the grid, and the sensors installed on the roof. On the right, it is shown a detail of the mounting of the sensors: a MOXA camera on the upper part and a Velodyne LiDAR below the camera.

[0029] FIG. 5 illustrates an example of the result of applying an image-based effective operation area using the experimental setup. On the left side, it is shown an image provided by one of the MOXA cameras of the prototype and on the right side it is shown an area provided by the generation of processing boundaries subsystem of the example embodiment.

[0030] FIG. 6 illustrates an example of an image labeled to be part of the database used for the rock segmentation training process performed by the visual rock segmentation subsystem of the example embodiment.

[0031] FIG. 7 illustrates an example of the determination of the region of interest around the rock breaker’s end effector using point clouds. On the left side, it is shown a raw visualization of the data provided by the Velodyne LiDARs, unfiltered, and on the right side it is shown a visualization of that measurement when they are filtered using the 3D region of interest around the rock breaker’s end effector of the example embodiment. On the right side can also be visualized the front edges of the region of interest, which is represented as a cuboid.

[0032] FIG. 8 illustrates an example of the region of interest provided by the determination of regions of interest around the rock breaker’s end effector subsystem of the example embodiment. The projection of the region of interest is represented as a rectangle. DETAILED DESCRIPTION

[0033] The following is a detailed description of exemplary embodiments to illustrate the principles of the invention. The embodiments are provided to illustrate aspects of the invention, but the invention is not limited to any embodiment. The scope of the invention encompasses numerous alternatives, modifications and equivalents, only limited by the embodiments of the claims.

[0034] During a fracturing process using rock breakers, the rock breaker hits a rock in a determined pose previously determined, then a rock fracture determination system and method are required in order to automatically indicate to the rock breaker controller if the rock was fractured or not. If the rock was actually fractured, then the rock breaker has to finish the fracturing process and go to a next stage, in order to determine a new rock target pose.

[0035] In this manner, the present invention is referred to a rock fracture determination system which implements a method that, using sensory information composed of images, point clouds or both of them, automatically determines, in real time, whether a rock which a rock breaker is attempting to break was fractured or not.

[0036] In a particular embodiment the system could also be used in an assisted teleoperation process of rock breakers.

[0037] The system is able to work using images provided by at least one monocular camera, point clouds provided by at least one range sensor, or using both images and point clouds. As used herein, a point cloud refers to a set of 3D data points acquired by at least one range sensor, and the range sensors that can be used for this application may include 2D / 3D LIDARs, RADARs, and stereo cameras, among others.

[0038] The proposed invention is based on the concept that the fracturing of a rock being hit by a rock breaker can be determined by observing changes in the rock's appearance using either images or point clouds, or both of them.

[0039] In the case of using images, when the rock is fractured, changes in the rock's appearance can be observed mainly through the contour of the rock which will change or the number of observed rocks which will momentarily increase. In fact, in case a rock fractures completely, two or more rocks instead of only one will be observed. Thus, the fracturing of a rock can be detected by using features such as variations on the rock’s perimeter, variations on the rock’s area, and variations on the number of detected rocks among others, or any other features that can measure variations of the rock appearance associated with the rock fracturing.

[0040] In the case of using a point cloud, changes in the rock's appearance can be observed mainly through either its shape or the density of the points or both of them, which will change when the rock is fractured. Taking into account these changes some features associated are computed.

[0041] Once the described features are determined, as will be explained in detail below, statistical classifiers are used to analyze the features obtained in order to make a decision and generate a Boolean signal that determines if the fracture of a rock took place or not.

[0042] The preferred embodiment of the rock fracture determination system proposed comprises six subsystems. A general diagram of this preferred embodiment system and the relationships among the subsystems is illustrated in FIG. 1. In this manner, the preferred embodiment of the rock fracture determination system (10) comprises the following subsystems: generation of processing boundaries (11), visual rock segmentation (12), determination of regions of interest around the rock breaker’s end effector (13), rock characterization using point clouds (14), feature generation (15) and fracture detection (16).

[0043] Generation of processing boundaries (11)

[0044] This subsystem receives the area of effective operation of the rock breaker, and using this information, provides processing boundaries for the sensory information utilized by the rest of the rock fracture determination system (10). The area of effective operation corresponds to the three-dimensional region of interest in which rock fractures are to be determined. Using this information, this subsystem outputs another three-dimensional region that delimits which points should be included in the processing of point clouds, and in the case of the images, the regions of interest as well, including the conversion of the appropriate image coordinates for each camera, in which rock fracturing could occur.

[0045] Visual rock segmentation (12) This subsystem segments the rocks contained on the image’s processing boundaries generated by the generation of processing boundaries subsystem (11), in image coordinates, and generates masks for each segmented rock. These masks individualize the detected rocks, i.e. their contours, by delimiting the region they occupy in the processed images. FIG. 2 illustrates examples of segmentation masks obtained when processing image regions within appropriate processing boundaries. The segmentation of rocks is performed by any standard instance segmentation model or algorithm, such as Mask R-CNN, YOLO v7, or other model or algorithm with similar functionality. Given a trained standard instance segmentation model, a fine-tuning process is performed on said model to achieve proper rock segmentation. This fine-tuning process consists of adjusting the parameters of the already pre-trained model by training it again on a database of segmented rocks. This database is specifically constructed by collecting and labeling data from the operation site.

[0046] Determination of regions of interest around the rock breaker’s end effector (13)

[0047] Under the assumption that rocks that are actively fractured will be in the neighborhood of the rock breaker's end effector, the required analysis for determining such fractures can be confined to this neighborhood area. This subsystem generates a three-dimensional region around the end effector, using information of the rock breaker’s configuration at a given time instant, and the direct kinematic model of the rock breaker. The configuration of the rock breaker is provided by measurements from encoders installed in its joints, as well as the rock breakers dimensions. The generated region is compared with the area generated by the generation of processing boundaries subsystem (11), in order to limit the vertexes located outside of this previous area. The final resulting region of interest is fed to the rock characterization using point clouds subsystem (14), and is also converted to the image coordinate systems that are used by the feature generation subsystem (15).

[0048] Rock characterization using point clouds (14)

[0049] This subsystem is in charge of processing the point clouds that are to be used for the rock fracture determination system. Firstly, the input point cloud is filtered to eliminate those points outside the processing delimitation regions generated by the determination of regions of interest around the rock breaker’s end effector subsystem (13). Subsequently, using an encoder specially designed for point cloud processing, this subsystem generates a fixed-size representation, herein called PT-vector, which encapsulates the information contained in the original point cloud, and then it provides it to the feature generation subsystem (15). The method used for encoding the point cloud being processed comprises any point cloud encoding system such as PointNet, FSPool, or other similar encoding systems.

[0050] Feature generation (15)

[0051] This subsystem is responsible for producing a set of features to be used for the determination of rock fractures. The preferred embodiment computes two types of features: features based on segmentation masks provided by the visual rock segmentation subsystem (12), and features computed using the PT-vector provided by the rock characterization using point clouds subsystem (14). Other embodiments can compute only one type of features, either based on segmentation mask or using PT-vector from point clouds.

[0052] Features based on using segmentation masks are obtained by processing the masks that are located within the region of interest around the rock breaker’s end effector. The decision to include on the feature calculation those masks that are not completely contained within this region of interest, is based on the calculation of the area of the masks that effectively is contained within the region of interest, and if this effective area exceeds a certain threshold, then that mask is included in the calculation.

[0053] Taking into account that when the rock is fractured either its contour will change or the number of observed rocks will increase, the preferred embodiment considers the following features, notwithstanding that other features of a similar nature may be used in other embodiments:

[0054] - Largest perimeters difference: this feature is obtained by calculating the difference between the average of the largest perimeters associated with the masks obtained for n frames at the beginning of a rock-breaking sequence, and the largest perimeter associated to the masks obtained using the current frame. Then, this feature measures variations on the rock’s perimeter.

[0055] - Average area difference: this feature is obtained by calculating the difference between the average areas of the masks generated for n frames at the beginning of a rock-breaking sequence, and the largest area associated to the masks obtained using the current frame. Then this feature measures variations on the rock’s area.

[0056] - Difference in the number of segmented rocks: this feature is obtained by calculating the difference between the average number of rock masks generated for n frames at the beginning of a rock-breaking sequence, and the number of rock masks generated using the current frame. Then, this feature measures variations on the number of detected rocks.

[0057] The fracturing of the rock can also be detected by analyzing variations on the point cloud. Thus, taking into account that when the rock is fractured the point cloud representation will change, either, in relation to the shape, the density of points, or both of them, the preferred embodiment consider the following features, notwithstanding that other features of similar nature may be used in other embodiments:

[0058] - PT-vector generated by the encoder of the point clouds: this feature corresponds to the PT-vector without modifications.

[0059] - L1 distance between PT-vectors: this feature corresponds to the L1 distance calculated between the PT-vector obtained from the point cloud at the beginning of the rock-breaking sequence, and the PT-vector obtained from the current point cloud. Then, this feature measures variations on the shape of the point cloud.

[0060] - L2 distance between PT-vectors: This feature is analogous to the previous feature, but using L2 distance. Then, this feature also measures variations on the shape of the point cloud.

[0061] - Density difference between point clouds: Corresponds to the difference between the number of elements composing the point cloud at the beginning of a rock-breaking sequence, and the number of elements of the current point cloud. This feature measures variations on the number of points of the point cloud.

[0062] Fracture detection (16)

[0063] The input for this subsystem is the set of features generated by the feature generation subsystem (15), which in the preferred embodiment uses at least three statistical classifiers for their analysis: one classifier for analyzing the features obtained by processing the image obtained by each camera (in case of using N cameras, then N classifiers are required), one classifier for analyzing the features obtained by processing the point cloud, and a third classifier, herein called global classifier, that makes a decision from the previous classifiers, and generates a Boolean signal that determines if the fracture of a rock took place or not.

[0064] In the preferred embodiment, the classifiers mentioned above comprise different statistical classifiers, considering the different inputs they have, and also the training process, which use different data in each case. In this manner, the image-based features and the point cloud features classifiers are trained independently using the features provided by the feature generation subsystem (15). The global classifier, on the other hand, is trained using the Boolean signals provided by the other classifiers, when they process a given set of corresponding images and point cloud.

[0065] In the preferred embodiment it should be noted that, although there is a single representation for the point cloud that is used by the fracture determination system, in the case of the image-based features, each camera calculates these features independently. Therefore, for a system embodiment using N cameras, N sets of image-based features are computed, and they are fed independently to N image-based classifiers, resulting in N Boolean signals. On the other hand, in the case of point-cloud based features, for a system embodiment using M range sensors, for example M LiDARs, only a single integrated point cloud is generated, then only a single set of point-cloud based features is computed and fed to the point cloud features classifier. Consequently, N+ 1 Boolean signals are fed to the global classifier to determine if a rock fracture took place or not. A general diagram of a preferred embodiment showing the classification process realized by the fracture detection subsystem (16) is illustrated in FIG. 3.

[0066] The classifiers used for this subsystem comprise any standard statistical classifier or a combination of them, such as Support Vector Machines (SVMs), Random Forest classifiers, Decision Tree, Bayesian classifiers, or artificial neural networks, among others.

[0067] Example of Application

[0068] One example embodiment was performed using a semi-industrial prototype. This prototype was realized using an experimental setup comprising a Bobcat E10 mini-excavator, with a hydraulic breaker as end effector, a scaled steel transfer grid, and sensors mounted on it, including two monocular cameras and two 3D LiDAR sensors. The purpose of this prototype was to emulate on a scale model an environment such as the one existing around underground mining grids, where there is a hydraulic breaker in charge of fracturing oversized rocks that fail to pass through a grid that regulates the size of the ore. The Bobcat E10 mini-excavator has a hydraulic arm with four degrees of freedom, similar to those commonly found in hydraulic breakers used in real mining operations. In this manner, taking into account the size of the hydraulic breaker and the grid as a whole, it is possible to simulate a realistic operating environment at a scale of 1 :2.

[0069] The sensors used for the prototype include two MOXA Vport P06HC-1MP-M12 Series cameras, and two Velodyne Puck VLP 16 3D LiDARs, which were installed above the grid. In addition, to obtain the instantaneous configuration of the Bobcat E10 hydraulic breaker arm, string encoders were installed on the hydraulic cylinders associated with each of its joints.

[0070] FIG. 4 illustrates the experimental setup described above for the example embodiment. On the left, it is shown the Bobcat E10 mini-excavator, the grid, and the sensors installed on the roof, i.e. above the grid. On the right, it is shown a detail of the mounting of the sensors, a MOXA camera on the upper part and a Velodyne LiDAR below the camera.

[0071] Additionally, the scaled prototype includes as part associated to the experimental setup system, visualization and simulation tools.

[0072] Generation of processing boundaries

[0073] It was determined a region of effective operation of the hydraulic breaker, providing boundaries for the data coming from the LiDARs. This region, which in this example embodiment was represented as a cuboid, was used to filter points coming from the LiDARs that lie outside of it.

[0074] On the other hand, it was determined a region for an image-based effective operation area, independent from the one that filters LiDAR measurements, applied to the images coming from the MOXA cameras through a binary mask. FIG. 5 shows an example of the result of applying an image-based effective operation area using the experimental setup. On the left side, FIG. 5 shows an image provided by one of the MOXA cameras of the prototype, and on the right side, FIG. 5 shows the area in coordinates of the image shown on left side, provided by the generation of processing boundaries subsystem of the example embodiment. Visual rock segmentation

[0075] In order to obtain a model capable of segmenting rocks using visual information, a fine-tuning was performed on a pre-trained model of YOLO v7, using the version adapted to perform visual segmentation. To carry out this task, a database was created using the semi-industrial prototype, where rocks of different sizes and in different poses were placed on the grid, and the resulting images were then labeled.

[0076] An example of an image labeled to be part of the database used for the rock segmentation training process performed by the visual rock segmentation subsystem of the example embodiment is shown in FIG. 6. This process resulted in a database of 402 labeled images.

[0077] The generated database was subsequently used to train and validate the model obtained. For this purpose, it was divided into a training and validation set, with proportions of 80% and 20% of the original database, respectively.

[0078] Table 1 shows a quantitative measure of the visual segmentation model performance. This table includes the usual metric to analyze the performance of object detection and segmentation processes, such as: precision, recall, and mAP (mean Average Precision) using 50 and 50-95 as loll (Intersection over Union) thresholds, i.e. mAP50 and mAP50-95 respectively.

[0079] Table 1. Performance of the YOLO v7 model, after the fine-tuning segmentation process for the prototype experimental setup.

[0080] Determination of regions of interest around the rock breaker’s end effector

[0081] The determination of the region of interest around the rock breaker’s end effector using point cloud can be visualized in FIG. 7. On the left side, FIG. 7 shows a raw visualization of the data provided by the Velodyne LiDARs, unfiltered. On the right side of FIG. 7 it is shown how the region of interest, which is generated in 3D, allows to filter the point cloud, which is subsequently encoded by the point cloud rock characterization subsystem. The region of interest is represented as a cuboid, whose front edges can also be visualized on the right side of FIG. 7. On the other hand, an example of the region of interest around the rock breaker’s end effector using images, which is generated to restrict the processing of feature calculation on the segmentation masks associated with the rocks, can be visualized in FIG. 8. This region is generated on the coordinates of the image provided by one of the MOXA cameras, and the projection of this region is represented as a rectangle in FIG. 8.

[0082] Rock characterization rocks using point clouds

[0083] The method used for encoding the point cloud in this example embodiment included a FSPool encoder. This encoder was trained using a database generated utilizing the described prototype and measurements provided by the Velodyne PUCK VLP 16 LiDARs. This database included examples of point clouds obtained from 30 fracturing attempts, including successful and unsuccessful attempts. Once the encoder was trained, it was able to generate fixed-size representations for the point clouds, of different cardinalities, that it has processed. This representation corresponds to a 20-component vector, given an input point cloud.

[0084] Feature generation and fracture detection subsystems

[0085] The training of the classifiers associated with fracture detection, was realized generating percussion sequences in which the Bobcat E10 mini-excavator attempted to break rocks. The database was created considering the requirement of variability in the success of the operation, since both fracture and non-fracture examples are required to train the classifiers of the fracture detection subsystem. Additionally, it was considered that images and point clouds for the same rock-breaking attempt have a temporal correlation, for this reason, the division into training and validation sets was done in terms of these attempts, and not in terms of the total number of data.

[0086] In this way, the fracture detection subsystem classifiers that directly process image or point cloud features were trained using data from 50 rock fracture attempts. In the case of the classifiers that process mask features, as a result of these 50 attempts, 679 images were generated, 201 of them correspond to frames where fracturing of a rock took place and 478 correspond to non-fracture frames. On the other hand, in the case of point clouds, considering the differences in sampling sensory data, as a result of these same 50 attempts 2048 point-cloud samples were obtained, 571 of them were associated with fracture situations and 1477 with non-fracture.

[0087] As noted above, to avoid correlation between data, the 50 fracture attempts were separated to generate training and validation sets for the classifiers. For the classifier that processes segmentation masks, 40 of these attempts were used for training and 10 for validation. For the classifier that processes point cloud features, only 20 of the 50 total attempts were used, because the remaining 30 attempts were destined to train and validate the FSPool encoder, as was explained before, and similarly as the case of images, the attempts were separated, 14 of them were used for training and 6 for validation.

[0088] It is important to note that the feature generation using the feature determination subsystem is a requirement for the training of these classifiers, taking into account that these features are the inputs to the classifiers.

[0089] The performance results obtained by the classifier that processed features based on segmentation masks provided by the visual rock segmentation subsystem, on its validation set, are shown in Table 2. In this case, the classification was realized using two different statistical classifiers: Random Forest and Decision Tree.

[0090] Table 2. Performance results of two classifiers using the features based on the segmentation masks

[0091] Likewise, the performance results obtained by the classifier processing point cloud-based features, also on its validation set, are shown in Table 3. In this case, the results were obtained by using as a feature, the code directly provided by the FSPool encoder for the identification of point clouds corresponding to rock fracture situations. In this case, the classifier was a Decision Tree.

[0092] Table 3. Performance results using the features directly provided by FSPool encoder for the identification of point clouds corresponding to rock fracture situations.

[0093] Classifier Class Precision Recall F1 -Score

[0094] Finally, to train the global classifier that unifies the decisions of the classifiers that compose the system, it was used a database consisting of 60 rock breaking attempts, which are independent of the 50 attempts previously used for the training of the classifiers that processed features. From the total of 60 attempts, 40 of them were used for the training of the global classifier, and 20 for its validation. The performing results obtained by evaluating this global classifier processing the outputs of the classifiers, which in turn processing features coming from the segmentation masks and the point cloud, are shown in Table 4. In this case, the classifier was a Decision Tree.

[0095] Table 4. Performance results obtained by the global classifier

[0096] The present invention has been described in terms of particular embodiments and applications, in both summarized and detailed forms, and it is not intended that these descriptions should limit in any way the scope of the invention. It will be also understood that many substitutions, changes and variations in the described embodiments, applications and details of the system and methods illustrated herein can be made by those skilled in the art without departing from the spirit of this invention.

Claims

CLAIMS1. A method for determining rock fractures during the rock breaking process using impact hammers, comprising the steps of:■ receiving sensory data consisting of images and / or point clouds of a plurality of rocks, wherein the images being obtained from one or more camera devices, and / or said point clouds being obtained from one or more range sensors;■ generating processing boundaries by creating a three-dimensional region of interest that limits the sensory information to be used during subsequent steps, wherein said information being selected from point clouds, images, or combinations thereof;■ performing visual rock segmentation, wherein said segmentation generates masks for each segmented rock corresponding to its boundaries;■ determining regions of interest around the end effector of the impact hammer, comprising the generation of a three-dimensional region of interest around the end effector using as input the state of the impact breaker at a given time and its kinematic model;■ characterizing one or more rocks using point clouds, which are first filtered to eliminate those points outside the processing boundary regions generated by the previous steps and then encoded using a fixed-size representation;■ generating a set of features for determining rock fractures and calculated using the previously obtained rock segmentation and fixed-size representation of point clouds; and■ determining the rock fractures using a set of statistical classifiers that process the set of features generated in the previous step.

2. The method of claim 1, wherein the range sensors are selected from a group of sensors including different types of cameras, 2D / 3D LIDAR sensors, RADAR sensors, other range sensors, and / or combinations of thereof.

3. The method of claim 1, wherein the visual rock segmentation is performed by computer vision object detection or object segmentation methods such as Convolutional Neural Networks, Mask-convolutional neural networks, Faster R- convolutional neural networks, YOLO models, or any other machine learning or deep learning architecture.

4. The method of claim 1, wherein the rock characterization using point clouds comprises eliminating the points outside the region of interest and subsequently, generating a fixed-size representation from the point clouds using an encoding method.

5. The method of claims 1 and 4, wherein the fixed-size encoding of the point cloud is performed by encoders based on PointNet++, F SPool, Graph Convolutional Networks, PointCNN, Dynamic Graph CNNs, Transformer-based models, Voxelbased approaches, 3D Shape Descriptors, or any other encoding method.

6. The method of claim 1, wherein the feature generation step comprises computing two types of features: features based on segmentation masks and features computed using the fixed size encoding vector of point clouds:- The features based on segmentation masks consist of differences of rock-shape properties such as area, perimeter, distance from the center of mass, and other related properties, calculated at the start of the breaking operation and at the current time.- The features based on the fixed size encoding vector of point clouds consist of the calculation of differences and / or vector distances between those encoding vectors at the start of the breaking operation and at the current time.

7. The method of claim 1, wherein said classifiers are selected from the group comprising Support Vector Machines (SVMs), Random Forest classifiers, Bayesian classifiers, Artificial Neural Networks, Decision Trees, Bayes Classifiers, Boosting Classifiers, or other statistical classifiers, and / or combinations thereof.

8. A system for determining rock fractures during the rock breaking process using impact hammers, comprising the steps of:■ receiving sensory data consisting of images and / or point clouds of a plurality of rocks, wherein the images being obtained from one or more camera devices, and / or said point clouds being obtained from one or more range sensors;■ generating processing boundaries by creating a three-dimensional region of interest that limits the sensory information to be used during subsequent steps, wherein said information being selected from point clouds, images, or combinations thereof;■ performing visual rock segmentation, wherein said segmentation generates masks for each segmented rock corresponding to its boundaries;■ determining regions of interest around the end effector of the impact hammer, comprising the generation of a three-dimensional region of interest around the end effector using as input the state of the impact breaker at a given time and its kinematic model;■ characterizing one or more rocks using point clouds, which are first filtered to eliminate those points outside the processing boundary regions generated by the previous steps and then encoded using a fixed-size representation;■ generating a set of features for determining rock fractures and calculated using the previously obtained rock segmentation and fixed-size representation of point clouds; and■ determining the rock fractures using a set of statistical classifiers that process the set of features generated in the previous step.

9. The system of claim 8, wherein the range sensors are selected from a group of sensors including different types of cameras, 2D / 3D LIDAR sensors, RADAR sensors, other range sensors, and / or combinations of thereof.

10. The system of claim 8, wherein the visual rock segmentation is performed by computer vision object detection or object segmentation methods such as Convolutional Neural Networks, Mask-convolutional neural networks, Faster R- convolutional neural networks, YOLO models, or any other machine learning or deep learning architecture.

11. The system of claim 8, wherein the rock characterization using point clouds comprises eliminating the points outside the region of interest and subsequently, generating a fixed-size representation from the point clouds using an encoding method.

12. The system of claims 8 and 11, wherein the fixed-size encoding of the point cloud is performed by encoders based on PointNet++, F SPool, Graph Convolutional Networks, PointCNN, Dynamic Graph CNNs, Transformer-based models, Voxelbased approaches, 3D Shape Descriptors, or any other encoding method.

13. The system of claim 8, wherein the feature generation step comprises computing two types of features: features based on segmentation masks and features computed using the fixed size encoding vector of point clouds:- The features based on segmentation masks consist of differences of rock-shape properties such as area, perimeter, distance from the center of mass, and other related properties, calculated at the start of the breaking operation and at the current time.- The features based on the fixed size encoding vector of point clouds consist of the calculation of differences and / or vector distances between those encoding vectors at the start of the breaking operation and at the current time.

14. The system of claim 8, wherein said classifiers are selected from the group comprising Support Vector Machines (SVMs), Random Forest classifiers, Bayesian classifiers, Artificial Neural Networks, Decision Trees, Bayes Classifiers, Boosting Classifiers, or other statistical classifiers, and / or combinations thereof.

Images