Hazard detection system and method

Abstract

A geotechnical hazard detection method includes creating a dataset of images using a plurality of devices, classifying each image in the dataset into one of four categories, performing a series of transformations on the images in the dataset to generate an expanded dataset, training a deep learning model using the expanded dataset, converting the model to a smaller framework suitable for a mobile device, implementing the framework on the mobile device, and classifying images taken by the mobile device into one of the four categories using the framework

Description

HAZARD DETECTION SYSTEM AND METHODCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Provisional Application US 63 / 557,588 filed February 25, 2024, hereby incorporated by reference in its entirety.BACKGROUND

[0002] Mining is an essential industry that plays a vital role in our daily lives, as it provides a wide range of critical resources and minerals, which are essential for modem technology7, as well as for enabling the transition to low-carbon energy7sources. Mineral resource extraction can be carried out through various methods, including underground mining, open-pit mining, and on water-based sources, depending on the location and type of the resource being extracted. Despite its many forms, all types of mining have inherent risks. Underground mining is often considered one of the most demanding and hazardous environments for workers, as they can be exposed to toxic emissions, dust, heat stress, and many hazards associated with cave-in and rockfalls. Government agencies like the Occupational Safety and Health Administration (OSHA) and the Mine Safety and Health Administration (MSHA) work hand in hand with mining companies to ensure that rigorous safety standards are met.

[0003] For this purpose, strict construction and mining standards are in place and regular inspections are conducted in underground mines to proactively identify and mitigate any potential hazards. These inspections, however, strongty rely on workers’ ability to identity7hazards, and are thus prone to human errors. In addition, hazard recognition using visual inspection requires extensive training and may be challenging given the extension of mine sites and illumination challenges existing in many underground environments.

[0004] One alternative to automate hazard recognition is the use of computer visionbased solutions. Several applications in mining environments commonly take advantage of computer vision systems, deployed through surveillance cameras, mobile equipment, or smartphone apps, to help identify hazards and keep work environments safe. Recent advances in machine learning and deep learning have contributed to these results, as increased progress has been made towards solutions that operate in industrial environments with more or less controlled illumination. In underground mining, however, the incorporation of these technologies has faced slower advances, due mostly to the challenging conditions inunderground mines, due to environmental variability as the operation progresses, challenging illumination, and a severe lack of representative data associated with structural hazards in particular.SUMMARY

[0005] In a first aspect, a method for identifying and mapping, in real-time, hazards and people in underground environments includes creating a dataset of images using a plurality of devices; classifying each image in the dataset into one of four categories; performing a series of transformations on the images in the dataset to generate an expanded dataset; training a deep learning model using the expanded dataset; converting the model to a smaller framework suitable for a mobile device; implementing the framework on the mobile device; and classifying images taken by the mobile device into one of the four categories using the framework.

[0006] In a further aspect, creating the dataset includes capturing each image by positioning any device of the plurality of devices approximately one meter aw ay from a target area to be inspected.

[0007] In another aspect, classifying each image includes classifying the images into categories of hazard, maintenance required, not hazard and not usable.

[0008] In yet another aspect, performing a series of transformations on the images includes applying data augmentation techniques by rotating, translating, flipping or zooming an image to create a transformed image; and resizing each image to a standardized dimension.

[0009] Further, performing a series of transformations on the images may also include re-scaling the channel values of each image to a range between 0 and 1. Yet further, the standardized dimension is 224 x 224 pixels;

[0010] In another aspect, the deep learning model is a convolutional neural network (CNN).

[0011] In another aspect, training the model further comprises training the model by freezing pre-trained feature extraction layers in the CNN and replacing dense layers of the CNN.BRIEF DESCRIPTION OF THE FIGURES

[0012] FIGS. 1 A - 1C depict images of mine areas that are a hazard, in embodiments.

[0013] FIGS. 2 A - 2C are images taken of a Maintenance Required area in a mine, in embodiments.

[0014] FIGS. 3A - 3C are images of a Not Hazard area in a mine, in embodiments.

[0015] FIGS. 4A - 4C are examples of images that are not usable.

[0016] FIG. 5 is a schematic diagram of a system for performing a geotechnical hazard detection method, in embodiments.

[0017] FIG. 6 is a flowchart of a geotechnical hazard detection method, in embodiments.DETAILED DESCRIPTION OF EMBODIMENTS

[0018] Embodiments disclosed herein include a method for identifying and mapping, in real-time, hazards and people in underground environments. The method includes capturing of data using, for example, visual, NIR, and lidar imaging, which is used to feed a machine learning module. The machine learning module assigns a category to the image / point cloud perceived. This information is then geo-referenced, categorized information is transmitted into a centralized center for mapping. The information in the map is time-stamped and it can be retrieved as a time-series.

[0019] Mine operators are required to develop and follow a roof control plan approved by the Mine Safety and Health Administration (MSHA). This plan should be tailored to the prevailing geological conditions and the mining system used at the mine, including details for managing the roof, face, ribs, and addressing potential rock bursts. Regular and thorough inspections are performed to ascertain that roof control plans are comprehensive and comply with safety regulations. These inspections are typically conducted periodically by a senior geotechnical specialist, but every miner (or worker) is also responsible for examining their work area and reporting any potential hazards. Key aspects evaluated during these inspections include the following:

[0020] 1. Correct installation of ground support elements: Inspectors check for appropriate bolt spacing, ensuring that bolts are evenly distributed and securely anchored. This also includes verifying that mesh overlap is sufficient to cover and support the rock surface and that shotcrete (a type of sprayed concrete) is applied at the correct thickness to provide adequate support and prevent rockfalls.

[0021] 2. Condition of ground support elements: This involves assessing the physical state of support structures. Inspectors look for signs of spalling (chipping or flaking of the shotcrete), any damaged shotcrete that may compromise structural integrity, and the condition of bolts and mesh, to ensure they are intact and not deformed or broken.

[0022] 3. Presence of corrosion or corrosive conditions: Corrosion can weaken metal support elements, such as bolts and mesh. Inspectors look for rust or other signs of corrosion and assess the surrounding environment for conditions that could accelerate corrosive processes, such as high humidity or exposure to water.

[0023] 4. Changes in rock mass quality: The quality of the rock mass is crucial for mine stability. Inspectors examine joint characteristics (e.g., spacing, orientation, and infill material), the extent and nature of fracturing, and types of alteration (chemical changes in the rock) to determine any changes that could affect stability.

[0024] 5. Lithological or geotechnical-domain alterations: Changes in the geological composition or properties of rock layers (lithology) can affect the stability' of the mine. Inspectors monitor for alterations in geotechnical domains (zones with distinct geological features) that could indicate potential instability or require changes in support strategies.

[0025] 6. Changes in water conditions or inflows: Water can significantly impact mine stability. Inspectors check for new or increased water inflows, changes in groundwater levels, and the presence of leaks, which can weaken rock and soil or lead to erosion and washout of support materials.

[0026] 7. Signs of structural movement or stress: Indicators of structural movement include cracking of shotcrete, deformation of support elements, and floor heave (upward movement of the mine floor). These signs suggest that the ground is shifting, which could lead to instability or collapse if not addressed.

[0027] 8. Presence of major geological structures: Faults, folds, and other major geological structures can create zones of weakness. Inspectors identity7and monitor these structures to assess their potential impact on mine stability' and plan appropriate support measures.

[0028] 9. Orientation of structures that could lead to wedge failure: Certain orientations of joints, faults, or bedding planes can create wedge-shaped blocks of rock that are prone to slipping or falling. Inspectors analyze the geometry and orientation of these structures to anticipate and prevent wedge failures.

[0029] 10. Evidence of mining irregularities: Over-break (excessive excavation beyond the planned boundaries) and poor scaling (failure to remove loose rock from walls and roofs) can compromise mine stability. Inspectors check for these irregularities to ensure that the mine is excavated and maintained according to plan.

[0030] 11. Visible tension cracks, raveling behind shotcrete, and other signs of instability: Inspectors look for surface cracks, especially those indicating tension, whichsuggest that the rock is under stress and may fail. Raveling (falling of small rock pieces behind shotcrete) and other signs of instability are also closely monitored to prevent larger rockfalls or collapses.

[0031] Although it is the responsibility of every miner to identify and report any of these characteristics, the reality is that most miners may not have the expertise to report this effectively. In embodiments disclosed herein, a portable tool assists workers in identifying hazards to improve safety conditions in underground mines, and potentially also in other underground environments, such as tunnels, subway infrastructure after an emergency, etc. This will enable every miner to identify preliminary possible hazards and promptly inform experts for a more in-depth inspection, thus contributing to reducing the occurrence of roof fall accidents in underground operations.

[0032] Current machine learning models used for roof fall hazard prediction are mounted on expensive sensors, computationally expensive, or lack the robustness for accurate prediction in the underground mining environment. A design methodology as disclosed herein provides for a robust, low-cost, deep learning-based algorithm for underground mine roof fall hazard prediction. A data sampling plan is developed to ensure the replicability and robustness of the developed model. In addition, feature engineering and transformation methods are described to identify relevant features for hazard identification. The new tool will detect roof fall hazards in real-time and contribute to a crowdsourcing approach for underground hazard detection.

[0033] A dataset of images is collected in an underground environment using a plurality of devices. Devices used to collect images may be portable devices such as mobile phones or tablets.

[0034] In embodiments, these images taken in a mine may be generally classified in four categories: Hazard, Maintenance Required, Not Hazard and Not Usable.

[0035] FIGS. 1 A - 1C are images taken of a Hazard area in a mine. This may be understood as an area where ground support is inadequate or failing, characterized by unsupported sections (FIGS. 1 A-1B) and visible discontinuities in the rock mass (FIG. 1C), protruding bolts, and bent plates. These signs indicate immediate danger, requiring prompt attention to prevent rockfalls or collapses.

[0036] FIGS. 2 A - 2C are images taken of a Maintenance Required area in a mine. This may be understood as an area where ground support elements show signs of wear or minor failure, such as bulging mesh or bent plates. These areas are not immediately hazardous but need maintenance to prevent future deterioration into more severe conditions.

[0037] FIGS. 3A - 3C are images of a Not Hazard area in a mine. This may be understood as an area where the rock mass is fully supported, and with no visible discontinuities. Bolts and mesh are flush with the rock surface, indicating that the ground support is functioning as intended and there are no immediate risks.

[0038] FIGS. 4A - 4C are examples of images that are not usable. In other words, it is not possible to assess the condition of the mine due to obstructions, such as objects or people in the frame, or due to poor quality, such as blurry or moving images. These images need to be retaken to provide a clear view of the area being inspected.

[0039] To enhance the dataset and ensure class balance, a series of transformations may be performed on an original set of images, for example, approximately 3000 images. Despite following specific criteria and developing guidelines for capturing these images, variations inevitably arise due to factors such as differences in the heights of the individuals taking the photos, and the fact that pictures are not always taken from exactly the same distance or position. Although these variations affect data capture, they were chosen for the data collection procedures as they mimic the conditions in which a diverse workforce may capture images for the intended application in real-life scenarios.

[0040] To simulate a broader range of conditions and improve the robustness of the deep learning model, transformations on the captured images are subject to data augmentation techniques such as rotations, translations, flipping, and zooming to create additional images for an expanded dataset. These transformations help mimic the diverse perspectives and distortions that occur in real mining environments, where conditions are not always ideal and can vary considerably. By expanding the dataset in this manner, the distribution of images to categories may be balanced and the total number of images in the expanded dataset may be increased, for example, to 10,000. While it is possible to use any number of images to enhance the dataset, 10,000 images were deemed sufficient to achieve accurate classification without imposing excessive computational demands. This number ensures that the model is both effective and efficient, providing reliable hazard detection in underground mining operations. Additionally, this expansion allows for an equal distribution of 2500 images in each class, enhancing the model’s exposure to varied data and thereby improving its predictive accuracy and reliability.

[0041] Furthermore, all images in our dataset were resized to a standardized dimension of 224 x 224 pixels. This uniformity is crucial as it streamlines the input process, ensuring that our deep learning application can efficiently handle the data during training. Additionally, we applied scaling and normalization techniques to the images. This processinvolves re-scaling the channel values of each image to a range between 0 and 1, which is a standard practice in image processing for neural networks. Such scaling and normalization help in reducing model training times and improve the convergence behavior during the learning process. By standardizing the pixel intensity values across all images, issues related to variations in lighting and exposure that can occur in different mining environments may be mitigated. This rigorous preprocessing ensures that the expanded dataset is well prepared, making it more conducive for the deep learning model to learn meaningful patterns and features effectively and accurately.

[0042] The machine learning model that executes the classification may use collaborative / federated learning based on the captured data. Examples of the captured data included user-provided images. These images may be targeted from the centralized center and may be used to tune the parameters of the machine learning model. The parameters are shared among users connected to the centralized center and they are used to improve a general model that will learn for the captures of all users. The information will be deployed using an HMI and it can also be retrieved in real-time to aid in localization and rescue missions.

[0043] Once data collection and augmentation is completed, deep learning-based model development is performed. Deep learning, a subset of machine learning, leverages algorithms inspired by the human brain’s structure and function, known as artificial neural networks. This approach is distinct from traditional machine learning techniques due to its use of multiple layers that can model complex patterns and relationships within large datasets. While some machine learning models also utilize multiple layers, the fundamental difference lies in the depth and complexity: machine learning models generally consist of a few layers, whereas deep learning models are distinguished by their many layers, sometimes hundreds. Another difference is that traditional machine learning models typically require manual feature extraction, where specific characteristics or features are identified and fed into the model. In contrast, deep learning models have the ability to automatically leam and extract relevant features from raw data during the training process. This deep architecture and feature-learning capability allow deep learning to excel at recognizing complex patterns in unstructured data, such as images.

[0044] Convolutional neural networks (CNNs), a specialized class of deep neural networks, are tailored for processing structured grid data like images. CNNs employ learnable filters to perform convolution operations on input data, making them highly effective for taskssuch as image classification, object detection, and segmentation. These tasks require precise identification and spatial distribution of objects within images.

[0045] As disclosed herein, the primary role of the CNN is to classify each image into one of four predefined categories, presenting a clear classification challenge. The images, once preprocessed and standardized, are fed into the CNN. The network then utilizes its learned filters to detect and categorize key features that indicate potential hazards or maintenance requirements in the mining environment.

[0046] A CNN for use in a real-time mobile application includes characteristics such as efficient feature extraction and reduced computational complexity. These characteristics may be provided, for example, by a CNN having architectural features such as depthwise separable convolutions, linear bottlenecks, and inverted residuals. In embodiments, a CNN such as MobileNet™, MobileNetV2™, or MobileNetV3™ may be used, although any CNN that is suitable for implementation on a mobile device may be used.

[0047] To illustrate principles disclosed herein, a description of MobileNetV2 is provided. MobileNetV2 is an advanced neural network architecture designed primarily for mobile and embedded vision applications, because of lightweight deep learning models that optimize speed and efficiency. Mobil eNetV2 includes several architectural features that significantly enhance its performance:

[0048] Depth wise separable convolutions: These are used to reduce the model size and computational cost by separating the convolution into a depthwise and a pointwise convolution, effectively filtering inputs and combining them to create new features.

[0049] Linear bottlenecks: These structures capture the important features of the network and compress the input information to reduce dimensions, which conserves processing power without significant loss of information.

[0050] Efficient model architecture: MobileNetV2 optimizes both the architectural design and the operation flow to maximize efficiency. This enables the model to deliver high accuracy while maintaining a small footprint, making it ideal for devices with limited computational resources.

[0051] The architecture of MobileNetV2, as illustrated in Table 1, has a design that balances high performance with low computational demands, making it ideal for resource constrained environments like mobile devices.Table 1. Summary of MobileNetV2 architecture.

[0052] To adapt MobileNetV2 for geotechnical hazard detection, transfer learning was applied by freezing the pre-trained feature extraction layers and replacing the final dense layers to suit the classification task. The architecture modifications are outlined in Table 2.Table 2. MobileNetV2 architecture details.

[0053] In embodiments, the trained deep learning model is converted to a reduced parameter model suitable for implementation on a mobile device, for example, TensorFlow™ Lite. This type of framework enables the deployment of machine learning models on mobileand embedded devices, supporting platforms like Android™ and iOS™, as well as loT devices.

[0054] FIG. 5 is a schematic of a hazard detection system 100. Embodiments of a hazard detection system 100 includes at least one of a node 110, a coordination mechanism 120, and a hazard mapper 130. The system may include additional nodes 110. Node 110 includes at least one of a data acquisition system 140, a computational model 114, and a labeling tool 115.

[0055] Data acquisition system 140 may include one or both of a sensor 142 and a data infrastructure 144. Sensor 142 may be or include a camera, and captures spatial data, which may includes images and / or point clouds. The spectral region of the spatial data may be visible or near-IR, and may be hyperspectral. Data infrastructure 144 collects the acquired data, timestamps it, and georeferences it, and stores it / transmits it to one or more electronic memory systems. Georeferencing the acquired data may include associating the acquired data with a predefined map or existing landscape reference such as waypoint, image-based reference, etc.

[0056] Computational model 114 receives information from data acquisition system 140, processes it, and generates a classification label (for instance hazard, no hazard, requires maintenance, other, etc.) to the captured data. Computational model 114 may be based on machine learning. It may include a deep learning model comprised of a structure and a set of tunable model parameters. The model parameters may be modified based on at least one of (i) values that are determined for a specific site or region based on historical information about hazards, and (ii) values transmitted by coordination mechanism 120.

[0057] Node 110 may include a processor 116 and memory 118 storing machine- readable instructions that, when executed by the processor 116, cause node 110 to execute functions of node 110 described herein. Memory 118 may store computational model 114.

[0058] Labeling tool 115 allows a user to validate the classification task and / or override it by providing feedback to the system.

[0059] Coordination mechanism 120 may include a computational model 124 and have networking capabilities. Coordination mechanism 120 receives information from nodes 110, computes parameters, and distributes, not necessarily in equal manner, the resulting information / parameters to computational models 114 of nodes 110. Coordination mechanism 120 may be a node 110 or be part of anode 110.

[0060] Coordination mechanism 120 may include a processor 126 and memory 128 storing machine-readable instructions that, when executed by processor 126, causecoordination mechanism 120 to execute functions ascribed to it herein. Memory' 128 may store computational model 124 as machine-readable instructions.

[0061] Hazard mapper 130 may include a computational model 134 and have networking capabilities. Hazard mapper 130 maps the information generated by nodes 110, using a predefined map or coordinate system, to associate a specific set of coordinates, including position and time, for each input classified in one or more specific categories (for instance, hazard). Hazard mapper 130 may also consider: (a) a data infrastructure (e.g.. data infrastructure 144) for information / data recording and (b) a platform for visualization and communication of information.

[0062] Hazard mapper 130 may include a processor 136 and memory 138 storing machine-readable instructions that, when executed by processor 136, cause hazard mapper 130 to execute functions ascribed to it herein. Memory 138 may store computational model 134 as machine-readable instructions.

[0063] In embodiments, hazard detection system 100 includes a processor 186 that executes the functions of at least two of processors 116. 126, and 136. Similarly, hazard detection system 100 may include a memory 108 that includes the machine-readable instructions of at least two of memory 1 18, memory 128, and memory 138.

[0064] Each of memory' 118, 128, and 138 may be transitory' and / or non-transitory and may include one or both of volatile memory (e.g.. SRAM, DRAM, computational RAM, other volatile memory, or any combination thereof) and non-volatile memory (e.g.. FLASH, ROM, magnetic media, optical media, other non-volatile memory', or any' combination thereol). Part or all of the memory' 118, 128, and 138 may be integrated into processor 116, processor 126, and processor 136, respectively.

[0065] FIG. 6 is a flowchart of a geotechnical hazard detection method 200. which may be implemented by geotechnical hazard detection system 100. Method 200 includes steps 210 - 270.

[0066] Step 210 includes creating a dataset of images. In an example of step 210, the images are collecting using a plurality of devices to simulate the real-life use case where images would be collected by many different workers in a mine.

[0067] Step 220 includes classifying each image. In an example of step 220, each image in the dataset into one of four categories: Hazard, Maintenance Required, Not Hazard, Not Usable.

[0068] Step 230 includes performing a series of transformations on the images in the dataset to generate an expanded dataset. In an example of step 230, these transformations mayinclude data augmentation techniques of rotating, translating, flipping or zooming an image to create a transformed image; and resizing each image to a standardized dimension.

[0069] Step 240 includes training a deep learning model using the expanded dataset.

[0070] Step 250 includes converting the model to a smaller framework suitable for a mobile device.

[0071] Step 260 includes implementing the framework on the mobile device.

[0072] Step 270 includes classifying images taken by user using the mobile device in an underground environment into one of the four categories using the framework.

[0073] Changes may be made in the above methods and systems without departing from the scope of the present embodiments, ft should thus be noted that the matter contained in the above description or shown in the accompanying drawings should be interpreted as illustrative and not in a limiting sense. Herein, and unless otherwise indicated the phrase “in embodiments” is equivalent to the phrase “in certain embodiments,” and does not refer to all embodiments.

[0074] Features described herein may be combined in various ways without departing from the scope hereof. The following enumerated examples illustrate some possible, nonlimiting combinations. Each mining operation has unique characteristics and challenges. These include vary ing geological conditions, different scales of operation, and diverse regulatory environments. By presenting a flexible and scalable methodology, any of the disclosed embodiments may be used by any mining operation to adapt and implement similar technologies tailored to their specific needs. The application’s architecture allows for modifications and tuning to address the particularities of different mining sites.

[0075] For example, embodiments may also be used for automatic detection of the use of personal protection equipment, authorization, and number of people in different areas of a mine. For PPE detection, an object detection model is developed, where the classes are: "Helmet", i.e., the head of a person wearing a helmet (3177 labels) and "No Helmet", i.e., the head of a person without a helmet (2134 labels). To perform well it is necessary7to include images of workers of various ages, genders and ethnic backgrounds wearing different types and brands of PPE.

[0076] Georeferencing capabilities may be used to not only detect hazards but also precisely map their locations within the underground mines. Integrating spatial mapping technology into the application will facilitate a more strategic approach to hazard management, allowing mine operators to prioritize interventions and optimize safety measures effectively.

[0077] Regarding instances of the terms “and / or” and “at least one of,” for example, in the cases of “A and / or B” and “at least one of A and B.” such phrasing encompasses the selection of (i) A only, or (ii) B only, or (iii) both A and B. In the cases of “A, B, and / or C” and “at least one of A, B, and C,” such phrasing encompasses the selection of (i) A only, or (ii) B only, or (iii) C only, or (iv) A and B only, or (v) A and C only, or (vi) B and C only, or (vii) each of A and B and C. This may be extended for as many items as are listed.

[0078] The following claims are intended to cover all generic and specific features described herein, as well as all statements of the scope of the present method and system, which, as a matter of language, might be said to fall therebetween.

Claims

CLAIMSWe claim:

1. A geotechnical hazard detection method, comprising: creating a dataset of images using a plurality of devices; classifying each image in the dataset into one of four categories; performing a series of transformations on the images in the dataset to generate an expanded dataset; training a deep learning model using the expanded dataset; converting the model to a smaller framework suitable for a mobile device; implementing the framework on the mobile device; and classify ing images taken by the mobile device into one of the four categories using the framework.

2. The method of claim 1, wherein creating the dataset further comprises: capturing each image by positioning any device of the plurality of devices approximately one meter away from a target area to be inspected.

3. The method of claim 1, wherein classifying each image further comprises classifying the images into categories of hazard, maintenance required, not hazard and not usable.

4. The method of claim 1 , wherein performing a series of transformations on the images further comprises: applying data augmentation techniques by rotating, translating, flipping or zooming an image to create a transformed image; and resizing each image to a standardized dimension.

5. The method of claim 4, wherein performing a series of transformations on the images further comprises re-scaling the channel values of each image to a range between 0 and 1.

6. The method of claim 4, wherein the standardized dimension is 224 x 224 pixels;7. The method of claim 1, wherein the deep learning model is a convolutional neural network (CNN).

8. The method of claim 7, wherein training the model further comprises training the model by freezing pre-trained feature extraction layers in the CNN.

9. The method of claim 8, wherein training the model further comprises replacing dense layers of the CNN.

Images