A multi-view image-based detection system and method for large boulders in a shaft

Abstract

The application discloses a kind of based on multi-view image's chute large lump ore detection system and method, belong to mine safety monitoring technical field.The system is through front and side monitoring camera, intelligent detection module, edge server and alarm cooperative work, realize the real-time intelligent monitoring of chute discharge port ore.When chute vibration platform triggers PLC action signal, double-view camera synchronously collects ore falling image, and is transmitted to intelligent detection module.Module built-in AI image instance segmentation algorithm carries out pixel-level segmentation to ore, calculates its maximum circumscribed rectangle size and generates detection table;Edge server judges ore size based on over-limit threshold, in combination with depth estimation algorithm to calculate ore three-dimensional coordinates, accurately locate the position of over-limit ore.The system supports image storage, scrolling broadcast unprocessed ore information and alarm triggering function, until operation and maintenance personnel are disposed.The application improves the accuracy and real-time performance of large lump ore detection through multi-view fusion and three-dimensional positioning technology, effectively prevents chute blockage or equipment damage, and ensures mine production safety.

Description

Technical Field

[0001] This invention relates to the field of mine safety monitoring technology, specifically to a system and method for detecting large ore blocks in ore passes based on multi-view images, aiming to solve the safety hazards and transportation efficiency problems caused by large ore blocks in ore pass operations. Background Technology

[0002] In underground metal mining, ore passes serve as crucial dynamic channels for ore transportation, and their safe operation directly impacts production efficiency and personnel safety. However, the internal environment of ore passes is complex, characterized by high magnetic fields, lack of light sources, high humidity, and high dust concentrations. Traditional manual inspection methods are inefficient, risky, and unable to comprehensively cover the entire cross-section of the ore pass. In recent years, with the development of instrument vision and sensor technologies, automated monitoring systems based on image analysis have shown engineering potential for application in ore pass scenarios. However, relevant technical solutions are still limited, and existing solutions have shortcomings.

[0003] Chinese invention patent (A fully automated ore loading and foreign object detection system for ore passes [Publication No.: CN116398239A]) detects large ore pieces by installing 3D cameras and LiDAR. However, the working conditions near the ore pass discharge port are harsh and dusty, resulting in significant noise in the point cloud data captured by the LiDAR. Furthermore, deploying LiDAR is costly and difficult to maintain. Chinese invention patent (A method for ore block size detection based on YOLOv5 image recognition algorithm and depth camera [Publication No.: CN115565024A]) uses an improved YOLOv5 algorithm to analyze ore block size in a blasting scenario. This patent focuses on ore identification in static scenes and does not address dynamic monitoring of ore passes. Chinese utility model patent (A mineral inventory monitoring device for ore passes [Publication No.: CN222559688U]) monitors mineral inventory by moving an infrared device via a slide. This patent also focuses on static measurement and lacks dynamic large ore identification capabilities.

[0004] Therefore, how to build a multi-view image acquisition system, design machine vision algorithms to efficiently and in real time detect large pieces of ore in ore passes, and issue alarm information based on abnormal ore size are key issues that urgently need to be solved by those skilled in the art. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention aims to provide a system and method for detecting large ore blocks in ore passes based on multi-view images. Through multi-view collaborative monitoring, intelligent image processing algorithms, and edge computing technology, it achieves efficient, real-time, and accurate detection and early warning of large ore blocks in ore passes, solving the following problems:

[0006] (1) Lack of multi-view data fusion: Establish a dual-view monitoring system from the front and sides to eliminate detection blind spots.

[0007] (2) Insufficient dynamic early warning capability: Develop a three-dimensional positioning algorithm based on depth estimation to realize dynamic tracking and real-time early warning of large ore blocks.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a multi-view image-based large ore detection system for ore chutes, comprising a front monitoring camera (1), a side monitoring camera (2), a front intelligent detection module (3), a side intelligent detection module (4), an edge server (5), and an alarm (6). The front intelligent detection module (3) is connected to the front monitoring camera (1), and the side intelligent detection module (4) is connected to the side monitoring camera (2). Both the front intelligent detection module (3) and the side intelligent detection module (4) are connected to the edge server (5) through a switch, and the edge server (5) is connected to the alarm (6).

[0009] The front monitoring camera (1), the side monitoring camera (2), the front intelligent detection module (3), the side intelligent detection module (4), the edge server (5), and the alarm (6) are all adjusted according to the actual working conditions of the chute, including their shape, quantity, and installation method. There are two front intelligent detection modules (3) and two front monitoring cameras (1); there are two side intelligent detection modules (4) and two side monitoring cameras (2).

[0010] The PLC action signal of the chute discharge port vibration table is collected to trigger the front monitoring camera (1) and the side monitoring camera (2) to collect images of ore falling from two perspectives. The collected images are respectively input into the front intelligent detection module (3) and the side intelligent detection module (4) connected to the front monitoring camera (1) and the side monitoring camera (2). The front intelligent detection module (3) and the side intelligent detection module (4) are embedded with a large ore size detection algorithm to detect the size of each ore in the image and transmit the detection results to the edge server (5). The edge server (5) is equipped with an over-limit threshold, a depth estimation algorithm, a database management system and an alarm system to determine whether the ore size exceeds the limit, give the specific location of the large ore with the over-limit size, store the image containing the large ore, output the alarm signal, and scroll the unprocessed large ore image and its three-dimensional position information. The alarm (6) is connected to the edge server (5). When the large ore size detection algorithm detects a large ore that exceeds the threshold, the edge server (5) triggers the alarm (6) to sound an alarm.

[0011] A method for detecting large ore blocks in ore passes based on multi-view images, characterized by comprising the following steps:

[0012] Step 1: When the vibrating table at the feed chute emits the first PLC action signal, it triggers the front monitoring camera (1) and the side monitoring camera (2) to collect images of ore falling from the feed chute from two different perspectives, and sends the collected images to the front intelligent detection module (3) and the side intelligent detection module (4) in real time.

[0013] Step 2: The frontal intelligent detection module (3) calls the large ore size detection algorithm to detect the ore images collected from the frontal view and outputs a detailed size table A for each ore; the side intelligent detection module (3) calls the large ore size detection algorithm to detect the ore images collected from the side view and outputs a detailed size table B for each ore. Tables A and B are then sent to the edge server.

[0014] Step 3: The edge server (5) determines whether there are large pieces of ore based on the over-limit threshold and the ore size tables recorded in tables A and B. If there are large pieces of ore exceeding the limit, the edge server (5) gives the three-dimensional position of the large pieces of ore according to the depth estimation algorithm and marks them on the image. Then, an alarm is triggered to remind the maintenance personnel to handle it in time. At the same time, the edge server (5) stores the image containing the large pieces of ore exceeding the limit and scrolls the unprocessed large pieces of ore image and its three-dimensional position information.

[0015] In step 1, when the vibrating table at the feed chute begins to vibrate, causing the ore to fall from the feed chute, the PLC signal of the first vibration action should be extracted to trigger the front monitoring camera (1) and the side monitoring camera (2) to start taking pictures of the ore.

[0016] In step 2, the large ore size detection algorithm is an AI-based image instance segmentation algorithm. This algorithm can segment the pixel region of each ore in the image. Based on the pixel region of each ore, the length and width of its maximum bounding rectangle can be calculated, thereby generating two tables, A and B. According to production requirements, the large ore size data that meets the requirements is input into the system. If the video scan data comparison shows a value greater than the set value, the system will issue an alarm.

[0017] In step 3, the edge server (5) compares the contents of tables A and B with the over-limit threshold. When it finds that there is an over-limit large piece of ore in the image, it records the image to the database management system and triggers the alarm (6) to remind the maintenance personnel. At the same time, it calculates the distance between the large piece of ore and the lens of the monitoring camera (1) in front of it from the image using a depth estimation algorithm. From the side image, the distance between the large ore and the lens of the side monitoring camera (2) was calculated using a depth estimation algorithm. Since the installation positions of the front monitoring camera (1) and the side monitoring camera (2) are fixed, the height information of the large ore in the real three-dimensional world can be calculated based on the center of the image pixel area of ​​the large ore and the installation height of the camera. Based on this, the coordinates of the large ore block in the real three-dimensional world can be obtained. This allows maintenance personnel to promptly monitor and take appropriate measures.

[0018] If maintenance personnel fail to handle large pieces of ore in a timely manner, the alarm information on the edge server will not be cleared and will continue to remind the maintenance personnel in a scrolling manner.

[0019] This invention comprises two parts: hardware architecture and software algorithm.

[0020] 1. Hardware Architecture

[0021] (1) Image acquisition layer: Configure a front monitoring camera (1) and a side monitoring camera (2) to capture front and side views of ore falling from the chute feed opening, respectively;

[0022] (2) Intelligent detection layer: The front intelligent detection module (3) and the side intelligent detection module (4) are embedded with a large ore size detection algorithm to analyze image data in real time;

[0023] (3) Data processing layer: Edge server (5) Deploys over-limit threshold judgment, depth estimation algorithm, database management system and alarm system to realize data fusion, decision and storage;

[0024] (4) Early warning execution layer: The alarm (6) is connected to the edge server (5) to receive alarm signals and execute early warning.

[0025] 2. Software Algorithm

[0026] (1) Trigger acquisition: The dual-view camera is triggered to acquire images synchronously by the action signal of the PLC action signal of the vibrating table at the chute discharge port;

[0027] (2) Size detection: The intelligent detection module calls the AI ​​image instance segmentation algorithm to output the ore size table (A, B);

[0028] (3) Comprehensive judgment: The edge server compares the over-limit threshold, calculates the three-dimensional coordinates of the large ore block by combining the depth estimation algorithm, stores the image and triggers an alarm.

[0029] (4) Dynamic early warning: If an over-limit ore is detected, the system will continuously broadcast its image and location information until the maintenance personnel handle it.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] (1) Multi-view collaborative monitoring: Dual-view layout eliminates detection blind spots and improves detection comprehensiveness.

[0032] (2) Dynamic early warning capability: Real-time detection of the status of large ore blocks, with three-dimensional positioning accuracy expected to reach the centimeter level, providing data support for proactive safety control.

[0033] (3) High cost-effectiveness: No need for expensive equipment such as lidar, significantly reducing system deployment and maintenance costs.

[0034] (4) High level of intelligence: The edge computing architecture enables localized data processing, reduces network latency, and improves real-time performance. It can flexibly adjust large parameters according to the requirements of mine production, simplify the data comparison process, reduce data redundancy, reduce network latency, and improve the practicality of production work. Attached Figure Description

[0035] Figure 1 This is a system framework diagram of the present invention;

[0036] In the diagram: 1. Frontal surveillance camera; 2. Side surveillance camera; 3. Frontal intelligent detection module; 4. Side intelligent detection module; 5. Edge server; 6. Alarm. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] I. System Components and Implementation Details

[0039] (a) Deployment of image acquisition and detection equipment

[0040] 1. Camera selection and installation - Frontal surveillance camera (1) and side surveillance camera (2)

[0041] (1) Hardware parameters: Use an industrial-grade dustproof and waterproof camera (such as IP67 protection level), with a resolution of not less than 1920×1080, a frame rate of ≥30fps, and equipped with an infrared fill light or a custom light source to adapt to the low light environment of the chute.

[0042] (2) Installation location: The front camera (1) is installed inside the protective cover directly opposite the feed port of the chute, with the lens axis perpendicular to the center line of the feed port and about 1-3 meters above the feed port to ensure complete capture of the ore falling trajectory. The side camera (2) is installed inside the protective box on the side wall of the chute, with the lens axis at a 45° angle to the direction of ore falling, covering the blind area on the side of the feed port;

[0043] (3) Parameter adjustment: Adjust the camera focal length and shooting angle according to the actual size of the chute. On-site calibration can ensure spatial alignment of the dual-view images.

[0044] 2. Intelligent detection module configuration - front intelligent detection module (3) and side intelligent detection module (4)

[0045] (1) Hardware configuration: FPGA or DSP is used;

[0046] (2) Algorithm deployment: Based on the YOLACT instance segmentation model, the pre-training dataset contains ore images under different lighting and dust concentrations, and the detection accuracy of the ore pass scene is optimized through transfer learning;

[0047] (3) Data transmission: It is directly connected to the camera via a gigabit Ethernet interface and uses the UDP protocol to receive image streams in real time with a processing delay of ≤100ms.

[0048] (ii) Edge server deployment

[0049] 1. Hardware Configuration

[0050] (1) Server selection: Deploy an industrial-grade edge server (such as Hikvision iDS-96256NX-I24), configured with 4 NVIDIA GPUs for real-time video analysis, input bandwidth of 768Mbps, and support containerized deployment;

[0051] (2) Network architecture: Connect the intelligent detection module through industrial-grade switches to build a ring topology network to ensure that the failure of a single node does not affect the availability of the system.

[0052] 2. Algorithm and Software Deployment

[0053] (1) Over-limit threshold judgment: preset ore size threshold (such as any dimension of length / width / height > 80cm), supports dynamic configuration to adapt to different mine needs;

[0054] (2) Depth estimation algorithm: Based on the principle of monocular vision to measure depth, spatial depth coordinates are calculated by combining camera calibration parameters (intrinsic parameter matrix, distortion coefficient);

[0055] (3) Database Management System: The SQLite embedded database is used to store information such as images of oversized ores, timestamps, and three-dimensional coordinates, and supports historical data query and export;

[0056] (4) Alarm system: Integrated alarm (6) supports graded alarm mechanism (e.g., the ore over-limit size is divided into three levels, corresponding to different light colors and alarm frequencies, etc.).

[0057] II. Implementation Steps and Methods

[0058] (a) Image acquisition triggering mechanism

[0059] 1. PLC signal acquisition

[0060] (1) Read the motion signals in real time through the open interface of the vibration table PLC control system (such as Modbus TCP protocol);

[0061] (2) De-jittering: Software filtering algorithms (such as moving average filtering) are used to eliminate signal jitter and ensure stable triggering.

[0062] 2. Synchronous shooting control

[0063] (1) When the first valid PLC action signal is detected, a synchronization pulse is sent to the camera to trigger real-time image acquisition;

[0064] (2) Frame synchronization mechanism: Hardware trigger signal is used to ensure that the time difference between dual-view image acquisition is <10ms, thus avoiding motion blur.

[0065] (II) Ore Size Inspection and Data Transmission

[0066] 1. Instance Splitting Process

[0067] (1) Image preprocessing: Histogram equalization and noise reduction (such as median filtering) are performed on the original image;

[0068] (2) Model inference: Input the preprocessed image into the YOLACT model and output the ore mask and bounding box coordinates;

[0069] (3) Size calculation: Calculate the minimum bounding rectangle based on the mask area, obtain the length, width, height and other dimension data, and generate tables A and B.

[0070] 2. Data transmission optimization

[0071] (1) Data compression: A lightweight compression algorithm is adopted to compress the image data to 10%~20% of the original size while maintaining the detection accuracy;

[0072] (2) Transmission protocol: The detection results are published to the edge server using the MQTT protocol, and QoS level 2 is supported to ensure data reliability.

[0073] (III) Comprehensive Judgment and Early Warning Implementation

[0074] 1. Over-limit judgment mechanism

[0075] (1) Threshold comparison: Traverse all ore size data in tables A and B, and determine that any dimension exceeds the threshold;

[0076] (2) False detection filtering: Combine continuous frame data and use the Kalman filtering algorithm to track the movement trajectory of the ore and eliminate instantaneous interference.

[0077] 2. Three-dimensional positioning and storage

[0078] (1) Depth of field calculation: Metric3D is called to calculate the depth of field for the over-limit ore, and the three-dimensional coordinates are obtained by combining the camera calibration parameters. ;

[0079] (2) Database storage: Store the images, coordinates and timestamps of the oversized ore into the SQLite database, and support querying by time and size.

[0080] 3. Alarm and broadcasting mechanism

[0081] (1) Alarm triggering: When an excess of ore is detected, the edge server sends a control signal to the alarm (6) to activate the audible and visual alarm;

[0082] (2) Scrolling broadcast: The monitoring terminal displays a list of unprocessed over-limit ores in real time, sorted by the degree of over-limit, and supports clicking to view details (including original image and three-dimensional coordinates).

[0083] Through the above specific implementation methods, the present invention can effectively solve the technical problem of detecting large pieces of ore in mine ore passes and provide intelligent monitoring methods for safe production in mines.

[0084] It should be noted that the above embodiments of the present invention are merely examples for clearly illustrating the content involved in the present invention, and are not intended to limit the implementation of the present invention. For example, the surveillance camera is not limited to visible light cameras, but also covers all instruments capable of imaging. The multi-view perspective is not limited to the front and side, but also includes all subdivided viewpoints in three-dimensional space. The instance segmentation algorithm not only covers convolutional neural networks, but also covers all unsupervised and supervised segmentation algorithms. Obvious system, method, functional, and application variations made by those skilled in the art based on the above embodiments are still within the protection scope of the present invention.

Claims

1. A multi-view image-based system for detecting large ore blocks in ore passes, characterized in that, The system includes a front-facing surveillance camera (1), a side-facing surveillance camera (2), a front-facing intelligent detection module (3), a side-facing intelligent detection module (4), an edge server (5), and an alarm (6). The front-facing intelligent detection module (3) is connected to the front-facing surveillance camera (1), and the side-facing intelligent detection module (4) is connected to the side-facing surveillance camera (2). Both the front-facing intelligent detection module (3) and the side-facing intelligent detection module (4) are connected to the edge server (5) via a switch. The edge server (5) is connected to the alarm (6).

2. The multi-view image-based large ore detection system for ore passes according to claim 1, characterized in that, There are two front intelligent detection modules (3) and two front monitoring cameras (1); there are two side intelligent detection modules (4) and two side monitoring cameras (2).

3. The multi-view image-based large ore detection system for ore passes as described in claim 2, characterized in that, The PLC action signal of the chute discharge port vibration table is collected to trigger the front monitoring camera (1) and the side monitoring camera (2) to collect images of ore falling from two perspectives. The collected images are respectively input into the front intelligent detection module (3) and the side intelligent detection module (4) connected to the front monitoring camera (1) and the side monitoring camera (2). The front intelligent detection module (3) and the side intelligent detection module (4) are embedded with a large ore size detection algorithm to detect the size of each ore in the image and transmit the detection results to the edge server (5). The edge server (5) is equipped with an over-limit threshold, a depth estimation algorithm, a database management system and an alarm system to determine whether the ore size exceeds the limit, give the specific location of the large ore with the over-limit size, store the image containing the large ore, output the alarm signal, and scroll the unprocessed large ore image and its three-dimensional position information. The alarm (6) is connected to the edge server (5). When the large ore size detection algorithm detects a large ore that exceeds the threshold, the edge server (5) triggers the alarm (6) to sound an alarm.

4. A method for detecting large ore blocks in ore passes based on multi-view images, characterized in that, Includes the following steps: Step 1: When the vibrating table at the feed chute emits the first PLC action signal, it triggers the front monitoring camera (1) and the side monitoring camera (2) to collect images of ore falling from the feed chute from two different perspectives, and sends the collected images to the front intelligent detection module (3) and the side intelligent detection module (4) in real time. Step 2: The frontal intelligent detection module (3) calls the large ore size detection algorithm to detect the ore images collected from the frontal view and outputs a detailed size table A for each ore; the side intelligent detection module (3) calls the large ore size detection algorithm to detect the ore images collected from the side view and outputs a detailed size table B for each ore. Tables A and B are then sent to the edge server. Step 3: The edge server (5) determines whether there are large pieces of ore based on the over-limit threshold and the ore size tables recorded in tables A and B. If there are large pieces of ore exceeding the limit, the edge server (5) gives the three-dimensional position of the large pieces of ore according to the depth estimation algorithm and marks them on the image. Then, an alarm is triggered to remind the maintenance personnel to handle it in time. At the same time, the edge server (5) stores the image containing the large pieces of ore exceeding the limit and scrolls the unprocessed large pieces of ore image and its three-dimensional position information.

5. The method for detecting large ore blocks in a ore pass based on multi-view images according to claim 4, characterized in that, In step 1, when the vibrating table at the feed chute starts to vibrate, causing the ore to fall from the feed chute, the PLC signal of the first vibration action should be taken out to trigger the front monitoring camera (1) and the side monitoring camera (2) to start taking pictures of the ore.

6. The method for detecting large ore blocks in a ore pass based on multi-view images according to claim 5, characterized in that, In step 2, the large ore size detection algorithm is an AI-based image instance segmentation algorithm. This algorithm can segment the pixel region of each ore in the image. Based on the pixel region of each ore, the length and width of its maximum bounding rectangle can be calculated, thereby generating two tables, A and B.

7. The method for detecting large ore blocks in a ore pass based on multi-view images according to claim 6, characterized in that, In step 3, the edge server (5) compares the contents of tables A and B with the over-limit threshold. When it finds that there is an over-limit large piece of ore in the image, it records the image to the database management system and triggers the alarm (6) to remind the maintenance personnel. At the same time, it calculates the distance between the large piece of ore and the lens of the monitoring camera (1) in front of it from the image in front using a depth estimation algorithm. ; The distance between the large ore block and the lens of the side surveillance camera (2) was calculated from the side image using a depth estimation algorithm. Since the installation positions of the front monitoring camera (1) and the side monitoring camera (2) are fixed, the height information of the large ore in the real three-dimensional world can be calculated based on the center of the image pixel area of ​​the large ore and the installation height of the camera. Based on this, the coordinates of the large ore block in the real three-dimensional world can be obtained. This allows maintenance personnel to promptly monitor and take appropriate measures. If maintenance personnel fail to handle large pieces of ore in a timely manner, the alarm information on the edge server will not be cleared and will continue to remind the maintenance personnel in a scrolling manner.

Images