Coal mine shaft inspection method and system

Abstract

The invention relates to a coal mine shaft intelligent inspection method and system based on multi-modal fusion, and the method comprises the steps: firstly, installing a sound pickup and a camera at corresponding positions in a shaft, collecting sound and video data, collecting shaft specification information through a sensor, and storing the shaft specification information in a database for judging an abnormality occurrence position; then, the collected data are stored as audio and video files through a control panel and sent to an FTP server, and the shaft specification information is called through an API interface and analyzed by a sensor data monitoring module; and finally, the abnormal video detection module receives the video file, analyzes the video content in real time by using an object detection large model technology, and refines and confirms the result through the reasoning post-processing module. Through combination of audio, video and sensor data, comprehensive, real-time and accurate inspection of shaft equipment is realized, the labor cost is reduced, and the inspection efficiency and safety are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of coal mines, and particularly relates to a method and system for inspecting coal mine shafts. Background Art

[0002] Mine hoisting and transportation is a key link in the coal mining process. The transportation of coal, gangue, personnel, materials, and equipment in the underground working face depends on the hoisting and transportation system. However, in daily operation, problems such as pipeline clip detachment, shaft wall subsidence, cracking, and deformation often occur, seriously threatening the safety of vertical shaft hoisting. Therefore, daily inspection of coal mine shafts is an important prerequisite for ensuring coal mine production safety and the personal safety of miners.

[0003] Currently, traditional inspection methods mainly rely on manual regular inspections, which are carried out by visual inspection and manual touch. The efficiency is low and the reliability is poor. It is impossible to conduct comparative analysis and real-time monitoring, and there are potential safety hazards. To solve these problems, the existing technology has begun to adopt an automated inspection method, using AI algorithms to detect the operating state of the shaft in real time. However, these technologies have problems such as few analysis dimensions and low inspection accuracy.

[0004] In the prior art, CN202563576U discloses a mobile inspection device for coal mine vertical shafts. Images are collected by a CCD camera or imaging device installed on the cage, and the image data is stored in combination with a depth measurement device and an industrial control computer. Then, the computer processes the images to judge potential safety hazards. However, this technology only relies on video acquisition information and does not integrate audio information, so it is impossible to alarm and monitor non-visual abnormal situations, and there is a possibility of missed detection.

[0005] CN102682492A provides a method and device for mobile inspection of coal mine vertical shafts, which uses an inspection robot to collect visual image information to realize real-time monitoring of the shaft environment. However, this technology still has the following problems: the collected modal data is limited to visual image information only, which cannot meet the monitoring requirements of non-visual abnormal scenarios; it is impossible to realize the centralized analysis of multi-scene and regional information and the rapid iterative optimization of algorithms; the cost of using a robot for inspection is high and the benefits are limited, so it is not feasible.

[0006] In summary, the prior art has problems such as single detection dimension, low accuracy, high cost, and difficulty in implementation in shaft inspection. There is an urgent need for an intelligent inspection solution that integrates multi-modalities to improve inspection efficiency and accuracy, reduce labor costs, and ensure mine safety. Summary of the Invention

[0007] (1) Object of the Invention

[0008] In order to overcome the above deficiencies, the object of the present invention is to provide a method and system for inspecting coal mine shafts to solve the above technical problems.

[0009] (2) Technical Solution

[0010] To achieve the above object, the technical solution provided by this application is as follows:

[0011] A method for inspecting a coal mine shaft includes the following steps:

[0012] Step 1: Multimodal data collection. Install a microphone and a camera at corresponding positions to collect sound and video data, and collect shaft specification information through sensors, and store it in a database for determining the location where an abnormality occurs;

[0013] Step 2: Data upload and forwarding. The collected sound and video data are stored as corresponding audio and video files through the control boards of the shaft wall and the cage guides, and sent to a specific ftp server, and then called to a subsequent algorithm processing module for processing; for the shaft specification information, by calling the API interface, the corresponding data source is searched in the database, and analyzed through the sensor data monitoring module;

[0014] Step 3: Abnormal video detection module. By receiving the observed video files from the shaft, using the object detection large model technology, real-time analysis of the video content for possible abnormal phenomena is realized. After detecting the abnormal phenomena, the abnormal phenomena include enlarged joints, bolt detachment, and misalignment of cage guides. The system further refines and confirms the detection results through an inference post-processing module.

[0015] Preferably, after the abnormal phenomena are detected in Step 3, the system will issue an alarm prompt, and on-site personnel will conduct verification and maintenance manually to ensure the accuracy and reliability of the detection results.

[0016] Preferably, the abnormal video detection in Step 3 includes the intelligent inspection scenario of the main shaft equipment and the intelligent inspection scenario of the auxiliary shaft equipment.

[0017] The intelligent inspection scenarios of the main shaft equipment and the auxiliary shaft equipment include the following monitoring schemes:

[0018] (1) Install a cage guide camera on the top platform of the skip. The camera faces the cage guide, records slow inspection, and when a change in the joint or misalignment of the joint is recognized, an alarm is sent to the application platform;

[0019] (2) Install a bolt camera on the top platform of the skip. The camera faces the bolt, records slow inspection, and when a bolt loss is recognized, an alarm is sent to the application platform;

[0020] (3) Install a microphone on the top platform of the skip, record production and inspection records, and when abnormal rubbing between the cage ear and the cage guide is recognized, an alarm is sent to the application platform;

[0021] The intelligent inspection scenario of the auxiliary shaft shaft equipment further includes: installing a spliced camera on the top platform of the skip to record slow inspection, identifying abnormal well walls, and alarming the application platform.

[0022] Preferably, the algorithm processing module in step 2 is a damage audio detection module. The steps of the damage audio detection module specifically include: downloading an audio file from a file source, then processing the noisy audio based on metrics such as roughness, fluctuation energy, and RMS value to eliminate various interfering sounds, then performing frame division processing on the audio signal, and extracting key features such as loudness, sharpness, roughness, and fluctuation intensity for the sound responses in different frequency bands to comprehensively extract the features in the audio signal.

[0023] Preferably, the sensor data monitoring module includes the following: the sensor records the running state of the shaft in real time, writes the state information and time information into the database, and the time alignment module reads relevant information from the database through the API during multi-modal fusion for fusion with the backend and other modules.

[0024] Preferably, step 3 further includes aligning the shaft image data, audio data, and sensors in the time dimension, and comprehensively analyzing whether there are abnormal situations in the shaft through time window constraints.

[0025] A coal mine shaft inspection system includes:

[0026] A data acquisition device, installed on the ceiling or top of the cage in the shaft, for real-time acquisition of sound signals, video data, and sensor data in the shaft. The sensor data includes shaft height information;

[0027] A data upload and forwarding module, for uploading the acquired sound signals, video data, and sensor data to the server;

[0028] An anomaly detection module, including an abnormal video detection module and a damage audio detection module. The abnormal video detection module uses object detection large model technology to perform real-time analysis on video data to detect abnormal phenomena in the shaft. The abnormal phenomena include enlarged joints and bolt loosening; the damage audio detection module processes the sound signal based on acoustic metrics, extracts key features, and determines whether there are abnormal situations;

[0029] A sensor data monitoring module, for capturing the running state of the shaft in real time and entering the state information into the database;

[0030] A time window alignment and fusion analysis module, for aligning the shaft image data, audio data, and sensor data in the time dimension, and comprehensively analyzing whether there are abnormal situations in the shaft through time window constraints;

[0031] The post - processing alarm module is used to summarize the inference results of all single modalities and the multi - modality fusion analysis results, push detailed alarm information to the mine - side central system, automatically save pictures or video evidence of abnormal situations, and upload alarm information to the group - side server.

[0032] Preferably, the data acquisition device includes:

[0033] A pick - up microphone, which is used to pick up abnormal vibration and friction sounds between the ear - filling and the roadway.

[0034] A camera, which is used to collect video data of the cage ceiling or the top of the cage, and detect abnormal images such as enlarged seams, bolt detachment in real - time.

[0035] A sensor, which is used to collect wellbore height information.

[0036] Beneficial effects:

[0037] The present invention forms a complete multi - modality fusion analysis method for coal mine wellbores. Compared with traditional manual inspection and single - modality wellbore inspection schemes, it can effectively reduce the cost of manual inspection, improve the automation rate, and improve the accuracy of inspection and monitoring. It overcomes the difficulties of low accuracy and difficult implementation cases in the existing technology, and is a wellbore inspection method that can be quickly applied and replicated. Description of the Drawings

[0038] Figure 1 It is a schematic flow chart of the present invention. Detailed Embodiments

[0039] To make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the specific embodiments and with reference to the attached Figure 1 , which is for further detailed description of the present invention. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well - known structures and technologies is omitted to avoid unnecessarily confusing the concept of the present invention.

[0040] A coal mine wellbore inspection method provided by the present invention includes the following steps:

[0041] Step 1: Multi - modality data acquisition

[0042] Install a pick - up microphone and a camera at the corresponding positions to collect sound and video data, and collect wellbore specification information through a sensor, and store it in a database for judging the location where an abnormality occurs.

[0043] Step 2: Data upload and forwarding

[0044] The collected voice and video data are stored as corresponding audio and video files through the control boards of the shaft wall and the guide rail, and sent to a specific FTP server, and then the subsequent algorithm processing module is called for processing. For the shaft specification information, the corresponding data source is searched from the database in the form of calling the API interface, and analyzed through the sensor data monitoring module.

[0045] Step 3: Abnormal video detection module

[0046] By receiving the observation video files from the shaft, using the object detection large model technology, the real-time analysis of the video content is realized, and the possible abnormal phenomena are detected, such as the seam becoming larger, the bolt falling off, and the guide rail misalignment. After detecting the abnormal phenomena, the system further refines and confirms the detection results through the inference post-processing module.

[0047] Preferably, after the abnormal phenomena are detected in step 3, the system will issue an alarm prompt, and on-site personnel will conduct verification and maintenance manually to ensure the accuracy and reliability of the detection results.

[0048] Preferably, the abnormal video detection in step 3 includes the intelligent inspection scenarios of the main shaft equipment and the auxiliary shaft equipment. The specific monitoring schemes are as follows:

[0049] Install a guide rail camera on the top platform of the skip. The camera faces the guide rail, records the slow inspection, identifies the seam change or misalignment, and alarms the application platform.

[0050] Install a bolt camera on the top platform of the skip. The camera faces the bolt, records the slow inspection, identifies the bolt loss, and alarms the application platform.

[0051] Install a microphone on the top platform of the skip, record the production and inspection records, identify the abnormal rubbing of the cage ear and the guide rail, and alarm the application platform.

[0052] The intelligent inspection scenario of the auxiliary shaft equipment also includes: installing a stitching camera on the top platform of the skip, recording the slow inspection, identifying the shaft wall abnormality, and alarming the application platform.

[0053] Preferably, the algorithm processing module in step 2 is a damage audio detection module. The specific steps include:

[0054] Download the audio file from the file source.

[0055] Process the noisy audio based on the indicators of roughness, fluctuation energy, and RMS value to eliminate various interference sounds.

[0056] Frame the audio signal, and extract the key features of loudness, sharpness, roughness, and fluctuation intensity for the sound responses in different frequency bands, and extract the features in the audio signal in all directions.

[0057] Preferably, the sensor data monitoring module includes the following:

[0058] The sensor records the running state of the wellbore in real time and writes the state information and time information into the database.

[0059] During multimodal fusion, the time alignment module reads relevant information from the database through the API for fusion with the backend and other modules.

[0060] Preferably, step 3 also includes aligning the wellbore image data, audio data, and sensors in the time dimension, and comprehensively analyzing whether there are abnormal conditions in the wellbore through time window constraints.

[0061] A coal mine shaft inspection system includes the following modules:

[0062] Data acquisition device

[0063] Installed on the ceiling or top of the cage in the wellbore, it is used to collect the sound signal, video data, and sensor data in the wellbore in real time. The sensor data includes the wellbore height information.

[0064] Data upload and forwarding module

[0065] Used to upload the collected sound signal, video data, and sensor data to the server.

[0066] Abnormality detection module

[0067] Includes an abnormal video detection module and a damaged audio detection module:

[0068] Abnormal video detection module: Using object detection large model technology to analyze the video data in real time and detect abnormal phenomena in the wellbore, such as enlarged seams and bolt loosening.

[0069] Damaged audio detection module: Processes the sound signal based on acoustic metrics, extracts key features, and determines whether there are abnormal conditions.

[0070] Sensor data monitoring module, used to capture the running state of the wellbore in real time and enter the state information into the database.

[0071] Time window alignment and fusion analysis module, used to align the wellbore image data, audio data, and sensor data in the time dimension, and comprehensively analyze whether there are abnormal conditions in the wellbore through time window constraints.

[0072] Post-processing alarm module, used to summarize all single-modal inference results and multimodal fusion analysis results, and push detailed alarm information to the mine-side central system. At the same time, it automatically saves pictures or video evidence of abnormal situations and uploads the alarm information to the group-side server.

[0073] Preferably, the data acquisition device includes:

[0074] A pickup: used to pick up abnormal vibrations and frictional sounds between the ear irrigation and the channel irrigation.

[0075] A camera: used to collect video data of the cage ceiling or the top of the cage, and to detect enlarged seams, bolt detachment or other abnormal images in real time.

[0076] A sensor: used to collect shaft height information.

[0077] The beneficial effects of the present invention include:

[0078] 1. Improve inspection efficiency: Through automated and intelligent multimodal data acquisition and analysis, the time and workload of manual inspection are reduced.

[0079] 2. Enhance detection accuracy: By combining multimodal fusion analysis of audio, video and sensor data, abnormal conditions in the shaft can be detected more comprehensively, reducing the missed detection rate.

[0080] 3. Real-time monitoring and alarm: The system can analyze data in real time and issue alarm prompts to promptly discover and handle potential safety hazards, ensuring the safety of the mine.

[0081] 4. Reduce labor costs: Reducing the dependence on manual inspection, lowering labor costs, while improving the reliability and efficiency of inspection.

[0082] 5. Strong scalability: The system architecture supports the fusion of multiple modal data and can be further expanded and optimized according to actual needs.

[0083] A method for inspecting a coal mine shaft, the operation process of which is as follows: First, install a pickup, a camera and a sensor on the cage ceiling or the top of the cage in the shaft to collect sound signals, video data and specification information such as the shaft height in real time, and store the sensor data in a database for subsequent determination of the location where an abnormality occurs; Next, the collected sound and video data are stored as corresponding audio and video files through the control board of the shaft wall and the guide rail, and sent to a specific FTP server, while the shaft specification information is retrieved from the database by calling the API interface to find the corresponding data source, and analyzed by the sensor data monitoring module; Then, the abnormal video detection module receives the observed video file from the shaft, and uses the object detection large model technology to analyze the video content in real time to detect whether there are abnormal phenomena, such as enlarged seams, bolt detachment, guide rail misalignment, etc. After detecting an abnormal phenomenon, the system further refines and confirms the detection result through the inference post-processing module; At the same time, the damage audio detection module downloads the audio file from the file source, processes the noisy audio based on acoustic indicators such as roughness, fluctuation energy and RMS value, eliminates the interference sound, then performs frame splitting on the audio signal, extracts key features such as loudness, sharpness, roughness and fluctuation intensity, and determines whether there is an abnormal situation; In addition, the sensor data monitoring module records the operation status of the shaft in real time, writes the status information and time information into the database, and the time alignment module reads the relevant information from the database through the API during multi-modal fusion for fusion with the backend and other modules; Finally, the time window alignment and fusion analysis module aligns the shaft image data, audio data and sensor data in the time dimension, and comprehensively analyzes whether there are abnormal situations in the shaft through time window constraints. The post-processing alarm module summarizes all the inference results of single modalities and the multi-modal fusion analysis results, and pushes the detailed alarm information to the mine-side central system. The mine-side system automatically saves the picture or video evidence of the abnormal situation, and synchronously uploads the alarm information to the group-side server. The on-site staff conducts manual verification and maintenance according to the alarm prompt to ensure the accuracy and reliability of the detection results.

[0084] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or device including the said element.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for inspecting a coal mine shaft, characterized in that, It includes the following steps: Step 1: Multimodal data collection. Install a pickup and a camera at the corresponding positions to collect sound and video data, and collect wellbore specification information through sensors and store it in a database for determining the location where an abnormality occurs. Step 2: Data upload and forwarding. The collected sound and video data are stored as corresponding audio and video files through the control boards of the shaft wall and the cage guides, and sent to a specific FTP server, and then call the subsequent algorithm processing module for processing; for the wellbore specification information, find the corresponding data source in the database by calling the API interface, and analyze it through the sensor data monitoring module. Step 3: Abnormal video detection module. By receiving the observed video files from the wellbore and using the object detection large model technology, it realizes the real-time analysis of the video content for possible abnormal phenomena. After detecting the abnormal phenomena, the abnormal phenomena include enlarged joints, bolt detachment, and misalignment of cage guides. The system further refines and confirms the detection results through the inference post-processing module.

2. The method for inspecting a coal mine shaft according to claim 1, wherein, After the abnormal phenomena are detected in Step 3, the system will issue an alarm prompt, and on-site personnel will conduct verification and maintenance manually to ensure the accuracy and reliability of the detection results.

3. The coal mine shaft inspection method according to claim 1, characterized in that, The abnormal video detection in Step 3 includes the intelligent inspection scenario of the main shaft wellbore equipment and the intelligent inspection scenario of the auxiliary shaft wellbore equipment. The intelligent inspection scenarios of the main shaft wellbore equipment and the auxiliary shaft wellbore equipment include the following monitoring schemes: (1) Install a cage guide camera on the top platform of the skip. The camera faces the cage guide, records the slow inspection, and when a joint change or joint misalignment is identified, it alarms the application platform. (2) Install a bolt camera on the top platform of the skip. The camera faces the bolts, records the slow inspection, and when a bolt loss is identified, it alarms the application platform. (3) Install a pickup on the top platform of the skip to record production and inspection records. When abnormal rubbing between the cage ear and the cage guide is identified, it alarms the application platform. The intelligent inspection scenario of the auxiliary shaft wellbore equipment further includes: Install a stitching camera on the top platform of the skip to record the slow inspection. When an abnormality of the shaft wall is identified, it alarms the application platform.

4. A coal mine shaft inspection method according to claim 1, characterized in that, The algorithm processing module in Step 2 is a damage audio detection module. The steps of the damage audio detection module specifically include: Download the audio file from the file source, and then process the noisy audio based on the indicators of roughness, fluctuation energy, and RMS value to eliminate various interference sounds. Then, perform frame processing on the audio signal, and extract the key features of loudness, sharpness, roughness, and fluctuation intensity for the sound responses in different frequency bands to extract the features in the audio signal comprehensively.

5. A coal mine shaft inspection method according to claim 1, characterized in that, The sensor data monitoring module includes the following: The sensor records the running state of the wellbore in real time, writes the state information and time information into the database, and the time alignment module reads the relevant information from the database through the API during multimodal fusion for fusion with the backend and other modules.

6. The coal mine shaft inspection method according to claim 1, wherein Step 3 further includes aligning the wellbore image data, audio data, and sensors in the time dimension, and comprehensively analyzing whether there are abnormal situations in the wellbore through time window constraints.

7. A coal mine shaft inspection system for implementing the coal mine shaft inspection method according to any one of claims 1 to 6, characterized in that, It includes: A data acquisition device is installed on the cage ceiling or the top of the cage in the shaft and is used to collect the sound signals, video data, and sensor data in the shaft in real time. The sensor data includes the shaft height information; A data upload and forwarding module is used to upload the collected sound signals, video data, and sensor data to the server; An anomaly detection module includes an abnormal video detection module and a damaged audio detection module. The abnormal video detection module uses object detection large model technology to analyze the video data in real time and detect abnormal phenomena in the shaft. The abnormal phenomena include enlarged seams and bolt detachment. The damaged audio detection module processes the sound signals based on acoustic metrics, extracts key features, and determines whether there are abnormal situations; A sensor data monitoring module is used to capture the operating state of the shaft in real time and record the status information in the database; A time window alignment and fusion analysis module is used to align the shaft image data, audio data, and sensor data in the time dimension and comprehensively analyze whether there are abnormal situations in the shaft through time window constraints; A post-processing alarm module is used to summarize all single-modal inference results and multi-modal fusion analysis results, push detailed alarm information to the mine-side central system, automatically save pictures or video evidence of abnormal situations, and upload the alarm information to the group-side server.

8. The coal mine shaft inspection system according to claim 7, characterized in that, The data acquisition device includes: A microphone is used to pick up abnormal vibration and friction sounds between the cage ears and the cage guides; A camera is used to collect video data of the cage ceiling or the top of the cage and detect enlarged seams, bolt detachment, or other abnormal images in real time; A sensor is used to collect the shaft height information.

Images