Heading machine early warning linkage system and method based on thermal imaging

By installing an AI system with thermal imaging camera and edge computing gateway modules in the coal mine, we can monitor the surrounding environment and temperature of the boring machine in real time, identify potential safety hazards and issue early warnings, solving the problems of boring machine environmental monitoring and equipment status detection, and improving the Coal mine underground safety and production efficiency.

CN118934054AActive Publication Date: 2024-11-12ZHENGZHOU HENGDA INTELLIGENT CONTROL TECHNOLOGY CO LTD +1

Patent Information

Application Number
CN202411146320.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-12
Estimated Expiration
2044-08-20

AI Technical Summary

Technical Problem

During the underground excavation work of coal mines, due to the limited dust and field of view, the existing video surveillance system is difficult to effectively monitor the surrounding environment of the excavator, which increases the risk of safety accidents. The high temperature of the excavator head poses a threat to the stability of the equipment, making it difficult to accurately detect personnel and equipment. state.

Method used

Using a thermal imaging-based boring machine early warning linkage system, the environment and temperature data of the boring machine are monitored in real time through multiple intrinsically safe mining thermal imaging cameras, and combined with the AI ​​inference engine of the edge computing gateway module to conduct real-time analysis to identify potential safety hazards. , and issue early warning and shutdown instructions through the sound and light alarm and equipment control unit to realize emergency stop and automatic reset of the boring machine.

Benefits of technology

It improves the perceived accuracy of the environment around the boring machine, reduces blind spots in the field of vision of staff, ensures safe and efficient production underground of coal mines, quickly identify and respond to safety hazards, and reduces production risks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118934054A_ABST
    Figure CN118934054A_ABST
Patent Text Reader

Abstract

The invention discloses a heading machine early warning linkage system and method based on thermal imaging, and relates to the technical field of coal mine underground intelligent monitoring. The heading machine early warning linkage system comprises a data acquisition module, an edge computing gateway module, a sound-light alarm module and a terminal monitoring module; the data acquisition module is used for monitoring thermal imaging video streams and temperature data around the underground heading machine in real time, and the edge computing gateway module is used for identifying potential safety hazards and immediately sending an instruction to the sound-light alarm module once an abnormal condition is found, so that the sound-light alarm module sends out a sound-light alarm signal and controls the running state of the heading machine at the same time; and the terminal monitoring module is used for receiving the early warning information sent by the edge computing gateway module and displaying surrounding monitoring information of the heading machine on a monitoring interface in real time. According to the invention, the working condition sensing of the area near the heading machine is effectively realized, the view blind area of workers during the heading operation is effectively reduced, and the safe and efficient production of the underground coal mine is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of underground intelligent monitoring in coal mines, and in particular relates to a roadheader early warning linkage system and method based on thermal imaging. Background Art

[0002] As an important energy source in my country, coal provides a basic guarantee for promoting my country's industrialization process. Faced with the complex environment of coal mines, the application of video-based AI monitoring technology in coal mines effectively separates workers from dangerous environments and ensures safe production in coal mines.

[0003] However, during tunneling work, a large amount of dust fills the air, making it difficult for ordinary visible light cameras to capture clear images, which poses a considerable challenge to the effective operation of the monitoring system. At the same time, during operation, the tunneling machine driver has difficulty in fully grasping the dynamic situation of the operating area due to limited vision. Once a worker mistakenly enters a dangerous area, the driver may not be able to discover it in time, thereby increasing the risk of safety accidents.

[0004] In addition, the tunneling head generates extremely high temperatures during operation, which poses a serious threat to the stability and life of the equipment. Once the equipment fails, it will not only affect production efficiency, but may also cause safety accidents. Therefore, how to accurately detect the status of personnel and equipment operation in the tunneling work area and intelligently link it with the tunneling equipment has become an urgent problem to be solved in the current coal mine safety production. Summary of the invention

[0005] In view of the defects and problems existing in the prior art, the present invention provides a tunnel boring machine early warning linkage system and method based on thermal imaging. The present invention effectively solves the problem of perception of working conditions in the area near the tunnel boring machine, effectively reduces the blind spots of vision of workers during tunneling operations, and ensures safe and efficient production in underground coal mines.

[0006] The solution adopted by the present invention to solve its technical problem is: a tunnel boring machine early warning linkage system based on thermal imaging, including a data acquisition module, an edge computing gateway module, an audible and visual alarm module and a terminal monitoring module; the data acquisition module is used to monitor the thermal imaging video stream and temperature data around the underground tunnel boring machine in real time, and transmit the data to the edge computing gateway module; the edge computing gateway module has a built-in inference engine for real-time analysis of the received data to identify potential safety hazards, and once an abnormal situation is found, the edge computing gateway module immediately issues an instruction to the audible and visual alarm module to make it issue an audible and visual alarm signal, and at the same time sends a control instruction to the equipment control unit in the module to control the operating status of the tunnel boring machine; the terminal monitoring module is communicatively connected to the edge computing gateway module, supports remote monitoring and real-time data transmission, is used to receive the early warning information sent by the edge computing gateway module, and displays the monitoring information around the tunnel boring machine in real time on the monitoring interface.

[0007] Furthermore, the data acquisition module relies on multiple mining intrinsically safe thermal imaging cameras installed on the tunnel boring machine. One camera is installed on each side of the front end of the tunnel boring machine according to the width of the camera's field of view, and is responsible for monitoring the equipment temperature and personnel status in the area near the head of the tunnel boring machine; one camera is also installed on each side of the rear of the tunnel boring machine, and is responsible for detecting the personnel status at the rear of the tunnel boring machine.

[0008] Furthermore, the edge computing gateway module is integrated into the artificial intelligence computing core board, including a model reasoning unit, a temperature acquisition unit, a business processing unit, a device control unit, a data storage unit and a web service unit; the model reasoning unit has a built-in AI reasoning engine to perform real-time intelligent analysis of the video stream, identify violations, and send the detected warning information to the business processing unit; the business processing unit receives these warning information, and performs logical judgment according to preset rules and thresholds, and then issues corresponding warnings or shutdown information; the temperature acquisition unit is used to obtain the temperature data matrix from the data acquisition module in real time, and synchronize time with the video stream data in the model reasoning unit. The model reasoning unit detects the position information of the target in the thermal imaging image, and this information is then used by the temperature acquisition unit to determine the specific position of the target in the temperature data matrix.

[0009] Furthermore, the business processing unit will communicate the early warning information issued with the sound and light alarm module through the RS485 protocol to realize the sound and light alarm; and send the shutdown information to the equipment control unit through the internal protocol; the business types of the business processing unit include personnel detection in the dangerous area at the front end of the tunnel boring machine, high temperature monitoring of the tunnel boring head, and personnel detection in the dangerous area at the rear end of the tunnel boring machine.

[0010] Furthermore, after receiving the shutdown information, the device control unit converts it into a DIDO switching signal and sends it to the control system of the tunnel boring machine to realize the emergency stop operation of the tunnel boring machine; at the same time, the device control unit will automatically perform a reset operation within a certain time after the emergency stop, allowing the tunnel boring machine to restart.

[0011] Furthermore, the data storage unit is used to establish and manage Mysql and Minio databases to realize the storage of alarm information, images and videos, and provide corresponding external query interfaces to ensure data retention and user historical data query needs.

[0012] Furthermore, the web-based service unit is used to display real-time analysis results of the video stream collected by the tunnel boring machine, and at the same time provides functions of real-time pop-up alarms, historical data queries, and alarm statistical analysis; the unit adopts a B / S architecture, and the client can directly access the WEB-based interface via a web page through the industrial ring network, thereby realizing real-time monitoring and browsing of the surrounding area of ​​the tunnel boring machine.

[0013] Furthermore, the terminal monitoring module includes an underground flat-panel super terminal, an above-ground monitoring host and a mine-used intrinsically safe monitoring host; wherein the underground flat-panel super terminal realizes communication with the edge computing gateway module through a WIFI module; the mine-used intrinsically safe monitoring host directly communicates using the RJ45 network port of the edge computing gateway module; the above-ground monitoring host communicates with the edge computing gateway module through an underground ring network; the terminal monitoring module can realize mobile terminal browsing of the original video stream, the analyzed AI video stream, the alarm message, and the statistical analysis information by accessing the WEB service of the edge computing gateway module.

[0014] The present invention also provides a tunnel boring machine early warning linkage method based on thermal imaging, comprising the following steps:

[0015] S1. Use the mine intrinsically safe thermal imaging camera installed on the tunnel boring machine to monitor and sense the temperature of the surrounding area of ​​the underground tunnel boring machine in real time, so as to obtain a real-time thermal imaging video stream (..., I n-2 , I n-1 , I n ) and the temperature data matrix (..., T n-2 , T n-1 , T n ) and transmit these data to the edge computing gateway module via Ethernet;

[0016] S2. The model reasoning unit in the edge computing gateway module uses the standard rtsp protocol to obtain video stream data in real time, and performs real-time intelligent analysis on the received thermal imaging video stream to detect the staff and tunneling heads in the four cameras; at the same time, the temperature acquisition unit obtains the temperature data matrix in real time, and uses the camera private protocol to update the temperature map of the camera monitoring area in real time; the model reasoning unit detects the position of the tunneling head and the staff, and synchronizes the video stream data with the temperature data matrix to determine the maximum temperature of the target;

[0017] S3. The business processing unit makes a logical judgment on the detection result of the model reasoning unit according to the preset rules to determine whether to issue a warning or shutdown information; if a dangerous situation is detected, the business processing unit sends a warning information to the sound and light alarm module through the RS485 protocol, and controls the equipment control unit to execute the emergency stop operation of the tunnel boring machine through the internal protocol;

[0018] S4. After receiving the warning information, the optical alarm module will alert the staff through voice broadcast and light warning; after receiving the emergency stop signal, the equipment control unit will convert the signal into a DIDO switch signal and send it to the tunnel boring machine control system to perform the emergency stop operation, and automatically reset after 10 seconds;

[0019] S5. At the same time, the WEB service unit displays the real-time analysis results and provides functions such as real-time pop-up alarm, historical data query, and alarm statistical analysis. Users can access this information through the web page. Underground workers and surface managers can access the WEB service through the terminal monitoring module to monitor the working status of the tunnel boring machine and receive alarm information in real time, thereby realizing centralized monitoring and management of the entire tunnel boring machine early warning linkage system.

[0020] Furthermore, the steps S5-S8 include the following three types of alarm processes:

[0021] The first type of alarm process: When a worker is detected in the screen of the two cameras at the front end of the tunnel boring machine and breaks into the set ROI dangerous area, an alarm message of personnel intrusion into the dangerous area and a shutdown message will be issued. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "personnel intrusion into the dangerous area" accompanied by flashing lights; after the equipment control unit obtains the shutdown information through the internal protocol, it will close the emergency stop switch circuit to implement the emergency stop operation of the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds.

[0022] The second type of alarm process: When the tunneling head is detected in the two camera images at the front of the tunnel boring machine and the temperature value exceeds the set temperature, an alarm message of over-temperature and shutdown information of the equipment will be issued. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "equipment temperature is too high" accompanied by flashing lights; after the equipment control unit obtains the shutdown information through the internal protocol, it will close the emergency stop switch circuit to realize the emergency stop operation of the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds;

[0023] The third type of alarm process: When the two cameras at the rear of the tunnel boring machine detect that someone has entered the dangerous area of ​​the drawn ROI, an alarm message will be issued that the person is approaching the tunnel boring machine. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "person approaching the tunnel boring machine" accompanied by flashing lights; and at the same time as the early warning is triggered, the business processing unit calls the internal interface to store the video frame that currently triggers the early warning, and at the same time, it will record the video for 10 seconds and upload the early warning data to the database.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The thermal imaging-based tunnel boring machine early warning linkage system and method provided by the present invention have significant advantages in improving mine operation safety, enhancing monitoring efficiency and reducing production risks. The present invention utilizes thermal imaging technology for environmental monitoring, which can penetrate dust and darkness and provide clearer images than traditional visible light cameras, especially in dusty and poorly lit environments such as coal mines. The system can monitor the environment and temperature around the tunnel boring machine in real time, and perform real-time analysis through the edge computing gateway module to quickly identify potential safety hazards. It can also achieve time synchronization between video stream data and temperature data matrix to ensure the accuracy and real-time performance of temperature detection.

[0026] The system of the present invention can automatically issue corresponding alarm information according to different monitoring results to achieve rapid response; through the automated equipment control unit, the system can realize emergency stop and automatic reset operations on the tunnel boring machine, thereby improving the efficiency of safety management; the system can realize centralized monitoring and management of the entire tunnel boring machine early warning linkage system, which is beneficial to improving management efficiency and system stability. Through automated and intelligent safety management, the system of the present invention can reduce production interruptions caused by misoperation or delayed response while ensuring safety, thereby improving production efficiency.

[0027] The method of the present invention uses a thermal imaging camera to obtain the thermal imaging video stream and temperature map of the monitoring points near the tunnel boring machine; by training the corresponding target detection inference model for the scene application, the real-time detection of the staff and the tunnel boring machine drill bit in the thermal imaging video stream is realized; according to the position of the tunnel boring head in the corresponding temperature map, the relevant temperature of the tunnel boring machine drill bit is obtained, and when the temperature is higher than the set threshold, the equipment high temperature alarm is issued; according to the identified staff position, the set dangerous area is judged to realize the corresponding personnel intrusion alarm; at the same time, the equipment control unit realizes the emergency stop control of the tunnel boring machine after receiving the issued warning message. The implementation of this method effectively solves the perception of the working conditions in the area near the tunnel boring machine, effectively reduces the blind spots of the staff when performing tunneling operations, and ensures safe and efficient production in coal mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the connection of the system of the present invention;

[0029] Figure 2 This is a system connection diagram of the edge computing gateway module of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of the artificial intelligence computing core board of the present invention;

[0031] Figure 4 It is a schematic diagram of the process structure of the method of the present invention;

[0032] Figure 5 This is a thermal imaging monitoring screen diagram of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0034] Embodiment 1:

[0035] This embodiment provides a tunnel boring machine early warning linkage system based on thermal imaging, such as Figure 1 As shown, it mainly includes a data acquisition module, an edge computing gateway module, an audio-visual alarm module and a terminal monitoring module. The data acquisition module monitors the surrounding environment and temperature data of the underground tunneling machine in real time, and transmits the data to the artificial intelligence edge computing gateway module. The edge computing gateway module uses a built-in algorithm to perform real-time analysis on the received data to identify potential safety hazards. Once an abnormal heat source or safety hazard is found, an alarm message will be immediately sent to the audio-visual alarm module to cause it to send an audio-visual alarm signal. At the same time, the terminal monitoring module can provide real-time tunneling machine monitoring information for underground staff and surface managers, and can also centrally monitor and manage the entire tunneling machine early warning linkage system to ensure the stable operation of the system. The specific contents are as follows.

[0036] The above data acquisition module relies on multiple mine-use intrinsically safe thermal imaging cameras installed on the tunnel boring machine, see Figure 5 This camera solves the problem that traditional visible light cameras cannot be cleaned and capture live images due to heavy dust on the excavation working surface. During the actual installation, one camera is installed on each side of the front end of the tunnel boring machine according to the width of the camera's field of view, responsible for monitoring the equipment temperature and personnel status in the area near the head of the tunnel boring machine; one camera is also installed on each side of the rear end of the tunnel boring machine, mainly responsible for detecting the personnel status at the rear end of the tunnel boring machine, ensuring that there are no blind spots around the tunnel boring machine equipment.

[0037] In this way, by installing the mine intrinsically safe thermal imaging camera at the corresponding positions of the head and tail ends, it is possible to collect visible light imaging, thermal imaging and ambient temperature data in the coal mine. The thermal imager supports communication protocols such as RTSP, SDK, and ONVIF. In this embodiment, the thermal imaging video stream and temperature data matrix collected by the mine intrinsically safe thermal imaging camera use Ethernet as the medium, and use the standard RTSP video stream protocol to realize data transmission with the model reasoning unit in the edge computing gateway module, and use the SDK (Software Development Kit) communication protocol to realize data transmission with the temperature acquisition unit in the edge computing gateway module. The resolution of the thermal imaging video stream is 1280X960, and the size of the temperature data matrix is ​​256X192.

[0038] like Figure 3 As shown, the edge computing gateway module is integrated into the artificial intelligence computing core board, and its external communication interface includes 1) RS-485 bus: used for data communication with the sound and light alarm module; 2) CAN bus: used to communicate with DIDO switch and other equipment, and to control equipment such as tunneling machines according to actual on-site needs; 3) RJ45 network port: used to obtain environmental data collected by the camera; 4) WIFI module: supports data transmission between wireless devices; 5) Optical port: used to access the 10G optical fiber ring network used in the mine, providing high-speed and stable data transmission. The above communication interfaces can meet the data exchange needs of different devices and systems.

[0039] like Figure 2As shown in the figure, the module contains a model reasoning unit, a temperature acquisition unit, a business processing unit, a device control unit, a data storage unit, and a web-side service unit. It has a built-in AI reasoning engine that can perform real-time intelligent analysis of video streams, thereby effectively identifying violations. At the same time, once a violation is detected, the alarm mechanism can be triggered immediately, and the data transmission with the sound and light alarm module will be used for early warning, and the alarm information will be thrown to the web-side service unit. At the same time, the emergency stop switch of the tunnel boring machine is controlled by the DIDO switch quantity to perform a shutdown operation to ensure safety. In addition, the video stream after AI analysis and processing can be forwarded through a variety of standard protocols, such as RTSP, RTMP, FLV, etc., according to needs to meet the access and playback requirements of different terminal devices or background servers.

[0040] Furthermore, the model reasoning unit is placed in the edge computing gateway module to provide a target detection reasoning engine for relevant personnel and tunneling heads. By collecting the on-site thermal imaging video stream and annotating the data set, a VOC (Visual Object Class) data set containing two categories of tunneling heads and staff is obtained, and the model is trained using the constructed YOLOv5m artificial neural network training environment. The trained YOLOv5m model can accurately identify tunneling heads and staff, and give the accurate position of the detected target in the image coordinate system. In order to adapt to the framework structure of the artificial intelligence computing core board, the real-time analysis of the thermal imaging video stream collected by the thermal imaging camera can be achieved by converting the trained YOLOv5m model into a corresponding target reasoning engine and loading it into the artificial intelligence computing core board.

[0041] The temperature acquisition unit is used to acquire the temperature data matrix output by the thermal imaging camera in real time, and realize time synchronization with the video stream data in the model reasoning unit. Specifically, the upper left corner position and size (x, y, w, h) in the coordinate system of the thermal imaging image detected by the above-mentioned model reasoning unit are obtained, and the position of the target in the temperature data matrix is ​​obtained by coordinate system conversion, and the data of the area where the target is located in the temperature data matrix is ​​screened, so as to obtain the highest temperature of the area where the target is located, which is the temperature value of the current target.

[0042] The business processing unit is mainly used to make logical judgments on relevant warning information and issue warning or shutdown information, where the warning information communicates with the sound and light alarm module through the RS485 protocol to realize the sound and light alarm, and the shutdown information exchanges data with the equipment control unit through the internal protocol. This unit contains the following three services.

[0043] (1) Personnel detection in the dangerous area at the front end of the tunnel boring machine

[0044] With respect to the two thermal imaging cameras at the front end of the tunnel boring machine, when the inference engine in the model inference unit detects that a worker has entered the set danger zone, the business processing unit immediately issues a warning message and a shutdown message.

[0045] (2) High temperature monitoring of tunneling head

[0046] For the two thermal imaging cameras at the front end of the tunnel boring machine, the inference engine will detect the position of the tunnel boring head in real time, and use the coordinate system conversion to obtain the tunnel boring head temperature in the temperature map. When the tunnel boring head temperature is monitored to be higher than the set temperature threshold, the business processing unit will issue a high temperature warning and shutdown information.

[0047] (3) Personnel detection in the dangerous area behind the tunnel boring machine

[0048] For the two thermal imaging cameras at the rear end of the tunnel boring machine, when the inference engine detects that relevant staff have entered the set ROI danger zone, the business processing unit will issue an early warning message of approaching personnel.

[0049] The unit can make accurate logical judgments on warning information based on preset rules and thresholds. Once a dangerous situation is detected, it can quickly issue a warning or shutdown information to achieve a rapid response to the dangerous situation. It also supports a variety of warning services and rule configurations to meet safety needs in different scenarios, effectively prevent potential safety accidents, and improve production efficiency and safety.

[0050] The device control unit obtains the device emergency stop information from the business processing unit through the internal protocol. After receiving the device emergency stop information, it will immediately convert this signal into a DIDO (Digital Input / Output) switch signal and send it to the control system of the tunnel boring machine. After receiving the DIDO switch signal, the tunnel boring machine control system will perform an emergency stop operation to ensure that the equipment stops running quickly, thereby ensuring the safety of personnel. After the tunnel boring machine stops suddenly, in order to avoid the staff being unable to restart the tunnel boring machine and affecting the work efficiency, the device control unit will automatically perform a reset operation after 10 seconds. The reset operation will restore the DIDO switch signal to the initial state, allowing the tunnel boring machine control system to re-accept the start-up command, ensuring that the equipment can restart normally and continue to work. In this way, through the DIDO switch signal, the device control unit can accurately control the operating state of the tunnel boring machine, thereby realizing emergency stop and reset operations, and at the same time has an automatic reset function, which can automatically perform a reset operation after the tunnel boring machine stops suddenly to avoid affecting work efficiency. Through the collaborative work of the equipment control unit and other modules such as the business processing unit and the model reasoning unit, the edge computing gateway module can realize real-time monitoring and safety management of the tunnel boring machine, effectively prevent potential safety accidents, and improve production efficiency and work safety.

[0051] The data storage unit is also placed in the edge computing gateway module. The unit mainly establishes and manages two types of databases: Mysql and Minio. The Mysql database is used to store structured data such as alarm information, device logs, configuration parameters, etc., to provide efficient data query, insertion, update and deletion operations; complex data analysis and statistics are performed through the SQL language. The Minio database is a high-performance object storage server that is compatible with the Amazon S3 cloud storage service interface. It is mainly used to store unstructured data such as images and videos, providing persistent storage and backup functions for data, and supporting distributed storage and expansion of data. The data storage unit provides a RESTful API or a similar interface, allowing external systems or users to query the data stored in the database through HTTP requests, ensuring data retention and user historical data query needs. The unit realizes the storage of alarm information, images and videos, and can preserve historical data for a long time to ensure the traceability and auditability of the data.

[0052] The web-side service unit is mainly used to display the real-time analysis results of the 4-channel video streams near the tunnel boring machine, and also provides functions such as real-time pop-up alarm, historical data query, and alarm statistical analysis. With the advanced B / S architecture, the client can directly access the WEB-side interface through the industrial ring network and the web page, thereby realizing real-time monitoring and browsing around the tunnel boring machine.

[0053] Each unit in the above edge computing gateway module is connected to external devices through internal protocols and structures, which can realize real-time acquisition, processing, analysis and control of data. Through close collaboration between the units, the entire system can be ensured to operate efficiently and safely, thereby realizing safe monitoring and safety management of the mine environment.

[0054] The sound and light alarm module relies on the sound and light alarm, which is mainly composed of RS-485 bus, micro control unit, speech synthesis, attack and defense, and lighting control (LED) units. It uses RS485 bus to obtain the warning information issued by the business processing unit, and can realize voice broadcast and light warning. The light includes three colors: red, yellow, and green; the voice broadcast supports local playback and uses the Chinese national standard code GB2312 to realize online customized warning broadcast.

[0055] The terminal monitoring module includes an underground flat-panel super terminal, an above-ground monitoring host, and a mine-used intrinsically safe monitoring host, wherein the underground flat-panel super terminal can communicate with the artificial intelligence computing gateway through a WIFI module, and by accessing the WEB-side service of the artificial intelligence edge computing gateway, the mobile terminal can browse the original video stream, the analyzed AI video stream, the alarm message, the statistical analysis and other information. The mine-used intrinsically safe monitoring host directly communicates with the RJ45 network port of the artificial intelligence edge computing gateway. And by accessing the WEB-side service of the artificial intelligence edge computing gateway, the mobile terminal can browse the original video stream, the analyzed AI video stream, the alarm message, the statistical analysis and other information. The above-ground monitoring host communicates with the artificial intelligence gateway through the underground ring network. And by accessing the WEB-side service of the artificial intelligence edge computing gateway, the mobile terminal can browse the original video stream, the analyzed AI video stream, the alarm message, the statistical analysis and other information.

[0056] Staff can access the WEB services in the artificial intelligence gateway via web pages through the configured tablet super terminal, well monitoring host, and mine intrinsically safe monitoring host. At this time, users can browse the original video stream, analyzed AI video stream, alarm messages, statistical analysis and other information.

[0057] The tunnel boring machine early warning linkage system based on thermal imaging provided by the present invention generates visible thermal radiation images through an infrared thermal imager, and generates clear images according to the temperature difference between objects, so that even small details can be seen, thereby providing high-precision monitoring. The system combines artificial intelligence technology to perform characteristic analysis on video images, generate abnormal data values, and automatically determine whether an alarm needs to be issued according to a preset alarm threshold, thereby improving the intelligence level of the system. The system provides a strong guarantee for coal mine safety production through its advantages of comprehensive detection, high precision, all-weather working capability, low missed reporting rate, efficient early warning linkage, improved operation safety and intelligent management, etc., effectively improves operation efficiency and the convenience of data management, and provides an effective technical solution for mine safety management.

[0058] Embodiment 2:

[0059] Based on Example 1, this example provides a tunnel boring machine early warning linkage method based on thermal imaging, see Figure 4 , which includes the following steps:

[0060] S1. Data collection: The mine intrinsically safe thermal imaging camera installed on the tunnel boring machine is used to monitor and sense the temperature of the surrounding area of ​​the underground tunnel boring machine in real time, so as to obtain a real-time thermal imaging video stream (..., I n-2 , I n-1 , I n ) and the temperature data matrix (..., Tn-2 , T n-1 , T n ), and transmit these data to the edge computing gateway module via Ethernet.

[0061] S2. Data transmission: The thermal imaging video stream and temperature data matrix collected by the thermal imaging camera are transmitted to the edge computing gateway module through the standard RTSP video streaming protocol and SDK communication protocol.

[0062] S3. Edge computing analysis: The model reasoning unit in the edge computing gateway module uses the standard RTSP protocol to obtain video stream data in real time, and performs real-time intelligent analysis on the received thermal imaging video stream to detect the staff and tunneling heads in the four cameras; at the same time, the temperature acquisition unit obtains the temperature data matrix in real time, that is, it uses the camera's private protocol to update the temperature map of the camera monitoring area in real time.

[0063] S4, target detection and temperature synchronization: The model reasoning unit detects the positions of the tunneling head and the staff, and synchronizes the video stream data with the temperature data matrix to determine the maximum temperature of the target. Furthermore, the coordinate system conversion formula is used to convert the position and size in the image coordinate system into the corresponding values ​​in the temperature data matrix, and the area where the target is located in the temperature data matrix is ​​screened to obtain the maximum temperature value as the temperature of the current target.

[0064] Specifically, taking the detection of workers as an example, when the model reasoning unit detects the position and size (x i ,y i , w i ,h i ), the position and size (x t ,y t , w t ,h t ). Given that the image resolution is 1280×960 and the size of the temperature data matrix is ​​256×192, we can see that the relationship between the two coordinate systems is as follows:

[0065]

[0066] From the above relationship, we can know the position and size (x t ,y t , w t ,h t ), by filtering the data of the target area in the temperature data matrix, the highest temperature of the target area is obtained and used as the temperature value Tp of the current target; when the detection target is the tunneling head, the tunneling head temperature T can be obtained in the same way dAt the same time, the model inference unit annotates the location, size, category and temperature of the target in the video stream.

[0067] S5. Business logic processing: The business processing unit performs logical judgment on the detection results of the model reasoning unit according to preset rules to determine whether to issue an early warning or shutdown information.

[0068] S6. Warning and shutdown operation: If a dangerous situation is detected, the business processing unit sends a warning message to the sound and light alarm module through the RS485 protocol, and controls the equipment control unit through the internal protocol to execute the emergency stop operation of the tunnel boring machine.

[0069] S7. Sound and light alarm: After receiving the early warning information, the sound and light alarm module will alert the staff through voice broadcast and light warning.

[0070] S8, equipment control: After receiving the emergency stop signal, the equipment control unit converts the signal into a DIDO switch signal and sends it to the tunnel boring machine control system to perform an emergency stop operation, and automatically resets after 10 seconds. Further, according to the contents of the above steps S5-S8, there are three types of alarm processes, as follows:

[0071] (1) The first type of alarm process: When a worker is detected in the images of the two cameras at the front of the tunnel boring machine and enters the set ROI danger zone, an alarm message of personnel intrusion into the danger zone and a shutdown message are issued. After the sound and light alarm obtains the alarm message through the standard RS485 protocol, it will issue a voice alarm of "personnel intrusion into the danger zone" accompanied by flashing lights; after the equipment control unit obtains the shutdown message through the internal protocol, it will close the emergency stop switch circuit to implement the emergency stop operation of the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds.

[0072] (2) The second type of alarm process: When the two cameras at the front of the tunnel boring machine detect the tunnel boring head and the temperature value T d When the set temperature is exceeded, the equipment will issue an alarm message indicating that the temperature is too high and a shutdown message will be sent. After the sound and light alarm obtains the alarm message through the standard RS485 protocol, it will issue a voice alarm of "equipment temperature is too high" accompanied by flashing lights. After the equipment control unit obtains the shutdown message through the internal protocol, it will close the emergency stop switch circuit to implement an emergency stop operation on the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds.

[0073] (3) When the two cameras at the rear of the tunnel boring machine detect that someone has entered the dangerous area of ​​the drawn ROI, an alarm message will be issued indicating that the person is approaching the tunnel boring machine. After the sound and light alarm obtains the alarm message through the standard RS485 protocol, it will issue a voice alarm of "person approaching the tunnel boring machine" accompanied by flashing lights.

[0074] When the warning is triggered, the business processing unit stores the video frame that currently triggers the warning by calling the corresponding internal interface, records the video for 10 seconds, and uploads the warning data to the database.

[0075] S9. Data storage: The data storage unit stores structured data such as alarm information, device logs, and configuration parameters in the MySQL database, and stores unstructured data such as images and videos in the Minio database.

[0076] S10. Web-side service display: The web-side service unit displays real-time analysis results and provides functions such as real-time pop-up alarms, historical data query, and alarm statistical analysis. Users can access this information through web pages.

[0077] S11. Terminal monitoring: Underground workers and surface managers can access the WEB service through the underground tablet super terminal, surface monitoring host and mine intrinsically safe monitoring host to monitor the working status of the tunnel boring machine in real time and receive alarm information.

[0078] S12. System management: The terminal monitoring module centrally monitors and manages the entire tunnel boring machine early warning linkage system to ensure the stable operation of the system.

[0079] The thermal imaging-based tunnel boring machine early warning linkage method provided in this embodiment effectively solves the problem of sensing the working conditions in the area near the tunnel boring machine, effectively reduces the blind spots of the workers during tunneling operations, and ensures safe and efficient production in underground coal mines.

[0080] The above description is only a preferred embodiment of the present invention and does not limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A tunnel boring machine early warning linkage system based on thermal imaging, characterized in that: It includes a data acquisition module, an edge computing gateway module, an sound and light alarm module and a terminal monitoring module; the data acquisition module is used to monitor the thermal imaging video stream and temperature data around the underground tunnel boring machine in real time, and transmit the data to the edge computing gateway module; the edge computing gateway module has a built-in inference engine for real-time analysis of the received data to identify potential safety hazards, and once an abnormal situation is found, the edge computing gateway module immediately sends an instruction to the sound and light alarm module to make it send out a sound and light alarm signal, and at the same time sends a control instruction to the equipment control unit in the module to control the operating status of the tunnel boring machine; the terminal monitoring module is communicatively connected to the edge computing gateway module, supports remote monitoring and real-time data transmission, is used to receive the early warning information sent by the edge computing gateway module, and displays the monitoring information around the tunnel boring machine in real time on the monitoring interface.

2. The tunnel boring machine early warning linkage system based on thermal imaging according to claim 1, characterized in that: The data acquisition module relies on multiple mining intrinsically safe thermal imaging cameras installed on the tunnel boring machine. One camera is installed on each side of the front end of the tunnel boring machine according to the width of the camera's field of view, responsible for monitoring the equipment temperature and personnel status in the area near the head of the tunnel boring machine; one camera is also installed on each side of the rear end of the tunnel boring machine, responsible for detecting the personnel status at the rear end of the tunnel boring machine.

3. The tunnel boring machine early warning linkage system based on thermal imaging according to claim 1, characterized in that: The edge computing gateway module is integrated in the artificial intelligence computing core board, including a model reasoning unit, a temperature acquisition unit, a business processing unit, a device control unit, a data storage unit and a web-side service unit; the model reasoning unit has a built-in AI reasoning engine to perform real-time intelligent analysis of the video stream, identify violations, and send the detected warning information to the business processing unit; the business processing unit receives these warning information, and performs logical judgment according to preset rules and thresholds, and then issues corresponding warnings or shutdown information; the temperature acquisition unit is used to obtain the temperature data matrix from the data acquisition module in real time, and synchronize the time with the video stream data in the model reasoning unit. The model reasoning unit detects the position information of the target in the thermal imaging image, and this information is then used by the temperature acquisition unit to determine the specific position of the target in the temperature data matrix.

4. The tunnel boring machine early warning linkage system based on thermal imaging according to claim 3 is characterized in that: The business processing unit will communicate the early warning information issued with the sound and light alarm module through the RS485 protocol to realize the sound and light alarm; and send the shutdown information to the equipment control unit through the internal protocol; the business types of the business processing unit include personnel detection in the dangerous area at the front end of the tunnel boring machine, high temperature monitoring of the tunnel boring head, and personnel detection in the dangerous area at the rear end of the tunnel boring machine.

5. The tunnel boring machine early warning linkage system based on thermal imaging according to claim 4 is characterized in that: After receiving the shutdown information, the equipment control unit converts it into a DIDO switch signal and sends it to the control system of the tunnel boring machine to realize the emergency stop operation of the tunnel boring machine; at the same time, the equipment control unit will automatically perform a reset operation within a certain time after the emergency stop, allowing the tunnel boring machine to restart.

6. The tunnel boring machine early warning linkage system based on thermal imaging according to claim 3 is characterized in that: The data storage unit is used to establish and manage Mysql and Minio databases to realize the storage of alarm information, images and videos, and provide corresponding external query interfaces to ensure data retention and user historical data query needs.

7. The tunnel boring machine early warning linkage system based on thermal imaging according to claim 3 is characterized in that: The web-based service unit is used to display the real-time analysis results of the video stream collected by the tunnel boring machine, and at the same time provides functions such as real-time pop-up alarms, historical data queries, and alarm statistical analysis; the unit adopts a B / S architecture, and the client can directly access the WEB-based interface via the industrial ring network using a web page, thereby realizing real-time monitoring and browsing of the area around the tunnel boring machine.

8. The tunnel boring machine early warning linkage system based on thermal imaging according to claim 1, characterized in that: The terminal monitoring module includes an underground flat-panel super terminal, an above-ground monitoring host and a mine-used intrinsically safe monitoring host; the underground flat-panel super terminal communicates with the edge computing gateway module through a WIFI module; the mine-used intrinsically safe monitoring host directly communicates using the RJ45 network port of the edge computing gateway module; the above-ground monitoring host communicates with the edge computing gateway module through an underground ring network; the terminal monitoring module can access the WEB service of the edge computing gateway module to enable the mobile terminal to browse the original video stream, the analyzed AI video stream, the alarm message, and the statistical analysis information.

9. A method for early warning linkage of a roadheader based on thermal imaging, using the early warning linkage system of a roadheader based on thermal imaging as described in any one of claims 1 to 8, characterized in that: The following steps are involved: S1. Use the mine intrinsically safe thermal imaging camera installed on the tunnel boring machine to monitor and sense the temperature of the surrounding area of ​​the underground tunnel boring machine in real time, so as to obtain a real-time thermal imaging video stream (..., I n-2 , I n-1 , I n ) and the temperature data matrix (..., T n-2 , T n-1 , T n ) and transmit these data to the edge computing gateway module via Ethernet; S2. The model reasoning unit in the edge computing gateway module uses the standard rtsp protocol to obtain video stream data in real time, and performs real-time intelligent analysis on the received thermal imaging video stream to detect the staff and tunneling heads in the four cameras; at the same time, the temperature acquisition unit obtains the temperature data matrix in real time, and uses the camera private protocol to update the temperature map of the camera monitoring area in real time; The model reasoning unit detects the positions of the tunneling head and the staff and synchronizes the video stream data with the temperature data matrix to determine the maximum temperature of the target; S3. The business processing unit makes a logical judgment on the detection result of the model reasoning unit according to the preset rules to determine whether to issue a warning or shutdown information; if a dangerous situation is detected, the business processing unit sends a warning information to the sound and light alarm module through the RS485 protocol, and controls the equipment control unit to execute the emergency stop operation of the tunnel boring machine through the internal protocol; S4. After receiving the warning information, the optical alarm module will alert the staff through voice broadcast and light warning; after receiving the emergency stop signal, the equipment control unit will convert the signal into a DIDO switch signal and send it to the tunnel boring machine control system to perform the emergency stop operation, and automatically reset after 10 seconds; S5. At the same time, the WEB service unit displays the real-time analysis results and provides functions such as real-time pop-up alarm, historical data query, and alarm statistical analysis. Users can access this information through the web page. Underground workers and surface managers can access the WEB service through the terminal monitoring module to monitor the working status of the tunnel boring machine and receive alarm information in real time, thereby realizing centralized monitoring and management of the entire tunnel boring machine early warning linkage system.

10. The method for early warning linkage of a roadheader based on thermal imaging according to claim 9, characterized in that: According to the steps S5-S8, there are three types of alarm processes: The first type of alarm process: When a worker is detected in the images of the two cameras at the front end of the tunnel boring machine and enters the set ROI dangerous area, an alarm message of personnel intrusion into the dangerous area and a shutdown message will be issued. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "personnel intrusion into the dangerous area" accompanied by flashing lights; after the equipment control unit obtains the shutdown information through the internal protocol, it will close the emergency stop switch circuit to implement the emergency stop operation of the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds. The second type of alarm process: When the tunneling head is detected in the two camera images at the front of the tunnel boring machine and the temperature value exceeds the set temperature, an alarm message of over-temperature and shutdown information of the equipment will be issued. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "equipment temperature is too high" accompanied by flashing lights; after the equipment control unit obtains the shutdown information through the internal protocol, it will close the emergency stop switch circuit to realize the emergency stop operation of the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds; The third type of alarm process: When the two cameras at the rear of the tunnel boring machine detect that someone has entered the dangerous area of ​​the drawn ROI, an alarm message will be issued that the person is approaching the tunnel boring machine. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "person approaching the tunnel boring machine" accompanied by flashing lights; and at the same time as the warning is triggered, the business processing unit calls the internal interface to store the video frame that currently triggers the warning, and at the same time, it will record the video for 10 seconds and upload the warning data to the database.

Citation Information

Patent Citations

  • Heading machine state monitoring device and system

    CN111367232A

  • Equipment health state video linkage online monitoring and early warning platform system

    CN111968345A

  • Cantilever heading machine motor protection control system and method

    CN114183158A

  • Safety early warning system and safety early warning method for personnel nearby belt conveyor

    CN118116142A

  • Personnel approaching sensing and early warning system applied to mine dangerous area and operation method

    CN118379837A

Cited By

  • Method and system for detecting unsafe behaviors of personnel on driving working face and medium

    CN119314229A

  • A method, system and medium for detecting unsafe behavior of personnel at a tunneling working face

    CN119314229B

  • Heading machine cutterhead full-disc cutter temperature detection system and heading machine

    CN119933733A

  • Intelligent gateway device and application method

    CN120956563A