A tunnel boring machine early warning linkage system and method based on thermal imaging

Abstract

The present invention discloses a tunnel boring machine early warning linkage system and method based on thermal imaging, which relates to the field of intelligent monitoring technology in underground coal mines. The system includes a data acquisition module, an edge computing gateway module, an audible and visual alarm module, and a terminal monitoring module. The data acquisition module is used to monitor the thermal imaging video stream and temperature data around the underground tunnel boring machine in real time. The edge computing gateway module is used to identify potential safety hazards and, upon detecting an abnormal situation, immediately sends a command to the audible and visual alarm module to cause it to emit an audible and visual alarm signal and simultaneously control the operating status of the tunnel boring machine. The terminal monitoring module is used to receive the early warning information sent by the edge computing gateway module and display the monitoring information around the tunnel boring machine in real time on the monitoring interface. The present invention effectively solves the problem of perceiving the working conditions in the area near the tunnel boring machine, effectively reduces the blind spots of the workers during tunneling operations, and ensures safe and efficient production in underground coal mines.

Description

Technical Field

[0001] The present invention belongs to the technical field of underground intelligent monitoring in coal mines, and in particular relates to a roadheader early warning linkage system and method based on thermal imaging. Background Art

[0002] As a vital energy source in my country, coal provides a fundamental foundation for the country's industrialization. Faced with the complex environment underground in coal mines, the application of video-based AI monitoring technology effectively removes workers from dangerous environments and ensures safe production underground.

[0003] However, during tunneling, a large amount of dust fills the air, making it difficult for ordinary visible light cameras to capture clear images, posing a significant challenge to the effective operation of the monitoring system. Furthermore, the tunneling machine driver's limited field of view makes it difficult to fully grasp the dynamic situation in the work area. If a worker strays into a dangerous area, the driver may not be able to detect them in time, increasing the risk of safety accidents.

[0004] Furthermore, the tunneling head generates extremely high temperatures during operation, posing a serious threat to the equipment's stability and lifespan. Equipment failure not only impacts production efficiency but can also lead to safety incidents. Therefore, accurately monitoring the status of personnel and equipment within the tunneling area and intelligently linking this with tunneling equipment has become a pressing issue for coal mine safety. Summary of the Invention

[0005] In response to the defects and problems existing in the existing technology, the present invention provides a tunnel boring machine early warning linkage system and method based on thermal imaging. The present invention effectively solves the problem of sensing the working conditions in the area near the tunnel boring machine, effectively reduces the blind spots of the workers during tunneling operations, and ensures safe and efficient production in coal mines.

[0006] The solution adopted by the present invention to solve its technical problems is: a tunnel boring machine early warning linkage system based on thermal imaging, including a data acquisition module, an edge computing gateway module, an sound and light alarm module and a terminal monitoring module; the data acquisition module is used to monitor the thermal imaging video stream and temperature data around the underground tunnel boring machine in real time, and transmit the data to the edge computing gateway module; the edge computing gateway module has a built-in inference engine for real-time analysis of the received data to identify potential safety hazards, and once an abnormal situation is found, the edge computing gateway module immediately issues an instruction to the sound and light alarm module to make it issue an sound and light alarm signal, and at the same time sends a control instruction to the equipment control unit in the module to control the operating status of the tunnel boring machine; the terminal monitoring module is communicatively connected to the edge computing gateway module, supports remote monitoring and real-time data transmission, is used to receive the early warning information sent by the edge computing gateway module, and displays the monitoring information around the tunnel boring machine in real time on the monitoring interface.

[0007] Furthermore, the data acquisition module relies on multiple mining intrinsically safe thermal imaging cameras installed on the tunnel boring machine. One camera is installed on each side of the front end of the tunnel boring machine according to the width of the camera's field of view, responsible for monitoring the equipment temperature and personnel status in the area near the head of the tunnel boring machine; one camera is also installed on each side of the tail of the tunnel boring machine, responsible for detecting the personnel status at the tail of the tunnel boring machine.

[0008] Furthermore, the edge computing gateway module is integrated into the artificial intelligence computing core board, including a model reasoning unit, a temperature acquisition unit, a business processing unit, a device control unit, a data storage unit and a web service unit; the model reasoning unit has a built-in AI reasoning engine, which performs real-time intelligent analysis of the video stream, identifies violations, and sends the detected warning information to the business processing unit; the business processing unit receives these warning information, and performs logical judgment according to preset rules and thresholds, and then issues corresponding warning or shutdown information; the temperature acquisition unit is used to obtain the temperature data matrix from the data acquisition module in real time, and synchronize it with the video stream data in the model reasoning unit. The model reasoning unit detects the position information of the target in the thermal imaging image, and this information is then used by the temperature acquisition unit to determine the specific position of the target in the temperature data matrix.

[0009] Furthermore, the business processing unit will communicate the early warning information issued with the sound and light alarm module through the RS485 protocol to realize the sound and light alarm; and send the shutdown information to the equipment control unit through the internal protocol; the business types of the business processing unit include personnel detection in the dangerous area at the front end of the tunnel boring machine, high temperature monitoring of the tunnel boring head, and personnel detection in the dangerous area at the rear end of the tunnel boring machine.

[0010] Furthermore, after receiving the shutdown information, the device control unit converts it into a DIDO switch signal and sends it to the control system of the tunnel boring machine to realize the emergency stop operation of the tunnel boring machine; at the same time, the device control unit will automatically perform a reset operation within a certain period of time after the emergency stop, allowing the tunnel boring machine to restart.

[0011] Furthermore, the data storage unit is used to establish and manage Mysql and Minio databases to realize the storage of alarm information, images and videos, and provide corresponding external query interfaces to ensure data retention and user historical data query needs.

[0012] Furthermore, the web-side service unit is used to display the real-time analysis results of the video stream collected by the tunnel boring machine, and at the same time provides real-time pop-up alarms, historical data queries, and alarm statistical analysis functions; the unit adopts a B / S architecture, and the client can directly access the WEB-side interface through the industrial ring network using a web page, thereby realizing real-time monitoring and browsing of the area around the tunnel boring machine.

[0013] Furthermore, the terminal monitoring module includes an underground flat-panel super terminal, an above-ground monitoring host and a mine-used intrinsically safe monitoring host; wherein the underground flat-panel super terminal communicates with the edge computing gateway module through a WIFI module; the mine-used intrinsically safe monitoring host directly communicates using the RJ45 network port of the edge computing gateway module; the above-ground monitoring host communicates with the edge computing gateway module through an underground ring network; the terminal monitoring module can access the WEB service of the edge computing gateway module to enable the mobile terminal to browse the original video stream, the analyzed AI video stream, the alarm message, and the statistical analysis information.

[0014] The present invention also provides a tunnel boring machine early warning linkage method based on thermal imaging, comprising the following steps:

[0015] S1. Using the mine intrinsically safe thermal imaging camera installed on the tunnel boring machine to monitor the temperature of the surrounding area of ​​the tunnel boring machine in real time, thereby obtaining a real-time thermal imaging video stream (..., I n-2 , I n-1 , I n ) and the temperature data matrix (..., T n-2 , T n-1 , T n ) and transmit these data to the edge computing gateway module via Ethernet;

[0016] S2. The model inference unit in the edge computing gateway module uses the standard RTSP protocol to acquire video stream data in real time, and performs real-time intelligent analysis on the received thermal imaging video stream to detect the workers and tunneling heads in the four cameras. At the same time, the temperature acquisition unit acquires the temperature data matrix in real time and uses the camera's private protocol to update the temperature map of the camera monitoring area in real time. The model inference unit detects the positions of the tunneling head and workers and synchronizes the video stream data with the temperature data matrix to determine the maximum temperature of the target.

[0017] S3. The business processing unit performs logical judgment on the detection results of the model reasoning unit according to preset rules to determine whether to issue an early warning or shutdown information; if a dangerous situation is detected, the business processing unit sends an early warning information to the sound and light alarm module via the RS485 protocol and controls the equipment control unit to execute an emergency stop operation of the tunnel boring machine through the internal protocol;

[0018] S4. After receiving the warning information, the optical alarm module alerts the staff through voice broadcast and light warning. After receiving the emergency stop signal, the equipment control unit converts the signal into a DIDO switch signal and sends it to the roadheader control system to execute the emergency stop operation. It automatically resets after 10 seconds.

[0019] S5. At the same time, the WEB service unit displays the real-time analysis results and provides functions such as real-time pop-up alarms, historical data query, and alarm statistical analysis. Users can access this information through the web page. Underground workers and surface managers can access the WEB service through the terminal monitoring module to monitor the working status of the tunnel boring machine in real time and receive alarm information, thereby realizing centralized monitoring and management of the entire tunnel boring machine early warning linkage system.

[0020] Furthermore, the steps S5-S8 include the following three types of alarm processes:

[0021] The first type of alarm process: When a worker is detected in the images of the two cameras at the front end of the tunnel boring machine and enters the set ROI danger zone, an alarm message of personnel intrusion into the danger zone and a shutdown message will be issued. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "personnel intrusion into the danger zone" accompanied by flashing lights; after the equipment control unit obtains the shutdown information through the internal protocol, it will close the emergency stop switch circuit to implement the emergency stop operation of the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds.

[0022] The second type of alarm process: When the two cameras at the front of the tunnel boring machine detect that the tunnel boring head has exceeded the set temperature, an over-temperature alarm and shutdown message will be issued. After the sound and light alarm receives the alarm message through the standard RS485 protocol, it will issue a voice alarm of "device temperature is too high" accompanied by flashing lights. After the equipment control unit receives the shutdown message through the internal protocol, it will close the emergency stop switch circuit to implement the emergency stop operation of the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds.

[0023] The third type of alarm process: When the two cameras at the rear of the tunnel boring machine detect that someone has entered the drawn ROI dangerous area, an alarm message will be issued that the person is approaching the tunnel boring machine. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "person approaching the tunnel boring machine" accompanied by flashing lights; and at the same time as the warning is triggered, the business processing unit calls the internal interface to store the video frame that currently triggers the warning, and at the same time, it will record the video for 10 seconds and upload the warning data to the database.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The thermal imaging-based tunnel boring machine early warning linkage system and method provided by the present invention have significant advantages in improving mine operation safety, enhancing monitoring efficiency and reducing production risks. The present invention uses thermal imaging technology for environmental monitoring, which can penetrate dust and darkness and provide clearer images than traditional visible light cameras, especially in dusty and poorly lit environments such as coal mines; the system can monitor the environment and temperature around the tunnel boring machine in real time, and perform real-time analysis through the edge computing gateway module to quickly identify potential safety hazards; it can also achieve time synchronization between video stream data and temperature data matrix to ensure the accuracy and real-time performance of temperature detection.

[0026] The system of the present invention can automatically issue corresponding alarm information according to different monitoring results to achieve rapid response; through the automated equipment control unit, the system can realize emergency stop and automatic reset operations of the tunnel boring machine, thereby improving the efficiency of safety management; the system can realize centralized monitoring and management of the entire tunnel boring machine early warning linkage system, which is conducive to improving management efficiency and system stability. Through automated and intelligent safety management, the system of the present invention can reduce production interruptions caused by misoperation or delayed response while ensuring safety, thereby improving production efficiency.

[0027] The method of the present invention uses a thermal imaging camera to obtain a thermal imaging video stream and temperature map of monitoring points near a tunnel boring machine. By training a corresponding target detection inference model for this scenario, real-time detection of workers and tunnel boring machine drill bits in the thermal imaging video stream is achieved. Based on the position of the tunnel boring machine drill bit in the corresponding temperature map, the relevant temperature of the tunnel boring machine drill bit is obtained, and when the temperature exceeds a set threshold, a high-temperature alarm is issued for the equipment. Based on the identified worker position and the set danger zone, a judgment is made to achieve a corresponding personnel intrusion alarm. Simultaneously, upon receiving the issued warning message, the equipment control unit implements emergency stop control of the tunnel boring machine. The implementation of this method effectively solves the problem of sensing the working conditions in the area near the tunnel boring machine, effectively reduces the blind spots of workers during tunneling operations, and ensures safe and efficient production in underground coal mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the system connection of the present invention;

[0029] Figure 2 This is a system connection diagram of the edge computing gateway module of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of the artificial intelligence computing core board of the present invention;

[0031] Figure 4 Schematic diagram of the process structure of the method of the present invention;

[0032] Figure 5 This is the thermal imaging monitoring screen of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below with reference to the accompanying drawings and examples.

[0034] Example 1:

[0035] This embodiment provides a roadheader early warning linkage system based on thermal imaging, such as Figure 1 As shown, it mainly includes a data acquisition module, an edge computing gateway module, an audio-visual alarm module and a terminal monitoring module. The data acquisition module monitors the surrounding environment and temperature data of the underground tunnel boring machine in real time, and transmits the data to the artificial intelligence edge computing gateway module. The edge computing gateway module uses a built-in algorithm to perform real-time analysis on the received data to identify potential safety hazards. Once an abnormal heat source or safety hazard is found, an alarm message will be immediately sent to the audio-visual alarm module to cause it to issue an audio-visual alarm signal. At the same time, the terminal monitoring module can provide real-time tunnel boring machine monitoring information for underground workers and surface managers, and can also centrally monitor and manage the entire tunnel boring machine early warning linkage system to ensure the stable operation of the system. The specific contents are as follows.

[0036] The above data acquisition module relies on multiple mine-used intrinsically safe thermal imaging cameras installed on the roadheader, see Figure 5 This camera overcomes the problem of traditional visible light cameras being unable to clean and capture live footage due to heavy dust in the tunneling work surface. During installation, one camera is installed on each side of the front end of the TBM, based on the camera's field of view. It monitors the temperature of the equipment and the status of personnel near the TBM head. Another camera is installed on each side of the rear end of the TBM, primarily to monitor the status of personnel at the rear end, ensuring that there are no blind spots around the TBM equipment.

[0037] In this way, by installing mine-use intrinsically safe thermal imaging cameras at corresponding positions at the head and tail ends, it is possible to collect visible light imaging, thermal imaging, and ambient temperature data in coal mines. The thermal imager supports communication protocols such as RTSP, SDK, and ONVIF. In this embodiment, the thermal imaging video stream and temperature data matrix collected by the mine-use intrinsically safe thermal imaging camera are transmitted over Ethernet as a medium, using the standard RTSP video streaming protocol to achieve data transmission with the model inference unit in the edge computing gateway module, and using the SDK (Software Development Kit) communication protocol to achieve data transmission with the temperature acquisition unit in the edge computing gateway module. The resolution of the thermal imaging video stream is 1280X960, and the size of the temperature data matrix is ​​256X192.

[0038] like Figure 3 As shown, the edge computing gateway module is integrated into the artificial intelligence computing core board. Its external communication interfaces include 1) RS-485 bus: for data communication with the sound and light alarm module; 2) CAN bus: for communication with DIDO switch devices and other devices, and to control equipment such as tunneling machines according to actual on-site needs; 3) RJ45 network port: for obtaining environmental data collected by cameras; 4) Wi-Fi module: for supporting data transmission between wireless devices; 5) optical port: for accessing the 10G fiber ring network used in the mine, providing high-speed and stable data transmission. These communication interfaces can meet the data exchange needs of different devices and systems.

[0039] like Figure 2As shown, the module contains a model inference unit, a temperature acquisition unit, a business processing unit, a device control unit, a data storage unit, and a web service unit. The built-in AI inference engine can perform real-time intelligent analysis of video streams, effectively identifying violations. Once a violation is detected, an alarm mechanism is immediately triggered, providing an early warning through data transmission with the sound and light alarm module, and the alarm information is sent to the web service unit. Simultaneously, the DIDO switch controls the tunnel boring machine's emergency stop switch to ensure safety. Furthermore, the video stream processed by AI analysis can be forwarded as needed through various standard protocols, such as RTSP, RTMP, and FLV, to meet the access and playback requirements of different terminal devices or backend servers.

[0040] Furthermore, the model inference unit is placed in the edge computing gateway module, which is used to provide a target detection inference engine for relevant personnel and tunneling heads. By collecting the on-site thermal imaging video stream and annotating the data set, a VOC (Visual Object Class) data set containing two categories of tunneling heads and staff is obtained, and the model is trained using the constructed YOLOv5m artificial neural network training environment. The YOLOv5m model obtained by training can accurately identify tunneling heads and staff, and give the accurate position of the detected target in the image coordinate system. In order to adapt to the framework structure of the artificial intelligence computing core board, real-time analysis of the thermal imaging video stream collected by the thermal imaging camera can be achieved by converting the trained YOLOv5m model into a corresponding target inference engine and loading it into the artificial intelligence computing core board.

[0041] The temperature acquisition unit is used to acquire the temperature data matrix output by the thermal imaging camera in real time, while simultaneously achieving time synchronization with the video stream data in the model inference unit. Specifically, the model inference unit detects the position and size (x, y, w, h) of the upper left corner of the thermal imaging image in the coordinate system, and uses the coordinate system transformation to obtain the position of the target in the temperature data matrix. By filtering the data of the target area in the temperature data matrix, the highest temperature in the target area is obtained, which is the current target temperature value.

[0042] The service processing unit is primarily responsible for performing logical analysis on relevant warning information and issuing warning or shutdown information. Warning information communicates with the sound and light alarm module via the RS485 protocol to generate an audible and visual alarm, while shutdown information is exchanged with the device control unit via an internal protocol. This unit includes the following three services:

[0043] (1) Personnel detection in the dangerous area at the front end of the tunnel boring machine

[0044] Regarding the two thermal imaging cameras at the front end of the tunnel boring machine, when the inference engine in the model inference unit detects that a worker has entered the set danger zone, the business processing unit immediately issues a warning message and a shutdown message.

[0045] (2) High temperature monitoring of tunneling head

[0046] For the two thermal imaging cameras at the front end of the tunnel boring machine, the inference engine will detect the position of the tunnel boring head in real time, use the coordinate system conversion, and then obtain the tunnel boring head temperature in the temperature map. When the tunnel boring head temperature is monitored to be higher than the set temperature threshold, the business processing unit will issue a high temperature warning message and a shutdown message.

[0047] (3) Personnel detection in the dangerous area behind the tunnel boring machine

[0048] For the two thermal imaging cameras at the rear end of the tunnel boring machine, when the inference engine detects that relevant staff have entered the set ROI danger area, the business processing unit will issue an early warning message of the approaching personnel.

[0049] This unit can make accurate logical judgments on warning information based on preset rules and thresholds. Once a dangerous situation is detected, it can quickly issue a warning or shutdown information to achieve a rapid response to the dangerous situation. At the same time, it supports multiple warning services and rule configurations to meet safety needs in different scenarios, effectively prevent potential safety accidents, and improve production efficiency and safety.

[0050] The device control unit obtains the device emergency stop information from the business processing unit through an internal protocol. After receiving the device emergency stop information, it will immediately convert this signal into a DIDO (Digital Input / Output) switch signal and send it to the tunnel boring machine control system. After receiving the DIDO switch signal, the tunnel boring machine control system will perform an emergency stop operation to ensure that the equipment stops running quickly, thereby ensuring the safety of personnel. After the tunnel boring machine stops suddenly, in order to prevent the staff from being unable to restart the tunnel boring machine and affecting work efficiency, the device control unit will automatically perform a reset operation after 10 seconds. The reset operation will restore the DIDO switch signal to its initial state, allowing the tunnel boring machine control system to re-accept the start command, ensuring that the equipment can restart normally and continue to operate. In this way, through the DIDO switch signal, the device control unit can accurately control the operating status of the tunnel boring machine, thereby realizing emergency stop and reset operations. It also has an automatic reset function, which can automatically perform a reset operation after the tunnel boring machine stops suddenly to avoid affecting work efficiency. Through the collaborative work of the equipment control unit and other modules such as the business processing unit and the model inference unit, the edge computing gateway module can realize real-time monitoring and safety management of the tunnel boring machine, effectively prevent potential safety accidents, and improve production efficiency and work safety.

[0051] The data storage unit is also placed in the edge computing gateway module. This unit mainly establishes and manages two types of databases: Mysql and Minio. The Mysql database is used to store structured data such as alarm information, device logs, configuration parameters, etc., to provide efficient data query, insertion, update and deletion operations; and complex data analysis and statistics are performed through the SQL language. The Minio database is a high-performance object storage server that is compatible with the Amazon S3 cloud storage service interface. It is mainly used to store unstructured data such as images and videos, providing persistent storage and backup functions for data, and supporting distributed storage and expansion of data. The data storage unit provides a RESTful API or a similar interface, allowing external systems or users to query the data stored in the database through HTTP requests, ensuring data retention and user historical data query needs. The unit realizes the storage of alarm information, images and videos, and can preserve historical data for a long time to ensure the traceability and auditability of the data.

[0052] The web-based service unit primarily displays real-time analysis results from four video streams near the roadheader, while also providing features like real-time pop-up alerts, historical data query, and alarm statistics analysis. Using an advanced B / S architecture, clients can directly access the web interface via the industrial ring network, enabling real-time monitoring and browsing of the area surrounding the roadheader.

[0053] Each unit in the above edge computing gateway module is connected to external devices through internal protocols and structures, which can realize real-time acquisition, processing, analysis and control of data. Through close collaboration between the units, the entire system can be ensured to operate efficiently and safely, thereby realizing safe monitoring and safety management of the mine environment.

[0054] The sound and light alarm module relies on an audible and visual alarm, which primarily consists of an RS-485 bus, a microcontroller unit, speech synthesis, attack and defense, and lighting control (LED) units. It utilizes the RS485 bus to receive warning information from the service processing unit and provides voice and light alerts. The lighting features three colors: red, yellow, and green. Voice announcements support local playback and online customization of warning announcements using the Chinese national standard GB2312.

[0055] The terminal monitoring module includes an underground flat-panel super terminal, an above-ground monitoring host, and a mine-used intrinsically safe monitoring host. The underground flat-panel super terminal can communicate with the artificial intelligence computing gateway through a WIFI module, and by accessing the WEB service of the artificial intelligence edge computing gateway, the mobile terminal can browse the original video stream, the analyzed AI video stream, alarm messages, statistical analysis and other information. The mine-used intrinsically safe monitoring host uses the RJ45 network port of the artificial intelligence edge computing gateway to communicate directly. And by accessing the WEB service of the artificial intelligence edge computing gateway, the mobile terminal can browse the original video stream, the analyzed AI video stream, alarm messages, statistical analysis and other information. The above-ground monitoring host communicates with the artificial intelligence gateway through the underground ring network. And by accessing the WEB service of the artificial intelligence edge computing gateway, the mobile terminal can browse the original video stream, the analyzed AI video stream, alarm messages, statistical analysis and other information.

[0056] Staff can access the WEB services in the artificial intelligence gateway through the configured tablet super terminal, well monitoring host, and mine intrinsically safe monitoring host using web pages. At this time, users can browse the original video stream, analyzed AI video stream, alarm messages, statistical analysis and other information.

[0057] The tunnel boring machine early warning linkage system based on thermal imaging provided by the present invention generates visible thermal radiation images through an infrared thermal imager, and generates clear images according to the temperature difference between objects. Even small details can be seen, thereby providing high-precision monitoring. The system combines artificial intelligence technology to perform characteristic analysis on video images, generate abnormal data values, and automatically determine whether an alarm needs to be issued based on a preset alarm threshold, thereby improving the intelligence level of the system. The system provides strong guarantees for coal mine safety production through its advantages of comprehensive detection, high precision, all-weather working capability, low missed reporting rate, efficient early warning linkage, improved operation safety and intelligent management, effectively improves operation efficiency and the convenience of data management, and provides an effective technical solution for mine safety management.

[0058] Example 2:

[0059] Based on Example 1, this example provides a tunnel boring machine early warning linkage method based on thermal imaging, see Figure 4 , specifically including the following steps:

[0060] S1. Data acquisition: The mine intrinsically safe thermal imaging camera installed on the tunnel boring machine is used to monitor the temperature of the surrounding area of ​​the tunnel boring machine in real time, thereby obtaining a real-time thermal imaging video stream (..., I n-2 , I n-1 , I n ) and the temperature data matrix (..., Tn-2 , T n-1 , T n ) and transmits this data to the edge computing gateway module via Ethernet.

[0061] S2. Data transmission: The thermal imaging video stream and temperature data matrix collected by the thermal imaging camera are transmitted to the edge computing gateway module through the standard RTSP video streaming protocol and SDK communication protocol.

[0062] S3. Edge computing analysis: The model inference unit within the edge computing gateway module uses the standard RTSP protocol to obtain video stream data in real time, and performs real-time intelligent analysis of the received thermal imaging video stream to detect the workers and tunneling heads in the four cameras. At the same time, the temperature acquisition unit obtains the temperature data matrix in real time, that is, it uses the camera's private protocol to update the temperature map of the camera monitoring area in real time.

[0063] S4. Target Detection and Temperature Synchronization: The model inference unit detects the positions of the tunneling head and personnel and synchronizes the video stream data with the temperature data matrix to determine the target's maximum temperature. Furthermore, using a coordinate system conversion formula, the position and size in the image coordinate system are converted to corresponding values ​​in the temperature data matrix. The target region in the temperature data matrix is ​​selected, and the highest temperature value is obtained as the current target temperature.

[0064] Specifically, taking the detection of workers as an example, when the model reasoning unit detects the position and size (x i ,y i , w i , h i ), the position and size (x t ,y t , w t , h t ). Given that the image resolution is 1280×960 and the size of the temperature data matrix is ​​256×192, we can see that the relationship between the two coordinate systems is as follows:

[0065]

[0066] From the above relationship, we can know the position and size (x t ,y t , w t , h t ), by filtering the target area data in the temperature data matrix, the highest temperature of the target area is obtained and used as the current target temperature value Tp; when the detection target is the tunneling head, the tunneling head temperature T can be obtained in the same way. dAt the same time, the model inference unit annotates the location, size, category and temperature of the target in the video stream.

[0067] S5. Business logic processing: The business processing unit performs logical judgment on the detection results of the model reasoning unit according to preset rules to determine whether to issue an early warning or shutdown information.

[0068] S6. Warning and shutdown operation: If a dangerous situation is detected, the business processing unit sends a warning message to the sound and light alarm module through the RS485 protocol, and controls the equipment control unit through the internal protocol to execute the emergency stop operation of the tunnel boring machine.

[0069] S7. Sound and light alarm: After receiving the early warning information, the sound and light alarm module will alert the staff through voice broadcast and light warning.

[0070] S8, Equipment Control: After receiving the emergency stop signal, the equipment control unit converts the signal into a DIDO switch signal and sends it to the roadheader control system to execute the emergency stop operation, and automatically resets after 10 seconds. Furthermore, according to the contents of the above steps S5-S8, there are three types of alarm processes, as follows:

[0071] (1) The first type of alarm process: When a worker is detected in the images of the two cameras at the front end of the tunnel boring machine and enters the set ROI dangerous area, an alarm message of personnel intrusion into the dangerous area and a shutdown message will be issued. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "personnel intrusion into the dangerous area" accompanied by flashing lights; after the equipment control unit obtains the shutdown information through the internal protocol, it will close the emergency stop switch circuit to implement the emergency stop operation of the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds.

[0072] (2) The second type of alarm process: When the tunneling head is detected in the two cameras at the front of the tunneling machine and the temperature value T d When the set temperature is exceeded, an alarm message indicating that the equipment temperature is too high and a shutdown message will be issued. After the sound and light alarm obtains the alarm message through the standard RS485 protocol, it will issue a voice alarm of "equipment temperature is too high" accompanied by flashing lights; after the equipment control unit obtains the shutdown message through the internal protocol, it will close the emergency stop switch circuit to implement the emergency stop operation of the tunnel boring machine, and reset the emergency stop circuit after an interval of 10 seconds.

[0073] (3) When the two cameras at the rear of the tunnel boring machine detect that a person has entered the dangerous area of ​​the drawn ROI, an alarm message will be issued indicating that the person is approaching the tunnel boring machine. After the sound and light alarm obtains the alarm message through the standard RS485 protocol, it will issue a voice alarm of "person is approaching the tunnel boring machine" accompanied by flashing lights.

[0074] When the warning is triggered, the business processing unit stores the video frame that currently triggers the warning by calling the corresponding internal interface, records the video for 10 seconds, and uploads the warning data to the database.

[0075] S9. Data storage: The data storage unit stores structured data such as alarm information, device logs, and configuration parameters in the MySQL database, and stores unstructured data such as images and videos in the Minio database.

[0076] S10. Web service display: The web service unit displays real-time analysis results and provides functions such as real-time pop-up alarms, historical data query, and alarm statistical analysis. Users can access this information through the web page.

[0077] S11. Terminal monitoring: Underground workers and surface management personnel can access the WEB service through the underground tablet super terminal, surface monitoring host and mine intrinsically safe monitoring host to monitor the working status of the tunnel boring machine in real time and receive alarm information.

[0078] S12. System management: The terminal monitoring module centrally monitors and manages the entire tunnel boring machine early warning linkage system to ensure stable operation of the system.

[0079] The thermal imaging-based tunnel boring machine early warning linkage method provided in this embodiment effectively solves the problem of sensing the working conditions in the area near the tunnel boring machine, effectively reduces the blind spots of the workers during tunneling operations, and ensures safe and efficient production in coal mines.

[0080] The above description is only a preferred embodiment of the present invention and does not limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A roadheader early warning linkage system based on thermal imaging, characterized by: Including data acquisition module, edge computing gateway module, sound and light alarm module and terminal monitoring module; The data acquisition module relies on multiple mining intrinsically safe thermal imaging cameras installed on the tunnel boring machine, which are used to monitor the thermal imaging video stream and temperature data around the underground tunnel boring machine in real time, and uses Ethernet as the medium to transmit the collected thermal imaging video stream and temperature data matrix to the model inference unit in the edge computing gateway module using the standard RTSP video streaming protocol, and uses the SDK communication protocol to realize data transmission with the temperature acquisition unit in the edge computing gateway module; the installation position of the thermal imaging camera is to install one on each side of the front end of the tunnel boring machine according to the width of the camera's field of view, responsible for monitoring the equipment temperature and personnel status in the area near the head of the tunnel boring machine; one is also installed on each side of the tail of the tunnel boring machine, responsible for detecting the personnel status at the tail of the tunnel boring machine; The edge computing gateway module has a built-in inference engine for real-time analysis of received data to identify potential safety hazards. Once an abnormal situation is found, the edge computing gateway module immediately sends an instruction to the sound and light alarm module to emit an sound and light alarm signal, and at the same time sends a control instruction to the device control unit in the module to control the operating status of the roadheader. The edge computing gateway module is integrated into the artificial intelligence computing core board, including a model inference unit, a temperature acquisition unit, a business processing unit, a device control unit, a data storage unit, and a web service unit. The model inference unit has a built-in AI inference engine that performs real-time intelligent analysis of video streams, identifies violations, and sends detected warning information to the business processing unit. Specifically, the model inference unit collects on-site thermal imaging video streams and annotates the data set to obtain a VOC data set containing two categories: tunneling heads and workers. It uses the trained YOLOv5m artificial neural network to accurately identify the tunneling heads and workers and provide the exact location of the detected targets in the image coordinate system. The business processing unit receives these warning information, performs logical judgment according to preset rules and thresholds, and then issues corresponding warning or shutdown information; The temperature acquisition unit is used to obtain the temperature data matrix from the data acquisition module in real time and synchronize it with the video stream data in the model inference unit. The model inference unit detects the position information of the target in the thermal imaging image, and this information is then used by the temperature acquisition unit to determine the specific position of the target in the temperature data matrix. Specifically, the position and size (x, y, w, h) of the upper left corner of the thermal imaging image coordinate system detected by the model inference unit are used to obtain the position of the target in the temperature data matrix through coordinate system conversion. By screening the data of the target area in the temperature data matrix, the highest temperature in the target area is obtained, which is the temperature value of the current target. The terminal monitoring module is communicatively connected to the edge computing gateway module, supports remote monitoring and real-time data transmission, is used to receive early warning information sent by the edge computing gateway module, and display monitoring information around the tunnel boring machine in real time on the monitoring interface.

2. The thermal imaging-based early warning linkage system for roadheaders according to claim 1, characterized in that: The business processing unit will communicate the early warning information issued with the sound and light alarm module through the RS485 protocol to realize the sound and light alarm; and send the shutdown information to the equipment control unit through the internal protocol; the business types of the business processing unit include personnel detection in the dangerous area at the front end of the tunnel boring machine, high temperature monitoring of the tunnel boring head, and personnel detection in the dangerous area at the rear end of the tunnel boring machine.

3. The thermal imaging-based early warning linkage system for roadheaders according to claim 2, characterized in that: After receiving the shutdown information, the device control unit converts it into a DIDO switching signal and sends it to the control system of the tunnel boring machine to realize the emergency stop operation of the tunnel boring machine; at the same time, the device control unit will automatically perform a reset operation within a certain period of time after the emergency stop, allowing the tunnel boring machine to restart.

4. The thermal imaging-based early warning linkage system for roadheaders according to claim 1, characterized in that: The data storage unit is used to establish and manage Mysql and Minio databases to realize the storage of alarm information, images and videos, and provide corresponding external query interfaces to ensure data retention and user historical data query needs.

5. The thermal imaging-based early warning linkage system for roadheaders according to claim 1, characterized in that: The web-side service unit is used to display the real-time analysis results of the video stream collected by the tunnel boring machine, and at the same time provides real-time pop-up alarms, historical data queries, and alarm statistical analysis functions; the unit adopts a B / S architecture, and the client can directly access the WEB-side interface using a web page through the industrial ring network, thereby realizing real-time monitoring and browsing of the area around the tunnel boring machine.

6. The thermal imaging-based early warning linkage system for roadheaders according to claim 1, characterized in that: The terminal monitoring module includes an underground flat-panel super terminal, an above-ground monitoring host and a mine-used intrinsically safe monitoring host; the underground flat-panel super terminal communicates with the edge computing gateway module through a WIFI module; the mine-used intrinsically safe monitoring host communicates directly using the RJ45 network port of the edge computing gateway module; the above-ground monitoring host communicates with the edge computing gateway module through an underground ring network; the terminal monitoring module can access the WEB service of the edge computing gateway module to enable the mobile terminal to browse the original video stream, the analyzed AI video stream, the alarm message, and the statistical analysis information.

7. A thermal imaging-based early warning linkage method for a roadheader, using the thermal imaging-based early warning linkage system for a roadheader according to any one of claims 1 to 6, characterized in that: The following steps are involved: S1. Use the intrinsically safe thermal imaging camera installed on the roadheader to monitor the temperature of the surrounding area of ​​the underground roadheader in real time, thereby obtaining a real-time thermal imaging video stream and temperature data matrix, and transmit this data to the edge computing gateway module via Ethernet; S2. The model inference unit in the edge computing gateway module uses the standard RTSP protocol to obtain video stream data in real time, and performs real-time intelligent analysis on the received thermal imaging video stream to detect the workers and tunneling head in the four cameras. At the same time, the temperature acquisition unit obtains the temperature data matrix in real time and uses the camera's private protocol to update the temperature map of the camera monitoring area in real time. The model inference unit detects the positions of the tunneling head and workers and synchronizes the video stream data with the temperature data matrix to determine the maximum temperature of the target; S3. The business processing unit performs logical judgment on the detection results of the model reasoning unit according to preset rules to determine whether to issue an early warning or shutdown information; if a dangerous situation is detected, the business processing unit sends an early warning information to the sound and light alarm module via the RS485 protocol and controls the equipment control unit to execute an emergency stop operation of the tunnel boring machine through the internal protocol; S4. After receiving the warning information, the sound and light alarm module alerts the staff through voice broadcast and light warning. After receiving the emergency stop signal, the equipment control unit converts the signal into a DIDO switch signal and sends it to the roadheader control system to execute the emergency stop operation. It automatically resets after 10 seconds. S5. At the same time, the WEB service unit displays the real-time analysis results, provides real-time pop-up alarms, historical data query, and alarm statistical analysis functions. Users can access this information through the web page. Underground workers and surface managers can access the WEB service through the terminal monitoring module to monitor the working status of the tunnel boring machine in real time and receive alarm information to achieve complete control of the entire tunnel. The aircraft entry warning linkage system is used for centralized monitoring and management.

8. The thermal imaging-based early warning linkage method for a roadheader according to claim 7, characterized in that: According to the steps S5-S8, there are three types of alarm processes: The first type of alarm process: When a worker is detected in the two cameras at the front of the roadheader and enters the set ROI danger zone, an alarm message indicating human intrusion into the danger zone and a shutdown message are issued. After the audible and visual alarm receives the alarm message via the standard RS485 protocol, it will issue a voice alarm indicating "human intrusion into the danger zone" and flash the lights. After the equipment control unit receives the shutdown message via the internal protocol, it closes the emergency stop switch circuit, implementing an emergency stop operation on the roadheader, and resets the emergency stop circuit after an interval of 10 seconds. The second type of alarm process: When the two cameras at the front of the tunnel boring machine detect that the tunnel boring head temperature exceeds the set temperature, an over-temperature alarm and shutdown message will be issued. After the sound and light alarm receives the alarm information through the standard RS485 protocol, it will issue a voice alarm "Device temperature is too high" accompanied by flashing lights. After the equipment control unit receives the shutdown information through the internal protocol, it will close the emergency stop switch circuit to implement the emergency stop operation of the tunnel boring machine and reset the emergency stop circuit after an interval of 10 seconds. The third type of alarm process: When the two cameras at the rear of the tunnel boring machine detect that someone has entered the drawn ROI danger zone, an alarm message indicating that the person is approaching the tunnel boring machine will be issued. After the sound and light alarm obtains the alarm information through the standard RS485 protocol, it will issue a voice alarm of "person approaching the tunnel boring machine" accompanied by flashing lights. At the same time as the warning is triggered, the business processing unit calls the internal interface to store the video frame that currently triggers the warning, and at the same time, it will record the video for 10 seconds and upload the warning data to the database.

Images