Mine disaster rescue robot based on multi-modal perception and intelligent identification

Abstract

The invention discloses a mine disaster rescue robot based on multi-modal sensing and intelligent recognition and a mine disaster monitoring method, and relates to the technical field of mine disaster rescue robots. The mine disaster rescue robot based on multi-modal sensing and intelligent recognition mainly comprises an image processing and edge calculation unit, a UWB positioning module, an inertial measurement unit, a laser radar, a binocular depth camera, a thermal imager, a sound sensor, a gas sensor, an RGB camera, a motion control unit and a power supply system. According to the mine disaster rescue robot and the mine disaster monitoring method based on the multi-mode perception and intelligent identification, the positioning precision and the autonomous navigation capability of the rescue robot in a complex mine environment can be improved, and the accuracy and the real-time performance of state judgment of trapped workers are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mine disaster rescue robots, and more specifically, to a mine disaster rescue robot and a mine disaster monitoring method based on multimodal perception and intelligent recognition. Background Art

[0002] Mine disaster rescue robots are used to improve rescue efficiency, reduce risks and shorten response times. However, there are still deficiencies in positioning and personnel identification in the prior art. In terms of positioning, traditional robots rely on inertial navigation or fixed beacons, and it is difficult to adapt to the complex terrain in mines, resulting in large navigation errors and limited mobility. In a disaster environment, the beacons may be damaged, further reducing rescue efficiency. In terms of personnel identification, traditional methods mainly rely on visible light or infrared cameras, but the insufficient light, severe dust and smoke interference in mines affect the detection accuracy. In addition, existing robots are difficult to accurately judge the position, posture and vital signs of workers, and rescue still requires manual analysis, delaying the opportunity.

[0003] The above content is only used to assist in understanding the technical solution of the present invention, and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The purpose of the present invention is to provide a mine disaster rescue robot and a mine disaster monitoring method based on multimodal perception and intelligent recognition, which can improve the positioning accuracy and autonomous navigation ability of the rescue robot in a complex mine environment, and improve the accuracy and real-time performance of judging the state of trapped workers.

[0005] The present invention provides a mine disaster rescue robot based on multimodal perception and intelligent recognition, including an image processing and edge computing unit, a UWB positioning module, an inertial measurement unit, a lidar, a binocular depth camera, a thermal imager, a sound sensor, a gas sensor, an RGB camera, a motion control unit and a power supply system; the image processing and edge computing unit is electrically connected to the UWB positioning module, the inertial measurement unit, the lidar, the binocular depth camera, the thermal imager, the sound sensor, the gas sensor, the RGB camera, the motion control unit and the power supply system.

[0006] The present invention also provides a mine disaster monitoring method, which uses the above-mentioned mine disaster rescue robot based on multimodal perception and intelligent recognition to monitor and give early warnings of mine disaster accidents in mines and mine tunnels.

[0007] Implementing the mine disaster rescue robot and the mine disaster monitoring method based on multimodal perception and intelligent recognition provided by the present invention has the following beneficial effects: In view of the problems existing in mine rescue robots in the prior art, such as insufficient positioning accuracy, poor mobility, and insufficient ability to identify workers and judge accidents, the present invention proposes a mine rescue robot based on multi-modal perception and intelligent recognition. By integrating technologies such as UWB positioning, visual positioning, gas sensing, infrared imaging, and deep learning, it realizes high-precision positioning, intelligent worker identification, and accident judgment, which can greatly improve the rescue efficiency and ensure the safety of trapped personnel. Specifically, the present invention integrates UWB+visual inertial SLAM positioning to achieve high-precision autonomous navigation and adapt to complex environments; fully combines UWB and visual positioning technologies to greatly improve the positioning accuracy and autonomous navigation ability of the rescue robot in complex mine environments. The present invention combines visible light, infrared, and deep learning algorithms to accurately identify trapped workers and analyze their vital signs, providing intelligent support for rescue decision-making and improving the rescue success rate. By introducing object detection and pose recognition technologies of deep learning, the robot can accurately identify trapped workers and judge their status in real time, greatly improving the rescue efficiency. Compared with traditional mine rescue methods, the robot provided by the present invention can effectively avoid high-risk areas, accurately obtain the location and condition of trapped workers, and provide real-time feedback through intelligent analysis to help rescue personnel make more scientific decisions; not only reducing the risk of manual rescue, but also improving the response speed and accuracy of mine rescue, providing a more efficient and safe auxiliary tool for rescue personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention will be further described below in conjunction with the drawings and embodiments. In the drawings: Figure 1 is a block diagram of the mine rescue robot based on multi-modal perception and intelligent recognition provided by the present invention; Figure 2 is the principle of the TDOA algorithm provided by the present invention; Figure 3 is the flow chart of the visual algorithm provided by the present invention; Figure 4 is the ES-EKF algorithm flow provided by the present invention; Figure 5 is a schematic diagram of the dimensional changes of each layer provided by the present invention; Figure 6 is the principle diagram of YOLOv8 provided by the present invention; Figure 7 is the visible light detection of YOLOv8 provided by the present invention; Figure 8 is the thermal imaging detection of YOLOv8 provided by the present invention; Figure 9It is the overall framework of OCR provided by the present invention and the equivalent Transformer process; Figure 10 It is the schematic diagram of human pose estimation by HRNet provided by the present invention; Figure 11 It is the timing diagram of human pose estimation by HRNet provided by the present invention; Figure 12 It is the flowchart of human pose estimation by HRNet provided by the present invention; Figure 13 It is the schematic diagram of joint recognition provided by the present invention; Figure 14 It is the schematic diagram of fall detection provided by the present invention. Specific Embodiments

[0009] For a clearer understanding of the technical features, objectives, and effects of the present invention, the specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0010] Figure 1 The schematic diagram of the mine disaster rescue robot based on multi-modal perception and intelligent recognition in this embodiment is shown. In this embodiment, the mine disaster rescue robot based on multi-modal perception and intelligent recognition includes an image processing and edge computing unit, a UWB positioning module, an inertial measurement unit, a lidar, a binocular depth camera, a thermal imager, a sound sensor, a gas sensor, an RGB camera, a motion control unit, and a power supply system; the image processing and edge computing unit is electrically connected to the UWB positioning module, the inertial measurement unit, the lidar, the binocular depth camera, the thermal imager, the sound sensor, the gas sensor, the RGB camera, the motion control unit, and the power supply system; In an exemplary embodiment, the UWB positioning module is used to communicate with UWB base stations deployed in the mine and collect the distance information of the mine accident rescue robot to each base station; the inertial measurement unit is used to collect the motion state information of the mine accident rescue robot, fuse with the UWB positioning module and visual SLAM data to improve the positioning accuracy; the lidar is used to construct a map of the mining area environment to achieve obstacle avoidance and autonomous navigation; the binocular depth camera is used to collect depth visual information and achieve high-precision positioning in combination with the visual SLAM algorithm; the thermal imager is used to detect high-temperature areas in the mining area to assist in identifying the body temperature characteristics of trapped personnel or the location of the fire source; the sound sensor is used to monitor abnormal sounds in the mine, analyze the miner's distress signal through an AI model to improve the rescue efficiency; the gas sensor is used to detect the concentration of combustible gases, toxic gases and oxygen in the mining area environment and trigger an alarm when the concentration exceeds the standard; the RGB camera is used to capture visible light images and achieve automatic recognition of personnel, obstacles and mine facilities in combination with the YOLOv8 target detection algorithm; the motion control unit is used to receive the motion instructions from the image processing and edge computing unit and convert the motion instructions into motor drive signals to achieve path following and obstacle avoidance strategies; the power supply system is used to supply power to each circuit of the mine accident rescue robot. In an exemplary embodiment, the image processing and edge computing unit is configured to: based on multiple UWB base stations and UWB positioning modules deployed in the mine, use the TDOA algorithm to determine the position of the mine accident rescue robot in three-dimensional space in real time. In an exemplary embodiment, the image processing and edge computing unit is further configured to: based on the inertial measurement unit and the binocular depth camera, use the Kalman filter algorithm under multi-state constraints of stereo vision to achieve visual positioning of the mine accident rescue robot. In an exemplary embodiment, the image processing and edge computing unit is further configured to: based on multiple UWB base stations, UWB positioning modules, inertial measurement units and binocular depth cameras deployed in the mine, use the extended Kalman filter algorithm for error states to perform information fusion of sensors to achieve positioning of the mine accident rescue robot. In an exemplary embodiment, the information fusion of sensors by the extended Kalman filter algorithm for error states includes: using the visual inertial odometry positioning data as the nominal value, setting the error value of the state vector at the current moment to zero, and calculating the time interval; predicting the error value and covariance matrix of the state vector at the current moment; obtaining the observation vector of the system at the current moment according to the error value of the state vector at the current moment; updating the error value and covariance matrix of the state vector according to the observation vector and covariance matrix of the system at the current moment; obtaining the optimal estimate of the state vector according to the updated error value of the state vector and the nominal value of the state vector. In an exemplary embodiment, the error value and covariance matrix of the current moment state vector are predicted, as shown in the formula: , , where is the position error of the system in the x direction relative to the reference state at the k-th moment, is the position error of the system in the y direction relative to the reference state at the k-th moment, is the velocity error of the system in the x direction relative to the reference state at the k-th moment, is the velocity error of the system in the x direction relative to the reference state at the k-th moment, represents the position coordinates of the combined system on the two-dimensional plane at the k-th moment; and respectively represent the velocities of the combined system in the x direction and y direction at the k-th moment; is the nominal value of the state vector at the k-th moment, is the error value, is the true state quantity of the system at the k-th moment, which consists of the estimated value and the error term; t is the sampling interval time of the combined system; and respectively represent the accelerations of the combined system in the x direction and y direction at the (k - 1)-th moment; and are the predictions of the error state and covariance at the k-th moment respectively; F is the state matrix of the system, is the noise vector of the system, is the process noise covariance matrix.

[0011] In an exemplary embodiment, the observation vector of the system at the current moment is obtained according to the error value of the current moment state vector, as shown in the formula: , where is the observation vector of the system at k moment, is k the error value of the state vector at the moment, H is the observation matrix, is the observation noise; In an exemplary embodiment, the error value and covariance matrix of the state vector are updated according to the observation vector and covariance matrix of the system at the current moment, as shown in the formula: , where is the Kalman gain at the k-th moment, is kThe observation covariance at the moment, where I is the identity matrix; In an exemplary embodiment, the image processing and edge computing unit is further configured to: based on the mine tunnel image, use a multi-layer perceptron model for classification and recognition, and extract and calculate the basic feature information of the mine tunnel to obtain a judgment on the presence or absence of an accident and a prediction of the type; In an exemplary embodiment, the image processing and edge computing unit is further configured to: based on the visible light image and the infrared thermal imaging image, use the YOLOv8 object detection model to determine the posture state of the worker; This embodiment provides a mine accident monitoring method, which uses a mine accident rescue robot based on multi-modal perception and intelligent recognition in Shangyu to monitor and warn of mine accident in mines and mine tunnels.

[0012] In some embodiments, the above mine accident rescue robot based on multi-modal perception and intelligent recognition can also be implemented in the following manner. In this embodiment, the mine accident rescue robot based on multi-modal perception and intelligent recognition is equipped with a variety of sensors such as a UWB positioning module, an inertial measurement unit (IMU), a high-definition camera, an infrared thermal imager, and a gas sensor to build a diversified information collection system to achieve comprehensive monitoring of complex situations in the mine accident environment; As an exemplary embodiment, the hardware composition of the rescue robot: The rescue robot uses NVIDIA Jetson Orin NX as the core unit for image processing and edge computing, runs the Ubuntu operating system, and is connected to the UWB module, Livox Mid-40 3D lidar, Intel Realsense D455 binocular depth camera, IMU module, FLIR Lepton 3.5 thermal imager, gas sensor, sound sensor, RGB camera, and motion control unit through interfaces such as USB, UART, SPI, I2C, and CAN. The UWB module is used to communicate with the UWB base stations deployed in the mine and collect the distance information from the rescue robot to each base station; the Livox Mid-40 3D lidar has a 360° field of view and is used to construct a map of the mining area environment to achieve obstacle avoidance and autonomous navigation; the Intel Realsense D455 binocular depth camera is used to collect depth information and achieve high-precision positioning in combination with the visual SLAM algorithm; the IMU module integrates a gyroscope, an accelerometer, and a magnetometer and is used to collect the motion state information of the robot, fuse with the UWB and visual SLAM data, and improve the positioning accuracy. The FLIR Lepton 3.5 thermal imager is used to detect high-temperature areas in the mining area and assist in identifying the body temperature characteristics of trapped personnel or the location of the fire source; the sound sensor is used to monitor abnormal sounds in the mine, analyze the miner's distress signal through an AI model, and improve the rescue efficiency; the gas sensor is used to detect the concentration of combustible gases, toxic gases, and oxygen in the mining area environment and trigger an alarm when the concentration exceeds the standard; the RGB camera is used to capture visible light images and achieve automatic recognition of personnel, obstacles, and mine facilities in combination with the YOLOv8 object detection algorithm. The main control board of the robot is connected to the motion control unit through the CAN bus. The motion control unit uses STM32F407 as the main controller to control the four-wheel independent drive system of the Mecanum wheels. The Mecanum wheels provide omnidirectional movement capabilities, enabling the robot to flexibly respond to narrow and complex environments. At the same time, the independent suspension system provides shock absorption functions to ensure stable operation on rough terrain. STM32F407 is responsible for receiving the motion instructions calculated by Jetson Orin NX and converting them into motor drive signals to achieve path following and obstacle avoidance strategies. In terms of the power supply system, the robot is equipped with a 24V lithium battery pack as the main power supply to supply the main control unit, lidar, camera, and motion system; an independent 12V lithium battery is used to supply power to low-power sensors (such as gas sensors, IMUs, sound sensors, etc.) to ensure that even when the main battery is running low, key environmental data can still be continuously collected. The power management module (PMIC - Texas Instruments BQ25703A) supports intelligent power scheduling. When the battery level is below 20%, it gives priority to ensuring the operation of core functions and sends a low-battery alarm to the terminal.

[0013] It should be noted that the rescue robot operates in a combination of autonomous navigation and remote control. Based on the multi-source fusion positioning of visual SLAM + UWB + IMU, it achieves high-precision autonomous navigation and supports remote operation at the same time, adapting to complex rescue environments. After the robot is started, the 3D lidar and binocular depth camera collect the mine environment information in real time, use LIO-SAM (Lidar-Inertial Odometry SLAM) for map construction, and based on A Path Planning Algorithm * Calculate the optimal inspection path and autonomously search for trapped personnel in the mine. During the inspection process, the robot combines thermal imaging, RGB cameras, and sound sensors for multi-modal detection, uses YOLOv8 object detection to identify trapped workers, combines thermal imaging data to confirm vital signs, listens for distress signals at the same time, and identifies abnormal sound sources through CNN + Mel spectrum analysis to improve the reliability of object detection. During the search process, the robot also continuously monitors the mine environment. Gas sensors are used to detect harmful gases. Once the concentration exceeds the standard, the robot will automatically adjust the inspection path and send an alarm to the terminal; the thermal imager monitors for fires to assist in mine accident emergency response; when the robot's battery level is lower than the threshold, it automatically returns to the charging station to ensure the smooth execution of subsequent tasks. When the robot detects trapped personnel, it will automatically generate the optimal rescue path and send the personnel location information, environmental conditions, and real-time video stream to the remote terminal to assist rescue personnel in making decisions. At the same time, the robot can release emergency lighting to provide light in dark environments, or comfort trapped personnel through voice broadcasts and provide evacuation guidance. The rescue robot integrates technologies such as visual SLAM, UWB positioning, and deep learning object detection to achieve efficient autonomous search, environmental monitoring, and remote control capabilities, providing intelligent, high-precision, safe, and reliable technical support for mine accident rescue.

[0014] It should be noted that in terms of positioning and navigation, in this embodiment, a combination of UWB and visual positioning is adopted to solve the accuracy and adaptability problems in the prior art. In terms of UWB positioning, multiple UWB base stations are deployed in the mine, and the rescue robot is equipped with a UWB tag to perform high-precision positioning using the UWB communication system. The robot determines its position in three-dimensional space in real time according to the TDOA algorithm by calculating the time difference of signal propagation between multiple base stations. This technology has the characteristics of high precision and low power consumption and is suitable for the complex spatial structure in the mine environment. As Figure 2 shown, on the premise of knowing the coordinates of three UWB base stations BS1, BS2, and BS3, a system of equations can be written based on the distances d1, d2, and d3 from the target object to the three UWB base stations to solve the coordinates of the target object.

[0015] In terms of visual positioning technology, the inspection robot is equipped with an IMU and a camera, which combines the Visual-Inertial-Odometry SLAM (VIO-SLAM) technology to achieve visual positioning. In this design, the Stereo-Multi-State Constraint Kalman Filter (S-MSCKF) algorithm that fuses IMU and visual information under the EKF framework is adopted. The S-MSCKF algorithm can be divided into two parts: front-end vision and back-end filtering. The front-end vision is mainly used to detect feature points in the image and publish data such as the coordinates of the feature points in the camera coordinate system to the back-end. During this process, the extracted feature points need to be tracked and binocular and frame-to-frame mismatched points need to be removed. The algorithm flow of the front-end vision is as Figure 3As shown in (a). The algorithm process mainly includes the following key links: Initializing the first frame: After the binocular camera obtains the first frame of image, it first detects the Feature from Accelerated Segment Test (FAST) feature points in the left-eye image, projects the left-eye feature points into the right eye for binocular feature point matching according to the internal parameters between the left and right eye cameras, and then performs processing such as Lucas-Kanade (LK) optical flow tracking and epipolar geometry constraint through the image pyramid, eliminates the feature points that are tracked outside the image, and finally divides the feature points into grids and outputs the FAST feature point information; Tracking feature points: During the movement of the camera, different frames of images will be received successively. It is necessary to track the feature points extracted from the previous frame of image in the next frame of image and extract new feature points. In this stage, the epipolar geometry constraint, LK optical flow tracking, and Two-points RANSAC (RANdom SAmple Consensus) method are also used to continuously eliminate the mis-matched feature points between the left and right eyes and the feature points that exceed the image boundary; Adding new feature points: As time goes by, some features in the video image will disappear, and some features may also have cumulative errors. Therefore, it is necessary to add some new features during the continuous tracking process to ensure that the program can keep running; The main function of the S-MSCKF backend is to perform extended Kalman filtering on the feature point coordinates transmitted from the visual front end and the IMU state to obtain the optimal estimated value of the current pose of the camera; First, receive the front-end image feature point information by subscribing to relevant topics, and then initialize relevant quantities such as gravity and deviation; Then process the IMU data to construct the state matrix and input matrix in the differential equation system, and obtain the state transition matrix; After that, perform state prediction, state augmentation, and state update in sequence, remove the historical camera state data, and obtain the current camera state; And finally publish the topic of the camera's pose information; Combining UWB with visual positioning technology, the UWB technology itself has the characteristics of high precision and low power consumption, and can provide ultra-wideband signals for data communication at the same time. However, within the short time of UWB signal refreshing, the base station cannot determine the position of the UWB positioning tag, which will lead to a decrease in the accuracy of dynamic positioning. Therefore, in this embodiment, visual positioning is introduced, and the visual positioning fills in the travel trajectory of the robot during the period when the UWB signal is missing, which can improve the positioning accuracy of the robot. At the same time, the visual positioning technology introduced in this embodiment enables the robot to have the functions of autonomous route planning and obstacle avoidance, and no longer needs to install guide rails. This can not only reduce costs and save the occupied area of the pipe gallery, but also adapt to mines with different layouts and improve the robustness of the robot's route planning.

[0016] The following introduces the UWB-VIO (Visual Inertial Odometry) combined positioning algorithm: To obtain accurate and reliable positioning data after long-term operation, it is necessary to fuse and calculate the positioning data respectively output by UWB and VIO. Sensor information fusion methods mainly include weighted average method, Kalman filtering method, Bayesian inference method, neural network algorithm, etc. In this embodiment, the Error State-Extended Kalman Filter (ES-EKF) algorithm for processing nonlinear problems is selected for sensor information fusion. In the UWB-VIO combined positioning algorithm, the independent coordinate system used in the UWB positioning process is used as the world coordinate system for VIO positioning. Then, the position and velocity of the mobile robot are used as the state vector of the system. Assuming that the UWB positioning observation value and the VIO positioning observation value are timestamp-aligned at time k, the state vector at time k is , where represents the position coordinates of the combined system on the two-dimensional plane at the k-th moment; and represent the velocities of the combined system in the x-direction and y-direction at the k-th moment respectively. Assuming that the nominal value of the state vector at time k is , and the error value is , . Then the error state model of the system can be expressed as Equation (1):

[0017] where: t is the sampling interval time of the combined system; and represent the accelerations of the combined system in the x-direction and y-direction at the (k - 1)-th moment respectively.

[0018] The ES-EKF algorithm process is as Figure 4 shown. Taking the VIO positioning data of the combined system received as the nominal value of the state vector , and then performing error state update. After sorting Equation (1), the prediction of the error state and covariance at time k is shown as Equation (2):

[0019] where: F is the state matrix of the system, is the noise vector of the system, which are respectively defined as: , ; where, is the optimal estimate of the error state covariance at the (k - 1)-th moment; is the noise covariance at time k-1. Considering that the state vector of the combined system is only used to track the position and velocity of the system, in this embodiment, the acceleration is used as the random noise of the system. The difference between the positioning information obtained from UWB and VIO is used as the observation vector of the system , then the observation equation of the combined system is:

[0020] where: is the observation noise; and The values are respectively: ,[[ID= , After that, the error state and covariance matrix can be updated, including calculating the Kalman gain , updating the error state and updating the covariance matrix . The specific equations are shown in Equation (4):

[0021] where: is k The observation covariance at time. Then according to get k The optimal estimated value result of the true state at time. The principle of the pseudocode of this algorithm is as follows:

[0022] The specific process is: using the visual inertial odometry (VIO) positioning data as the nominal value ; let k The error value of the state vector at time ; calculate the time interval: ; according to Equations (1) and (2) for k Predict the error value and covariance matrix of the state vector at time, and calculate to get and ; according to Equation (3), get the observation vector of the system at k time ; according to Equation (4) for k Update the error value and covariance matrix of the state vector at time, and calculate to get and ; let According to = , calculate to get k The optimal estimate of the state vector at time; output the optimal state estimate of the system at time k .

[0023] To design a method capable of classifying and recognizing mine tunnel images and extracting and calculating basic feature information of mine tunnels, in this embodiment, a deep convolutional neural network model is used to learn pipeline image data. The algorithm design mainly relies on the Python language and uses the relatively convenient Keras as the development framework. First, a large number of images of mine tunnel collapses and other accidents are collected and subjected to grayscale conversion, labeling, and size normalization processing to form a sufficient training set. Then, the trained model is used to judge the presence or absence of accidents and predict the types of test set data; the following contents are included: Original data collection: Machine learning for image recognition requires a large amount of data sets to train the model. Generally speaking, the amount of data required to train a neural network is at least more than 500. However, compared with other complex images such as human faces, mine tunnel images are relatively simple, with fewer features and relatively simple recognition. Experimental results prove that a data volume of about 500 can basically meet the requirements for training the model. Therefore, 600 images of each type are collected for training.

[0024] Data preprocessing: The various types of collapse images collected cannot be directly used to train the model and must be processed to generate a labeled data set before they can be used. To reduce the amount of calculation and accelerate the training speed of the convolutional neural network, it is necessary to perform size normalization and grayscale conversion on the data. 1) Image size normalization: Since the resolution of the collected data images is different, and the input layer of the convolutional neural network requires data of the same size, normalization processing must be performed. To reduce the training time and improve the training speed of the model, the images are scaled and the size is normalized to 80×80. After the resolution is reduced, it will not affect the distribution characteristics of the pixel point groups with relatively low grayscale in the image. 2) Grayscale conversion: The collected mine tunnel images are color pictures, and the convergence speed is relatively slow during training. To accelerate the convergence of the neural network model, the pipeline pictures are subjected to grayscale conversion. 3) Labeling. The data collected cannot be directly used for training. Each type of pipeline accident image must be labeled with its respective category before the training results can be compared with the true classification results and the parameters can be continuously updated. 4) Generating a data set. The labeled images are in the jpg image format, and it is necessary to read the dot matrix data in the images. Using the interface provided by OpenCV, the image data is read, and the labels of the images are corresponded with the image data information and written into a csv file. When training the convolutional neural network, this csv file can be directly read.

[0025] Building a Multilayer Perceptron Model: A multilayer perceptron (MLP) is a neural network composed of fully connected layers with at least one hidden layer. In this network, each node is an artificial neuron, and the output of one artificial neuron can be used as the input of another. Moreover, the output of each hidden layer neuron is transformed through an activation function. A remarkable feature of the MLP is that, when the network is large enough, the model can represent any mathematical function. Model Parameters: The structure of the multilayer perceptron has an input layer, hidden layers, and an output layer. Layers are fully connected to each other, and the output value of the previous layer serves as the input value of the next layer. Select a multilayer perceptron with 3 hidden layers: The input layer takes in a two-dimensional grayscale image with a size of 80×80×1. After unfolding it into a one-dimensional tensor, it is passed into the hidden layers. The number of neurons in each hidden layer is 256, 512, and 256 respectively, and the output layer outputs 4 classification results. Training and Parameter Update: A fixed number of data samples are randomly selected from the training set as a batch for training. After each batch training, the model parameters are updated once until all data has been parameter-updated. This process is called one epoch of training. The batch size is set to 32 data samples, and training and parameter update end after 3,000 epochs.

[0026] Building a Convolutional Neural Network Structure: Generally speaking, the more layers a convolutional neural network has, the better the data fitting effect, but the longer the computing time and the greater the probability of overfitting. For small batches of data, the number of layers does not need to be too high. In this embodiment, a 5-layer network structure model is selected for training. Specific Network Structure Parameters: Build a 5-layer convolutional neural network, and the parameters of each layer are shown in Table 1. The first three layers are convolutional layers (Conv layers) for extracting image features, and each convolutional layer is followed by a pooling layer. The last two layers are fully connected layers (Fc layers), and a SoftMax classifier is connected after the fully connected layers to achieve image classification. To prevent overfitting during training, a dropout layer is connected after each pooling layer to randomly deactivate some neurons to ensure good results on the test set.

[0027] Table 1: Neural Network Model Parameters

[0028] The dimensional changes of the input data after passing through each layer of the convolutional neural network are as Figure 5 shown.

[0029] Neural network training: The processed images are used as the dataset for model training. 20% of the data is randomly selected as the validation set to improve the generalization ability of the model. To select a suitable optimizer, different optimizers are tested under the same epoch and batch size; Testing the recognition results of two models: To obtain a recognition model with high accuracy, the model is experimentally verified. 400 images are randomly selected from the validation set, and the model is used to predict these 400 images. The recognition results are plotted through a confusion matrix.

[0030] In terms of intelligent recognition, this embodiment adopts deep learning technology to achieve precise recognition of trapped workers in a complex mine environment. In a mine accident environment, there is insufficient light and severe interference from smoke and dust, and traditional visual detection is prone to failure. Therefore, this embodiment adopts a multi-modal deep learning model, combining visible light, infrared thermal imaging, and depth sensor data to improve the recognition accuracy.

[0031] To ensure that the mine accident rescue robot can accurately identify trapped workers in a complex environment, this embodiment adopts a deep learning object detection algorithm that fuses visible light and infrared thermal imaging. The mine accident environment usually has adverse factors such as low light, smoke, and dust coverage, and traditional single-modal visual detection methods are prone to failure in such environments. Therefore, this embodiment introduces the YOLOv8 object detection model, Figure 6 which is the schematic diagram of YOLOv8; by combining visible light and infrared thermal imaging data, the object detection ability of the robot in a harsh environment is improved.

[0032] The dual-light camera (visible light + infrared thermal imaging) carried by the robot continuously collects mine environment images and uses a multi-modal fusion algorithm for object detection. First, the YOLOv8 model performs object detection on the visible light image and the thermal imaging image respectively. The former is used to identify the worker's form in a normal environment, and the latter is used to extract the worker's body temperature characteristics in a low-light or high-smoke environment to enhance the detection ability of trapped workers. As Figure 7 and Figure 8 shown, in a low-light environment, the visible light camera may not be able to identify the worker, but the infrared thermal imaging can clearly capture the body temperature distribution of the worker, thereby improving the reliability of detection.

[0033] The data processing flow is as follows. First, the robot obtains the visible light image $I_v$ and the infrared thermal imaging image $I_t$, which are used as the inputs of YOLOv8 respectively. Subsequently, YOLOv8 uses a deep neural network to extract multi-scale features and fuse the information of different modality images to ensure effective target detection even in occlusion, low light or smoky environments. The model finally outputs the bounding box $(x, y, w, h)$ and class labels of the workers (such as "worker", "obstacle"), filters out the misdetected targets with low temperature by combining the thermal imaging threshold, and adopts an adaptive weight fusion strategy to weighted merge the detection results of visible light and infrared thermal imaging to improve the stability of detection.

[0034] After accurately identifying the workers, the next step is to estimate their postures to judge the states of the workers (e.g., standing, fallen or trapped). To achieve this goal, this embodiment adopts the human pose estimation technology, combines the HRNet model to extract the joint point information of the workers, and further analyzes their postures.

[0035] First, HRNet extracts the key point positions of the human body, such as the head, torso, limbs, etc., through a multi-stage convolutional neural network. Through the relative position relationship of these key points, this embodiment can judge the posture of the workers. For example, when a worker stands, the key points of the legs and torso show a straight line arrangement; while if the worker falls to the ground, the key points of the body will show an obvious horizontal or nearly horizontal arrangement, which is convenient to distinguish whether the worker has fallen or is trapped. Accurately identifying the state of the workers and quickly providing a real-time judgment on whether the workers are in a dangerous state (such as fallen or trapped) provides an important basis for subsequent rescue decisions. This technical solution is particularly important in the process of mine accident rescue because the posture of the workers can directly affect the formulation and implementation efficiency of rescue strategies.

[0036] It should be noted that this embodiment is based on deep learning and multi-modal data fusion, and organically combines the YOLOv8 object detection model and the HRNet human pose estimation model. The whole process mainly includes two major stages: Object detection stage: The visible light image and the infrared thermal imaging image collected by the dual-light camera are input into YOLOv8, and the bounding box of the workers is detected in real time by using multi-scale feature extraction and adaptive weight fusion strategy; Pose estimation and fall judgment stage: Conduct HRNet key point detection on the detected worker area, and determine the pose state of the workers (such as standing, falling, lying prone, etc.) by analyzing the relative positions and angles of each key point of the human body. This embodiment makes full use of the feature extraction ability of the deep neural network in complex backgrounds and low light environments, and effectively improves the robustness and accuracy of object detection and pose recognition in mine accident environments. The following is a detailed description of the YOLOv8 object detection module: Multi-modal Image Acquisition and Feature Extraction: In a mine accident environment, due to insufficient lighting, heavy dust, and smoke obstruction, the image information of a single modality often has limitations. Therefore, in this embodiment, a dual-camera system is used to synchronously acquire visible light images \(I_v\) and infrared thermal imaging images \(I_t\); the former captures the morphological features of workers, and the latter provides temperature distribution information, which helps to detect targets under low-light or occlusion conditions. For these two types of image data, the system uses a Convolutional Neural Network (CNN) as a feature extractor. After each image undergoes multiple layers of convolution, pooling, and non-linear activation, feature maps of different scales are obtained. Using a Feature Pyramid Network (FPN) structure, detailed information about the target can be captured at various scales. Its mathematical description is as follows:

[0037] Subsequently, the two feature maps are fused through an adaptive fusion layer:

[0038] where \(\alpha\) and \(\beta\) are learnable weight parameters, which are optimized through the backpropagation mechanism to adaptively adjust the contributions of the two modalities of data in different environments.

[0039] Bounding Box Regression and Object Classification: After feature fusion, the YOLOv8 network uses a fully connected layer to process the fused features and outputs the bounding box and class probability of the detected object. The bounding box parameters \((x, y, w, h)\) represent the center position, width, and height of the object in the image, respectively. During the object detection process, a regression method is used to fit the actual bounding box, and its loss function usually uses Smooth L1 loss, defined as:

[0040] The corresponding bounding box loss is:

[0041] At the same time, the classification loss part uses the cross-entropy loss function to ensure the accurate classification of the detected objects. The object detection module adopts an end-to-end joint optimization strategy during the training phase to improve the accuracy of both bounding box regression and classification simultaneously.

[0042] The advantages of this module are as follows: Multi-modal fusion: By simultaneously using visible light and infrared data, the detection accuracy under low-light or occlusion conditions is greatly improved; Multi-scale feature extraction: The FPN structure can capture detailed information of targets of different sizes and is suitable for complex on-site scenarios; Real-time performance: The network is designed in a lightweight manner to meet the requirements of real-time data processing in mine accident rescue.

[0043] The following is a detailed description of the HRNet pose estimation and fall detection module: HRNet network structure and key point extraction principle: HRNet (High-Resolution Network) has outstanding advantages in human pose estimation. Its main feature is to maintain high-resolution features and fuse multi-scale information. For the worker images in the detection area, HRNet directly outputs the heatmaps of each key point of the human body, and then locates the key point positions through the maximum response values of the heatmaps. Assuming that HRNet outputs K key points, their two-dimensional coordinates can be expressed as:

[0044] Common key points include the head, shoulders, elbows, wrists, hips, knees, and ankles, etc. These points constitute the complete human skeleton information. The multi-resolution parallel design of HRNet ensures high-precision key point localization results even in complex backgrounds. In addition, in this embodiment, HRNet is further combined with Object Contextual Representation, achieving higher performance on three datasets. The code of RNet+OCR is included in this branch. Figure 9 is the overall framework of OCR and the equivalent Transformer process; Figure 10 is the schematic diagram of HRNet human pose estimation; Calculation of the human body center of gravity and angle analysis: After detecting the human body key points, in order to further judge the state of the worker, it is necessary to calculate the center of gravity of the human body and the angles between the connections of some key points and the horizontal line. The center of gravity calculation formula is:

[0045] Next, taking the shoulders and hips as an example, calculate the included angle θ between their connection line and the horizontal line:

[0046] If this angle θ significantly deviates from the vertical direction (for example, close to the horizontal state, that is, |θ| is small), it may indicate that the worker is in a fallen or lying state.

[0047] Fall detection criterion and time series analysis: To further improve the accuracy of fall detection, this module not only relies on the pose information of a single frame of image, but also combines multi-frame time series data to analyze the movement trend of the worker. The fall detection criterion is set as follows: Where: θt is the angle threshold (e.g., 10° - 15°), which is satisfied when the worker's posture is close to horizontal; Δd is the displacement of the center of gravity position between consecutive frames. When the displacement exceeds the set threshold dth, it indicates that the worker's position has changed significantly. Multi-frame time series analysis can reduce the false detection rate and eliminate false judgments caused by single-frame noise or instantaneous posture changes; Figure 11 is the time series diagram of HRNet human pose estimation; Figure 12 is the flow chart of HRNet human pose estimation; Figure 13 is the schematic diagram of joint recognition, Figure 14 is the schematic diagram of fall detection; The following introduces the system integration and real-time feedback mechanism: In the mine disaster rescue scenario, the system needs to face extreme lighting and environmental interference problems. The multi-modal data acquisition module not only relies on dual-light cameras to collect images, but also combines IMU data to achieve dynamic correction, ensuring the stability and continuity of image acquisition. The preprocessing module uses technologies such as image enhancement, denoising filtering, and size normalization to provide high-quality input for subsequent object detection and pose estimation. In terms of model collaborative inference and result fusion, the system is divided into two main inference processes during operation: YOLOv8 inference process: The preprocessed image data first enters the object detection module. After multi-modal feature fusion, it real-time outputs candidate target regions and bounding boxes; HRNet pose estimation process: The detected target regions are key processed. Key point information is extracted through HRNet, and the center of gravity and joint angles of the human body are calculated; Subsequently, the multi-frame time series data is used to analyze the target movement trend to determine whether there is a fall or abnormal situation. The detection results of the two major modules are fused in the system background. A certain confidence weighting mechanism is used to correct different detection results, and the final judgment result is transmitted to the on-site control center through a low-latency data channel. Rescue personnel can view the worker's status, location information, and fall alarms in real time through the monitoring platform, providing a basis for emergency rescue decisions.

[0048] Real-time Feedback and Alarm Mechanism: To ensure timely response in critical moments, a real-time feedback mechanism with low latency is designed in this embodiment. Once the detection module detects an abnormality (such as a fall or being in a dangerous posture for a long time), it will immediately generate an alarm signal and upload the worker's location information, posture data, and chronological change records to the rescue dispatch center. The dispatch center conducts automatic analysis based on the data and also supports manual intervention to ensure that the rescue plan can be implemented quickly and effectively. Its features include: Multi-modal data fusion: Effectively overcomes the deficiencies of a single visual sensor in extreme environments, improving detection robustness and accuracy. Precise target detection and key point extraction: Utilizes advanced YOLOv8 and HRNet models to ensure accurate positioning and identification of human portrait features in complex environments. Dynamic chronological analysis: Judges the worker's motion state through multi-frame data fusion to achieve accurate judgment of abnormal states such as falls. Real-time alarm and data feedback: The system supports a low-latency real-time feedback mechanism, providing accurate and timely information support for on-site rescue decisions.

[0049] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope protected by the claims of the present invention. All of these are within the protection scope of the present invention.

Claims

1. A mine rescue robot based on multimodal perception and intelligent recognition, characterized by: It includes an image processing and edge computing unit, a UWB positioning module, an inertial measurement unit, a lidar, a binocular depth camera, a thermal imager, a sound sensor, a gas sensor, an RGB camera, a motion control unit, and a power supply system; the image processing and edge computing unit is electrically connected to the UWB positioning module, the inertial measurement unit, the lidar, the binocular depth camera, the thermal imager, the sound sensor, the gas sensor, the RGB camera, the motion control unit, and the power supply system.

2. The mine disaster rescue robot based on multi-modal perception and intelligent recognition according to claim 1, wherein, The UWB positioning module is used to communicate with the UWB base stations deployed in the mine and collect the distance information of the mine accident rescue robot to each base station; the inertial measurement unit is used to collect the motion state information of the mine accident rescue robot, fuse with the UWB positioning module and visual SLAM data to improve the positioning accuracy; the lidar is used to construct a mine environment map to achieve obstacle avoidance and autonomous navigation; the binocular depth camera is used to collect depth visual information and achieve high-precision positioning in combination with the visual SLAM algorithm; the thermal imager is used to detect high-temperature areas in the mine and assist in identifying the body temperature characteristics of trapped personnel or the location of the fire source; the sound sensor is used to monitor abnormal sounds in the mine, analyze the miner's distress signal through an AI model to improve the rescue efficiency; the gas sensor is used to detect the concentration of combustible gases, toxic gases and oxygen in the mine environment and trigger an alarm when the concentration exceeds the standard; the RGB camera is used to capture visible light images and achieve automatic recognition of personnel, obstacles and mine facilities in combination with the YOLOv8 target detection algorithm; the motion control unit is used to receive the motion instructions from the image processing and edge computing unit and convert the motion instructions into motor drive signals to achieve path following and obstacle avoidance strategies; the power supply system is used to supply power to each circuit of the mine accident rescue robot.

3. The mine disaster rescue robot based on multi-modal perception and intelligent recognition according to claim 1, characterized in that, The image processing and edge computing unit is configured to: utilize the TDOA algorithm to determine the position of the mine accident rescue robot in three-dimensional space in real time according to multiple UWB base stations and the UWB positioning module deployed in the mine.

4. The mine accident rescue robot based on multi-modal perception and intelligent recognition according to claim 1, characterized in that, The image processing and edge computing unit is configured to: achieve visual positioning of the mine accident rescue robot by using the Kalman filter algorithm under multi-state constraints of stereo vision according to the inertial measurement unit and the binocular depth camera.

5. The mine rescue robot based on multimodal perception and intelligent recognition according to claim 1, characterized in that: The image processing and edge computing unit is configured to: perform information fusion of sensors by using the error state extended Kalman filter algorithm according to multiple UWB base stations, the UWB positioning module, the inertial measurement unit and the binocular depth camera deployed in the mine to achieve positioning of the mine accident rescue robot.

6. The mine disaster rescue robot based on multi-modal perception and intelligent recognition according to claim 5, characterized in that, The error state extended Kalman filtering algorithm is used for information fusion of sensors, including: using visual inertial odometry positioning data as the nominal value, setting the error value of the state vector at the current moment to zero, and calculating the time interval; predicting the error value and covariance matrix of the state vector at the current moment; obtaining the observation vector of the system at the current moment according to the error value of the state vector at the current moment; updating the error value and covariance matrix of the state vector according to the observation vector and covariance matrix of the system at the current moment; and obtaining the optimal estimate of the state vector according to the updated error value of the state vector and the nominal value of the state vector.

7. The mine disaster rescue robot based on multi-modal perception and intelligent recognition according to claim 6, wherein, The prediction of the error value and covariance matrix of the state vector at the current moment is as follows: ， ， in, is the position error of the system in the x direction relative to the reference state at the kth moment, is the position error of the system in the y direction relative to the reference state at the kth moment, is the velocity error of the system in the x direction relative to the reference state at the kth moment, is the velocity error of the system in the x direction relative to the reference state at the kth moment, Represents the position coordinates of the combined system at the kth moment on the two-dimensional plane; and Respectively represent the velocities of the combined system in the x and y directions at the kth moment; is the nominal value of the state vector at time k, is the error value, The actual state of the system at the kth moment is composed of the estimated value and the error term; t is the sampling interval of the combined system; and Respectively represent the acceleration of the combined system in the x-direction and y-direction at the k-1th moment; and are the predictions of the error state and covariance at time k; F is the state matrix of the system, is the noise vector of the system, is the process noise covariance matrix.

8. The mine disaster rescue robot based on multi-modal perception and intelligent recognition according to claim 1, characterized in that, The image processing and edge computing unit is further configured to: classify and identify according to the mine tunnel image by using a multi-layer perceptron model, extract and calculate the basic feature information of the mine tunnel, and obtain the judgment of the presence or absence of an accident and the prediction of the type.

9. The mine accident rescue robot based on multimodal perception and intelligent recognition according to claim 1, characterized in that, The image processing and edge computing unit is further configured to: determine the posture state of the worker according to the visible light image and the infrared thermal imaging image by using the YOLOv8 object detection model.

10. A mine accident monitoring method, characterized in that, Use the mine disaster rescue robot based on multi-modal perception and intelligent recognition as described in any one of claims 1-9 to monitor and warn of mine disasters in mines and mine tunnels.

Images