Early warning method and system for dangerous area of heading machine fusing IMU and visual features

Abstract

The invention provides a heading machine dangerous area early warning method and system fusing IMU and visual characteristics, and relates to the field of video image processing, and the method comprises the steps: employing a checkerboard calibration plate to calibrate an infrared camera, so as to obtain the parameters of the camera, and the parameters comprise an internal parameter, an external parameter, and a distortion coefficient; according to the parameters of the camera, performing synchronous processing on the camera and the IMU to obtain synchronous data of the camera and the IMU; correcting the image according to the synchronous data of the camera and the IMU to obtain a corrected image; according to the corrected image, identifying and positioning the position of the miner, and obtaining the size information of the miner in the image to obtain key data of the image; and calculating the actual distance between the miner and the heading machine according to the key data of the image to obtain the actual distance. Whether the mine is in a dangerous area or not is judged according to the distance, if the mine is in the dangerous area, the heading machine stops and gives an alarm, and safety of miners is protected.

Description

Technical Field

[0001] The present invention relates to the field of video image processing, and in particular to a dangerous area early warning method and system for a tunnel boring machine integrating IMU and visual features. Background Art

[0002] Due to the narrow space underground in coal mines and the limited range of human activities, when the tunnel boring machine is operating, if a person breaks into the operating area, a crushing accident is likely to occur, causing casualties. Therefore, it is necessary to monitor the dangerous operating area of ​​the tunnel boring machine to prevent people from entering by mistake. Using machine vision technology to perceive the surrounding environment is an important way, which has the advantages of low cost and strong portability. The surrounding environment image is collected by an infrared camera, and visual ranging is realized based on the principle of similar triangles. When a person enters the designated dangerous area, the equipment will sound an alarm. However, during the movement of the tunnel boring machine, due to the unevenness of the coal mine tunnel, the onboard camera may move, rotate or vibrate, thereby affecting the accuracy of the monocular ranging based on vision and causing the measurement error to increase. The accuracy of visual ranging can be further improved by correcting the image with the posture data collected by the onboard IMU sensor of the tunnel boring machine. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a dangerous area warning method and system for a tunnel boring machine that integrates IMU and visual features. After obtaining accurate height information of the miner's image through the first three modules, the distance measurement module calculates the distance from the tunnel boring machine's onboard camera to the miner, and determines whether it is in the dangerous area based on the distance. If it is in the dangerous area, the tunnel boring machine will stop and alarm to protect the safety of the miners.

[0004] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0005] In a first aspect, a dangerous area warning method for a roadheader is provided by integrating IMU and visual features, the method comprising:

[0006] The infrared camera is calibrated using a checkerboard calibration plate to obtain the camera's parameters, including intrinsic parameters, extrinsic parameters, and distortion coefficients;

[0007] According to the camera parameters, the camera and IMU are synchronized to obtain the synchronization data of the camera and IMU;

[0008] According to the synchronization data of the camera and the IMU, the image is corrected to obtain a corrected image;

[0009] According to the corrected image, the position of the miner is identified and located, and its size information in the image is obtained to obtain the key data of the image;

[0010] Based on the key data of the image, the actual distance between the miner and the tunnel boring machine is calculated to obtain the actual distance;

[0011] Preset the safety distance threshold and compare the actual distance with the safety distance threshold to achieve early warning of dangerous areas.

[0012] Furthermore, the infrared camera is calibrated using a checkerboard calibration plate to obtain the camera's parameters, including intrinsic parameters, extrinsic parameters, and distortion coefficients, including:

[0013] By making a checkerboard calibration plate, and using an infrared camera to take multiple sets of images of the checkerboard calibration plate at different angles and distances;

[0014] Detect the corner points on the calibration plate in each set of images, and refine the detected corner point positions at the sub-pixel level to obtain the corner point information;

[0015] According to the corner point information, the camera is calibrated to obtain the camera parameters, including intrinsic parameters, extrinsic parameters and distortion coefficients.

[0016] Furthermore, according to the parameters of the camera, the camera and the IMU are synchronized to obtain synchronization data of the camera and the IMU, including:

[0017] Make sure the camera and IMU have the same timestamp source;

[0018] For each camera image, record the timestamp of its capture. For IMU data, record the timestamp of each data point.

[0019] For each camera frame timestamp, calculate the difference between it and all IMU timestamps to get all timestamp differences;

[0020] According to all timestamp differences, find the smallest difference, and obtain the closest IMU data point of each camera frame according to the found minimum time difference to obtain the synchronization data of the camera and IMU.

[0021] Furthermore, the image is corrected according to the synchronization data of the camera and the IMU to obtain a corrected image, including:

[0022] Obtain IMU attitude data based on the synchronization data between the camera and IMU;

[0023] Convert the attitude data (pitch angle, roll angle, yaw angle) provided by IMU into a rotation matrix;

[0024] According to the rotation matrix, through Fe t =T×Fe i Perform a reverse rotation transformation on the image captured by the camera to obtain the rectified image, where Fe i is the video image of the i-th frame, Fet is the video frame after affine transformation compensation, where is the affine transformation matrix for rotation compensation.

[0025] Furthermore, the position of the miner is identified and located based on the corrected image, and its size information in the image is obtained to obtain key data of the image, including:

[0026] Preprocessing the rectified image to obtain a preprocessed image;

[0027] The preprocessed image is subjected to feature extraction using the YOLOv5 backbone network to obtain a feature image, and the feature image is fused through the YOLOv5 neck network to obtain a fused image;

[0028] Based on the fused feature map, bounding box prediction and category probability prediction are performed to obtain the detection result of YOLOv5;

[0029] Extract the same direction coordinates (x1, y1) and (x2, y2) of the target boundary vertex position from the detection results of YOLOv5, and use Calculate the size information of the miner in the image to obtain the key data of the image, where H o Indicates the height of the original image; H c Indicates the height of the current image; W o Indicates the width of the original image; W c Indicates the width of the current image; S cd Indicates the size of the target after correction; S cb Indicates the size of the target before calibration.

[0030] Furthermore, the actual distance between the miner and the tunnel boring machine is calculated based on the key data of the image to obtain the actual distance, including:

[0031] According to the key data of the image and the parameters of the camera, the focal length of the camera and the size information of the miners in the image are obtained, and the average height of the miners is obtained through measurement;

[0032] Based on the focal length and the average height of the miners, the size information of the miners in the image is obtained by Calculate the actual distance between the miner and the tunnel boring machine, where f is the focal length, di is the actual distance between the miner and the tunnel boring machine, w is the average height of the miners, sr is the possible scaling parameter of the image, if the image is not scaled, this value is 1; cor is due to the correction of errors caused by factors such as light and posture, determined by experimental data or historical data, dt wis the pixel width of the target object in the image, p is the physical size of a single pixel of the image sensor, k1 and k2 are the radial distortion coefficients, and r is the distance from the center of the image to the projection point of the miner on the image.

[0033] Furthermore, a safety distance threshold is preset, and the actual distance is compared with the safety distance threshold to achieve early warning of dangerous areas, including:

[0034] Preset safety distance threshold;

[0035] Compare the actual distance with the safety distance threshold to obtain a comparison result, including when the actual distance is less than the safety distance threshold, the personnel enter the dangerous area where the tunnel boring machine is working, the equipment alarms and stops working, otherwise, it works normally;

[0036] Based on the comparison results, early warning of dangerous areas is achieved.

[0037] Second, a dangerous area warning system for roadheaders that integrates IMU and visual features, including:

[0038] The camera calibration module is used to calibrate the infrared camera using a checkerboard calibration plate to obtain the camera's parameters, including intrinsic parameters, extrinsic parameters, and distortion coefficients;

[0039] The image correction module is used to synchronize the camera and the IMU according to the parameters of the camera to obtain synchronization data of the camera and the IMU; and to correct the image according to the synchronization data of the camera and the IMU to obtain a corrected image;

[0040] The personnel detection module is used to identify and locate the miners according to the rectified images, and obtain their size information in the images to obtain the key data of the images;

[0041] The distance calculation module is used to calculate the actual distance between the miner and the tunnel boring machine based on the key data of the image to obtain the actual distance; preset the safety distance threshold, and compare the actual distance with the safety distance threshold to achieve early warning of the dangerous area.

[0042] According to a third aspect, a computing device includes:

[0043] one or more processors;

[0044] The storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method described.

[0045] In a fourth aspect, a computer-readable storage medium stores a program, and when the program is executed by a processor, the method described is implemented.

[0046] The above solution of the present invention includes at least the following beneficial effects:

[0047] The attitude data (rotation matrix or rotation vector) provided by IMU can be used to correct the attitude change of the camera. Correcting the image involves adjusting the position of the object in the image through IMU data to ensure that the image data is consistent with the actual attitude of the camera, further improving the accuracy of distance calculation using the similar triangle method; image correction through the attitude data of the IMU sensor can avoid the problem of difficulty in extracting image feature points using the image feature method; using the monocular vision ranging method fused with IMU, by calculating the distance between the target object and the airborne equipment, and judging according to the set safety threshold, the equipment will stop and alarm when entering the dangerous area, realizing early warning of the dangerous area underground. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of a dangerous area warning method for a tunnel boring machine that integrates IMU and visual features, provided by an embodiment of the present invention.

[0049] Figure 2 It is a schematic diagram of a dangerous area warning system for a tunnel boring machine that integrates IMU and visual features, provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in a form and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0051] like Figure 1 As shown, an embodiment of the present invention proposes a dangerous area warning method for a tunnel boring machine integrating IMU and visual features, and the method comprises the following steps:

[0052] Step 11, calibrating the infrared camera using a checkerboard calibration plate to obtain the camera's parameters, including intrinsic parameters, extrinsic parameters, and distortion coefficients;

[0053] Step 12, synchronizing the camera and the IMU according to the parameters of the camera to obtain synchronization data of the camera and the IMU;

[0054] Step 13, correcting the image according to the synchronization data of the camera and the IMU to obtain a corrected image;

[0055] Step 14, based on the corrected image, identify and locate the position of the miner, and obtain its size information in the image to obtain key data of the image;

[0056] Step 15, calculating the actual distance between the miner and the tunnel boring machine based on the key data of the image to obtain the actual distance;

[0057] Step 16, preset a safety distance threshold, and compare the actual distance with the safety distance threshold to achieve a dangerous area warning.

[0058] In an embodiment of the present invention, the intrinsic parameters, extrinsic parameters and distortion coefficients of the infrared camera can be accurately obtained through calibration, and the synchronous processing of the camera and the IMU can ensure that the timestamps of the image data and the IMU data are consistent, thereby improving the accuracy of data fusion. Through image correction, the distortion in the image can be eliminated, and the image quality can be improved, so that the subsequent identification and positioning of the miner are more accurate; accurately identifying and locating the position of the miner and obtaining its size information in the image are the basis for calculating the actual distance between the miner and the tunnel boring machine. At the same time, these key data are also helpful for further analyzing the working status and environment of the miner; by calculating the actual distance between the miner and the tunnel boring machine, an accurate reference can be provided for the operation of the tunnel boring machine to prevent safety accidents caused by misjudgment of the distance; by presetting a safety distance threshold and comparing the actual distance with it, potential safety hazards can be discovered in time, and the operator can be reminded through the early warning system, thereby effectively preventing miners from entering dangerous areas and improving the safety of mining operations.

[0059] In a preferred embodiment of the present invention, the above step 11 may include:

[0060] By making a checkerboard calibration plate, and using an infrared camera to take multiple sets of images of the checkerboard calibration plate at different angles and distances;

[0061] Detect the corner points on the calibration plate in each set of images, and refine the detected corner point positions at the sub-pixel level to obtain the corner point information;

[0062] According to the corner point information, the camera is calibrated to obtain the camera parameters, including intrinsic parameters, extrinsic parameters and distortion coefficients.

[0063] In an embodiment of the present invention, making a checkerboard calibration plate can provide a stable and easily identifiable reference for camera calibration. By taking images at different angles and distances, the performance of the camera can be comprehensively evaluated, and more comprehensive and accurate calibration data can be obtained, which helps to ensure that the camera can maintain high precision and stability under various working conditions. The accuracy of corner point detection directly affects the accuracy of camera calibration. The accuracy of calibration can be significantly improved by refining the corner point position at the sub-pixel level. Through camera calibration, the precise parameters of the camera can be obtained, including intrinsic parameters, extrinsic parameters and distortion coefficients, which are the basis of tasks such as image processing, three-dimensional reconstruction and measurement. It is crucial to improve the accuracy and stability of the entire system. At the same time, accurate camera parameters also help to optimize the fusion effect of the camera and other sensors such as IMU, and improve the performance of the entire system.

[0064] In an embodiment of the present invention, a checkerboard pattern is created by graphic design software, the checkerboard is composed of black and white squares, forming a grid, the size and number of the checkerboard are determined, which depends on the calibration accuracy you want and the field of view of the camera, and the checkerboard pattern is printed. The printed checkerboard calibration plate is fixed on a flat hard board, such as a wooden board or a plastic board, to ensure its stability. Ensure that the infrared camera is in good working condition, and adjust the focal length and exposure settings of the camera so that the checkerboard calibration plate can be clearly captured at different angles and distances. The checkerboard calibration plate is placed at different angles and distances to ensure that the camera can capture the full view of the calibration plate. For each position, use an infrared camera to take multiple images, at least 10-20 images at different angles and distances, and the lighting conditions are kept consistent during the shooting process; use the cv2.findChessboardCorners function in OpenCV to detect the corners on the calibration plate in each group of images, and adjust the parameters of the corner detection algorithm to ensure that each corner of the checkerboard can be accurately identified. For each detected corner point, the cv2.cornerSubPix function is used to refine the detected corner point at the sub-pixel level to improve the accuracy of the corner point position.

[0065] In another embodiment of the present invention, all sub-pixel refined corner point position information is organized into a data format required for calibration, including the coordinates of each corner point in the image and the corresponding world coordinates, and the world coordinates can be determined according to the actual size of the checkerboard calibration plate and the position during shooting. The cv2.calibrateCamera function is used to process the calibration data to obtain the camera's intrinsic parameters (such as focal length, principal point coordinates, etc.), extrinsic parameters (such as rotation matrix and translation vector) and distortion coefficients (such as radial distortion and tangential distortion coefficients).

[0066] In a preferred embodiment of the present invention, the above step 12 may include:

[0067] Make sure the camera and IMU have the same timestamp source;

[0068] For each camera image, record the timestamp of its capture. For IMU data, record the timestamp of each data point.

[0069] For each camera frame timestamp, calculate the difference between it and all IMU timestamps to get all timestamp differences;

[0070] According to all timestamp differences, find the smallest difference, and obtain the closest IMU data point of each camera frame according to the found minimum time difference to obtain the synchronization data of the camera and IMU.

[0071] In the embodiment of the present invention, ensuring the consistency and accuracy of timestamps is the basis of data synchronization. By determining the same timestamp source, the time deviation caused by the difference in clocks of different devices can be eliminated; accurately recording the timestamp of each data point is the key to achieving data synchronization, which can ensure that the camera image and IMU data can be accurately matched in subsequent processing, thereby obtaining accurate synchronized data; by calculating the timestamp difference, the time deviation between the camera image and the IMU data can be quantified, which helps to find the closest matching point and improve the accuracy of data synchronization; by finding the minimum time difference, the high temporal consistency of the camera image and the IMU data can be ensured. At the same time, accurate synchronized data also helps to reduce errors and uncertainties caused by time deviations, and improve the robustness and stability of the system.

[0072] In an embodiment of the present invention, if the camera and IMU are integrated into one device and have been synchronized by hardware, the device will provide a common timestamp source for marking the capture time of the camera image and IMU data. Or use the computer's system time as a reference, and assume that the drivers of the camera and IMU can accurately report the time of data capture. When the camera captures a frame of image, its driver or SDK usually provides a timestamp associated with the frame. This timestamp indicates the exact moment when the image was captured. Ensure that this timestamp is stored in a format compatible with the IMU data, and ensure that the timestamp of the IMU data is compatible with the timestamp of the camera image in format and clock source. For the timestamp of each camera frame, traverse the timestamps of all IMU data points and calculate the difference (absolute value) between them. This difference represents the temporal proximity of the camera frame and the IMU data point. A smaller time difference means that the camera frame and the IMU data point are captured at a similar time.

[0073] In another embodiment of the present invention, after calculating all possible timestamp differences, the smallest difference is found, and the IMU data point corresponding to the smallest difference is the data point closest in time to the current camera frame. In this way, an IMU data point synchronized with each camera frame can be found, thereby obtaining a synchronized data set of the camera and the IMU.

[0074] In a preferred embodiment of the present invention, the above step 13 may include:

[0075] Obtain IMU attitude data based on the synchronization data between the camera and IMU;

[0076] Convert the attitude data (pitch angle, roll angle, yaw angle) provided by IMU into a rotation matrix;

[0077] According to the rotation matrix, through Fe t =T×Fe i Perform a reverse rotation transformation on the image captured by the camera to obtain the rectified image, where Fe i is the video image of the i-th frame, Fe t is the video frame after affine transformation compensation, where is the affine transformation matrix for rotation compensation.

[0078] In an embodiment of the present invention, the IMU attitude data provides information about the direction of the camera in space. The attitude data obtained by synchronization data can ensure temporal consistency with the camera image. The rotation matrix is ​​a convenient way to describe the rotation of an object in three-dimensional space. The attitude data of the IMU is converted into a rotation matrix, which can be conveniently applied to image correction. The precise rotation transformation of the image is achieved through mathematical operations. Image correction can eliminate image distortion and deformation caused by changes in the camera attitude and restore the real scene of the image. Through the reverse rotation transformation, the image can be accurately corrected according to the attitude data of the IMU to improve the quality and usability of the image. The corrected image more accurately reflects the actual scene, which is helpful for subsequent tasks such as target detection, positioning and analysis. By applying the affine transformation matrix of rotation compensation, each frame of video image can be accurately corrected to keep it consistent with the actual attitude of the camera.

[0079] In an embodiment of the present invention, these posture data are extracted from a synchronized data set. Usually, these data are given in the form of Euler angles, namely pitch angle, roll angle and yaw angle, and the rotation matrix R is a 3x3 matrix used to describe the rotation in three-dimensional space. When converting the Euler angles to the rotation matrix, according to the posture representation of the IMU, follow a specific rotation order (such as ZYX or XYZ, etc.), use a library function (such as the cv2.Rodrigues function in OpenCV, but it is used for the conversion between rotation vectors and rotation matrices, or manually implement the conversion from Euler angles to rotation matrices), convert the pitch angle, roll angle and yaw angle into the rotation matrix R, and construct an affine transformation matrix T based on the rotation matrix R. The affine transformation matrix T is a 4x4 matrix used to describe rotations and translations in three-dimensional space; Where R is a 3x3 rotation matrix, 0 is a 3x1 zero vector, and 1 is a scalar 1. For each camera frame, perform the inverse rotation transformation by multiplying the image coordinates by the inverse matrix of T. Use an image processing library (such as OpenCV) to apply the affine transformation matrix T to the image Fe using the cv2.warpPerspective function. i .

[0080] In a preferred embodiment of the present invention, the above step 14 may include:

[0081] Preprocessing the rectified image to obtain a preprocessed image;

[0082] The preprocessed image is subjected to feature extraction using the YOLOv5 backbone network to obtain a feature image, and the feature image is fused through the YOLOv5 neck network to obtain a fused image;

[0083] Based on the fused feature map, bounding box prediction and category probability prediction are performed to obtain the detection result of YOLOv5;

[0084] Extract the same direction coordinates (x1, y1) and (x2, y2) of the target boundary vertex position from the detection results of YOLOv5, and use Calculate the size information of the miner in the image to obtain the key data of the image, where H o Indicates the height of the original image; H c Indicates the height of the current image; W o Indicates the width of the original image; W c Indicates the width of the current image; S cd Indicates the size of the target after correction; S cb Indicates the size of the target before calibration.

[0085] In the embodiment of the present invention, preprocessing can enhance the useful information in the image, suppress irrelevant information and noise, thereby improving the quality of the image, facilitating subsequent image analysis and target detection, making feature extraction more accurate, and improving the sensitivity and accuracy of detection; the backbone network of YOLOv5 can effectively extract key features from the preprocessed image. The neck network can fuse the features of different levels extracted by the backbone network to obtain a richer and more comprehensive feature representation, which helps the model to better understand and identify the target in the image and improve the accuracy and robustness of detection. Through bounding box prediction, the position of the target object in the image can be accurately located; and the category probability prediction can give the confidence that the target object belongs to a certain category. These two prediction steps together constitute the detection result of YOLOv5. Accurately extracting the size information of the miner in the image is the basis for the subsequent calculation of the actual distance between the miner and the tunneling machine. The key data of the image can be obtained, and then these data can be used to deduce the actual distance, providing important support for the safety warning of mining operations.

[0086] In an embodiment of the present invention, the rectified image is normalized, usually by scaling the pixel value to between 0 and 1 (if the original pixel value is 0-255), and the image is adjusted to a specific size according to the input requirements of the YOLOv5 model, and the pre-processed image is input into the backbone network of YOLOv5 for feature extraction, and the extracted feature map is further fused through the neck network of YOLOv5 (such as the PANet structure). In YOLOv5, a set of anchor boxes are pre-defined, and these anchor boxes serve as references for bounding box prediction; based on the fused feature map, YOLOv5 predicts the offset of each anchor box relative to the real target object, thereby obtaining the position information of the bounding box, and at the same time, YOLOv5 also predicts the category probability of each bounding box, and the prediction result of YOLOv5 usually includes the position information of the bounding box and the corresponding category probability, and these detection results are parsed to extract relevant information. From the parsed bounding box coordinates, by calculating the width and height of the bounding box, the size of the target can be obtained, and the size information of the target object in the image can be calculated.

[0087] In a preferred embodiment of the present invention, the above step 15 may include:

[0088] According to the key data of the image and the parameters of the camera, the focal length of the camera and the size information of the miners in the image are obtained, and the average height of the miners is obtained through measurement;

[0089] Based on the focal length and the average height of the miners, the size information of the miners in the image is obtained by Calculate the actual distance between the miner and the tunnel boring machine, where f is the focal length, di is the actual distance between the miner and the tunnel boring machine, w is the average height of the miners, sr is the possible scaling parameter of the image, if the image is not scaled, this value is 1; cor is due to the correction of errors caused by factors such as light and posture, determined by experimental data or historical data, dt w is the pixel width of the target object in the image, p is the physical size of a single pixel of the image sensor, k1 and k2 are the radial distortion coefficients, and r is the distance from the center of the image to the projection point of the miner on the image.

[0090] In an embodiment of the present invention, the focal length of the camera and the size information of the miners in the image are obtained, and the average height of the miners is obtained by measurement. These data are the basis for calculating the actual distance, ensuring the accuracy and reliability of subsequent calculations. The focal length is an important parameter of the camera, which determines the imaging effect of the image; and the size information and average height of the miners in the image are the key to converting the image data into actual space data. Through this calculation method, the actual distance between the miner and the tunneling machine can be accurately obtained, which is crucial to ensure the safety of mining operations. Accurate distance information can help operators make judgments and adjustments in time to avoid possible safety accidents, considering possible scaling parameters of the image, correcting errors caused by factors such as light and posture, radial distortion coefficients, and the distance from the center of the image to the projection point of the miner on the image. These factors will affect the imaging effect of the image, and then affect the accuracy of the distance calculation. By considering these factors and making corresponding corrections, the accuracy and robustness of the distance calculation can be improved, so that the system can maintain high performance under various conditions. Specifically, the consideration of scaling parameters ensures the correspondence between the image size and the actual size; the light and posture correction reduces the impact of the external environment on the image; the consideration of the radial distortion coefficient eliminates the influence of lens distortion on the image; and the distance from the image center to the miner's projection point reflects the miner's specific position in the image, which helps to calculate the actual distance more accurately.

[0091] In an embodiment of the present invention, the parameters of the camera are obtained, including the focal length, the physical size of a single pixel of the image sensor, etc. From the bounding box information extracted from the previous YOLOv5 detection result, the size information of the miner in the image, such as the pixel width and height of the miner, can be obtained. In the actual scene, the height of multiple miners is measured and the average value is calculated to obtain the average height of the miners. In addition to the previously obtained camera focal length, the size information of the miner in the image and the average height of the miners, it is also necessary to consider the possible scaling parameters of the image (if the image is not scaled, this value is 1), the error correction factor (determined by experimental data or historical data), the radial distortion coefficient (if the camera is distorted), and the distance from the center of the image to the projection point of the miner on the image; the camera imaging principle is based on the pinhole imaging model, and the principle of similar triangles can be used to calculate the actual distance. The ratio between the average height of the miner and the pixel height of the miner in the image is equal to the ratio between the focal length of the camera and the actual distance between the miner and the camera (tunneling machine). If the image has been scaled, the scaling parameters need to be applied to the calculation. At the same time, in order to correct the errors caused by factors such as light and posture, a correction factor determined by experiments or historical data needs to be multiplied. Combining all the above parameters and principles, a mathematical equation can be established to calculate the actual distance between the miner and the tunnel boring machine. If the camera has obvious distortion (such as radial distortion), the distortion coefficient of the camera is used to further correct the calculation process.

[0092] In a preferred embodiment of the present invention, the above step 16 may include:

[0093] Preset safety distance threshold;

[0094] Compare the actual distance with the safety distance threshold to obtain a comparison result, including when the actual distance is less than the safety distance threshold, the personnel enter the dangerous area where the tunnel boring machine is working, the equipment alarms and stops working, otherwise, it works normally;

[0095] Based on the comparison results, early warning of dangerous areas is achieved.

[0096] In an embodiment of the present invention, the preset safety distance threshold can clearly define the boundary between the safe area and the dangerous area, which helps to identify potential safety risks in a timely manner. By comparing the actual distance with the safety distance threshold, it can be quickly determined whether the miner has entered the dangerous area. This instant comparison mechanism helps to discover safety hazards in a timely manner and provide key information for taking emergency measures. Once the miner is detected to enter the dangerous area, the system immediately triggers an alarm and stops the work of the tunnel boring machine. The rapid response mechanism can effectively prevent safety accidents caused by people mistakenly entering the dangerous area and protect the lives of miners. When the actual distance is greater than or equal to the safety distance threshold, it means that the miner is in a safe area and the tunnel boring machine can work normally. This intelligent working mode ensures both work efficiency and work safety. By realizing the early warning of the dangerous area, the system can issue a warning in time when the miner approaches the dangerous area, reminding the operator to pay attention to safety, which helps to improve the safety awareness of the operators, reduce safety accidents caused by negligence, and create a safer working environment for mining operations.

[0097] In the embodiment of the present invention, according to the historical data, the safety distance threshold is preset, and the determined safety distance is converted into a specific value, namely, the safety distance threshold. Through the calculation process of step 15, the actual distance between the miner and the tunnel boring machine is obtained, and the actual distance obtained is compared with the safety distance threshold. If the actual distance is less than the safety distance threshold, it means that the miner has entered the dangerous area where the tunnel boring machine works. When it is judged to be a dangerous situation, the system should immediately trigger an alarm signal. This alarm signal can be a sound, light or other form of prompts, aiming to attract the attention of the on-site personnel. At the same time, the tunnel boring machine should receive a stop working instruction to ensure that no operation that may cause harm is performed before the danger is eliminated. When the alarm is triggered, the system should also record the event and notify the relevant management personnel. This can help the management personnel to understand the on-site situation in a timely manner and take necessary measures to solve the problem and prevent similar incidents from happening again. If the actual distance is greater than or equal to the safety distance threshold, it means that the miner is in a safe area. In this case, the tunnel boring machine can work normally without triggering any alarm or stop instruction. The system should continuously monitor the distance change to ensure that the safety state is maintained.

[0098] like Figure 2 As shown, an embodiment of the present invention further provides a dangerous area warning system 20 for a roadheader integrating IMU and visual features, comprising:

[0099] The camera calibration module 21 is used to calibrate the infrared camera using a checkerboard calibration plate to obtain the camera's parameters, including intrinsic parameters, extrinsic parameters, and distortion coefficients;

[0100] The image correction module 22 is used to synchronize the camera and the IMU according to the parameters of the camera to obtain synchronization data of the camera and the IMU; and to correct the image according to the synchronization data of the camera and the IMU to obtain a corrected image;

[0101] The personnel detection module 23 is used to identify and locate the position of the miner according to the corrected image, and obtain the size information of the miner in the image to obtain the key data of the image;

[0102] The distance calculation module 24 is used to calculate the actual distance between the miner and the tunnel boring machine based on the key data of the image to obtain the actual distance; preset a safety distance threshold, and compare the actual distance with the safety distance threshold to achieve early warning of the dangerous area.

[0103] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A dangerous area warning method for a roadheader integrating IMU and visual features, characterized in that: The method comprises: The infrared camera is calibrated using a checkerboard calibration plate to obtain the camera's parameters, including intrinsic parameters, extrinsic parameters, and distortion coefficients; According to the camera parameters, the camera and IMU are synchronized to obtain the synchronization data of the camera and IMU; According to the synchronization data of the camera and the IMU, the image is corrected to obtain a corrected image; According to the corrected image, the position of the miner is identified and located, and its size information in the image is obtained to obtain the key data of the image; Based on the key data of the image, the actual distance between the miner and the tunnel boring machine is calculated to obtain the actual distance; Preset the safety distance threshold and compare the actual distance with the safety distance threshold to achieve early warning of dangerous areas.

2. The method for early warning of dangerous areas of a tunnel boring machine by integrating IMU and visual features according to claim 1 is characterized in that: Use the checkerboard calibration plate to calibrate the infrared camera to obtain the camera parameters, including intrinsic parameters, extrinsic parameters and distortion coefficients, including: By making a checkerboard calibration plate, and using an infrared camera to take multiple sets of images of the checkerboard calibration plate at different angles and distances; Detect the corner points on the calibration plate in each set of images, and refine the detected corner point positions at the sub-pixel level to obtain the corner point information; According to the corner point information, the camera is calibrated to obtain the camera parameters, including intrinsic parameters, extrinsic parameters and distortion coefficients.

3. The method for early warning of dangerous areas of a tunnel boring machine by integrating IMU and visual features according to claim 2 is characterized in that: According to the camera parameters, the camera and IMU are synchronized to obtain the synchronization data of the camera and IMU, including: Make sure the camera and IMU have the same timestamp source; For each camera image, record the timestamp of its capture. For IMU data, record the timestamp of each data point. For each camera frame timestamp, calculate the difference between it and all IMU timestamps to get all timestamp differences; According to all timestamp differences, find the smallest difference, and obtain the closest IMU data point of each camera frame according to the found minimum time difference to obtain the synchronization data of the camera and IMU.

4. The method for early warning of dangerous areas of a roadheader integrating IMU and visual features according to claim 3 is characterized in that: According to the synchronization data of the camera and IMU, the image is corrected to obtain a corrected image, including: Obtain IMU attitude data based on the synchronization data between the camera and IMU; Convert the attitude data provided by IMU into a rotation matrix. The attitude data includes pitch angle, roll angle and yaw angle. According to the rotation matrix, through Fe t =T×Fe i Perform a reverse rotation transformation on the image captured by the camera to obtain the rectified image, where Fe i is the video image of the i-th frame, Fe t is the video frame after affine transformation compensation, where is the affine transformation matrix for rotation compensation.

5. The method for early warning of dangerous areas of a roadheader integrating IMU and visual features according to claim 4 is characterized in that: According to the corrected image, the miner’s position is identified and located, and the size information of the miner in the image is obtained to obtain the key data of the image, including: Preprocessing the rectified image to obtain a preprocessed image; The preprocessed image is subjected to feature extraction using the YOLOv5 backbone network to obtain a feature image, and the feature image is fused through the YOLOv5 neck network to obtain a fused image; Based on the fused feature map, bounding box prediction and category probability prediction are performed to obtain the detection result of YOLOv5; Extract the same direction coordinates (x1, y1) and (x2, y2) of the target boundary vertex position from the detection results of YOLOv5, and use Calculate the size information of the miner in the image to obtain the key data of the image, where H o Indicates the height of the original image; H c Indicates the height of the current image; W o Indicates the width of the original image; W c Indicates the width of the current image; S cd Indicates the size of the target after correction; S cb Indicates the size of the target before calibration.

6. The method for early warning of dangerous areas of a roadheader integrating IMU and visual features according to claim 5 is characterized in that: Based on the key data of the image, the actual distance between the miner and the tunnel boring machine is calculated to obtain the actual distance, including: According to the key data of the image and the parameters of the camera, the focal length of the camera and the size information of the miners in the image are obtained, and the average height of the miners is obtained through measurement; Based on the focal length and the average height of the miners, the size information of the miners in the image is obtained by Calculate the actual distance between the miner and the tunnel boring machine, where f is the focal length, di is the actual distance between the miner and the tunnel boring machine, w is the average height of the miners, sr is the possible scaling parameter of the image, if the image is not scaled, this value is 1; cor is due to the correction of errors caused by factors such as light and posture, determined by experimental data or historical data, dt w is the pixel width of the target object in the image, p is the physical size of a single pixel of the image sensor, k1 and k2 are the radial distortion coefficients, and r is the distance from the center of the image to the projection point of the miner on the image.

7. The method for early warning of dangerous areas of a roadheader integrating IMU and visual features according to claim 6 is characterized in that: Preset the safety distance threshold and compare the actual distance with the safety distance threshold to achieve early warning of dangerous areas, including: Preset safety distance threshold; Compare the actual distance with the safety distance threshold to obtain a comparison result, including when the actual distance is less than the safety distance threshold, the personnel enter the dangerous area where the tunnel boring machine is working, the equipment alarms and stops working, otherwise, it works normally; Based on the comparison results, early warning of dangerous areas is achieved.

8. A dangerous area warning system for a roadheader integrating IMU and visual features, characterized in that: include: The camera calibration module is used to calibrate the infrared camera using a checkerboard calibration plate to obtain the camera's parameters, including intrinsic parameters, extrinsic parameters, and distortion coefficients; The image correction module is used to synchronize the camera and the IMU according to the parameters of the camera to obtain synchronization data of the camera and the IMU; and to correct the image according to the synchronization data of the camera and the IMU to obtain a corrected image; The personnel detection module is used to identify and locate the miners according to the rectified images, and obtain their size information in the images to obtain the key data of the images; The distance calculation module is used to calculate the actual distance between the miner and the tunnel boring machine based on the key data of the image to obtain the actual distance; Preset the safety distance threshold and compare the actual distance with the safety distance threshold to achieve early warning of dangerous areas.

9. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images