Method for detecting and measuring distance of personnel in front of underground unmanned electric locomotive

Abstract

The invention discloses a method for detecting and measuring the distance of a person in front of an underground unmanned electric locomotive. The method comprises the following steps of: S1, synchronizing data time; s2, performing camera external parameter calibration through a multi-modal calibration device; s3, generating multi-modal data; s4, performing personnel detection on the multi-modal data; s5, track area segmentation; and S6, determining a longitudinal distance and a transverse distance. According to the method, the track line region can be segmented more accurately by fusing visible light and thermal imaging image information, so that the accuracy of distance measurement is improved. Through remote transmission of visible light and thermal imaging images and distance measurement data, a manager can know the condition in front of the unmanned electric locomotive in real time in a ground monitoring center and handle abnormal conditions in time, remote monitoring and management of underground operation are achieved, and the workload and risk of manual inspection are reduced.

Description

Technical Field

[0001] This invention relates to the field of unmanned underground locomotives, and more particularly to a method for detecting and measuring the distance to personnel in front of an unmanned underground locomotive. Background Technology

[0002] Mining locomotives are used for transporting ore between the mining area and the unloading station. The underground mining environment is harsh, so unmanned driving technology is often used to achieve automatic driving between these two points. During loading, unloading, and automatic operation of these unmanned locomotives, factors such as non-standard human operations, sudden personnel movement, and poor visibility can seriously affect their operational safety.

[0003] In the field of autonomous driving, the mainstream perception sensors include LiDAR, millimeter-wave radar, and visible light cameras. LiDAR obtains distance information of objects by emitting laser beams and measuring the time it takes for the reflected light to travel. It can accurately construct a 3D point cloud map of the surrounding environment, providing accurate information on object position, shape, and contour, and has high accuracy for tasks such as obstacle detection. However, in environments with high dust and moisture content, the sensor's performance is significantly affected, and the large amount of data is difficult to process at the device. Millimeter-wave radar uses electromagnetic waves in the millimeter-wave band for detection, possessing strong penetrating power and operating normally in adverse weather conditions such as rain, fog, snow, and sandstorms, with less susceptibility to environmental influences. However, it lacks texture and color information, has lower resolution compared to LiDAR, limited angle measurement accuracy, is sensitive to moving targets and metal, and is easily lost in background noise when people are stationary. Visible light cameras can capture rich visual details of the vehicle's surrounding environment, including surrounding colors, pedestrian appearances, and movements. This information is crucial for recognizing and understanding complex traffic scenes, helping autonomous driving systems make accurate decisions. However, monocular cameras lose depth information during imaging and cannot obtain distance information of targets in front. Furthermore, the distance estimation accuracy of monocular cameras is low and cannot meet practical requirements.

[0004] In the actual operating environment of unmanned locomotives in mines, lidar will fail in scenarios such as dust, water seepage, and fog. The complex distribution of metal objects such as anchor bolts, rails, and brakes around the tunnels results in noisy background noise for millimeter-wave radar, making it difficult to distinguish targets. In backlit or dimly lit sections of the tunnel, the detection capability of visible light cameras is significantly reduced. To ensure the safety of workers and the normal operation of the unmanned locomotives, high demands are placed on personnel detection and ranging; the perception capability of a single sensor is no longer sufficient to meet the operational needs of unmanned locomotives in mines. Summary of the Invention

[0005] The present invention aims to address the shortcomings of the prior art by providing a method for detecting and measuring the distance to personnel in front of an unmanned electric locomotive in a mine.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for detecting and measuring the distance to personnel in front of an unmanned electric locomotive in an underground mine, comprising the following steps:

[0008] S1. Data time synchronization: The visible light camera and thermal imaging camera are used with multi-threading technology to store the latest data into the buffer queue in real time. The algorithm reads the latest data pair in the queue to achieve time synchronization.

[0009] S2. Camera extrinsic parameter calibration using a multimodal calibration device: Turn on the power of the multimodal calibration device, collect image information from multiple different distances and angles, apply a self-optimizing calibration algorithm, calculate the homography transformation relationship from thermal imaging image to visible light image, and complete the calibration;

[0010] S3. Multimodal data generation: Multimodal data is generated through extrinsic parameter matrix transformation;

[0011] S4. Person detection on multimodal data: Using a deep learning-based object detection algorithm, personnel detection is performed on multimodal data to obtain the coordinate information of the personnel;

[0012] S5. Track Region Segmentation: Track region segmentation is performed on multimodal data using a deep learning-based instance segmentation algorithm;

[0013] S6. Determine longitudinal and lateral distances: Determine the longitudinal and lateral distances of the person in front by using a fusion ranging algorithm.

[0014] In step S3, the multimodal data is generated by converting the thermal imaging image into a light-emitting image using an extrinsic parameter matrix, and then cropping it to a fixed size.

[0015] The operation steps of the fusion ranging algorithm in step S6 are as follows:

[0016] P1. Target detection in multimodal images, outputting corresponding detection information, and obtaining the y-coordinate of the bottom position of the person in the image based on the detection box information;

[0017] P2. Semantic segmentation of multimodal images, outputting segmentation information of multiple tracks, and obtaining the track on which the locomotive is currently running based on the positional relationship of the segmented tracks;

[0018] P3. Based on the current track region information, use a region growth algorithm to intelligently grow the top of the current track region to expand the longitudinal region of the track segmentation;

[0019] P4. Based on the current track area information, the skeleton of the track line segmentation area is extracted, and polynomial curve fitting is performed to obtain the curvature of the track line at each position.

[0020] P5. Based on the personnel's coordinate information, track width, pixel width of the track area, and the curvature of the track at the personnel's coordinate point, obtain the longitudinal distance between the personnel and the locomotive and the lateral distance between the personnel and the track edge.

[0021] In step S2, data acquisition begins 5 minutes after the power to the multimodal calibration device is turned on. A visible light camera and a thermal imaging camera are used to capture images of the heated calibration plate. The calibration plate remains stationary at each position for more than 10 seconds.

[0022] In step S2, a multi-line fitting algorithm is applied to the acquired distance and angle image information to detect straight lines in visible light and thermal imaging, thereby obtaining corner points and forming multiple corner point pairs.

[0023] In step S2, the camera extrinsic parameter calibration of the multimodal calibration device only needs to be performed once, and there is no need to repeat the calibration.

[0024] The beneficial effects of this invention are as follows: By fusing visible light and thermal imaging information, this invention can more accurately segment the track line area, thereby improving the accuracy of distance measurement. Through remote transmission of visible light and thermal imaging images and distance measurement data, management personnel can monitor the situation in front of the unmanned locomotive in real time at the ground monitoring center, promptly handle any abnormalities, and achieve remote monitoring and management of underground operations, reducing the workload and risks of manual inspections. Attached Figure Description

[0025] Figure 1 This is a flowchart of the present invention.

[0026] The following will describe in detail, with reference to the accompanying drawings, embodiments of the invention. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0028] like Figure 1 As shown, a method for detecting and measuring the distance to personnel in front of an unmanned underground locomotive is described, and its steps are as follows:

[0029] S1. Data Time Synchronization: Multi-threading technology is applied to the visible light camera and thermal imaging camera to store the latest data in a buffer queue in real time. The algorithm reads the latest data pair from the queue to achieve time synchronization. Multi-threading technology is also used for soft synchronization of data. Image data from both modes of cameras is acquired in real time, stamped with a system timestamp, and a queue data structure is created to cache multiple pairs of data from both modes. Subsequent processing programs read the latest data pair from the queue to complete the time synchronization of visible light image data and thermal imaging image data.

[0030] S2. Camera extrinsic parameter calibration using a multimodal calibration device: Turn on the power to the multimodal calibration device, and start data acquisition 5 minutes after power-on. Use a visible light camera and a thermal imaging camera to acquire images of the heated calibration plate. Keep the calibration plate stationary at each position for more than 10 seconds, acquiring image information from multiple distances and angles. Apply a multi-line fitting algorithm to detect straight lines in the visible light and thermal images respectively, thereby obtaining corner points and forming multiple corner point pairs. Apply a self-optimizing calibration algorithm to calculate the homography transformation relationship from the thermal imaging image to the visible light image, completing the calibration. Once each device is calibrated once, no repeated calibration is required.

[0031] S3. Multimodal data generation: Through external parameter matrix transformation, the thermal imaging image is converted into a visible light image, and then cropped to a fixed size to generate multimodal data;

[0032] S4. Person detection on multimodal data: Person detection is performed on multimodal data using a deep learning-based object detection algorithm. The detection box information is used as a parameter for subsequent use to obtain the coordinate information of the person, that is, the y coordinate of the bottom position of the person in the image.

[0033] S5. Track Region Segmentation: Track region segmentation is performed on multimodal data using a deep learning-based instance segmentation algorithm; semantic segmentation of multimodal images outputs segmentation information for multiple tracks; based on the positional relationship of the segmented tracks, the track on which the locomotive is currently running is obtained.

[0034] S6. Determine longitudinal and lateral distances: Determine the longitudinal and lateral distances of the person in front by using a fusion ranging algorithm; Based on the current track area information, intelligently grow the top of the current track area using a region growing algorithm to expand the longitudinal area of ​​the track segmentation;

[0035] Based on the current track area information, the skeleton of the segmented track line area is extracted and polynomial curve fitting is performed to obtain the curvature of the track line at each position.

[0036] Based on the coordinates of the personnel, the track width, the pixel width of the track area, and the curvature of the track at the personnel's coordinates, the longitudinal distance of the personnel from the locomotive and the lateral distance from the edge of the track are obtained.

[0037] This method can output the longitudinal distance of personnel in front of the locomotive and the lateral distance from personnel to the track line, assisting in the safe operation of the unmanned locomotive and preventing accidents. Visible light cameras and thermal imaging cameras realize data perception of the scene in front and return the data to the controller. The controller implements data fusion and algorithm implementation of the two modes and communicates with the locomotive's main controller through the network port.

[0038] The invention has been described above with reference to the accompanying drawings. Obviously, the specific implementation of the invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution, or direct application to other situations without modification, are all within the scope of protection of the invention.

Claims

1. A method for detecting and measuring the distance to personnel in front of an unmanned electric locomotive in a mine, characterized in that, The steps are as follows: S1. Data time synchronization: The visible light camera and thermal imaging camera are used with multi-threading technology to store the latest data into the buffer queue in real time. The algorithm reads the latest data pair in the queue to achieve time synchronization. S2. Camera extrinsic parameter calibration using a multimodal calibration device: Turn on the power of the multimodal calibration device, collect image information from multiple different distances and angles, apply a self-optimizing calibration algorithm, calculate the homography transformation relationship from thermal imaging image to visible light image, and complete the calibration; S3. Multimodal data generation: Multimodal data is generated through extrinsic parameter matrix transformation; S4. Person detection on multimodal data: Using a deep learning-based object detection algorithm, personnel detection is performed on multimodal data to obtain the coordinate information of the personnel; S5. Track Region Segmentation: Track region segmentation is performed on multimodal data using a deep learning-based instance segmentation algorithm; S6. Determine longitudinal and lateral distances: Determine the longitudinal and lateral distances of the person in front by using a fusion ranging algorithm; In step S3, the multimodal data is generated by converting the thermal imaging image into a light-emitting image through an external parameter matrix, and then cropping it to a fixed size. In step S2, data acquisition begins 5 minutes after the power to the multimodal calibration device is turned on. A visible light camera and a thermal imaging camera are used to capture the heated calibration plate. The calibration plate is kept stationary at each position for more than 10 seconds. In step S2, a multi-line fitting algorithm is applied to the acquired distance and angle image information to detect straight lines in visible light and thermal imaging, thereby obtaining corner points and forming multiple corner point pairs.

2. The method for detecting and measuring personnel in front of an unmanned underground locomotive according to claim 1, characterized in that, The operation steps of the fusion ranging algorithm in step S6 are as follows: P1. Target detection in multimodal images, outputting corresponding detection information, and obtaining the y-coordinate of the bottom position of the person in the image based on the detection box information; P2. Semantic segmentation of multimodal images, outputting segmentation information of multiple tracks, and obtaining the track on which the locomotive is currently running based on the positional relationship of the segmented tracks; P3. Based on the current track region information, use a region growth algorithm to intelligently grow the top of the current track region to expand the longitudinal region of the track segmentation; P4. Based on the current track area information, the skeleton of the track line segmentation area is extracted, and polynomial curve fitting is performed to obtain the curvature of the track line at each position. P5. Based on the personnel's coordinate information, track width, pixel width of the track area, and the curvature of the track at the personnel's coordinate point, obtain the longitudinal distance between the personnel and the locomotive and the lateral distance between the personnel and the track edge.

3. The method for detecting and measuring the distance to personnel in front of an unmanned underground locomotive as described in claim 1, characterized in that, In step S2, the camera extrinsic parameter calibration of the multimodal calibration device only needs to be performed once, and there is no need to repeat the calibration.

Images