A blast hole target recognition and guidance method and system for a hole protection robot

Abstract

The application discloses a blast hole target recognition and guidance method and system of a blast hole robot, and relates to the technical field of intelligent mine operation. The blast hole is recognized and the position is solved through deep learning target detection, and the advancing posture is adjusted in real time; the number of visual target losses is counted in real time, and if the number of missed detections does not reach a threshold value, the blast hole is continuously guided and segmented to advance, and if the threshold value is reached, the radar positioning mode is switched; a two-dimensional laser radar is started to collect point clouds, the blast hole is recognized through ground recess features, and a near-distance offset is solved; the visual position and the radar offset data are respectively subjected to median filtering and smoothing processing, and the turning and step distance are adjusted in stages according to the position information after filtering; the above-mentioned sensing, positioning and control processes are circularly executed until the robot reaches a blast hole operation position. The application adopts camera and laser radar fusion sensing, sets a blind area transition mechanism to avoid target loss, filters and suppresses environmental detection disturbance, and realizes stable and reliable positioning through cooperation of far and near fields, thereby improving the safety and automation level of mine blast hole cleaning operation.

Description

Technical Field

[0001] This application relates to the field of intelligent mining operation technology, and more specifically, to a method and system for identifying and guiding borehole targets for a borehole protection robot. Background Technology

[0002] In the blasting process of mining, before the loading and sealing operations are carried out, loose soil, gravel, and debris easily accumulate at the blast hole openings, clogging them. The openings must be cleared before subsequent blasting preparations can begin. Currently, this clearing operation generally relies on manual entry into the blasting preparation area. However, the blasting area is a high-risk work site, with significant safety risks such as falling rocks, slope collapses, and sudden blasts. Manual operation poses extremely high personal safety hazards.

[0003] To mitigate the risks of manual operations, the industry has gradually introduced borehole protection robots adapted to mining scenarios. These robots are suitable for the uneven and complex terrain of mining areas, equipped with high-clearance chassis to achieve autonomous movement, and rely on integrated sensing devices, supporting navigation and control algorithms to complete autonomous identification and approach to boreholes. However, existing borehole protection robot perception and guidance solutions still have significant shortcomings: First, they mostly use a single high-precision LiDAR or high-end industrial vision equipment to complete full-range perception, resulting in high procurement and deployment costs for the entire set of sensing hardware, making it difficult to achieve cost reduction and efficiency improvement in large-scale deployment in mines; Second, the perception range of a single sensor is limited, long-range LiDAR point clouds are sparse and have poor recognition accuracy, and close-range cameras are prone to exceeding the field of view and losing targets. There are blind spots in the perception range of both, and the robot is prone to losing the borehole target in the blind spots, making it impossible to smoothly connect long-range coarse guidance and close-range fine positioning; Third, the lighting in mines is variable and there is a lot of interference from gravel and soil. Existing perception algorithms lack multi-frame data smoothing and noise reduction mechanisms, resulting in large fluctuations in single detections, causing the robot to frequently correct its course, resulting in poor travel stability and seriously affecting the efficiency of borehole alignment.

[0004] In summary, existing hole-clearing robot hole identification and guidance solutions are insufficient in balancing cost, environmental adaptability, and seamless multi-sensor collaboration, making them unsuitable for large-scale autonomous hole cleaning operations in mines. There is an urgent need for a low-cost hole target identification and guidance solution with blind zone transition strategies and stable perception data. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for identifying and guiding borehole targets on a borehole protection robot. To achieve environmental perception and localization for borehole target identification and guidance, the system is equipped with a planar camera and a two-dimensional lidar as the main sensors. The camera is used to acquire image information of the work area and perform target identification, while the lidar is used to acquire target distance information and assist in target localization.

[0006] It should be noted that both sensors in this application are mounted on the front of the robot's body, with their mounting plane perpendicular to the robot's forward direction and perpendicular to the ground. During system operation, the camera primarily undertakes the initial target identification and direction guidance tasks, adjusting the movement direction of the mobile chassis in real time based on visual detection results to ensure the target at the borehole remains directly in front of the robot. When the target distance further decreases and exceeds the camera's field of view, the robot continues to move in a straight line while maintaining its current direction of movement. Simultaneously, a two-dimensional LiDAR is activated for close-range detection. By analyzing the radar scan data, the position of the borehole is identified, and the final attitude adjustment is completed, thereby achieving precise positioning and guidance of the target.

[0007] This application provides a method for identifying and guiding borehole targets for a borehole protection robot, including: Step 1: Real-time acquisition of images of the surrounding environment using a planar camera positioned at the front of the hole-protecting robot; identification of boreholes in the images based on deep learning target detection; calculation of the borehole's orientation relative to the robot; and real-time adjustment of the robot's movement posture based on this orientation information. Step 2: Continuously count the number of target loss in visual recognition. If the number of target loss does not reach the preset threshold, maintain the current direction of travel and move forward in segments, continuing to execute the visual guidance in Step 1; if the number of target loss reaches the preset threshold, enter the radar positioning transition process. Step 3: Maintain the original forward direction and approach the target area in segments. At the same time, activate the two-dimensional lidar located at the front of the robot to collect environmental point clouds. Analyze the point cloud data to identify the gun holes corresponding to the ground depressions and calculate the close-range offset information of the gun holes relative to the robot. Step 4: Filter and smooth the visual orientation information output in Step 1 and the radar offset information output in Step 3 respectively. Adjust the robot's steering and stepping distance in stages according to the filtered target position information. Step 5: Repeat steps 1 to 4 until the robot reaches the preset working position of the blast hole.

[0008] In one embodiment, step one, which involves identifying and recognizing gun holes in an image based on deep learning target detection, calculating the gun hole's orientation relative to the robot, and adjusting the robot's movement posture in real time based on this orientation information, includes: A lightweight YOLO model is used to perform real-time target detection on images acquired by the camera, and the target with the highest confidence is selected as the effective gun hole. The horizontal deflection angle of the borehole relative to the camera optical axis is calculated by combining the image pixel coordinates and the camera field of view, and the deflection angle data of multiple consecutive frames are filtered and smoothed. Based on the smoothed deflection angle, corresponding control operations are performed to either correct the turn in place or move in a straight line with a fixed step size.

[0009] In one embodiment, the lightweight YOLO model includes: The YOLO11n lightweight network was selected as the backbone for object detection; A sample dataset was constructed by collecting images of mine blast holes from different angles and distances, and the blast hole targets in the dataset were labeled with rectangular bounding boxes. The labeled dataset is divided into training, validation and test sets according to a fixed ratio; Load general pre-trained weights to fine-tune the YOLO11n network using transfer learning, and complete iterative training by setting a fixed number of training rounds and batch size. The trained and converged YOLO11n model is deployed to the robot to process images acquired by the planar camera in real time and output the borehole bounding box and confidence data.

[0010] In one embodiment, step two, continuously counting the number of target loss instances in visual recognition, includes: The visual detection results of each round are judged in real time. When no valid blast hole is detected, the visual continuous missed detection count is accumulated. When a valid blast hole is detected, the visual continuous missed detection count is cleared to zero. When the visual continuous missed detection count does not reach the set threshold, the robot is determined to be in the perception transition blind zone between the planar camera and the two-dimensional LiDAR. It maintains its current direction of travel and continues to approach the target area in a segmented stepping manner without switching to the LiDAR positioning mode. When the cumulative count of continuous visual missed detections reaches the set threshold, it is determined that the borehole has exceeded the effective field of view of the planar camera. The system automatically terminates the vision-led guidance process and switches to the two-dimensional lidar close-range positioning process.

[0011] In one embodiment, step three involves identifying the boreholes corresponding to ground depressions through point cloud data analysis and calculating the near-range offset information of the boreholes relative to the robot, including: Two-dimensional lidar is used to periodically acquire environmental scanning point clouds and filter effective ranging points within the ground elevation range; based on the statistics of effective ranging points, a ground reference height is constructed, and the difference between the ranging of each scanning point and the ground reference height is compared to identify depression areas; When multiple consecutive scanning points meet the concavity determination criteria, they are identified as borehole areas, and the borehole center and lateral offset are calculated based on the scanning angle of the borehole area.

[0012] In one embodiment, step three further includes: The robot stops moving before each point cloud detection operation. After completing point cloud recognition, determine whether a blast hole has been detected based on the recognition results; When no blast hole is detected, the robot is controlled to move in segments and the number of undetected blast holes is accumulated. When the accumulated number of blast holes reaches the preset limit, the robot stops moving. After a blast hole is detected, the robot's posture is adjusted according to the lateral offset information of the blast hole.

[0013] In one embodiment, step four involves filtering and smoothing the visual orientation information output in step one and the radar offset information output in step three, respectively, including: Construct fixed-length cache queues to store the visual horizontal tilt angle and radar lateral offset of multiple consecutive frames, respectively. The real-time output of the visual horizontal deflection angle and radar lateral offset is sequentially stored into the corresponding cache queue; Extract the median of the data in each cache queue, and use the obtained median as the corresponding target position output data.

[0014] In one embodiment, adjusting the robot's steering and stepping distance in stages based on the filtered target position information includes: During the visual guidance phase, if the horizontal deflection angle of the smoothed bore exceeds the preset angle threshold, the robot is rotated in place to correct its posture; if the horizontal deflection angle of the smoothed bore is less than or equal to the preset angle threshold, the robot is controlled to move in a straight line with a first fixed step size. During the radar positioning phase, the deviation of the angle between the robot and the borehole is calculated based on the preset working reference distance and the arctangent function. When the angle deviation exceeds the preset angle threshold, the robot performs in-situ turning correction. When the angle deviation is less than or equal to the preset angle threshold, the robot is controlled to move forward in segments with a second fixed step size that is less than the first fixed step size.

[0015] Based on the same inventive concept, this application also proposes a borehole target recognition and guidance system for a borehole protection robot, the system comprising: Robot chassis; The perception module includes a planar camera and a two-dimensional lidar. The planar camera and the two-dimensional lidar are both located at the front of the robot chassis and are used to collect images of the environment in front and point cloud data of the environment, respectively. The vision processing module is connected to the planar camera signal and is configured with a trained lightweight YOLO model for recognizing gun holes based on images and calculating the orientation information of the gun holes relative to the robot. The blind zone transition control module is electrically connected to the vision processing module and the perception module respectively. It is used to count the number of target loss in visual recognition and switch the perception working mode according to the counting threshold. The radar point cloud processing module is connected to the two-dimensional lidar signal and is used to analyze the point cloud data, identify ground depressions, and calculate the close-range offset information of the blast holes. The filtering module is connected to the vision processing module and the radar point cloud processing module, respectively, and is used to perform median filtering and smoothing on the visual orientation information and radar offset information. The motion control module, connected to the filtering module and the robot chassis, is used to control the chassis to turn and step in stages based on the filtered target position information, and to repeatedly execute the recognition and guidance process until the robot reaches the preset working position of the blast hole.

[0016] In one embodiment, the system further includes: The system also includes a model training and deployment unit and a data storage unit; The model training and deployment unit is used to construct a dataset of mine borehole images, complete the YOLO11n lightweight network transfer learning training, and send the converged detection model to the vision processing module. The data storage unit has multiple fixed-length cache queues built in, which cache multiple consecutive frames of visual horizontal deflection data and radar lateral offset data, respectively, for the filtering module to read and complete the median calculation.

[0017] The beneficial effects of the embodiments in this application compared with the prior art are: This application employs a multi-sensor solution combining a planar camera and a two-dimensional LiDAR to achieve target detection at both near and far distances. The vision module uses the YOLO11n deep learning model for borehole detection, calculates the target's horizontal deflection angle using image bounding boxes, and introduces a sliding window midpoint filter to improve detection stability. The two-dimensional LiDAR utilizes the analysis of the scanning point's height relative to the ground reference to achieve precise pit location, and filters the lateral offset to enhance the system's robustness. Furthermore, the hardware cost is low, resulting in significant cost reduction and efficiency improvement. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart of a borehole target identification and guidance method for a borehole protection robot, provided as an embodiment of the present invention.

[0020] Figure 2 This is an embodiment of the present invention showing the actual installation location and field of view of the radar and camera.

[0021] Figure 3 The training results of a model in one embodiment of the present invention are (a) training loss and (b) detection performance metrics.

[0022] Figure 4 The diagram shows (a) a flowchart of a radar identification algorithm according to an embodiment of the present invention, and (b) a diagram showing the identification of point cloud distribution.

[0023] Figure 5 This is a visual guidance flowchart of one embodiment of the present invention.

[0024] Figure 6 This is a flowchart of the radar positioning stage according to an embodiment of the present invention. Detailed Implementation

[0025] It should be understood that, when used in this application specification and the appended claims, the term includes indicating the presence of the described feature, integral, step, operation, element and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0026] It should also be understood that the terms used in this application specification and the appended claims refer to any combination of one or more of the associated listed items and all possible combinations, and include such combinations.

[0027] Example 1: like Figure 1 As shown, this application provides a method for target recognition and guidance of blast holes for a hole protection robot, which is applicable to the autonomous approach and clearing of blast holes before blasting operations in mines. The specific steps are as follows: Step 1: Long-distance visual perception guidance: The planar camera placed at the front of the hole-protecting robot collects real-time images of the mining environment in front of it. Based on deep learning target detection, the target of the blast hole in the image is identified, the orientation information of the blast hole relative to the robot is calculated, and the robot's walking posture is dynamically adjusted according to the real-time orientation information to achieve long-distance coarse positioning guidance.

[0028] Specifically, this step employs a lightweight YOLO model to perform image-based blast hole detection. The YOLO11n lightweight network is preferred as the backbone network for target detection, as it has a small number of network parameters and fast inference speed, making it suitable for the real-time computing needs of the robot's embedded terminal. Before model deployment, a dedicated dataset for mine blast holes is constructed: real-world images of blast holes in the mining area are collected from different shooting angles and distances, including those obscured by loose soil and gravel, forming the original sample dataset. Rectangular bounding boxes are used to accurately label the blast hole targets within the images. The labeled dataset is then divided into training, validation, and test sets according to a fixed ratio. Subsequently, general pre-trained weights are loaded to perform transfer learning fine-tuning on the YOLO11n network. Fixed training epochs and batch sizes are set to complete iterative training until the model loss converges. Finally, the trained YOLO11n model is deployed on the robot's vision processing terminal.

[0029] During real-time inference, each frame of environmental image captured by the planar camera is input into the YOLO11n model for target detection. The target with the highest built-in confidence level is selected as the valid borehole, while invalid interfering targets such as rubble, soil mounds, and potholes are removed. Simultaneously, combining the image pixel coordinates with the camera's inherent field of view parameters, the horizontal deflection angle of the borehole center relative to the camera's optical axis is geometrically calculated, and the deflection angle data from multiple consecutive frames undergoes preliminary filtering and smoothing. Finally, differentiated control is executed based on the smoothed horizontal deflection angle value: when the deflection angle is greater than a preset angle threshold, the robot is controlled to rotate in place to complete attitude correction; when the deflection angle is less than or equal to the preset angle threshold, the robot is controlled to maintain a straight line movement with a first fixed step size, achieving stable long-distance visual guidance.

[0030] Step 2: Visual Miss Detection Counting and Blind Zone Transition: During the robot's vision-guided movement, the system continuously and in real-time determines the image detection results of each round, counts the number of visual targets lost, and achieves a smooth transition between perception modes. The specific determination logic is as follows: if a valid borehole is not detected in a single round of image detection, the visual missing detection count is incremented by one; if a valid borehole is successfully detected, the missing detection count is immediately reset to zero, and the vision-guided mode is maintained continuously.

[0031] When the continuous visual missed detection count does not reach the system's preset threshold, the robot is determined to be in the perception transition blind zone between the planar camera and the 2D LiDAR. At this time, the perception mode is not switched, the robot's current direction of travel remains unchanged, and it continues to approach the target area of ​​the borehole using a segmented stepping method, continuing to complete the visual guidance operation based on step one. When the cumulative continuous visual missed detection count reaches the preset threshold, the borehole is determined to be completely outside the effective field of view of the planar camera, visual guidance fails, the system automatically terminates the vision-led guidance process, triggers a mode switching command, and enters the radar positioning transition process.

[0032] Step 3: Precise positioning using close-range lidar point cloud: After switching to the radar positioning mode, the robot continues to approach the target area in segments while maintaining its original forward direction. At the same time, the two-dimensional lidar located at the front of the robot is activated to periodically collect point cloud data of the environment in front. By analyzing the point cloud data, the robot accurately identifies the blast holes corresponding to the ground depressions and calculates the close-range lateral offset information of the blast holes relative to the robot.

[0033] The specific point cloud processing method is as follows: The original point cloud obtained by the lidar scan is filtered, and only the effective ranging points within the preset elevation range on the ground are retained, while high-altitude and invalid long-range noise points are removed; Based on the filtered effective ranging points, a ground reference height for the current area is constructed by statistical fitting; The difference between the ranging height of each effective scanning point and the ground reference height is compared one by one to identify ground depression areas; When multiple consecutive scanning points meet the depression height determination condition, the depression area is determined to be an effective borehole area; Then, based on the radar scanning angle corresponding to the borehole area, the center position of the borehole and the lateral offset relative to the robot body are geometrically calculated to complete the close-range borehole positioning.

[0034] Meanwhile, the radar localization phase in this step adopts a stable operation logic of stop-check-go: before each point cloud scan detection, the robot is controlled to completely stop moving to avoid point cloud distortion caused by movement vibration; after completing a round of complete point cloud recognition and analysis, the corresponding action is executed according to the recognition result. If no valid gun hole is detected, the robot is controlled to move forward in segments with a fixed small step size, and the number of times the radar fails to detect it is accumulated at the same time. When the accumulated number of times the radar fails to detect it reaches the preset limit, the target is determined to be lost, and the robot is controlled to stop moving; if the gun hole is successfully detected, the robot's moving posture is adjusted in real time according to the lateral offset information of the gun hole to achieve precise alignment at close range.

[0035] Step 4: Multi-source data filtering and smoothing and phased motion control: The system filters and smooths the visual horizontal azimuth information output from Step 1 and the radar lateral offset information output from Step 3 to eliminate detection noise and data jumps caused by the complex environment of the mine. Then, based on the filtered and accurate target position information, the robot's turning angle and step distance are adjusted in stages.

[0036] The specific filtering method is as follows: The system constructs fixed-length buffer queues for visual data and radar data respectively to store the visual horizontal deflection angle and radar lateral offset of multiple consecutive frames; each frame of detection data output in real time is stored into the corresponding buffer queue in sequence to keep the queue data updated in real time; the median of the data in the two buffer queues is extracted respectively, and the median is used as the effective target position data of the final output to achieve the median filtering smoothing effect and effectively avoid the impact of abnormal data in a single detection on the guidance accuracy.

[0037] After filtering, differentiated phased control is executed: In the long-range visual guidance stage, the horizontal deflection angle of the filtered borehole is used as the basis. If the deflection angle exceeds the threshold, the robot turns and corrects itself in place. If it does not exceed the threshold, the robot moves in a straight line with a first fixed large step size. In the short-range radar positioning stage, the robot calculates the real-time angle deviation between the robot and the borehole based on the preset working reference distance and the arctangent function. If the angle deviation exceeds the threshold, the robot performs in-place attitude correction. If the angle deviation meets the requirements, the robot is controlled to move in segments with a second fixed small step size smaller than the first fixed step size to ensure the stability and accuracy of the close approach.

[0038] Step 5, Closed-loop Approach Operation: The robot continuously executes the perception, judgment, localization, filtering and motion control processes of Steps 1 to 4 above, continuously iteratively corrects the traveling posture and traveling distance, and gradually reduces the relative distance between the robot and the blast hole until the robot reaches the preset working position of the blast hole, completing the entire autonomous recognition and guidance operation.

[0039] This application utilizes a low-cost multi-sensor fusion architecture combining a planar camera and a 2D LiDAR, along with lightweight visual inspection using YOLO11n, a blind zone transition strategy for missed detection counting, point cloud depression recognition and localization, median filtering from dual data sources, and differentiated step control for near and far fields. This enables stable long-range guidance and precise short-range positioning of borehole targets under complex mining conditions. It solves the problems of large perception blind zones, easy target loss, poor detection stability, and high hardware costs associated with traditional solutions. It balances operational accuracy, operational stability, and economic feasibility for engineering implementation, and is suitable for the needs of large-scale autonomous borehole protection operations in mines.

[0040] In one embodiment, such as Figure 2 The diagram shows the actual installation positions and field of view of the radar and camera. Both sensors are mounted on the front of the robot's body, with their mounting planes perpendicular to the robot's direction of travel and perpendicular to the ground. During system operation, the camera primarily handles initial target identification and direction guidance, adjusting the movement direction of the mobile chassis in real time based on visual detection results to ensure the target borehole remains directly in front of the high-clearance mobile work device. When the target distance decreases further and exceeds the camera's field of view, the robot continues to move in a straight line while maintaining its current direction of travel. Simultaneously, a two-dimensional lidar is activated for close-range detection. By analyzing the radar scan data, the location of the borehole is identified, and the final attitude adjustment is completed, thereby achieving precise positioning and guidance of the target.

[0041] The target recognition dataset used in this application originates from blast hole images in a simulated soil clearing experiment. During data acquisition, a planar camera was used to photograph the work area, obtaining image data containing blast hole targets. Different shooting angles and distances were covered as much as possible during the acquisition process to improve the diversity of the dataset and the generalization ability of the model.

[0042] During the data annotation phase, the LabelImg annotation tool was used to manually annotate the borehole targets in the images. The annotation method employed a rectangular bounding box (BoundingBox) to select the bounding rectangular region of each borehole target and generate a corresponding annotation file for subsequent training and validation of the target detection model. Figure 3As shown, manual frame-by-frame annotation ensures the accuracy of the annotation results, providing a reliable data foundation for model training. This study constructed a dataset of 400 gun hole images. To ensure the rationality of the model training and testing process, the dataset was divided in an 8:1:1 ratio: 320 images were used for training and learning model parameters; 40 images were used for evaluating model performance and optimizing parameters during training; and 40 images were used for independent testing of the final model's recognition performance. This reasonable dataset division effectively evaluates the generalization ability of the target recognition model on unseen data. After data collection, annotation, and division, a complete gun hole target recognition dataset was finally formed, providing fundamental data support for the subsequent training and inference of the target detection model.

[0043] The target recognition dataset used comes from blast hole images in a simulated soil clearing experiment. During data acquisition, a planar camera was used to photograph the work area, obtaining image data containing blast hole targets. Different shooting angles and distances were covered as much as possible during the acquisition process to improve the diversity of the dataset and the generalization ability of the model.

[0044] In the data annotation stage, the LabelImg annotation tool was used to manually annotate the borehole targets in the images. The annotation method employed rectangular bounding boxes, selecting the bounding rectangle region of each borehole target and generating corresponding annotation files for subsequent training and validation of the target detection model. Manual frame-by-frame annotation ensured the accuracy of the annotation results, providing a reliable data foundation for model training. This study constructed a dataset of 400 borehole images. To ensure the rationality of the model training and testing process, the dataset was divided in an 8:1:1 ratio: 320 images were used for training and learning model parameters; 40 images were used for evaluating model performance and parameter tuning during training; and 40 images were used for independent testing of the final model's recognition performance. This reasonable dataset division effectively evaluated the generalization ability of the target recognition model on unseen data. After data collection, annotation, and division, a complete borehole target recognition dataset was finally formed, providing fundamental data support for the subsequent training and inference of the target detection model.

[0045] After constructing the borehole image dataset, the target detection model needs to be trained to achieve automatic identification of the target. This paper uses the YOLO target detection framework provided by Ultralytics for model training. YOLO (YouOnlyLookOnce) is a single-stage target detection algorithm based on deep convolutional neural networks. It can simultaneously complete target localization and category recognition in a single forward propagation, featuring fast detection speed and strong real-time performance, making it suitable for real-time perception tasks of mobile robots. For model selection, this paper uses YOLO11n as the target detection network. YOLO11n is a lightweight model in the YOLO11 series, maintaining detection accuracy while having low computational complexity, meeting the real-time recognition requirements of robot systems. During training, the model weights pre-trained on a large-scale dataset are first loaded, and the network is fine-tuned using transfer learning to improve the model's recognition ability in borehole detection tasks. Model training is implemented using Python and the Ultralytics deep learning framework. The training parameters are set as follows: 20 epochs and 8 batch sizes. The model is evaluated on the validation set after each training round to monitor its detection performance.

[0046] Training results as follows Figure 3 As the number of training epochs increases, the model's bounding box loss (BoxLoss), classification loss (ClassLoss), and distribution focus loss (DFLLoss) gradually decrease and stabilize in the later stages of training, indicating that the model has essentially converged. After training, the model exhibits good detection performance on the validation set, with a precision of 0.998, a recall of 1.000, and a mean AP50 of 0.995 under an IoU threshold of 0.5, while the mAP50-95 under a more stringent evaluation criterion is 0.773. Experimental results demonstrate that the trained target detection model can accurately identify borehole targets in images and exhibits good detection stability. The model can stably detect borehole locations under different background conditions and provide accurate target bounding boxes, providing reliable visual information for subsequent LiDAR-based target localization and operation guidance.

[0047] To improve the stability of the detection results, the system performs sliding window filtering on the borehole center angle and lateral distance obtained from multiple consecutive scans, and takes the median as the final output. This method can effectively reduce the impact of single measurement errors and radar noise. Experimental results show that the algorithm can stably identify the borehole positions in the area in front of the device, and output relatively accurate target angles and lateral offsets, providing reliable positioning information for subsequent device movement control and operation execution.

[0048] In one embodiment, to achieve accurate identification and spatial positioning of the target, this application proposes a method for calculating the horizontal deflection angle of the target relative to the camera's optical axis based on deep learning target detection algorithms and imaging geometry. The system uses a trained YOLO model to perform real-time detection on images captured by the camera, obtaining the bounding box position of the target and its corresponding confidence information. Among the detection results, the target with the highest confidence is selected as the current valid target to improve the stability and reliability of the identification results.

[0049] Let the coordinates of the top left and bottom right corners of the detection box be ( )and( If the target's horizontal center position in the image is: ; To convert pixel positions in an image into orientation information in real space, a mapping relationship between pixel coordinates and the camera's field of view needs to be established. Let the image width be... The x-coordinate of the target center is The normalized offset for the image center is: ; This normalization factor reflects the relative horizontal position of the target in the image, and its value ranges from [value missing]. Combined with the camera's horizontal field of view The horizontal deflection angle of the target relative to the camera's optical axis can be expressed as: ; in, The horizontal deflection angle of the target relative to the camera's optical axis is given. To meet the requirements of the control system, the calculation result is ultimately converted into an angle-based output. Considering that the visual detection process is susceptible to changes in ambient lighting, false detections in the model, and image noise, the angle result calculated in a single frame may fluctuate. To improve the stability of the system output, this paper employs a sliding window midpoint filtering method to smooth the calculation results of multiple consecutive frames. Specifically, a fixed-length angle buffer queue is constructed, and the angle calculation results of consecutive frames are stored in the queue. The median value is then taken as the current output value, effectively suppressing the influence of outliers on the results and improving the robustness and stability of the angle estimation. Through the above method, an effective conversion from image target detection to spatial angle estimation is achieved. While ensuring computational efficiency, it can stably output the horizontal deflection angle of the target relative to the camera's optical axis, providing a reliable basis for subsequent attitude adjustment and motion control of the device.

[0050] Two-dimensional LiDAR can periodically scan the surrounding environment and output distance information. By analyzing the scan data, changes in ground height can be identified, thereby enabling the detection of target objects. Compared with visual recognition, LiDAR has advantages such as being less affected by lighting conditions and having higher distance measurement accuracy, thus providing stable spatial position information for robot operations. The algorithm flow is as follows: Figure 4 As shown.

[0051] The lidar communicates with the host computer via a serial port and periodically outputs scan data. Each frame contains several measurement points, each consisting of a scan angle and its corresponding distance. First, the data is filtered based on the scan angle, retaining only points within a ±30° range facing the ground as the detection area, thus reducing interference from irrelevant data. Within the detection area, since most scan points normally correspond to the ground plane, the ground reference height can be estimated using statistical methods. Specifically, the forward distances of all valid scan points are statistically analyzed, calculating their mean and standard deviation, and outliers significantly deviating from the mean are removed to reduce noise. Then, the median of the remaining data is used as the ground reference height to obtain a stable ground reference. After obtaining the ground reference height, the borehole area is identified by comparing the distance values ​​of each scan point with the reference value. If the distance of a point is significantly greater than the ground reference value, it indicates the presence of a depression structure in that direction. By setting a distance threshold, points exceeding the threshold are marked as candidate borehole points. To reduce false detections caused by noise, the algorithm requires at least two consecutive outliers to be considered valid borehole areas. After detecting the borehole region, its angular range is further calculated to determine the borehole center position. Let the minimum and maximum angles corresponding to this region be and , respectively, then the borehole center angle is taken as the average of the two. Simultaneously, based on the scanning point distance information corresponding to the center angle, the lateral offset of the borehole in the robot coordinate system can be calculated, thereby obtaining the target's spatial position.

[0052] In the visual guidance stage, such as Figure 5As shown, the system enters a visual search state, acquiring images through the camera and calling a target detection algorithm to identify the location of the pothole. During the visual guidance phase, the system employs a "stop-look-walk" step-by-step control strategy. "Stop" means stopping the chassis movement upon entering the detection cycle to avoid image blur affecting the detection results; "look" means continuously acquiring multiple frames of images and performing target detection while stationary, improving detection reliability through multi-frame information; "walk" involves executing corresponding motion control based on the detected target's angle. Upon detecting a target, the system makes a decision based on the target's angle relative to the camera's optical axis. When the angle exceeds a set threshold (5°), the system enters a visual steering state, driving the chassis to rotate to align the target; when the angle is smaller, it directly enters a stepping forward state, moving towards the target at fixed step lengths (0.20m). After completing one stepping motion, the system returns to the visual steering state to detect and confirm the target again. This process constitutes a visual closed-loop control, enabling the robot to continuously correct its posture as it approaches the target.

[0053] like Figure 6 As shown, there is a transition zone between the sensing ranges of the two types of sensors. When a target enters this zone, visual detection may fail, while the radar has not yet stably detected the target. To address this blind spot problem, when the visual module fails to detect the target multiple times consecutively, the system does not immediately rely on radar for localization. Instead, it maintains its current direction of movement and gradually approaches the target area using a segmented approach. To this end, the system employs a visual failure counting mechanism. When the number of consecutive detection failures reaches a threshold (10 times), it is determined that the target has exceeded the effective visual sensing range or the visual module has failed. The system then transitions from the visual guidance phase to the radar localization phase, switching to radar search, thus achieving a dynamic transition in sensing methods. After entering the radar search state, the system scans using lidar to acquire spatial structure information of the environment and identifies the location of potholes through point cloud analysis. During the radar guidance phase, the system employs a progressive "stop-check-go" search mechanism. This involves controlling the chassis to advance intermittently in fixed steps (0.1m), triggering radar detection after each stop to determine if the target has entered the radar's effective sensing range. This ensures the device maintains a reasonable direction of movement even after visual guidance is lost, gradually transitioning to a radar-dominated precise positioning phase. Upon target detection, the system extracts the lateral offset distance and relative azimuth angle of the pit center. Based on geometric relationships, it calculates the turning angle, using a preset fixed distance of 64.635cm between the work point and the sensor as a reference. The arctangent function is used to calculate the angle between the device's current pose and the target, thus obtaining a precise turning angle. When the calculated angle deviation exceeds 5°, the system adjusts its attitude and begins turning; if the angle deviation is small, the current direction is considered basically aligned, and the system directly enters the target approach phase.

[0054] Example 2: This application also proposes a borehole target identification and guidance system for a borehole protection robot, the system comprising: Robot chassis; The perception module includes a planar camera and a two-dimensional lidar. The planar camera and the two-dimensional lidar are both located at the front of the robot chassis and are used to collect images of the environment in front and point cloud data of the environment, respectively. The vision processing module is connected to the planar camera signal and is configured with a trained lightweight YOLO model for recognizing gun holes based on images and calculating the orientation information of the gun holes relative to the robot. The blind zone transition control module is electrically connected to the vision processing module and the perception module respectively. It is used to count the number of target loss in visual recognition and switch the perception working mode according to the counting threshold. The radar point cloud processing module is connected to the two-dimensional lidar signal and is used to analyze the point cloud data, identify ground depressions, and calculate the close-range offset information of the blast holes. The filtering module is connected to the vision processing module and the radar point cloud processing module, respectively, and is used to perform median filtering and smoothing on the visual orientation information and radar offset information. The motion control module, connected to the filtering module and the robot chassis, is used to control the chassis to turn and step in stages based on the filtered target position information, and to repeatedly execute the recognition and guidance process until the robot reaches the preset working position of the blast hole.

[0055] In one embodiment, the system further includes: The system also includes a model training and deployment unit and a data storage unit; The model training and deployment unit is used to construct a dataset of mine borehole images, complete the YOLO11n lightweight network transfer learning training, and send the converged detection model to the vision processing module. The data storage unit has multiple fixed-length cache queues built in, which cache multiple consecutive frames of visual horizontal deflection data and radar lateral offset data, respectively, for the filtering module to read and complete the median calculation.

[0056] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for identifying and guiding borehole targets in a borehole protection robot, characterized in that, include: Step 1: Real-time acquisition of images of the surrounding environment using a planar camera positioned at the front of the hole-protecting robot; identification of boreholes in the images based on deep learning target detection; calculation of the borehole's orientation relative to the robot; and real-time adjustment of the robot's movement posture based on this orientation information. Step 2: Continuously count the number of times the visually recognized target is lost. If the number of times the target is lost does not reach the preset threshold, continue to move forward in segments in the current direction and continue to execute the visual guidance in Step 1. If the number of times a target is lost reaches a preset threshold, the radar positioning transition process will begin. Step 3: Maintain the original forward direction and approach the target area in segments. At the same time, activate the two-dimensional lidar located at the front of the robot to collect environmental point clouds. Analyze the point cloud data to identify the gun holes corresponding to the ground depressions and calculate the close-range offset information of the gun holes relative to the robot. Step 4: Filter and smooth the visual orientation information output in Step 1 and the radar offset information output in Step 3 respectively. Adjust the robot's steering and stepping distance in stages according to the filtered target position information. Step 5: Repeat steps 1 to 4 until the robot reaches the preset working position of the blast hole.

2. The method for identifying and guiding borehole targets in a borehole protection robot according to claim 1, characterized in that, Step one, which involves identifying and recognizing boreholes in an image using deep learning object detection, calculating the borehole's orientation relative to the robot, and adjusting the robot's movement posture in real time based on this orientation information, includes: A lightweight YOLO model is used to perform real-time target detection on images acquired by the camera, and the target with the highest confidence is selected as the effective gun hole. The horizontal deflection angle of the borehole relative to the camera optical axis is calculated by combining the image pixel coordinates and the camera field of view, and the deflection angle data of multiple consecutive frames are filtered and smoothed. Based on the smoothed deflection angle, corresponding control operations are performed to either correct the turn in place or move in a straight line with a fixed step size.

3. The method for identifying and guiding borehole targets for a borehole protection robot according to claim 2, characterized in that, The lightweight YOLO model includes: The YOLO11n lightweight network was selected as the backbone for object detection; A sample dataset was constructed by collecting images of mine blast holes from different angles and distances, and the blast hole targets in the dataset were labeled with rectangular bounding boxes. The labeled dataset is divided into training, validation and test sets according to a fixed ratio; Load general pre-trained weights to fine-tune the YOLO11n network using transfer learning, and complete iterative training by setting a fixed number of training rounds and batch size. The trained and converged YOLO11n model is deployed to the robot to process images acquired by the planar camera in real time and output the borehole bounding box and confidence data.

4. The method for identifying and guiding borehole targets in a borehole protection robot according to claim 3, characterized in that, Step 2: Continuously count the number of times the visually recognized target is lost, including: The visual detection results of each round are judged in real time. When no valid blast hole is detected, the visual continuous missed detection count is accumulated. When a valid blast hole is detected, the visual continuous missed detection count is cleared to zero. When the visual continuous missed detection count does not reach the set threshold, the robot is determined to be in the perception transition blind zone between the planar camera and the two-dimensional LiDAR. It maintains its current direction of travel and continues to approach the target area in a segmented stepping manner without switching to the LiDAR positioning mode. When the cumulative count of continuous visual missed detections reaches the set threshold, it is determined that the borehole has exceeded the effective field of view of the planar camera. The system automatically terminates the vision-led guidance process and switches to the two-dimensional lidar close-range positioning process.

5. The method for identifying and guiding borehole targets in a borehole protection robot according to claim 4, characterized in that, Step three involves analyzing point cloud data to identify the boreholes corresponding to ground depressions and calculating the near-range offset information of the boreholes relative to the robot, including: Two-dimensional lidar is used to periodically acquire environmental scanning point clouds and filter effective ranging points within the ground elevation range; based on the statistics of effective ranging points, a ground reference height is constructed, and the difference between the ranging of each scanning point and the ground reference height is compared to identify depression areas; When multiple consecutive scanning points meet the concavity determination criteria, they are identified as borehole areas, and the borehole center and lateral offset are calculated based on the scanning angle of the borehole area.

6. The method for identifying and guiding borehole targets in a borehole protection robot according to claim 5, characterized in that, Step three also includes: The robot stops moving before each point cloud detection operation. After completing point cloud recognition, determine whether a blast hole has been detected based on the recognition results; When no blast hole is detected, the robot is controlled to move in segments and the number of undetected blast holes is accumulated. When the accumulated number of blast holes reaches the preset limit, the robot stops moving. After a blast hole is detected, the robot's posture is adjusted according to the lateral offset information of the blast hole.

7. The method for identifying and guiding borehole targets in a borehole protection robot according to claim 6, characterized in that, Step 4: Perform filtering and smoothing processing on the visual orientation information output from Step 1 and the radar offset information output from Step 3, including: Construct fixed-length cache queues to store the visual horizontal tilt angle and radar lateral offset of multiple consecutive frames, respectively. The real-time output of the visual horizontal deflection angle and radar lateral offset is sequentially stored into the corresponding cache queue; Extract the median of the data in each cache queue, and use the obtained median as the corresponding target position output data.

8. The method for identifying and guiding borehole targets for a borehole protection robot according to claim 7, characterized in that, The step of adjusting the robot's steering and stepping distance in stages based on the filtered target position information includes: During the visual guidance phase, if the horizontal deflection angle of the smoothed bore exceeds the preset angle threshold, the robot is rotated in place to correct its posture; if the horizontal deflection angle of the smoothed bore is less than or equal to the preset angle threshold, the robot is controlled to move in a straight line with a first fixed step size. During the radar positioning phase, the deviation of the angle between the robot and the borehole is calculated based on the preset working reference distance and the arctangent function. When the angle deviation exceeds the preset angle threshold, the robot performs in-situ turning correction. When the angle deviation is less than or equal to the preset angle threshold, the robot is controlled to move forward in segments with a second fixed step size that is less than the first fixed step size.

9. A guidance system for a borehole target identification and guidance method for a borehole protection robot according to any one of claims 1-8, characterized in that, The system includes: Robot chassis; The perception module includes a planar camera and a two-dimensional lidar. The planar camera and the two-dimensional lidar are both located at the front of the robot chassis and are used to collect images of the environment in front and point cloud data of the environment, respectively. The vision processing module is connected to the planar camera signal and is configured with a trained lightweight YOLO model for recognizing gun holes based on images and calculating the orientation information of the gun holes relative to the robot. The blind zone transition control module is electrically connected to the vision processing module and the perception module respectively. It is used to count the number of target loss in visual recognition and switch the perception working mode according to the counting threshold. The radar point cloud processing module is connected to the two-dimensional lidar signal and is used to analyze the point cloud data, identify ground depressions, and calculate the close-range offset information of the blast holes. The filtering module is connected to the vision processing module and the radar point cloud processing module, respectively, and is used to perform median filtering and smoothing on the visual orientation information and radar offset information. The motion control module, connected to the filtering module and the robot chassis, is used to control the chassis to turn and step in stages based on the filtered target position information, and to repeatedly execute the recognition and guidance process until the robot reaches the preset working position of the blast hole.

10. The guiding system according to claim 9, characterized in that, The system also includes: The system also includes a model training and deployment unit and a data storage unit; The model training and deployment unit is used to construct a dataset of mine borehole images, complete the YOLO11n lightweight network transfer learning training, and send the converged detection model to the vision processing module. The data storage unit has multiple fixed-length cache queues built in, which cache multiple consecutive frames of visual horizontal deflection data and radar lateral offset data, respectively, for the filtering module to read and complete the median calculation.

Images