Obstacle detection method and system for autonomous operation engineering machinery

Abstract

The invention discloses an autonomous operation engineering machinery-oriented obstacle detection method and system, and relates to the technical field of obstacle detection, and the method comprises the following steps: S1, constructing and preprocessing a surface mine driving obstacle data set, and dividing the preprocessed data set into a training set, a verification set and a test set according to a preset proportion; s2, the YOLOv8 network is improved, and an obstacle detection model is built; s3, performing model training on the obstacle detection model by using the training set and the verification set until the loss function is converged, and obtaining an optimized obstacle detection model; and S4, performing obstacle detection on the test set through the optimized obstacle detection model to obtain a detection result. The real-time requirement in engineering practical application is met, meanwhile, high target detection performance is achieved, and therefore reliable technical support is provided for surface mine driving obstacle detection, and safe and efficient driving of autonomous operation engineering machinery in surface mines is assisted.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle detection, and more particularly to an obstacle detection method and system for autonomous operation construction machinery. Background Art

[0002] With the vigorous development of intelligent technologies, traditional mines have been transformed into smart mines, and driverless technologies have gradually been applied to open-pit mines. However, during the driving process of autonomous operation construction machinery on unstructured roads in open-pit mines, it will not only encounter pedestrians, cars, and other construction machinery (such as mining trucks, loaders, excavators, etc.), but also encounter potholes and falling rocks. These obstacles pose great potential safety hazards to driving in open-pit mines, and are prone to collisions, vehicle bogging, and rollovers, resulting in property losses and even threatening the lives and safety of personnel. Therefore, it has become crucial to study the accuracy and real-time performance of obstacle detection for driving in open-pit mines, which is directly related to the safety and efficiency of mine operations.

[0003] Currently, most in the field of construction machinery use multi-sensor fusion technology for obstacle detection. As a part of it, the image processing technology based on deep learning shows significant advantages due to the powerful feature learning ability and self-adaptability of convolutional neural networks. YOLO (You Only Look Once) is the current mainstream single-stage object detection algorithm, which has excellent real-time performance, a simple and efficient structure, can achieve multi-scale detection, can make full use of global context information, and supports multi-task learning. These characteristics make it have extremely excellent performance in fast object detection tasks and real-time application scenarios. However, in actual engineering applications, the obstacle detection algorithm model has long had the defect that it is difficult to balance detection accuracy and detection speed. Either the model is too large to be deployed, or it meets the real-time requirements but has a high false detection and missed detection rate.

[0004] Therefore, how to meet the real-time requirements in engineering practical applications while the model has high object detection performance has become a hot issue and the key in the current research field, and it is also a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides an obstacle detection method and system for autonomous operation construction machinery, which solves the problems existing in the background art.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An obstacle detection method for autonomous operation construction machinery, comprising the following steps:

[0008] S1. Construct and preprocess an open-pit mine driving obstacle dataset, and divide the preprocessed dataset into a training set, a validation set, and a test set according to a preset ratio;

[0009] S2. Improve the YOLOv8 network and build an obstacle detection model;

[0010] S3. Train the obstacle detection model using the training set and the validation set until the loss function converges to obtain an optimized obstacle detection model.

[0011] S4. Perform obstacle detection on the test set using the optimized obstacle detection model to obtain detection results.

[0012] In the above method, optionally, the specific steps of S1 are:

[0013] S11, collecting image data of obstacles to autonomous construction machinery in open-pit mines, removing unqualified images, screening qualified images, and obtaining an open-pit mine obstacle data set;

[0014] S12. Use the image annotation tool LabelImg to annotate various obstacles in the open-pit mine driving obstacle dataset image in YOLO format, and save the annotation information to a txt file corresponding to the image name;

[0015] S13, dividing the annotated images and the txt files storing the annotated information into a training set, a validation set, and a test set according to a preset ratio of 8:1:1.

[0016] In the above method, optionally, in S12, the marked obstacle types include: cars, pedestrians, loaders, excavators, mining trucks, fallen rocks, and potholes.

[0017] In the above method, optionally, S1 further includes:

[0018] S14, adding rain noise, fog noise, and Gaussian noise to the images in the training set and performing flipping, rotation, and random cropping to obtain an enhanced data set and input the enhanced data set into the obstacle detection model for model training.

[0019] In the above method, optionally, in S2, the improvement method of the YOLOv8 network is specifically as follows:

[0020] S21. Embed the SE attention mechanism in the YOLOv8 backbone network;

[0021] S22. In the feature fusion of the backbone network, the improved SPPF_FPSC module is used to replace the original SPPF module;

[0022] S23, use a suitable dynamic head network at the output;

[0023] S24. Adopt WIoU as the bounding box loss function;

[0024] S25. Adopt a pruning algorithm based on the LAMP score to prune redundant weight parameters and adjust the model structure.

[0025] In the above method, optionally, in S22, the improved SPPF_FPSC module enables the network to adaptively integrate the feature information of obstacles at different scales, specifically:

[0026] In the module initialization stage, reasonably divide the number of channels of the input data, and determine the number of hidden channels through a specific calculation method;

[0027] In the forward propagation process of the module, use the cv1 convolutional layer to integrate and optimize the channel information of the input features, adjust the distribution of the feature map, and obtain the basic features; for different preset dilation rates, repeatedly use the shared convolutional layer for convolutional operations multiple times, and splice and integrate the features extracted under different receptive fields in the channel dimension; use the cv2 convolutional layer to integrate and optimize the spliced features, and output the feature representation of the open-pit mine obstacle features.

[0028] In the above method, optionally, in S25, the pruning algorithm based on the LAMP score obtains the LAMP score of each connection by calculating the square of the magnitude of each target connection weight and normalizing it to the sum of the squared magnitudes of all surviving weights in the same layer; prune the connections with the LAMP score lower than the preset threshold to complete the compression and optimization of the model.

[0029] A system applying the obstacle detection method for autonomous operation construction machinery described in any one of the above, including:

[0030] A dataset construction module, used to construct an open-pit mine driving obstacle dataset and perform preprocessing, and divide the preprocessed dataset into a training set, a validation set, and a test set according to a preset ratio;

[0031] A model construction module, used to improve the YOLOv8 network and build an obstacle detection model;

[0032] A model training module, used to train the obstacle detection model through the training set and the validation set until the loss function converges, and obtain an optimized obstacle detection model;

[0033] A detection module, used to perform obstacle detection on the test set through the optimized obstacle detection model to obtain detection results.

[0034] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a method and system for obstacle detection of autonomous operation construction machinery, which has the following technical beneficial effects:

[0035] (1) In terms of detection accuracy, to overcome the problem of insufficient feature extraction of obstacles in the complex scenarios of open-pit mines in current technologies, the present invention embeds an attention mechanism and enhances the feature of the backbone network of YOLOv8, enabling it to more accurately capture the feature information of small targets, occluded targets, and targets of different scales, and significantly reducing the probability of missed detection and false detection when facing small falling rocks on unstructured roads or partially occluded construction machinery; by selecting a more suitable dynamic head network and loss function, the missed detection rate of small targets is reduced, and the distinction between the features of various obstacles is enhanced. Even when pedestrians and small mining trucks are at a long distance and the image is blurred, the two categories can be accurately distinguished and located;

[0036] (2) In terms of detection speed, aiming at the situation that some high-precision models cannot meet the real-time requirements of open-pit mines due to the excessive computational complexity and slow detection speed caused by complex network structures, the present invention adopts an optimization method of model compression, effectively reducing the number of model parameters and computational complexity without significantly reducing the detection accuracy. At the same time, the data preprocessing and model inference processes are optimized specifically to remove redundant calculations;

[0037] (3) In terms of adaptability to practical applications, the present invention fully integrates data simulating the harsh environment of mines during the training process, enhances the adaptability of the model to complex environments, reduces false alarms and missed alarms, and provides continuous and reliable safety guarantees for mine operations. Aiming at the problem that it is difficult to deploy the open-pit mine model on the device side, the present invention improves the model by taking into account both accuracy and speed, reducing the complexity and cost of the system while ensuring the accuracy and real-time performance of obstacle detection for autonomous operation construction machinery during driving, thus promoting the development of intelligent mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0039] Figure 1 is the step flow chart provided by the present invention;

[0040] Figure 2 is a schematic diagram of a partial data set provided by the present invention;

[0041] Figure 3 It is the flow chart of dataset construction provided by the present invention;

[0042] Figure 4 It is the overall structure diagram of the improved YOLOv8 provided by the present invention;

[0043] Figure 5 It is the schematic diagram of the SE attention mechanism module provided by the present invention;

[0044] Figure 6 It is the schematic diagram of the SPPF and SPPF_FPSC modules provided by the present invention;

[0045] Figure 7 It is the schematic diagram of the Dyhead module provided by the present invention;

[0046] Figure 8 It is the flow chart of pruning based on the LAMP score provided by the present invention;

[0047] Figure 9 It is the schematic diagram of channel comparison before and after pruning provided by the present invention;

[0048] Figure 10 It is the training result graph after finetuning after pruning provided by the present invention;

[0049] Figure 11 It is the comparison graph of the mAP_0.5 index of the training results between the improved YOLOv8 model and YOLOv8 provided by the present invention;

[0050] Figure 12 It is the schematic diagram of the detection results of obstacles for autonomous construction machinery driving in open-pit mines before and after improvement provided by the present invention. Specific embodiments

[0051] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0052] Regarding the obstacle detection technology in the field of construction machinery, the problems and defects existing in the prior art are understood from the following aspects:

[0053] 1) Analysis from the perspective of detection accuracy: When traditional obstacle detection algorithms are faced with obstacle detection in the complex scenarios of open-pit mines, due to their limited ability to extract features of small targets, occluded targets, and targets of different scales, the detection accuracy is difficult to meet the actual requirements. For example, when detecting small falling rocks in a mine pit or construction machinery partially obscured by dust, there are prone to missed detections or false detections. Existing deep learning-based obstacle detection methods often lack targeted feature optimization when dealing with multi-category obstacles in open-pit mine scenarios, resulting in insufficient feature differentiation between various obstacles, thereby affecting the overall detection accuracy. For example, it is difficult to accurately distinguish the categories of pedestrians and small mining trucks and precisely locate them when they are at a long distance and the image is blurred.

[0054] 2) Analysis from the perspective of detection speed: In order to pursue excellent detection accuracy, some high-precision object detection models have constructed extremely complex network structures, which has led to a large number of model parameters and extremely high computational complexity, and cannot meet the real-time requirements in the actual obstacle detection for autonomous construction machinery driving in open-pit mines. For example, although some deep networks based on multi-scale feature fusion have certain advantages in detection accuracy, the time required to detect one frame of image is too long to provide effective obstacle warning information for autonomous construction machinery in a timely manner. When traditional deep learning-based image processing technologies are applied in open-pit mine scenarios, the data preprocessing and model inference processes are not optimized specifically, resulting in a large amount of redundant calculations, which seriously affects the detection speed. For example, when processing high-resolution mine images, preprocessing operations such as image scaling and cropping are not carried out reasonably, causing the model to spend too much time on calculating irrelevant regions when processing image data. When some existing obstacle detection algorithms are deployed on open-pit mine driving equipment, due to insufficient consideration of hardware resource limitations, they cannot make full use of the parallel computing power of the hardware, resulting in a significant reduction in detection speed. For example, on some in-vehicle embedded devices with limited computing power, the algorithm is not optimized for the GPU or specific hardware architecture, resulting in low model operation efficiency.

[0055] 3) From the perspective of practical applications: In the actual operation scenarios of open-pit mines, there are strict and multi-dimensional requirements for the obstacle detection technology of autonomous operation construction machinery. On the one hand, accuracy is the core element to ensure safe operation. In open-pit mines, whether it is construction machinery such as large mining trucks, loaders, excavators, or scattered pedestrians, irregular potholes, and unpredictable falling rocks, they must be accurately identified. Even a tiny detection error may lead to serious collision accidents, damage expensive equipment, and endanger the lives of operating personnel. For example, on narrow transportation roads, if the positions of mining trucks and other engineering vehicles cannot be accurately distinguished, it is extremely easy to cause scratches or even more serious collisions, resulting in traffic jams on the transportation line and equipment shutdowns, bringing huge losses to mine production. On the other hand, speed is also indispensable for the efficient operation of open-pit mine operations. The operation process of open-pit mines is compact, and construction machinery needs to quickly shuttle through complex roads to ensure the continuity of ore mining, transportation, and other links. If the obstacle detection speed is too slow, the system will not be able to provide accurate detection results in the face of a rapidly changing driving environment, making it difficult for the autonomous driving system or operators to react in time. For example, during the driving of a high-speed mining truck, if the detection algorithm has a high delay, it may be too late to brake or avoid when a suddenly appearing falling rock is detected ahead, resulting in catastrophic consequences. However, the existing technologies often focus on one at the expense of the other between accuracy and speed. Some high-precision detection algorithms, due to the complex model and huge consumption of computing resources, result in slow detection speeds and cannot meet the requirements of real-time operations in open-pit mines; while some algorithms that pursue fast detection, due to excessive simplification of the model structure, greatly reduce the accuracy, frequently resulting in false detections and missed detections, posing a huge hidden danger to mine safety.

[0056] Therefore, to meet the requirements of accurate and real-time obstacle detection for autonomous operation construction machinery when driving on unstructured roads in open-pit mines and ensure the safety and efficiency of mine operations, the embodiments of the present invention disclose an obstacle detection method for autonomous operation construction machinery, including the following steps:

[0057] S1. Construct an open-pit mine driving obstacle data set and perform preprocessing, and divide the preprocessed data set into a training set, a validation set, and a test set according to a preset ratio;

[0058] S2. Improve the YOLOv8 network and build an obstacle detection model;

[0059] S3. Use the training set and the validation set to train the obstacle detection model until the loss function converges to obtain an optimized obstacle detection model;

[0060] S4. Perform obstacle detection on the test set through the optimized obstacle detection model to obtain detection results.

[0061] Reference Figure 1 Shown in the flowchart, this embodiment is based on YOLOv8. By embedding an attention mechanism, improving the feature fusion method of the backbone network, using a more suitable dynamic head network, a loss function more suitable for the actual open-pit mine environment, and means of model compression, it effectively improves its detection accuracy and reduces the number of parameters and computational complexity, enabling it to meet the real-time requirements in engineering practical applications while having high object detection performance, thus providing reliable technical support for the detection of driving obstacles in open-pit mines and assisting the safe and efficient driving of autonomous operation construction machinery in open-pit mines.

[0062] Next, the Figure 1 shown process will be elaborated in detail to further understand the technical solution of the present invention.

[0063] I. Construct a dataset

[0064] Reference Figure 3 Shown in the dataset construction flowchart, the specific steps of S1 provided in this embodiment are as follows:

[0065] S11. Collect image data of driving obstacles of autonomous operation construction machinery in open-pit mines, eliminate unqualified images, and screen qualified images to obtain a dataset of driving obstacles in open-pit mines, as Figure 2 shown;

[0066] S12. Use the image annotation tool LabelImg to perform yolo format annotation on 7 types of obstacles in the images of the driving obstacle dataset in open-pit mines, and save the annotation information to a txt file corresponding to the image name; among them, the annotated obstacle types include: car, pedestrian, loader, excavator, mining truck, falling rock, pothole;

[0067] S13. Divide the annotated images and the txt files saving the annotation information into a training set, a validation set, and a test set according to a preset ratio of 8:1:1;

[0068] S14. Add rain noise, fog noise, Gaussian noise to the images in the training set, and perform flipping, rotation, and random cropping processing, obtain an enhanced dataset and input the enhanced dataset into the obstacle detection model for model training, which can simulate the harsh environment of real open-pit mines and improve the generalization ability of the dataset.

[0069] II. Improve the YOLOv8 model

[0070] As Figure 4 shown is the overall structure diagram of the improved YOLOv8 proposed in this embodiment. It can be seen that in S2, the improvement method of the YOLOv8 network is specifically as follows:

[0071] S21. Embed the SE attention mechanism into the backbone network of YOLOv8, enabling the network to pay more attention to the important features related to obstacles and reducing the interference of irrelevant information. Therefore, in the case of occlusion, multiple targets, small targets, etc. in the open-pit mine scenario, the targets can be more accurately located and identified.

[0072] S22. In the feature fusion of the backbone network, replace the original SPPF module with the improved SPPF_FPSC module, enabling the network to adaptively integrate the feature information of obstacles at different scales, thus possessing excellent feature extraction capabilities.

[0073] S23. Use a suitable dynamic head network (Dyhead) at the output end to enhance the learning ability of obstacles of different sizes in the open-pit mine.

[0074] S24. Adopt WIoU as the bounding box loss function, taking into account the problems of complex backgrounds of obstacle images in open-pit mines, uneven scales and distributions of obstacles. Specifically, WIoU adopts a dynamic non-monotonic focusing mechanism, uses outlier degree instead of IoU to evaluate the quality of anchor boxes, and proposes a gradient gain allocation strategy. This strategy not only reduces the competitiveness of high-quality anchor boxes but also reduces the harmful gradients generated by low-quality samples, enabling WIoU to focus on anchor boxes of ordinary quality and improve the overall performance of the detection algorithm.

[0075] S25. Prune redundant weight parameters using a pruning algorithm based on LAMP scores, adjust the model structure, reduce the computational load, speed up the model running speed, improve real-time performance, and make it more conducive to deployment on the autonomous operation engineering machinery equipment side.

[0076] (1) Embedding attention mechanism method

[0077] During the driving process of autonomous construction machinery in the open-pit mine scenario, obstacle detection faces many challenges. Due to the complex mine environment, there are obstacles of various shapes, sizes, and materials, such as falling rocks, potholes, other construction machinery, etc., which require high feature extraction capabilities from the network model. As Figure 5 shown in the schematic diagram of the SE attention mechanism module embedded in the YOLOv8 backbone network in this embodiment, it has a significant effect:

[0078] First of all, the SE attention mechanism can evaluate and weight the importance of feature channels. In a complex background like an open-pit mine, the contribution degrees of feature information in different channels to obstacle detection are different. Through the SE attention mechanism, the network can adaptively learn and strengthen the channels that play important roles in identifying key features of various obstacles (such as the texture of falling rocks, the contour of construction machinery, etc.), and suppress the influence of noise or irrelevant feature channels, thereby enhancing the effectiveness and representativeness of features and enabling the network to more accurately locate and identify obstacles.

[0079] Secondly, it helps to enhance the model's detection ability for obstacles of different scales. The scales of obstacles in open-pit mines vary widely, ranging from small falling rocks to large autonomous construction machinery. The SE attention mechanism can optimize the feature representation on feature maps at different levels by recalibrating the feature channels, enabling the network to focus more on obstacle-related information when processing multi-scale features, avoiding feature loss or confusion caused by scale changes, thereby improving the detection accuracy for obstacles of different sizes and reducing the missed detection of small target obstacles.

[0080] Furthermore, the SE attention mechanism can improve the robustness of the model. The lighting conditions in the mine environment are complex and changeable, such as strong light irradiation, shadow areas, etc. These factors may interfere with the accuracy of feature extraction. The SE attention mechanism can reduce the feature fluctuations caused by changes in external factors such as lighting by emphasizing key feature channels, enabling the network to stably extract reliable obstacle features under different lighting conditions, enhancing the model's resistance to complex environmental interference, ensuring that autonomous construction machinery can more reliably detect obstacles during driving, avoiding collision accidents, and improving the safety and efficiency of operations.

[0081] (2) Improve the SPPF_FPSC module

[0082] The operating environment of open-pit mines is extremely complex and harsh, filled with various obstacles of different shapes, sizes, and rich textures, which poses strict requirements on the feature extraction ability of obstacle detection models. The traditional SPPF module is unable to cope with such complex features. It is difficult to fully cover and capture the unique features of these obstacles during the feature extraction process, resulting in the detection accuracy being difficult to reach the ideal level. In practical applications, false detections and missed detections are prone to occur, and it cannot provide accurate and reliable visual perception information for autonomous construction machinery in open-pit mines, seriously affecting driving safety and operation efficiency.

[0083] As Figure 6 shown in the schematic diagrams of the SPPF module and the improved SPPF_FPSC module, the SPPF_FPSC module aims to significantly enhance the feature extraction efficiency, specifically:

[0084] a) In the module initialization stage, the number of channels of the input data is reasonably divided, and the number of hidden channels is determined through a specific calculation method; for example, the input channel number c1 is halved (c = c1 / 2). The number of hidden channels is like a bridge, laying a solid foundation for a series of subsequent complex and orderly multi-level feature transformation operations, ensuring that features at different levels can smoothly transmit and interact with information on the appropriate channel dimension.

[0085] One of the core components of the SPPF_FPSC module is the shared convolutional layer (share_conv), which has carefully set convolutional kernel parameters. The size of the convolutional kernel is 3×3. This size of convolutional kernel can, while extracting local features, take into account the information association of surrounding pixels to a certain extent, providing strong support for the accuracy and integrity of feature extraction. More importantly, it has a flexibly adjustable dilation rate setting, and these dilation rate parameters constitute an important feature extraction strategy library.

[0086] b) During the forward propagation process of the module, its operation process is rigorous and efficient. The specific operation process is as follows:

[0087] First, the cv1 convolutional layer is used to perform preliminary in-depth processing on the input features. As the first process of feature processing, the cv1 convolutional layer uses a 1×1 convolutional kernel for convolution operation. Its main role is to integrate and optimize the channel information of the input features, adjust the distribution of the feature map, so that the features can better adapt to the subsequent processing process in the channel dimension. After this operation, the basic features are obtained, laying a solid foundation for more detailed and multi-dimensional feature extraction work;

[0088] Immediately afterwards, it enters the core multi-dilation rate convolution link. For different pre-set dilation rates, the shared convolutional layer (share_conv) is repeatedly used for convolution operation: when a smaller dilation rate (such as 1) is used for convolution, its receptive field is relatively small, and it can focus on the local detailed information in the feature map. Just like in an open-pit mine scenario, for the detailed texture features of small-sized obstacles nearby, such as the surface texture of fallen rocks and the structure of construction machinery, it can accurately capture and extract them; while when the dilation rate increases (such as 3 or 5), the receptive field expands accordingly. At this time, the convolution operation can span a larger area, thus effectively capturing the overall contour features of large-sized obstacles in the distance, such as the overall shape of large construction machinery and the general contour of pedestrians in the distance. In this way, the convolution operations with different dilation rates cooperate with each other and work together to comprehensively mine the rich information in the feature map from multiple scales and multiple perspectives without changing the size of the feature map, and collect and store the features extracted under different receptive fields in sequence. After the multi-dilation rate convolution operation is completed, these features from different receptive fields are spliced and integrated in the channel dimension. This process can organically combine the multi-scale feature information scattered in different channels to form an overall containing rich feature information;

[0089] Finally, the cv2 convolutional layer is used to integrate and optimize the concatenated features, and the feature representation of the open-pit mine obstacle features is output. Among them, the cv2 convolutional layer also uses a 1×1 convolutional kernel, which can perform deep fusion and refinement on the multi-channel features after concatenation, remove redundant information, and strengthen key features, so as to output a more discriminative feature representation that can accurately represent the open-pit mine obstacle features.

[0090] (3) Dynamic head network

[0091] During the driving process of construction machinery, various types of obstacles will be encountered. Among them, compared with obstacles such as pedestrians and falling rocks, the sizes of construction vehicles (such as loaders, excavators, and mining trucks) show extremely significant differences. Moreover, as the camera shooting angle changes continuously, when capturing the same obstacle target, the shape and position it presents will also change significantly. Given these complex challenging situations, for the obstacle detection model, its head network must have a certain spatial perception ability to effectively respond. The original detection head of YOLOv8 adopts a single-scale prediction structure, and this structure has obvious limitations when dealing with multi-scale targets. Due to its lack of global vision and relying only on a limited number of parameters for prediction, it is difficult to deeply and comprehensively mine the spatial structure information in the features, and thus it is unable to accurately identify obstacles of different scales in the multi-scale obstacle detection task.

[0092] As Figure 7 shown, in sharp contrast, there is the Dyhead (Dynamic head) module. This module has excellent characteristics. It can provide size perception, task perception, and spatial perception capabilities simultaneously. Through these unique perception capabilities, the detection performance of the model for obstacles of different scales, different shapes, and small targets has been greatly improved in a complex and changing environment. The Dyhead module also has the ability to dynamically adjust the weights of different feature layers, performs well in extracting multi-scale features, and can more efficiently fuse context information, thus significantly enhancing the recognition ability of multi-scale obstacle targets and providing more accurate and reliable technical support for obstacle detection during the driving process of construction machinery.

[0093] (4) Loss function

[0094] In the task of detecting obstacles during driving in open-pit mines, the constructed dataset faces numerous challenges. Due to the special environment of open-pit mines, the background of obstacle images is extremely complex, containing a large number of interfering factors such as rockfall textures, pothole shadows, and topographical changes under different lighting conditions. At the same time, the scales of obstacles vary greatly, ranging from tiny rockfalls, pedestrians to huge engineering vehicles, etc., and the distribution of these obstacles in the images is irregular and uneven. Traditional bounding box loss functions, such as IoU (Intersection over Union) and its variants, have limitations when dealing with such complex scenarios. IoU only simply considers the overlap ratio between the predicted box and the target box. For situations with large scale differences and complex backgrounds, it may not be able to accurately guide the model to learn the precise position of the bounding box.

[0095] However, the WIoU (Weighted Intersection over Union) loss function adopted in this embodiment demonstrates unique advantages. WIoU assigns different weights to targets of different scales. When calculating the loss, it can pay more attention to obstacles with smaller scales and those that are difficult to detect, preventing them from being masked by the losses of large-scale obstacles during the training process. For obstacles such as small-sized rockfalls and pedestrians that are abundant in open-pit mines, WIoU can ensure that the model gives them sufficient attention when learning bounding box regression, improving the accuracy of small target detection. In addition, WIoU can also effectively handle the occlusion problem under complex backgrounds. When there is mutual occlusion between obstacles or they are partially occluded by the mine background, WIoU, through its weighting mechanism and comprehensive consideration of the geometric relationship of the bounding boxes, can more reasonably evaluate the difference between the predicted box and the target box, enabling the model to learn how to accurately locate the bounding box under complex occlusion situations during the training process, thereby reducing detection errors caused by occlusion and improving the accuracy and robustness of the entire model for detecting driving obstacles in open-pit mines.

[0096] (5) Model Compression

[0097] Since deep neural network models generally require a large amount of computing resources and storage space, there are problems of low detection speed and poor real-time performance when deployed on mobile devices. In order to meet the requirements of construction machinery for real-time obstacle detection during driving operations, the redundant weights of the model are pruned to reduce the number of model parameters and the amount of computation as much as possible while meeting the accuracy requirements.

[0098] The pruning algorithm based on the LAMP (Layer-Adaptive Magnitude-based Pruning) score is a novel optimization method for neural network models. By calculating the squared magnitude of each target connection weight and normalizing it to the sum of the squared magnitudes of all "surviving weights" in the same layer, the LAMP score for each connection is obtained. The LAMP pruning method first identifies the connections that contribute less to the model performance (i.e., the connections with smaller weight magnitudes), and then, according to a preset threshold, prunes those connections with LAMP scores lower than the preset threshold. This method can adaptively determine which connections to prune based on the weight distribution of each layer, helping to reduce the number of model parameters and computational volume while minimizing the impact on the model performance, thereby achieving model compression and optimization. Its pruning method is as Figure 8 shown.

[0099] Corresponding to the foregoing method, an embodiment of the present invention also provides an obstacle detection system for autonomous operation construction machinery, which is used to Figure 1 For the specific implementation of the method in, an obstacle detection system for autonomous operation construction machinery provided by an embodiment of the present invention can be applied to computer terminals or various mobile devices, and specifically includes:

[0100] A dataset construction module, which is used to construct an open-pit mine driving obstacle dataset and perform preprocessing, and divide the preprocessed dataset into a training set, a validation set, and a test set according to a preset ratio;

[0101] A model construction module, which is used to improve the YOLOv8 network and build an obstacle detection model;

[0102] A model training module, which is used to train the obstacle detection model through the training set and the validation set until the loss function converges, and obtain an optimized obstacle detection model;

[0103] A detection module, which is used to perform obstacle detection on the test set through the optimized obstacle detection model to obtain detection results.

[0104] As Figure 9 shown is the channel comparison diagram before and after pruning the improved model. It can be seen from the original (base) and prune ratios of each layer structure in the figure that after optimizing the improved YOLOv8 model structure with the layer-adaptive amplitude-based pruning method, a large amount of parameter redundancy is reduced. As Figure 10 shown is the fine-tuning process after pruning the improved model. It can be seen that each performance index is basically the same as that of the improved model performance index. As Figure 11The performance metric mAP comparison between the final model and the baseline model YOLOv8 is shown. It can be seen that after the training becomes stable, the final model has an improvement compared to the baseline model YOLOv8. As Figure 12 The figure shows the comparison chart of the visualization results. It can be seen that while ensuring high detection accuracy, the final model greatly reduces the missed detection rate, which is beneficial to ensuring the safety of autonomous operation construction machinery.

[0105] In summary, the present invention proposes an obstacle detection method and system for autonomous operation construction machinery. An open-pit mine driving obstacle dataset is constructed through network collection and on-site acquisition; an SE attention mechanism is embedded in the YOLOv8 backbone network to make the network pay more attention to the important features related to obstacles and reduce the interference of irrelevant information, enabling more accurate target localization and recognition; in the feature fusion of the backbone network, the improved module SPPF_FPSC is used to replace the original SPPF module, enabling the network to adaptively integrate the feature information of obstacles of different scales, thus having excellent feature extraction capabilities; at the output end, a more suitable dynamic head network (Dyhead) is used to strengthen the learning ability for obstacles of different sizes in open-pit mines; considering the problems of complex backgrounds of open-pit mine obstacle images, uneven scales and distributions of obstacles, the WIoU is used as the bounding box loss function, which can focus on the anchor boxes of ordinary quality and improve the overall performance of the detection algorithm; a pruning algorithm based on the LAMP score is used to prune redundant weight parameters, accelerating the model running speed and making it more conducive to being deployed on the autonomous operation construction machinery equipment side.

[0106] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0107] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting obstacles in autonomous engineering machinery, characterized in that: The following steps are involved: S1. Construct and preprocess an open-pit mine driving obstacle dataset, and divide the preprocessed dataset into a training set, a validation set, and a test set according to a preset ratio; S2. Improve the YOLOv8 network and build an obstacle detection model; S3. Use the training set and the validation set to train the obstacle detection model until the loss function converges to obtain an optimized obstacle detection model. S4. Perform obstacle detection on the test set using the optimized obstacle detection model to obtain detection results.

2. The obstacle detection method for autonomous operation engineering machinery according to claim 1, characterized in that: The specific steps of S1 are: S11, collecting image data of obstacles to autonomous construction machinery in open-pit mines, removing unqualified images, screening qualified images, and obtaining an open-pit mine obstacle data set; S12. Use the image annotation tool LabelImg to annotate various obstacles in the open-pit mine driving obstacle dataset image in YOLO format, and save the annotation information to a txt file corresponding to the image name; S13, dividing the annotated images and the txt files storing the annotated information into a training set, a validation set, and a test set according to a preset ratio of 8:1:

1.

3. The obstacle detection method for autonomous operation engineering machinery according to claim 2, characterized in that: In S12, the marked obstacle types include: cars, pedestrians, loaders, excavators, mining trucks, fallen rocks, and potholes.

4. The obstacle detection method for autonomous operation engineering machinery according to claim 1, characterized in that: S1 also includes: S14, adding rain noise, fog noise, and Gaussian noise to the images in the training set and performing flipping, rotation, and random cropping to obtain an enhanced data set and input the enhanced data set into the obstacle detection model for model training.

5. The obstacle detection method for autonomous operation engineering machinery according to claim 1, characterized in that: In S2, the improvement methods of the YOLOv8 network are as follows: S21. Embed the SE attention mechanism in the YOLOv8 backbone network; S22. In the feature fusion of the backbone network, the improved SPPF_FPSC module is used to replace the original SPPF module; S23, use a suitable dynamic head network at the output; S24, using WIoU as the bounding box loss function; S25. Use the LAMP score-based pruning algorithm to prune redundant weight parameters and adjust the model structure.

6. The obstacle detection method for autonomous operation engineering machinery according to claim 5, characterized in that: In S22, the improved SPPF_FPSC module enables the network to adaptively integrate the feature information of obstacles of different scales, specifically: In the module initialization stage, the number of channels of the input data is reasonably divided, and the number of hidden channels is determined by a specific calculation method; In the forward propagation process of the module, the cv1 convolution layer is used to integrate and optimize the channel information of the input features, adjust the distribution of feature mapping, and obtain the basic features; For different pre-set expansion rates, the shared convolutional layer is repeatedly used to perform convolution operations, and the features extracted under different receptive fields are spliced ​​and integrated in the channel dimension; the spliced ​​features are integrated and optimized through the cv2 convolutional layer to output the feature representation of the open-pit mine obstacle features.

7. The obstacle detection method for autonomous operation engineering machinery according to claim 5, characterized in that: In S25, the LAMP score-based pruning algorithm calculates the square amplitude of each target connection weight and normalizes it to the sum of the square amplitudes of all surviving weights in the same layer to obtain the LAMP score of each connection; the connections with LAMP scores below the preset threshold are pruned to complete the compression and optimization of the model.

8. A system using the obstacle detection method for autonomous engineering machinery as described in any one of claims 1 to 7, characterized in that: include: The data set construction module is used to construct and preprocess the open-pit mine driving obstacle data set, and divide the preprocessed data set into a training set, a validation set, and a test set according to a preset ratio; Model building module, used to improve the YOLOv8 network and build an obstacle detection model; The model training module is used to train the obstacle detection model through the training set and the validation set until the loss function converges to obtain the optimized obstacle detection model; The detection module is used to perform obstacle detection on the test set through the optimized obstacle detection model to obtain the detection results.

Images