A mine electric locomotive anti-collision method and system based on edge deployment

By deploying anti-collision visual detection model on the edge of the mine motor locomotive, combining image acquisition and position measurement units, the problems of poor flexibility and high error detection rate in complex environments are solved, and more efficient obstacle detection and motors are achieved The vehicle operates anti-collision, reducing the risk of accidents.

CN118711160BActive Publication Date: 2024-12-06HUNAN INSTITUTE OF ENGINEERING +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411206830.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-06
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

Traditional mine motor locomotive anti-collision methods have poor flexibility and high error detection rates in complex mine environments. Due to the humid and dusty environments under the mine, it is difficult to effectively detect inevitable obstacles on the track, resulting in frequent motor locomotive accidents.

Method used

Adopting an anti-collision method based on edge deployment, by setting up an image acquisition unit and a position measurement unit on the motor vehicle, presetting and training of anti-collision visual detection model, the model is optimized using transfer learning method, combining backbone network, feature fusion network and Output detection head, detect obstacles in real time and control the motor vehicle to avoid collisions.

Benefits of technology

It improves the accuracy and safety of obstacle detection in complex environments of motor vehicles, reduces the false detection rate, enhances the anti-collision ability of motor vehicles, and reduces the occurrence of mine accidents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118711160B_ABST
    Figure CN118711160B_ABST
Patent Text Reader

Abstract

The present invention discloses a mine electric locomotive anti-collision method and system based on edge deployment, wherein an image acquisition unit and a position measurement unit are arranged on the mine electric locomotive, an anti-collision visual detection model is preset and trained, and the trained anti-collision visual detection model is edge-deployed on the mine electric locomotive, the image acquisition unit acquires images of the mine electric locomotive when it is running on the track in real time and inputs them into the trained anti-collision visual detection model for processing, and obtains a bounding box image with obstacle categories and confidence levels, and the position measurement unit obtains the distance information between the obstacle and the mine electric locomotive; the control module receives and processes the bounding box image with obstacle categories and confidence levels, and the distance information between the obstacle and the mine electric locomotive, and controls the anti-collision operation of the mine electric locomotive. The method and system sense the surrounding environment through intelligent device detection technology, and use artificial intelligence algorithms to detect targets, thereby controlling the mine electric locomotive to achieve anti-collision operation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of anti-collision of mine electric locomotives, and in particular to an anti-collision method and system for mine electric locomotives based on edge deployment. Background Art

[0002] As one of the main equipment for mine transportation, electric locomotives are responsible for transporting personnel, materials, ores and related equipment. With the rapid development of mine rail transportation, electric locomotive accidents are common. In the past five years, one-third of mine accidents were caused by electric locomotives, and most of the accidents were caused by the electric locomotive drivers' improper operation and failure to identify the intruding obstacles in front and behind the vehicle. Safe and reliable mine electric locomotive anti-collision methods and systems can help reduce mine accidents, protect the lives of miners, and maintain the normal production of enterprises. At the same time, intelligent transportation in mines is also an important part of the development of smart mines. At present, traditional anti-collision methods are mainly based on mechanical devices and sensor detection, and can only be fixed in a specific position. They are easily affected by the humid and dusty environment in the mine, with poor flexibility and high false detection rate. Due to the complex and changeable mine environment, the communication cost and bandwidth consumption in the mine are high. At the same time, obstacles such as rocks, coal blocks, workers, and building materials will inevitably appear on the electric locomotive tracks, which brings a series of challenges to the safe driving of mine electric locomotives. Mine electric locomotives need to detect surrounding obstacles while maintaining safe driving to prevent collisions with obstacles when driving too fast.

[0003] The mine electric locomotive collision avoidance system uses sensors to perceive the surrounding environment and uses artificial intelligence algorithms to detect targets. It then determines whether there is a collision risk based on the type of obstacle and the positional relationship between the locomotive and the obstacle, and controls the locomotive to make corresponding driving actions. Summary of the invention

[0004] The present invention provides a mine electric locomotive anti-collision method and system based on edge deployment to solve the technical problems mentioned in the background technology.

[0005] On the one hand, the present invention provides a mine electric locomotive anti-collision method based on edge deployment, the method comprising the following steps:

[0006] S1. An image acquisition unit and a position measurement unit are arranged on a mine electric locomotive;

[0007] S2, preset an anti-collision visual detection model, use the transfer learning method to train the anti-collision visual detection model, and deploy the trained anti-collision visual detection model edge on the mine electric locomotive; the anti-collision visual detection model in S2 includes a backbone network, a feature fusion network and an output detection head connected in sequence, the backbone network receives the image to be detected when the mine electric locomotive is running on the track and performs feature extraction, and outputs several feature maps of different scales, the feature fusion network receives several feature maps of different scales and performs feature fusion and optimization, and outputs the same number of fused feature maps as the feature maps of several different scales, and the output detection head maps the fused feature map to the preset target detection frame, and generates the same number of boundary box images with obstacle categories and confidence as the feature maps of several different scales.

[0008] S3, the image acquisition unit collects images of the mine electric locomotive running on the track in real time, inputs the images into the anti-collision visual detection model trained and deployed on the edge of the mine electric locomotive for processing, obtains a bounding box image with obstacle category and confidence, and the position measurement unit obtains the distance information between the obstacle on the mine track and the mine electric locomotive;

[0009] S4, the control module receives and processes the bounding box image with the obstacle category and confidence level, and the distance information between the obstacle and the mine electric locomotive, and controls the anti-collision operation of the mine electric locomotive.

[0010] Preferably, the image acquisition unit in S1 is specifically two zoom cameras, which are respectively arranged on the front and rear sides of the top of the mine electric locomotive, and the position measurement unit specifically includes an ultrasonic radar and a laser radar, which are both arranged in the front center of the mine electric locomotive.

[0011] Preferably, the backbone network includes a first CBS module, a second CBS module, a third CBS module, a fourth CBS module, a first ELAN module, a first MP module, a second ELAN module, a second MP module, a third ELAN module, a third MP module and a C3ST module connected in sequence. The backbone network receives the image to be detected when the mine electric locomotive is running on the track and performs feature extraction, and outputs the first feature map, the second feature map and the third feature map through the second ELAN module, the third ELAN module and the C3ST module.

[0012] Preferably, the feature fusion network includes a SPPCS PC-L module, a fifth CBS module, a first Upsample module, a fourth ELAN module, a sixth CBS module, a second Upsample module, a first ELAN-E module, a fourth MP module, a second ELAN-E module, a fifth MP module and a third ELAN-E module connected in sequence, and also includes a seventh CBS module and an eighth CBS module. The seventh CBS module, the eighth CBS module and the SPPCS PC-L module of the feature fusion network respectively receive the first feature map, the second feature map and the third feature map output by the backbone network and process them, splice the feature output by the seventh CBS module with the feature output by the second Upsample module, and input them into the first ELAN-E module for processing, and output a first fused feature map, splice the feature output by the eighth CBS module with the feature output by the first Upsample module, and input them into the fourth ELAN module for processing, splice the feature output by the fourth ELAN module with the feature output by the fourth MP module, and input them into the second ELAN-E module for processing, and output a second fused feature map, splice the feature output by the SPPCS PC-L module with the feature output by the fifth MP module, and input them into the third ELAN-E module for processing, and output a third fused feature map.

[0013] Preferably, in S2, a transfer learning method is used to train the anti-collision visual detection model, and the trained anti-collision visual detection model is edge-deployed on a mine electric locomotive, specifically including the following:

[0014] S21, using a data set with the same image features as an initial training data set, using the initial training data set to pre-train the anti-collision visual detection model for the source task, and then using the mine track obstacle data set to formally train the pre-trained anti-collision visual detection model for the target task, to obtain a trained anti-collision visual detection model;

[0015] S22, performing sparse training on the scaling factor of the BN layer in the trained anti-collision visual detection model using L1 regularization, and performing pruning processing on the trained anti-collision visual detection model according to the sparse training result to obtain a compressed anti-collision visual detection model;

[0016] S23, presetting the number of iterative training times, and performing parameter fine-tuning training on the compressed anti-collision visual detection model within the preset number of iterative training times to obtain a fine-tuned anti-collision visual detection model;

[0017] S24. Deploy the fine-tuned anti-collision visual detection model in the edge computing platform of the mine electric locomotive.

[0018] Preferably, the control module in S4 includes a storage unit, a computing unit and a control unit connected in sequence, and S4 specifically includes the following:

[0019] S41, the storage unit receives and stores the bounding box image with the obstacle category and confidence level, and the distance information between the obstacle and the mine electric locomotive;

[0020] S42, the calculation unit receives the distance information between the obstacle and the mine electric locomotive and uses the Kalman filter method to perform data fusion to obtain the final distance information;

[0021] S43, the control unit determines the collision risk according to the bounding box image with the obstacle category and confidence level and the final distance information, and controls the mine electric locomotive to achieve anti-collision operation according to the collision risk.

[0022] Preferably, the distance information between the obstacle and the mine electric locomotive in S41 includes ultrasonic radar distance information and laser radar distance information, and S42 specifically includes the following:

[0023] S421, the calculation unit receives ultrasonic radar distance information and laser radar distance information between the obstacle and the mine electric locomotive;

[0024] S422, using the ultrasonic radar distance information in the prediction phase, and using the ultrasonic radar distance information as a priori state estimation value of the observation state vector;

[0025] S423, using the laser radar distance information for the correction and update phase, using the laser radar distance information to correct and update the prior state estimate of the observation state vector to obtain a final state vector, and using the final state vector as the final distance information.

[0026] On the other hand, the present invention provides a mine electric locomotive anti-collision system, which adopts the above-mentioned mine electric locomotive anti-collision method based on edge deployment to realize the anti-collision operation of the mine electric locomotive. The system includes a mine electric locomotive, which is provided with an image acquisition unit, a position measurement unit, a control module and an edge computing device. The image acquisition unit, the position measurement unit and the control module are respectively connected to the edge computing device, and the anti-collision visual detection model is deployed in the edge computing device, wherein:

[0027] The image acquisition unit is used to acquire images containing obstacles when the mine electric locomotive runs along the track;

[0028] The position measurement unit is used to collect the distance information between the obstacle and the mine electric locomotive;

[0029] The anti-collision visual detection model is used to receive and process images containing obstacles to obtain bounding box images with obstacle categories and confidence levels.

[0030] The control module is used to receive and store bounding box images with obstacle categories and confidence levels, and distance information between obstacles and mine locomotives, calculate the final distance information and collision risk, and control the mine locomotive to achieve anti-collision operation based on the collision risk.

[0031] Preferably, the control module includes a storage unit, a computing unit and a control unit connected in sequence, the storage unit is used to store a bounding box image with obstacle category and confidence, and the distance information between the obstacle and the mine electric locomotive; the computing unit is used to receive the distance information between the obstacle and the mine electric locomotive and use the Kalman filtering method to perform data fusion to obtain the final distance information; the control unit is used to receive a bounding box image with obstacle category and confidence, the final distance information between the obstacle and the mine electric locomotive and calculate the collision risk, and control the mine electric locomotive according to the collision risk to achieve anti-collision operation.

[0032] The above-mentioned method and system for anti-collision of mine electric locomotives based on edge deployment, an image acquisition unit and a position measurement unit are set on the mine electric locomotive, an anti-collision visual detection model is preset and trained, and the trained anti-collision visual detection model is edge-deployed on the mine electric locomotive. The image acquisition unit collects images of the mine electric locomotive when it is running on the track in real time and inputs them into the anti-collision visual detection model trained by edge deployment on the mine electric locomotive for processing, and a bounding box image with obstacle category and confidence is obtained. The position measurement unit obtains the distance information between the obstacle and the mine electric locomotive; the control module receives the bounding box image with obstacle category and confidence, and the distance information between the obstacle and the mine electric locomotive and processes it, and controls the anti-collision operation of the mine electric locomotive. The method and system perceive the surrounding environment through intelligent device detection technology, and use algorithms to detect targets, and control the mine electric locomotive to achieve anti-collision operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a mine electric locomotive anti-collision method based on edge deployment in one embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of the installation positions of the image acquisition unit and the position measurement unit on the mine electric locomotive in one embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of a network structure of an anti-collision visual detection model in one embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of the network structure of C3ST in one embodiment of the present invention;

[0037] Figure 5 is a schematic diagram of the network structure of ELAN-E in one embodiment of the present invention;

[0038] Description of reference numerals:

[0039] 1. Zoom camera; 2. Ultrasonic radar; 3. LiDAR. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0041] In one embodiment, see Figure 1 , a mine electric locomotive anti-collision method based on edge deployment, specifically comprising the following steps:

[0042] S1. An image acquisition unit and a position measurement unit are set on the mine electric locomotive.

[0043] Furthermore, the image acquisition unit in S1 is specifically two mining explosion-proof variable-zoom cameras, which are respectively arranged on the front and rear sides of the top of the mine electric locomotive. The position measurement unit specifically includes an ultrasonic radar and a laser radar, which are both arranged in the front center of the mine electric locomotive.

[0044] Specifically, see Figure 2 , install the laser mine 3 and the ultrasonic radar 2 in the middle position in front of the mine electric locomotive, and the ultrasonic radar 2 is set directly below the laser mine 3, and install two zoom cameras 1 on the front and rear sides of the top of the locomotive respectively, and install the zoom camera 1 at a downward tilt of 30° in the vertical direction to better obtain the obstacle information in front and rear of the mine electric locomotive track at the corresponding viewing angle.

[0045] S2. Preset an anti-collision visual detection model, use the transfer learning method to train the anti-collision visual detection model, and deploy the trained anti-collision visual detection model edge on the mine electric locomotive.

[0046] Furthermore, the anti-collision visual detection model in S2 includes a backbone network, a feature fusion network, and an output detection head connected in sequence. The backbone network receives the image to be detected when the mine electric locomotive is running on the track and performs feature extraction, and outputs several feature maps of different scales. The feature fusion network receives several feature maps of different scales and performs feature fusion and optimization, and outputs the same number of fused feature maps as the feature maps of several different scales. The output detection head maps the fused feature maps to a preset target detection frame, and generates the same number of bounding box images with categories and confidence as the feature maps of several different scales.

[0047] Figures 3 to 5 The relevant English abbreviations are explained as follows:

[0048] W-MSA (Windows Multi-head Self-Attention); SW-MSA (Shifted Windows Multi-Head Self-Attention); LN (LayerNorm); MLP (Multilayer Perceptron), CBS 1 Represents 3X3 convolution with a step size of 1, mainly responsible for feature extraction, CBS 2 Represents 3X3 convolution with a step size of 2, mainly used for downsampling, CBS 3 It represents 1X1 convolution with a step size of 2, which is mainly used to change the number of channels. Conv represents convolutional neural unit, concat represents concatenation operation, bottleneck represents bottleneck layer, and Upsample represents upsampling.

[0049] Specifically, see Figure 3 The anti-collision visual detection model uses the target detection model YOLOv7-tiny as the framework, and mainly includes three modules: backbone network, feature fusion network, and output detection head. The specific working process is as follows:

[0050] The input of the backbone network is a 3-channel image of 640x640 pixels. The 3-channel image is input into the backbone network. After a series of convolution and pooling operations, the input 3-channel image is feature extracted and three scale feature maps are output, namely the first feature map 80x80x512, the second feature map 40x40x1024, and the third feature map 20x20x1024.

[0051] The first feature map 80x80x512, the second feature map 40x40x1024, and the third feature map 20x20x1024 are input into the feature fusion network. The feature fusion network outputs three fused feature maps containing fused feature information through feature fusion and optimization operations. They are the first fused feature map 80x80x128, the second fused feature map 40x40x256, and the third fused feature map 20x20x512. The three fused feature maps containing fused feature information can help the anti-collision visual detection model better understand and utilize the input feature maps, thereby improving the accuracy and efficiency of target detection.

[0052] The first fused feature map 80x80x128, the second fused feature map 40x40x256, and the third fused feature map 20x20x512 are input into the output detection head. The output detection head uses a convolutional layer to map the three scales of the input fused feature maps to the preset target detection frame, thereby generating three different bounding boxes. Each bounding box contains location information and the corresponding category probability of the target object. The bounding boxes are confidence filtered and non-maximum suppressed. Finally, the bounding box images are output, which are the first bounding box image 80x80x255, the second bounding box image 40x40x255, and the third bounding box image 20x20x255. The bounding box images contain information such as the bounding box, target category, category probability, and location.

[0053] Furthermore, the backbone network includes a first CBS module, a second CBS module, a third CBS module, a fourth CBS module, a first ELAN module, a first MP module, a second ELAN module, a second MP module, a third ELAN module, a third MP module and a C3ST module connected in sequence. The backbone network receives the image to be detected when the mine electric locomotive is running on the track and performs feature extraction, and outputs the first feature map, the second feature map and the third feature map through the second ELAN module, the third ELAN module and the C3ST module.

[0054] Specifically, in the application scenario of mine electric locomotive transportation, different detection targets are often of different scales, and the model needs to have good robustness to multi-scale features. At the same time, due to the high speed of the locomotive, it is necessary to detect obstacle targets at a distance to reserve a certain braking distance. However, the obstacle targets in the distance are often very small in scale in the camera, which requires improving the model's sensitivity to small target obstacles. However, with the increase in model depth, after multiple convolution operations, most of the feature information of small target objects will be lost in the deep feature map, which will lead to low detection accuracy of the model, insensitivity to small target detection, and accompanied by false detection and missed detection. Therefore, the original backbone network based on the existing target detection model YOLOv7-tiny was improved, see Figure 3The backbone network in the anti-collision visual detection model includes 4 CBS modules, 3 ELAN modules, 3 MP modules and 1 C3ST module, of which 4 CBS modules are connected in sequence, and after the 4 CBS modules, 3 ELAN modules and 3 MP modules are connected alternately in sequence, and the C3ST module is located in the last layer of the backbone network. Compared with the original backbone network in the YOLOv7-tiny network model, the last layer (the 10th layer, corresponding to the number 10 in the upper right corner of the C3ST module) in the original backbone network is replaced by the C3ST module from the ELAN module. The C3ST module is specifically formed by inserting the Swin Transformer module into the C3 module to capture the local and global image context feature information of the input feature map, improve the feature extraction and generalization capabilities of the target detection model, and reduce the network calculation amount.

[0055] The backbone network in the anti-collision visual detection model takes a 3-channel image of 640x640 pixels as input and passes through the 6th layer (corresponding to Figure 3 6, which is the second ELAN module), layer 8 (corresponding to Figure 3 8, which is the third ELAN module) and layer 10 (corresponding to Figure 3 10 in (that is, C3ST module) outputs the first feature map 80x80x512, the second feature map 40x40x1024, and the third feature map 20x20x1024.

[0056] The original C3 module consists of three CONVs and a Bottleneck component. In this embodiment, the Bottleneck component is replaced by a Swin Transformer module, thereby forming a C3ST module. Figure 4 , Figure 4 Concat in the figure represents tensor concatenation operation. The Swin Transformer module is an image classification model based on the transformer architecture. It has achieved good results in image classification tasks. The characteristics of the Swin Transformer are used to improve the feature expression and generalization capabilities of the target detection model. The Swin Transformer consists of two Swin Transformer Blocks connected in sequence. These two Swin Transformer Blocks include W-MSA, LN, SW-MSA, and MLP modules, among which:

[0057] MSA is the self-attention mechanism of Swin Transformer. Its main function is to perform attention calculation on image blocks. The image input into Swin Transformer is divided into a bunch of small image blocks. W-MSA is called a windowed self-attention mechanism. It uses the attention mechanism to connect the relationship between each small window, capture and fuse local features and overall information up and down; SW-MSA is improved based on the W-MSA self-attention mechanism. It adds a sliding translation operation on the basis of W-MSA to enhance the correlation between windows. In the SW-MSA self-attention mechanism, each image block not only performs attention calculation on itself, but also the information is interacted by adjacent image blocks. This translation operation enhances the model's capture of information between image blocks and improves the contextual correlation between local features and overall features. It can further optimize the model and improve the model's ability to extract and generalize features.

[0058] LN is a layer normalization operation that can accelerate the convergence of the model;

[0059] The MLP module can transform and transfer features to improve the feature expression and modeling capabilities of the model.

[0060] The connection of two consecutive Swin Transformer blocks allows Swin Transformer to alternately use two different self-attention mechanisms. The image entering Swin Transformer is divided into small image blocks, and then features are extracted through the window self-attention mechanism (W-MSA) and the sliding window self-attention mechanism (SW-MSA). The image is normalized before and after entering the two self-attention modules. Swin Transformer's unique windowed feature extraction and window translation interactive calculation method reduce the network calculation amount of the model.

[0061] Further, the feature fusion network includes a SPPCS PC-L module, a fifth CBS module, a first Upsample module, a fourth ELAN module, a sixth CBS module, a second Upsample module, a first ELAN-E module, a fourth MP module, a second ELAN-E module, a fifth MP module and a third ELAN-E module connected in sequence, and also includes a seventh CBS module and an eighth CBS module. The seventh CBS module, the eighth CBS module and the SPPCS PC-L module of the feature fusion network respectively receive the first feature map, the second feature map and the third feature map output by the backbone network and process them, splice the feature output by the seventh CBS module with the feature output by the second Upsample module, and input them into the first ELAN-E module for processing, and output a first fused feature map, splice the feature output by the eighth CBS module with the feature output by the first Upsample module, and input them into the fourth ELAN module for processing, splice the feature output by the fourth ELAN module with the feature output by the fourth MP module, and input them into the second ELAN-E module for processing, and output a second fused feature map, splice the feature output by the SPPCS PC-L module with the feature output by the fifth MP module, and input them into the third ELAN-E module for processing, and output a third fused feature map.

[0062] Specifically, the original feature fusion network in the existing target detection model YOLOv7-tiny makes extensive use of the efficient layer aggregation network ELAN, which can control the shortest and longest gradient paths, allowing the network to learn more features and have stronger robustness. Figure 3 The feature fusion network in the anti-collision visual detection model is specifically in the original feature fusion network in the YOLOv7-tiny network model, which is located at the 21st, 24th, and 27th layers (corresponding to Figure 3 The ELAN modules in 21, 24, and 27 are replaced by ELAN-E modules. Each ELAN-E module is obtained by embedding the ECA attention mechanism into the ELAN module.

[0063] The 19th layer of the feature fusion network in the anti-collision visual detection model (that is, the seventh CBS module, corresponding to Figure 3 19), the 14th layer (that is, the eighth CBS module, corresponding to Figure 3 14) and layer 11 (that is, SPPCS PC-L module, corresponding to Figure 3 11) receives and processes the first feature map 80x80x512, the second feature map 40x40x1024, and the third feature map 20x20x1024 respectively, and passes through the 21st layer (the first ELAN-E module, corresponding to Figure 3 21), 24th floor (the second ELAN-E module, corresponding to Figure 324), 27th floor (the third ELAN-E module, corresponding to Figure 3 27) Output the first fused feature map 80x80x128, the second fused feature map 40x40x256, and the third fused feature map 20x20x512.

[0064] See also Figure 5 , the efficient layer aggregation network ELAN consists of 7 groups of convolutional layers (corresponding to Figure 5 The first branch passes through a 1×1 convolution layer to adjust the number of channels. The second branch first passes through a 1×1 convolution layer, and then passes through two 3×3 convolution layers for feature extraction. The third branch first passes through a 1×1 convolution layer, and then passes through four 3×3 convolution layers for feature extraction. Finally, the features extracted by these three branches are superimposed together and then processed by the last 1x1 convolution layer to form the final extracted features.

[0065] On this basis, the ECA attention mechanism is embedded after the last 1×1 convolutional layer of the efficient layer aggregation network ELAN, thereby constructing the layer attention module ELAN-E. The layer attention module ELAN-E effectively improves the performance and efficiency of the convolutional neural network by improving the representation ability of the feature map in the channel dimension, enabling the network to better capture and utilize important information in the image when processing complex tasks.

[0066] The output detection head in the anti-collision visual detection model is the same as the original detection head in the target detection model YOLOv7-tiny. The output detection head in the anti-collision visual detection model includes three detection heads, which are connected to the three ELAN-E modules in the feature fusion module one by one. The three detection heads receive and process the first fused feature map 80x80x128, the second fused feature map 40x40x256, and the third fused feature map 20x20x512, and output the first bounding box image 80x80x255, the second bounding box image 40x40x255, and the third bounding box image 20x20x255.

[0067] After making the above improvements to the original backbone network and feature fusion network in the existing target detection model YOLOv7-tiny, an anti-collision visual detection model is obtained.

[0068] Furthermore, in S2, the anti-collision visual detection model is trained using a transfer learning method, and the trained anti-collision visual detection model is deployed on the mine electric locomotive, specifically including the following:

[0069] S21, using a data set with the same image features as an initial training data set, using the initial training data set to pre-train the anti-collision visual detection model for the source task, and then using the mine track obstacle data set to formally train the pre-trained anti-collision visual detection model for the target task, to obtain a trained anti-collision visual detection model;

[0070] S22, performing sparse training on the scaling factor of the BN layer in the trained anti-collision visual detection model using L1 regularization, and performing pruning processing on the trained anti-collision visual detection model according to the sparse training result to obtain a compressed anti-collision visual detection model;

[0071] S23, presetting the number of iterative training times, and performing parameter fine-tuning training on the compressed anti-collision visual detection model within the preset number of iterative training times to obtain a fine-tuned anti-collision visual detection model;

[0072] S24. Deploy the fine-tuned anti-collision visual detection model in the edge computing platform of the mine electric locomotive.

[0073] Specifically, deep learning often requires a large amount of labeled data and time to train neural network models. However, it is very difficult to obtain large-scale labeled data in many tasks, especially in specific scenario tasks such as mining tracks. Transfer learning can use the knowledge learned on a source task to reduce the dependence on large-scale labeled data by fine-tuning the model on the target task, especially when data is scarce. At the same time, due to the richness and diversity of data set information, training a model from scratch requires a lot of time and continuous adjustment and optimization to converge on the new task; transfer learning uses the model parameters trained on the source task as initialization, which can accelerate the convergence of the model on the target task, especially when there are shared features between the source task and the target task.

[0074] In this embodiment, when transfer learning is applied to the mine track obstacle detection task, the VisDrone2019 dataset with the same image features is used as the source task training dataset. The VisDrone2019 dataset has the characteristics of occlusion, small targets, and low illumination, which has the same image features as the mine track obstacle dataset, that is, shared features.

[0075] 1) First, the large-scale VisDrone2019 dataset is used for pre-training of the source task, so that the model corresponding to the source task can learn some common features, and its backbone network weights are used as the pre-training weights of the target task model, thereby providing optimization parameters for the target task; on this basis, the mine track obstacle dataset is used for formal training of the target task, which can achieve better training results when mine track obstacle data is scarce. The source task model and the target task model share parameters and feature information, reducing the training cost, improving the feature extraction capability of the target network, alleviating the overfitting problem, and improving the detection capability of the target network.

[0076] 2) A model lightweight strategy based on BN (Batch Normalization) layer is used to solve the problem of poor edge deployment caused by too many redundant parameters in the anti-collision visual detection model. In convolutional neural networks, the BN layer is usually placed between the convolution layer and the activation function layer. The convolution layer, BN layer, and activation function layer are located in the same channel. Therefore, this paper uses the scaling factor of the BN layer in the anti-collision visual detection model structure as the scaling factor in the pruning method. In the pruning process, the scaling factor of the BN layer is first sparsely trained using L1 regularization, so as to automatically identify channels whose scaling factor distribution of the BN layer is close to 0, that is, channels that contribute less to the model learning ability. Then, according to the results of sparse training, the channels with smaller scaling factors are pruned to obtain a compressed anti-collision visual detection model.

[0077] Furthermore, the above “L1 regularization is used for sparse training of the scaling factor of the BN layer”, the sparse training loss function can be specifically expressed as:

[0078] (1)

[0079] in, is the sparse training loss function, represents the original loss function of the anti-collision visual detection model, Indicates the scaling factor in the BN layer Regularization of represents the weight factor, and R represents a constant not equal to 0;

[0080] During the sparse training process, the output value of the BN layer will change because the BN layer is regularized. The output y of the BN layer after regularization is ij It is expressed as follows:

[0081] (2)

[0082] Among them, the parameters , They represent the mean and variance of the activation function, respectively. is a value infinitely close to 0. is the deviation, represents the original value of the i-th sample on the j-th feature, Represents the output value of the BN layer after regularization of the i-th sample on the j-th feature.

[0083] 3) Preset the number of iterative training times and adjust the training parameters, and perform fine-tuning training on the compressed anti-collision visual detection model to restore the model expression ability affected by the pruning operation, thereby obtaining a fine-tuned anti-collision visual detection model.

[0084] 4) Deploy the fine-tuned anti-collision visual detection model on the edge computing platform of the mining locomotive.

[0085] The edge computing platform uses the ARM-based Jetson TX2 edge computing platform developed by NVIDIA, which has its own GPU+CPU, 8GB of memory, 32GB of storage, and 256 NVIDIA CUDA cores. In order to facilitate the edge deployment of the fine-tuned anti-collision visual detection model, the Ubuntu 18.04 operating environment suitable for deep learning is configured. The fine-tuned anti-collision visual detection model is deployed in the edge computing platform of the mine electric locomotive. Edge deployment can directly process data on the electric locomotive device without relying on a specific network connection or geographic location, effectively reducing network communication costs, thereby reducing latency, and has high data privacy and security.

[0086] S3. The acquisition unit collects images of the mine electric locomotive running on the track in real time, inputs the images into the anti-collision visual detection model trained and deployed on the edge of the mine electric locomotive for processing, and obtains a bounding box image with obstacle category and confidence. The position measurement unit obtains the distance information between the obstacle on the mine track and the mine electric locomotive.

[0087] Ultrasonic radar ranging calculation formula:

[0088] (3)

[0089] LiDAR ranging calculation formula:

[0090] (4)

[0091] In the formula, , They are ultrasonic radar distance information and laser radar distance information. is the propagation speed of ultrasound in the medium, is the speed of light, It is the time from ultrasonic emission to receiving echo. It is the time from laser emission to receiving reflected light.

[0092] Since the two sensors have different acquisition frequencies, the collected data will be different in time and space, so coordinate conversion is required to convert the collected data to the same time and space to facilitate the subsequent information fusion. Establish the coordinate conversion relationship between the laser radar, ultrasonic radar and the vehicle's own coordinate system, convert the coordinates of the two sensors, laser radar and ultrasonic radar, to the same coordinate system to achieve the unification of sensor fusion in space. Convert the laser radar and ultrasonic radar coordinate systems to the prior map coordinate system:

[0093] (5)

[0094] (6)

[0095] In the formula, Indicates from Coordinate system to The coordinate transformation matrix of the coordinate system, , , and its multiplication adopts the chain rule, I is the electric locomotive center of mass coordinate system, M is the prior map coordinate system, L is the laser radar coordinate system, and U is the ultrasonic radar coordinate system.

[0096] S4, the control module receives and processes the bounding box image with the obstacle category and confidence level, and the distance information between the obstacle and the mine electric locomotive, and controls the anti-collision operation of the mine electric locomotive.

[0097] Furthermore, the control module in S4 includes a storage unit, a computing unit and a control unit connected in sequence, and S4 specifically includes the following:

[0098] S41. The storage unit receives and stores a bounding box image with obstacle categories and confidence levels, and distance information between the obstacle and the mine electric locomotive.

[0099] S42, the calculation unit receives the distance information between the obstacle and the mine electric locomotive and uses the Kalman filter method to perform data fusion to obtain the final distance information between the obstacle and the mine electric locomotive.

[0100] Furthermore, the distance information between the obstacle and the mine electric locomotive in S41 includes ultrasonic radar distance information and laser radar distance information, and S42 specifically includes the following:

[0101] S421, the calculation unit receives ultrasonic radar distance information and laser radar distance information between the obstacle and the mine electric locomotive;

[0102] S422, using the ultrasonic radar distance information in the prediction phase, and using the ultrasonic radar distance information as a priori state estimation value of the observation state vector;

[0103] S423, using the laser radar distance information for the correction and update link, using the laser radar distance information to correct and update the prior state estimate of the observed state vector to obtain the final state vector, and using the final state vector as the final distance information between the obstacle and the mine electric locomotive.

[0104] Specifically, laser radar has high detection accuracy and is not affected by sudden changes in light and dust in mines. Ultrasonic radar has high safety, low cost, and good close-range detection performance. By utilizing the different advantages of ultrasonic radar and laser radar, the accuracy and reliability of perception information can be improved. A data fusion method based on Kalman filtering is adopted, and the ultrasonic radar distance information is used in the prediction link, and the ultrasonic radar distance information is used as the prior state estimate of the observation state vector; the laser radar distance information is used in the correction and update link, and the prior state estimate of the observation state vector is corrected and updated to obtain the final state vector, and the final state vector is used as the final distance information between the obstacle and the mine electric locomotive. Among them, the timestamp and frequency of the observation state vector are determined by the laser radar distance information. When the distance information sent by the laser radar sensor is received, the update iteration is generated to correct the observation state vector.

[0105] S43, the control unit determines the collision risk according to the bounding box image with the obstacle category and confidence level and the final distance information, and controls the mine electric locomotive to achieve anti-collision operation according to the collision risk.

[0106] Specifically, the control module receives a bounding box image with obstacle category and confidence, laser radar distance information and ultrasonic radar distance information between the obstacle and the mine electric locomotive, converts the obstacle information, laser radar distance information and ultrasonic radar distance information from original analog signals into digital signals and stores them in a storage unit to facilitate subsequent storage calculations and processing, calculates the collision risk, and controls the anti-collision operation of the mine electric locomotive according to the obstacle type and the degree of collision risk. Specifically, electric locomotives, mining tools, workers, and large ores with high collision risks need to be braked urgently, and small fibers need to be slowed down to pass.

[0107] In another embodiment, a mine electric locomotive anti-collision system adopts the above-mentioned mine electric locomotive anti-collision method based on edge deployment to control the mine electric locomotive to achieve anti-collision operation. The system includes a mine electric locomotive, and the mine electric locomotive is provided with an image acquisition unit, a position measurement unit, a control module and an edge computing device. The image acquisition unit, the position measurement unit and the control module are respectively connected to the edge computing device, and the anti-collision visual detection model is deployed in the edge computing device, wherein:

[0108] The image acquisition unit is used to acquire images containing obstacles when the mine electric locomotive runs along the track;

[0109] The position measurement unit is used to detect the distance information between the obstacle and the mine electric locomotive;

[0110] The anti-collision visual detection model is used to receive and process images containing obstacles to obtain bounding box images with categories and confidence levels.

[0111] The control module is used to receive and store bounding box images with obstacle categories and confidence levels, and distance information between obstacles and mine locomotives, calculate the final distance information and collision risk, and control the mine locomotive to achieve anti-collision operation based on the collision risk.

[0112] Furthermore, the control module includes a storage unit, a computing unit and a control unit connected in sequence, the storage unit is used to store a bounding box image with a category and a confidence level, and the distance information between the obstacle and the mine electric locomotive; the computing unit is used to receive the distance information between the obstacle and the mine electric locomotive and use the Kalman filtering method to perform data fusion to obtain the final distance information; the control unit is used to receive the bounding box image with a category and a confidence level, the final distance information and calculate the collision risk, and control the mine electric locomotive to achieve anti-collision operation according to the collision risk.

[0113] For the specific limitations of a mine electric locomotive anti-collision system based on edge deployment, please refer to the limitations of a mine electric locomotive anti-collision method based on edge deployment mentioned above, which will not be repeated here.

[0114] The above-mentioned method and system for anti-collision of mine electric locomotives based on edge deployment, an image acquisition unit and a position measurement unit are set on the mine electric locomotive, an anti-collision visual detection model is preset and trained, and the trained anti-collision visual detection model is edge-deployed on the mine electric locomotive, the image acquisition unit collects images of the mine electric locomotive when it is running on the track in real time, and the image is input into the anti-collision visual detection model trained on the edge deployed on the mine electric locomotive for processing, and a bounding box image with category and confidence is obtained, and the position measurement unit obtains the distance information between the obstacle and the mine electric locomotive; the control module receives the bounding box image with category and confidence, and the distance information between the obstacle and the mine electric locomotive and processes it, and controls the anti-collision operation of the mine electric locomotive. The method and system perceive the surrounding environment through intelligent device detection technology, and use artificial intelligence algorithms to detect targets, and control the mine electric locomotive to achieve anti-collision operation.

[0115] The above is a detailed introduction to the mine electric locomotive anti-collision method and system based on edge deployment provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A mine electric locomotive anti-collision method based on edge deployment, It is characterized in that The method comprises the following steps: S1. An image acquisition unit and a position measurement unit are arranged on a mine electric locomotive; S2, preset an anti-collision visual detection model, use the transfer learning method to train the anti-collision visual detection model, and deploy the trained anti-collision visual detection model edge on the mine electric locomotive; the anti-collision visual detection model in S2 includes a backbone network, a feature fusion network and an output detection head connected in sequence, the backbone network receives the image to be detected when the mine electric locomotive is running on the track and performs feature extraction, outputs several feature maps of different scales, the feature fusion network receives several feature maps of different scales and performs feature fusion and optimization, outputs the same number of fused feature maps as the feature maps of several different scales, the output detection head maps the fused feature map to the preset target detection frame, and generates the same number of boundary box images with obstacle categories and confidence as the feature maps of several different scales; S3, the image acquisition unit collects images of the mine electric locomotive running on the track in real time, inputs the images into the anti-collision visual detection model trained and deployed on the edge of the mine electric locomotive for processing, obtains a bounding box image with obstacle category and confidence, and the position measurement unit obtains the distance information between the obstacle on the mine track and the mine electric locomotive; S4, the control module receives and processes the bounding box image with the obstacle category and confidence level, and the distance information between the obstacle and the mine electric locomotive, and controls the anti-collision operation of the mine electric locomotive; The backbone network includes a first CBS module, a second CBS module, a third CBS module, a fourth CBS module, a first ELAN module, a first MP module, a second ELAN module, a second MP module, a third ELAN module, a third MP module and a C3ST module, which are connected in sequence. The backbone network receives an image to be detected when a mine electric locomotive is running on a track and performs feature extraction, and outputs a first feature map, a second feature map and a third feature map through the second ELAN module, the third ELAN module and the C3ST module; wherein the C3ST module includes three CONV modules and a Swin Transformer module, the Swin Transformer module is composed of two Swin Transformer Blocks connected in sequence, the input feature map is subjected to feature extraction by the first CONV module, the extracted features are respectively input into the second CONV module and the Swin Transformer module for processing, the features extracted by the second CONV module and the Swin Transformer module are superimposed and then processed by the third CONV module, and the final extracted features are output; The feature fusion network includes a sequentially connected SPPCS PC-L module, a fifth CBS module, a first Upsample module, a fourth ELAN module, a sixth CBS module, a second Upsample module, a first ELAN-E module, a fourth MP module, a second ELAN-E module, a fifth MP module and a third ELAN-E module, and also includes a seventh CBS module and an eighth CBS module. The seventh CBS module, the eighth CBS module and the SPPCS PC-L module of the feature fusion network respectively receive the first feature map, the second feature map and the third feature map output by the backbone network and process them, splice the feature output by the seventh CBS module with the feature output by the second Upsample module, and input them into the first ELAN-E module for processing, and output a first fused feature map, splice the feature output by the eighth CBS module with the feature output by the first Upsample module, and input them into the fourth ELAN module for processing, splice the feature output by the fourth ELAN module with the feature output by the fourth MP module, and input them into the second ELAN-E module for processing, and output a second fused feature map, and splice the feature output by the SPPCS PC-L module with the feature output by the fifth MP module, and input them into the third ELAN-E module Block processing is performed to output the third fused feature map; wherein, each ELAN-E module includes an efficient layer aggregation network ELAN and an ECA attention mechanism. The ELAN module consists of 7 convolutional layers and is divided into three branches. The first branch adjusts the number of channels through the first convolutional layer, the second branch extracts features through the second to fourth convolutional layers, and the third branch extracts features through the second to sixth convolutional layers. Finally, the features extracted by these three branches are superimposed together and processed through the seventh convolutional layer to form the final extracted features. The ECA attention mechanism is embedded after the seventh convolutional layer. The convolution kernel size of the first, second and seventh convolutional layers is 1×1, and the convolution kernel size of the third to sixth convolutional layers is 3×3.

2. The mine electric locomotive anti-collision method based on edge deployment as claimed in claim 1, It is characterized in that The image acquisition unit in S1 is specifically two zoom cameras, which are respectively arranged on the front and rear sides of the top of the mine electric locomotive. The position measurement unit specifically includes an ultrasonic radar and a laser radar, which are both arranged in the front center of the mine electric locomotive.

3. The mine electric locomotive anti-collision method based on edge deployment as claimed in claim 2, It is characterized in that In S2, the anti-collision visual detection model is trained using the transfer learning method, and the trained anti-collision visual detection model is deployed on the mine electric locomotive. Specifically, it includes the following: S21, using a data set with the same image features as an initial training data set, using the initial training data set to pre-train the anti-collision visual detection model for the source task, and then using the mine track obstacle data set to formally train the pre-trained anti-collision visual detection model for the target task, to obtain a trained anti-collision visual detection model; S22, performing sparse training on the scaling factor of the BN layer in the trained anti-collision visual detection model using L1 regularization, and performing pruning processing on the trained anti-collision visual detection model according to the sparse training result to obtain a compressed anti-collision visual detection model; S23, presetting the number of iterative training times, and performing parameter fine-tuning training on the compressed anti-collision visual detection model within the preset number of iterative training times to obtain a fine-tuned anti-collision visual detection model; S24. Deploy the fine-tuned anti-collision visual detection model in the edge computing platform of the mine electric locomotive.

4. The mine electric locomotive anti-collision method based on edge deployment as claimed in claim 3, It is characterized in that The control module in S4 includes a storage unit, a computing unit and a control unit connected in sequence, and S4 specifically includes the following: S41, the storage unit receives and stores the bounding box image with the obstacle category and confidence level, and the distance information between the obstacle and the mine electric locomotive; S42, the calculation unit receives the distance information between the obstacle and the mine electric locomotive and uses the Kalman filter method to perform data fusion to obtain the final distance information; S43, the control unit determines the collision risk according to the bounding box image with the obstacle category and confidence level and the final distance information, and controls the mine electric locomotive to achieve anti-collision operation according to the collision risk.

5. The mine electric locomotive anti-collision method based on edge deployment as claimed in claim 4, It is characterized in that The distance information between the obstacle and the mine electric locomotive in S41 includes ultrasonic radar distance information and laser radar distance information, and S42 specifically includes the following: S421, the calculation unit receives ultrasonic radar distance information and laser radar distance information between the obstacle and the mine electric locomotive; S422, using the ultrasonic radar distance information in the prediction phase, and using the ultrasonic radar distance information as a priori state estimation value of the observation state vector; S423, using the laser radar distance information for the correction and update phase, using the laser radar distance information to correct and update the prior state estimate of the observation state vector to obtain a final state vector, and using the final state vector as the final distance information.

6. A mine electric locomotive anti-collision system, which uses the mine electric locomotive anti-collision method based on edge deployment as described in any one of claims 1 to 5 to realize the anti-collision operation of the mine electric locomotive, It is characterized in that The system includes a mine electric locomotive, which is provided with an image acquisition unit, a position measurement unit, a control module and an edge computing device. The image acquisition unit, the position measurement unit and the control module are respectively connected to the edge computing device, and the anti-collision visual detection model is deployed in the edge computing device, wherein: The image acquisition unit is used to acquire images containing obstacles when the mine electric locomotive runs along the track; The position measurement unit is used to collect the distance information between the obstacle and the mine electric locomotive; The anti-collision visual detection model is used to receive and process images containing obstacles to obtain bounding box images with obstacle categories and confidence levels. The control module is used to receive and store bounding box images with obstacle categories and confidence levels, and distance information between obstacles and mine locomotives, calculate the final distance information and collision risk, and control the mine locomotive to achieve anti-collision operation based on the collision risk.

7. The mine electric locomotive anti-collision system based on edge deployment as claimed in claim 6, It is characterized in that The control module includes a storage unit, a computing unit and a control unit connected in sequence, the storage unit is used to store a bounding box image with obstacle categories and confidence levels, and distance information between obstacles and mine electric locomotives; The calculation unit is used to receive the distance information between the obstacle and the mine electric locomotive and use the Kalman filter method to perform data fusion to obtain the final distance information; The control unit is used to receive a bounding box image with obstacle category and confidence, final distance information between the obstacle and the mine electric locomotive, and calculate the collision risk, and control the mine electric locomotive to achieve anti-collision operation according to the collision risk.

Citation Information

Patent Citations

  • Urban road traffic sign detection and identification method based on improved LEP-YOLO v7

    CN117636296A

  • Obstacle detection method for unmanned electric locomotive based on edge vision

    CN118485997A

  • Automatic obstacle avoidance system for automobile

    CN216942982U