Small target detection method for strip mine area

Abstract

The invention relates to the technical field of target detection in automatic driving, in particular to a small target detection method for an open-pit mine area, and the method comprises the steps: constructing a mine area small target detection data set which comprises laser radar point cloud data, a long-focus camera image and an annotation label; establishing a mining area small target detection network, wherein the small target detection network performs feature extraction, cross attention fusion and three-dimensional target detection on the laser radar point cloud data and the telephoto camera image; the detection precision of training is set, and the mining area small target detection network is trained in a mode of dynamically adjusting the MixUp proportion; the trained mining area small target detection network is deployed on a mine car of the strip mine area, the mine car inputs information collected by the sensor into the trained mining area small target detection network, and a small target detection result is obtained; the method can effectively improve the detection precision of small and medium targets in the strip mine area.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection in autonomous driving, and in particular to a small target detection method for open-pit mines. Background Art

[0002] Artificial intelligence is penetrating into various application fields. Especially in the development of autonomous driving technology, open-pit mines have become an important application scenario. The small target detection technology of autonomous driving mine cars is an important technical link in the construction of smart mines.

[0003] The perception technology of mine truck autonomous driving in open-pit mines is particularly important. Only good perception results can support downstream decision-making and control modules. The mining environment is relatively complex, and small targets such as stones, cones, and equipment parts are easily missed. In mining operation scenarios, LiDAR and cameras are the two most commonly used sensors in the process of autonomous driving. Since the point cloud data of small targets is sparse, small targets such as stones, cones, and equipment parts are easily missed in the process of LiDAR single-mode perception. Camera images have dense semantic information, but the lack of depth information makes them susceptible to background noise interference. Stones not only cause vehicle bumps and tire wear to mining trucks, but also have the risk of causing vehicle rollover. For mine truck autonomous driving, accurately detecting small targets that may cause unexpected risks is necessary to improve transportation efficiency and avoid accidents. Perceiving small target scenes within the safe braking distance can effectively support subsequent path planning and vehicle control, which can not only improve the safe transportation efficiency of mines, but also significantly reduce labor costs, with significant economic and social benefits.

[0004] The Chinese invention patent application with publication number CN116524356A discloses a method and system for detecting rock targets in mining areas. Based on the set of ore image samples, an ore annotation database is obtained. A target detection framework is constructed based on sample enhancement and attention generation network. The target detection framework is fine-tuned and trained using the annotation database to obtain a target detection model. The image to be detected is input into the target detection model to obtain the ore target detection result. This method only uses the data collected by the image sensor. The image naturally lacks depth information, which is easy to cause recognition and positioning deviations in the three-dimensional detection process, and cannot provide accurate small target location information for the vehicle decision module.

[0005] The Chinese invention patent application with publication number CN117593723A discloses a method and system for detecting obstacles in mining areas. Based on continuous vehicle position information and continuous single-frame 3D semantic point cloud data of the vehicle, a local semantic point cloud map and a local mining area obstacle map are constructed to obtain information about obstacles in the mining area in front of the vehicle. At the same time, the category of obstacles in the mining area in front of the vehicle is detected to achieve real-time and accurate detection of the location, boundary and height information of obstacles in mining areas of various categories. This method ignores the class imbalance problem of small targets in the data set during the entire detection process.

[0006] The Chinese invention patent application with publication number CN118072282A discloses an obstacle recognition method in an open-pit mine area, which obtains the point cloud spatial features and reflection intensity corresponding to the obstacle to be recognized, and further, according to the initial recognition result of the target distance point cloud of the geometric features, thereby determining the target recognition result of the obstacle to be recognized according to the initial recognition result of the point cloud, the reflection intensity and the target distance. This method does not consider factors such as obstacle occlusion and unstable sensor information in the target recognition process. Summary of the invention

[0007] In view of the above problems, the present invention provides a small target detection method for open-pit mines, which solves the technical problems in the prior art that the mine target detection network model has poor robustness, the ratio of positive and negative samples is unbalanced, and the small target detection box predicted by the network has low confidence and is easily suppressed.

[0008] The present invention provides a small target detection method for open-pit mines, comprising the following steps:

[0009] Step S1, constructing a mining area small target detection dataset, wherein the small target detection dataset includes: lidar point cloud data, telephoto camera images and annotation labels;

[0010] Step S2, establishing a small target detection network in a mining area, wherein the small target detection network performs feature extraction, cross-attention fusion, and three-dimensional target detection on the laser radar point cloud data and the telephoto camera image respectively;

[0011] Step S3, setting the detection accuracy of training, training the mining area small target detection network, and dynamically adjusting the MixUp ratio according to the detection accuracy during the training of the mining area small target detection network;

[0012] Step S4: deploy the trained mining area small target detection network on a mine car in an open-pit mine area. The mine car inputs the information collected by the sensor into the trained mining area small target detection network to obtain a small target detection result.

[0013] Preferably, in step S1, the annotation labels of the small target detection data set include: object categories and the center point, size and orientation angle of the object bounding box; the object categories include pedestrians, cars, cones, stones and equipment parts.

[0014] Preferably, the step S2 specifically includes:

[0015] Step S2-1: In the small target detection network, the telephoto camera image is input into CSPNet to extract image features, and then input into a feature pyramid network to obtain multi-scale image features;

[0016] Step S2-2, voxelizing the laser radar point cloud data, and extracting global features from the voxelized point cloud data through a sparse coding network to obtain point cloud features;

[0017] Step S2-3, performing cross attention mechanism processing on the multi-scale image features and the point cloud features to obtain fusion features;

[0018] Step S2-4: perform three-dimensional target detection on the fused features, use weighted DIoU to deduplicate the three-dimensional target detection results, and finally obtain the parameters and category information of the bounding box of the small target to be detected.

[0019] Preferably, the step S2-3 specifically includes:

[0020] The multi-scale image features and the point cloud features are calculated by the multi-head attention mechanism, and the calculation formula is:

[0021]

[0022] Among them, X (k) represents the image features of the lth layer, F(Q,X (k) ) is the fusion feature, M is the total number of split heads of the multi-head attention mechanism, Q is the query vector, A m (Q) represents the attention weight of the query vector Q of the mth head, W m represents the learnable weight matrix of the mth head;

[0023] The calculation formula of the query vector Q is:

[0024] Q = f(X (l) ,P)=Concat(head 1 ,…,head l )W 0

[0025]

[0026] Among them, P is the point cloud feature, f(X(l) ,P) indicates that the query vector Q is composed of X (l) and P is determined, Concat(haad 1 ,…,haad l ) means to splice l heads together, W 0 is the linear transformation weight matrix, head i represents the i-th head feature, is the query vector weight matrix of the i-th head, is the key vector weight matrix of the i-th head, is the value vector weight matrix of the i-th head, and Attention(·,·,·) represents the attention mechanism calculation.

[0027] Preferably, the step S3 specifically includes:

[0028] Step 3-1: Determine the initial value R of the detection accuracy 0 , smoothing factor β and initial value α of MixUp ratio 0 ;

[0029] Step 3-2: Perform num_iterations iterations. The iteration process includes: obtaining the detection accuracy R on the validation set. t , based on the R at the current tth iteration t The value and R at the t-1th iteration t The value determines the MixUp ratio α at the t+1th iteration t+1 ;

[0030] If R t <R t-1 , then increase the MixUp ratio. If R t ≥R t-1 , then reduce the MixUp ratio and obtain the MixUp ratio α′ at the t+1th iteration after the ratio adjustment t+1 , for α′ t+1 Apply the smoothing factor to get the MixUp ratio α at the t+1th iteration t+1 .

[0031] Preferably, the calculation formula for increasing the MixUp ratio is:

[0032]

[0033] The calculation formula for reducing the MixUp ratio is:

[0034]

[0035] The calculation formula for applying the smoothing factor is:

[0036] α t+1 =min(max(α′ t+1 ×β,min α ),max α )

[0037] Among them, R t is the detection accuracy at the tth iteration, α t is the MixUp ratio at the tth iteration, β is the smoothing factor, α′ t+1 Indicates the MixUp ratio at the t+1th iteration after the ratio adjustment, min α and max α are the minimum and maximum values ​​of MixUp ratio, γ is the hyperparameter adjustment step, n inc and n dec The thresholds for the number of consecutive increases and decreases in the MixUp ratio, respectively.

[0038] Preferably, the calculation expression of the weighted DIoU is:

[0039]

[0040] Among them, Area of ​​Overlap is the overlapping area, which represents the intersection of the predicted box and the true box, Area of ​​Union is the joint area, which represents the union of the predicted box and the true box, DIoU combined represents the weighted DIoU, d represents the center distance between the real box and the predicted box, c represents the diagonal length of the minimum closure area of ​​the minimum rectangular box that contains both the predicted box and the real box, W size is the inverse of the sum of the areas of the predicted box and the true box, W shape Represents the cosine similarity of the feature vectors of the predicted box and the true box, V 1 ,V 2 are the feature vectors of the predicted box and the true box respectively.

[0041] Compared with the prior art, the present invention has at least the following beneficial effects:

[0042] (1) The present invention provides a new query vector initialization strategy in the process of laser radar and camera feature fusion. Better query initialization can not only effectively reduce learning bias, but also better utilize the prior information of input data. Effectively utilizing the prior knowledge of input data for query initialization can efficiently detect targets. Through dynamic cross-modal relationship modeling, the network can more fully utilize the advantages of dense semantic information in the image to detect small targets.

[0043] (2) The present invention provides a new multimodal data enhancement method and designs a dynamic adjustment strategy to adjust the MixUp ratio according to the performance of the model on the validation set. The MixUp ratio is adaptively adjusted according to the performance of the model on the validation set, and a regularized smoothing mechanism and hyperparameter optimization technology are added in the process of MixUp ratio adjustment, thereby improving the robustness of the model to occlusion and depth changes.

[0044] (3) The present invention provides a new evaluation index in the non-maximum suppression process. Taking into account that small targets are more easily occluded by other objects in three-dimensional space, the size factor and shape factor are added to alleviate the low-confidence suppression of small targets in the post-processing process. At the same time, it can give the small target detection frame more retention possibilities when the small target is occluded. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0046] Figure 1 This is a flowchart for implementing the multimodal data enhancement provided by the present invention.

[0047] Figure 2 This is the overall architecture diagram of the small target detection network provided by the present invention.

[0048] Figure 3 The present invention provides a schematic diagram of feature fusion of extracted laser point cloud features and image features through a cross-attention mechanism.

[0049] Figure 4 A schematic diagram of changes in IoU values ​​of objects of different sizes provided by the present invention.

[0050] Figure 5 A flow chart of the small target detection method for open-pit mines provided by the present invention. DETAILED DESCRIPTION

[0051] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0052] The present invention provides a small target detection method for open-pit mines. The laser radar point cloud features and camera image features are fused under the Transform framework for target detection. During the training process, the MixUp ratio in multimodal data enhancement is adaptively adjusted according to the performance of the algorithm model on the validation set. At the same time, smoothing factors and hyperparameter optimization are added. In addition, new measurement indicators are added in the generation process of three-dimensional detection frames, namely, the non-maximum suppression strategy, so as to effectively improve the detection accuracy of small and medium-sized targets in open-pit mines.

[0053] In order to illustrate the effectiveness of the method proposed by the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment. Figure 5 As shown, a small target detection method for open-pit mines is disclosed, and the specific implementation steps are as follows:

[0054] Step S1, constructing a mining area small target detection dataset, wherein the small target detection dataset includes: lidar point cloud data, telephoto camera images and annotation labels.

[0055] Collect and organize data sets for small target detection, including LiDAR point cloud data and telephoto camera image data. During the data collection process, high-precision LiDAR sensors and telephoto cameras are used for synchronous acquisition. LiDAR can provide point cloud data in three-dimensional space to capture the depth information of the object, while the telephoto camera provides high-resolution two-dimensional images that can clearly record the appearance characteristics of the object. This combination of multimodal data helps to improve the accuracy and robustness of small target detection.

[0056] For the annotation labels of the data, the annotation labels of the present invention include attributes such as object category, center point of the object bounding box, size (length, width, height) and direction angle. The categories of the objects may include pedestrians, cars, cones, stones and equipment parts (such as cardboard boxes). In addition, to ensure high-precision matching between the LiDAR and camera data, an offline sensor fusion calibration is performed to generate a projection matrix between the LiDAR and camera data. The projection matrix can map the three-dimensional point cloud data to a two-dimensional image plane.

[0057] The data elements in the data set for small target detection finally obtained include: lidar point cloud data, telephoto camera images and annotation labels.

[0058] Through the above steps, the mining area small target detection data set constructed by the present invention can be used for subsequent network training.

[0059] Step S2: establishing a small target detection network in a mining area, wherein the small target detection network performs feature extraction, cross-attention fusion, and three-dimensional target detection on the laser radar point cloud data and the telephoto camera image respectively.

[0060] like Figure 2 As shown, in the mining area small target detection network of the present invention, the laser radar point cloud data and the telephoto camera image are subjected to feature fusion after passing through their respective feature extraction branches.

[0061] (1) For the image branch, the present invention adopts a lightweight backbone network CSPNet as the backbone network, that is, a feature extractor. The camera image is input into CSPNet to extract image features, and then input into the Feature Pyramid Network (FPN) to obtain multi-scale image features as the output of the image branch.

[0062] Considering that the computing resources on the vehicle side are very valuable and easily disturbed by the harsh environment, the present invention adopts a lightweight feature extraction network CSPNet to alleviate the problem of a large number of inference calculations. CSPNet splits the image features into two parts, one part is convolution operation, and the other part is connected with the result of the convolution operation of the previous part. Not all feature maps are passed in each stage, but useful feature maps are selectively passed to the next stage, and then input into the FPN network to generate multi-scale image features, generate effective receptive fields for targets of different sizes, capture small target information at high resolution, and improve the robustness of feature representation.

[0063] (2) For the point cloud branch, the present invention defines the point cloud range and voxel size, and voxelizes the input point cloud data. There are multiple voxel units in the voxelized point cloud. Each voxel unit can contain one or more points, and these points are divided into the same voxel unit in space. The voxelized point cloud data is subjected to global feature extraction through a sparse coding network to obtain point cloud features. Sparse convolution only calculates convolution operations on non-empty voxels, avoiding additional space and computational overhead, thereby significantly reducing the amount of calculation for the point cloud branch. In addition, parallel computing resources such as GPUs can effectively accelerate the sparse convolution and feature extraction process.

[0064] (3) Cross-attention fusion. The present invention provides a cross-attention mechanism for fusing multi-scale image features and point cloud features. The specific steps are as follows:

[0065] The multi-scale image features X obtained by the image branch and the point cloud branch are (l) And point cloud features P∈R N×c To calculate the query vector in the cross-attention mechanism, where X (l) represents the image features of the lth layer, R N×crepresents a real number matrix with a dimension of N×c, representing the features of the point cloud after voxelization, N is the number of voxel units in the voxelized point cloud, c is the dimension of the feature vector contained in each voxel unit, and the multi-scale image features are also used as key (Key)-value (Value) vectors. The fusion feature calculation formula of the present invention is:

[0066]

[0067] Among them, F(Q,X (l) ) is the fusion feature, M is the total number of split heads of the multi-head attention mechanism, Q is the query vector, A m (Q) represents the attention weight of the query vector Q of the mth head, W m represents the learnable weight matrix of the m-th head.

[0068] The calculation formula of the query vector Q is:

[0069] Q = f(X (l) ,P)=Concat(head 1 ,…,head l )W 0

[0070]

[0071] Among them, f(X (l) ,P) indicates that the query vector Q is composed of X (l) and P is determined, Concat(head 1 ,…,head l ) means to splice l heads together, W 0 is the linear transformation weight matrix, head i represents the i-th head feature, W i is the query vector weight matrix of the i-th head, is the key vector weight matrix of the i-th head, is the value vector weight matrix of the i-th head, and Attention(·,·,·) represents the attention mechanism calculation.

[0072] The calculation process of fusion features is as follows: Figure 3 As shown, the present invention adopts a multi-head attention mechanism including M split heads, and the query vector Q is jointly generated by multi-scale image features and point cloud features. Effectively utilizing the prior knowledge of the input data for query initialization can efficiently detect targets. The query initialization guided by multi-scale image features can make full use of the rich semantic information transmitted by the FPN layer. Through dynamic cross-modal relationship modeling, the network can more fully utilize the multi-scale features of the image to detect small targets.

[0073] (4) 3D object detection

[0074] Finally, DGCNN is used as the three-dimensional target detector of fused features, and the fused features are used for three-dimensional target detection. DGCNN dynamically constructs the graph structure of the fused features in each convolutional layer. After post-processing such as removing redundant detection boxes, the parameters and category information of the bounding box of the small target to be detected are finally obtained.

[0075] Specifically, in the process of 3D object detection, the model may predict multiple bounding boxes for the same object, and these bounding boxes may have a high degree of overlap. The present invention uses non-maximum suppression to select the best bounding box and remove unnecessary redundant boxes, thereby improving the accuracy and efficiency of detection.

[0076] An important step in the non-maximum suppression process is to calculate the intersection over union (IoU). The distribution of IoU values ​​varies from category to category, such as Figure 4 As shown in the figure, compared with large targets, small targets are more sensitive to the deviation of the bounding box. When the same offset is 2 pixels, the IoU value of large targets drops from 1 to 0.85, while the small target drops sharply to 0.64. The situation will be further aggravated when the offset is larger. The algorithm model usually generates redundant prediction results. Since the size of small targets in the actual three-dimensional space is relatively small, they may have lower confidence and are more easily ignored and suppressed in the traditional non-maximum suppression strategy.

[0077] The IoU metric is too strict for small objects. Sometimes the primary task of small object detection is to identify the object and obtain its approximate location, rather than being entangled in the accuracy of the object.

[0078] The specific calculation formula of IoU metric is:

[0079]

[0080] Among them, Area of ​​Overlap is the overlapping area, which represents the intersection of two bounding boxes, and Area of ​​Union is the union area, which represents the union of two bounding boxes.

[0081] Considering that small targets are more easily blocked by other objects in three-dimensional space, the DIoU indicator is improved compared with the traditional IoU indicator in small target detection, mainly because in the NMS process, not only the size of the overlapping area is considered, but also the penalty term of the center point distance is added. The specific calculation formula of the DIoU metric is:

[0082]

[0083] Where b is the center of the prediction box, b gt is the center of the ground-truth box, ρ 2(b,b gt ), that is, d 2 , represents the square of the distance between the center of the real box and the predicted box, and c represents the diagonal length of the minimum closure area of ​​the minimum rectangular box that contains both the predicted box and the real box.

[0084] The present invention aims at DIoU, and continues to add the size factor and the shape factor as the weight of the penalty term, so that the detection frame of the small target is more likely to be retained in the process of removing the redundant detection frame, and obtains the weighted DIoU, which is determined as DIoU combined , the calculation formula is:

[0085]

[0086] W size is the reciprocal of the sum of the areas of the two bounding boxes B1 and B2. When the bounding box area is very small, the protection of the small target detection box will be increased in the weight calculation process. shape The cosine similarity of the feature vectors representing two bounding boxes can effectively improve the detection of small objects that are easily occluded.

[0087] For bounding boxes B1 and B2, the shape feature vectors are V 1 =[l 1 ,w 1 ,h 1 ] and V 2 =[l 2 ,w 2 ,h 2 ] Then the cosine similarity W shape The calculation expression is:

[0088]

[0089] Among them, ∥·∥ represents the modulus of the vector.

[0090] Step S3, setting the detection accuracy of training, training the mining area small target detection network, and dynamically adjusting the MixUp ratio according to the detection accuracy during the training of the mining area small target detection network.

[0091] MixUp technology is a data enhancement method that mixes different samples and their labels by linear interpolation to generate new training data. In the present invention, MixUp technology is applied to point cloud and image data to generate virtual small target information.

[0092] In the process of training the small target detection network, the present invention adopts a multimodal data enhancement method, which can alleviate the class imbalance problem of small targets in the data set. Specifically, the present invention designs a dynamic adjustment strategy to adjust the MixUp ratio according to the performance of the model on the validation set, and determines the mean Average Precision (mAP) as the model detection accuracy to measure the performance of the model on the validation set. If the accuracy of the model is lower than this threshold, the MixUp ratio is increased to provide more data changes.

[0093] During the training process, the performance of the model is regularly evaluated on the validation set, and the MixUp ratio is dynamically adjusted based on the evaluation results, that is, the MixUp ratio is regularly iterated and updated. The MixUp ratio can be regarded as an adaptive learning rate. When the model encounters difficulties in certain tasks, the MixUp ratio is increased to "increase the learning rate" to help the model escape the local optimum. In order to avoid frequent and drastic changes in the MixUp ratio, a smoothing mechanism can be introduced to set the maximum and minimum MixUp ratio adjustment times. Use hyperparameter optimization to automatically find the best MixUp ratio adjustment strategy. Figure 1 As shown, the specific implementation steps are:

[0094] (1) Determine the initial value R of detection accuracy 0 , smoothing factor β and initial value α of MixUp ratio 0 ;

[0095] (2) Perform num_iterations iterations. The iteration process includes: obtaining the detection accuracy R on the validation set t , based on the R at the current tth iteration t The value and R at the t-1th iteration t-1 The value determines the MixUp ratio α at the t+1th iteration t+1 ;

[0096] If R t <R t-1 , then increase the MixUp ratio. If R t ≥R t-1 , then reduce the MixUp ratio and obtain the MixUp ratio α′ at the t+1th iteration after the ratio adjustment t+1 , for α′ t+1 Apply the smoothing factor to get the MixUp ratio α at the t+1th iteration t+1 .

[0097] The specific calculation formula for the increase in MixUp ratio when performance degrades is:

[0098]

[0099] The specific calculation formula for reducing the MixUp ratio when performance is improved is:

[0100]

[0101] The specific calculation formula of the smoothing factor is:

[0102] α t+1 =min(max(α′ t+1 ×β,min α ),max α )

[0103] Among them, R t is the detection accuracy at the tth iteration, α t is the MixUp ratio at the tth iteration, β is the smoothing factor, α′ t+1 Indicates the MixUp ratio at the t+1th iteration after the ratio adjustment, min α and max α are the minimum and maximum values ​​of MixUp ratio, γ is the hyperparameter adjustment step, n inc and n dec The number of times the MixUp ratio is increased or decreased respectively. After an iterative change of the MixUp ratio is increased or decreased, 1 is added. and The thresholds for the number of consecutive increases and decreases in the MixUp ratio, respectively.

[0104] Step S4: The trained mining area small target detection network is deployed on a mine car in an open-pit mine area. The mine car inputs the information collected by the sensor into the trained mining area small target detection network to obtain a small target detection result.

[0105] In this step, the trained mining area small target detection network is deployed on the mining cart in the open-pit mine. The mining cart is equipped with sensors such as high-precision laser radar and telephoto camera. These sensors collect mining area environmental data in real time. The laser radar provides three-dimensional point cloud data to capture spatial depth information, while the telephoto camera provides high-resolution images to record the detailed features of the scene.

[0106] The data collected by the sensor is input into the small target detection network in the mining area to identify and detect small targets in the scene. The output results can include information such as the bounding box and category label of each target. This information is used to judge the potential risks and operational requirements in the mining environment in real time, help the mine car to navigate and make decisions autonomously, improve the safety and efficiency of mining operations, and effectively improve the intelligence level of open-pit mines.

[0107] Although the specific embodiments of the present invention depict various actions or steps in a specific order, this should be understood as requiring such actions or steps to be performed in the specific order shown or in a sequential order, or requiring all illustrated actions or steps to be performed to obtain the desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination. The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0108] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A small target detection method for open-pit mines, characterized in that: The following steps are involved: Step S1, constructing a mining area small target detection dataset, wherein the small target detection dataset includes: lidar point cloud data, telephoto camera images and annotation labels; Step S2, establishing a small target detection network in a mining area, wherein the small target detection network performs feature extraction, cross-attention fusion, and three-dimensional target detection on the laser radar point cloud data and the telephoto camera image respectively; Step S3, setting the detection accuracy of training, training the mining area small target detection network, and dynamically adjusting the MixUp ratio according to the detection accuracy during the training of the mining area small target detection network; Step S4: deploy the trained mining area small target detection network on a mine car in an open-pit mine area. The mine car inputs the information collected by the sensor into the trained mining area small target detection network to obtain a small target detection result.

2. The small target detection method for open-pit mines according to claim 1, characterized in that: In step S1, the annotation labels of the small target detection data set include: object categories and the center point, size and direction angle of the object bounding box; the object categories include pedestrians, cars, cones, stones and equipment parts.

3. The small target detection method for open-pit mines according to claim 2, characterized in that: The step S2 specifically includes: Step S2-1: In the small target detection network, the telephoto camera image is input into CSPNet to extract image features, and then input into a feature pyramid network to obtain multi-scale image features; Step S2-2, voxelizing the laser radar point cloud data, and extracting global features from the voxelized point cloud data through a sparse coding network to obtain point cloud features; Step S2-3, performing cross attention mechanism processing on the multi-scale image features and the point cloud features to obtain fusion features; Step S2-4: perform three-dimensional target detection on the fused features, use weighted DIoU to deduplicate the three-dimensional target detection results, and finally obtain the parameters and category information of the bounding box of the small target to be detected.

4. The small target detection method for open-pit mines according to claim 3 is characterized in that: The step S2-3 specifically includes: The multi-scale image features and the point cloud features are calculated by the multi-head attention mechanism, and the calculation formula is: Among them, X (l) represents the image features of the lth layer, F(Q, X (l) ) is the fusion feature, M is the total number of split heads of the multi-head attention mechanism, Q is the query vector, A m (Q) represents the attention weight of the query vector Q of the mth head, W m represents the learnable weight matrix of the mth head; The calculation formula of the query vector Q is: Q=f(X (l) ,P)=Concat(head1,...,head l )W 0 where head i =Attention(PW i ,X (l) W i k ,X (l) W i v ) Among them, P is the point cloud feature, f(X (l) , P) indicates that the query vector Q is composed of X (l) and P determines, Concat(head1, ..., head l ) means to splice l heads together, W 0 is the linear transformation weight matrix, head i represents the i-th head feature, W i is the query vector weight matrix of the i-th head, W i k is the key vector weight matrix of the i-th head, W i v is the value vector weight matrix of the i-th head, and Attention(·,·,·) represents the attention mechanism calculation.

5. The small target detection method for open-pit mines according to claim 4, characterized in that: The step S3 specifically includes: Step 3-1, determine the initial value R0 of the detection accuracy, the smoothing factor β and the initial value α0 of the MixUp ratio; Step 3-2: Perform num_iterations iterations. The iteration process includes: obtaining the detection accuracy R on the validation set. t , based on the R at the current tth iteration t The value and R at the t-1th iteration t-1 The value determines the MixUp ratio α at the t+1th iteration t+1 ; If R t <R t-1 , then increase the MixUp ratio. If R t ≥R t-1 , then reduce the MixUp ratio and obtain the MixUp ratio α′ at the t+1th iteration after the ratio adjustment t+1 , for α′ t+1 Apply the smoothing factor to get the MixUp ratio α at the t+1th iteration t+1 .

6. The small target detection method for open-pit mines according to claim 5, characterized in that: The calculation formula for increasing the MixUp ratio is: The calculation formula for reducing the MixUp ratio is: The calculation formula for applying the smoothing factor is: a t+1 =min(max(α′ t+1 ×β, min α ),max α ) Among them, R t is the detection accuracy at the tth iteration, α t is the MixUp ratio at the tth iteration, β is the smoothing factor, α′ t+1 Indicates the MixUp ratio at the t+1th iteration after the ratio adjustment, min α and max α are the minimum and maximum values ​​of MixUp ratio, γ is the hyperparameter adjustment step, n inc and n dec The thresholds for the number of consecutive increases and decreases in the MixUp ratio, respectively.

7. The small target detection method for open-pit mines according to claim 6, characterized in that: In step S2-4, the calculation expression of the weighted DIoU is: Among them, Area of ​​Overlap is the overlapping area, which represents the intersection of the predicted box and the true box, Area of ​​Union is the joint area, which represents the union of the predicted box and the true box, DIoU combined represents the weighted DIoU, d represents the center distance between the real box and the predicted box, c represents the diagonal length of the minimum closure area of ​​the minimum rectangular box that contains both the predicted box and the real box, W size is the inverse of the sum of the areas of the predicted box and the true box, W shape Represents the cosine similarity of the feature vectors of the predicted box and the true box, V1 and V2 are the feature vectors of the predicted box and the true box respectively.

Images