Multi-target tracking method and system for coal gangue and sundries

Abstract

The invention provides a multi-target tracking method for coal gangue and sundries. The multi-target tracking method comprises the following steps: acquiring a coal gangue and sundries image target detection data set and performing data enhancement; cutting coal gangue and sundry targets to generate an image classification data set; inputting samples of the target detection data set and the data enhancement data set into a YOLOV5S network model to obtain a target detection model; inputting samples in the image classification data set into the VIT image classification network model to obtain an image feature extraction network model; inputting the real-time image into a target detection model, obtaining a category and a coordinate frame, obtaining feature data in combination with the image, performing target trajectory tracking, and generating a coal gangue and sundry trajectory; and for the target in the determined state track, whether the same target of the front and back frames crosses the detection boundary line for the first time is judged, and coordinates of the crossing boundary line and the current timestamp are stored in a to-be-grabbed matrix of the mechanical arm. According to the method, the multi-target tracking robustness, the detection precision and the grabbing accuracy of the mechanical arm are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of coal mine technology, and specifically but not limited to, to a multi-target tracking method and system for coal gangue and debris. Background Art

[0002] In the process of coal mining, with the mining and transportation of coal, a large amount of gangue and various debris are often accompanied. Among them, coal gangue is a low-carbon solid waste, which not only increases the difficulty and cost of coal processing in the later stage, but also reduces the calorific value of coal and increases the emission of pollutants. The debris mainly includes metal anchors, fiberglass anchors, wooden products, woven bags, waste oil pipes, etc. These debris will cause blockage in the production system of the coal preparation plant, causing equipment damage or delaying production, resulting in production losses to the coal preparation plant.

[0003] At present, the sorting of gangue mainly includes manual sorting, dry sorting and wet sorting. Manual sorting relies on a large amount of human resources, with a poor working environment, high labor intensity, and high missed selection and misselection rates. The wet sorting method is mainly divided into heavy medium method and jigging method, which mainly utilizes the density difference between coal and gangue, and separates the coal and gangue by placing the raw coal in the solution. The equipment relied on by this method is huge, the process is complex, the sorting efficiency is low, and it will cause certain pollution to the surrounding environment. The dry sorting method mainly includes drum crushing and visible light recognition sorting. The former uses the different strengths of coal and gangue, which leads to the destruction of the shape of the coal block during the sorting process, and the sorting effect is poor and the equipment is huge. The latter is based on the different transmittance of coal and gangue to light, but the sorting equipment is expensive and the sorting efficiency is low.

[0004] In recent years, with the rise of AI technology, deep learning-based visual inspection algorithms have made rapid progress in the detection of coal gangue, but there are relatively few methods for sorting debris in coal flow. At present, the coal gangue sorting method based on image recognition generally uses the alternate frame detection method to send the coordinates of coal gangue. Since this method only relies on a single frame image and ignores the data association of coal gangue in adjacent frames, once a false detection occurs, the detection result will be directly fed back to the motion controller, resulting in a decrease in the accuracy of the robot arm's grasping, and the alternate frame detection method is difficult to solve problems such as repeated detection and missed detection.

[0005] In view of this, it is necessary to provide a new method and system to solve at least part of the above problems. Summary of the invention

[0006] In response to one or more problems in the prior art, the present invention proposes a multi-target tracking method and system for gangue and debris, so as to solve the problems of low grasping accuracy and high false grasping and missing grasping rate of gangue and debris in the process of linkage between the visual system and the robotic arm.

[0007] The technical solution to achieve the purpose of the present invention is:

[0008] A multi-target tracking method for gangue and debris, comprising:

[0009] Step S1: Obtain a coal gangue and debris image target detection dataset.

[0010] Step S2: correspondingly crop out the gangue and debris targets in the target detection data set, and generate an image classification data set, where the category in the sample is each instance individual.

[0011] Step S3: data enhancement is performed on the gangue and debris image target detection dataset by rotation, image scaling, and nonlinear gamma.

[0012] Step S4: Input the samples in the target detection data set and the data enhancement data set into the YOLOV5S network model with enhanced global feature perception capability for training to obtain a gangue and debris target detection model.

[0013] Step S5: Input the samples in the image classification data set into the VIT image classification network model for training to obtain a network model for extracting features of gangue and debris images.

[0014] Step S6: adjust the real-time image to a predetermined resolution and input it into the target detection model. After model inference, a two-stage non-maximum suppression (NMS) is used to obtain the categories and coordinate frames of gangue and debris in the image.

[0015] Step S7: Input the image, category and coordinate frame of gangue and debris into the gangue and debris image feature extraction network model, obtain feature data after model inference, perform cascade matching and IoU matching through the image feature data, generate gangue and debris trajectories, and assign the corresponding category ID to the determined state trajectory.

[0016] Step S8: For the target in the determined trajectory, determine whether the same target in the previous and next frames crosses the detection boundary line for the first time through cross-border detection. Store the coordinates of the crossing boundary line and the current timestamp into the robot arm to be grasped matrix, and delete the corresponding trajectory.

[0017] Furthermore, the data enhancement specifically includes:

[0018] The image is randomly rotated to simulate the inspected object at different angles. The image is randomly scaled to simulate the inspected object of different sizes. The random rotation and scaling of the image must keep the original image size. The image is nonlinearly gamma-varied to simulate the influence of coal ash, dust, etc. on the lighting conditions of the inspected object.

[0019] Furthermore, the YOLOV5S network model with enhanced global feature perception capability specifically includes:

[0020] In the process of extracting features by downsampling the input image, the CBL module is used to focus more on local detail features for low-level features. For high-level features, the ACMIX module is combined to extract local features through convolution, and intermediate features through 1x1 convolution. Convolution and self-attention are used to reuse and aggregate intermediate features, so that high-level features not only have local features, but also have the ability to perceive global features.

[0021] For the three layers of effective feature extracted after feature fusion, CA coordinated attention modules are added respectively. This module can comprehensively consider the relationship between feature map channels and spatial position information, and enhance the model's ability to perceive global features without adding additional computational overhead.

[0022] Furthermore, the VIT image classification network model specifically includes:

[0023] For the large-scale image classification dataset segmented from the target detection dataset, the Transformer structure is used for image feature extraction. The attention mechanism can be effectively used to capture the global features in the image. Compared with the CNN network structure, richer and more diverse feature representations can be extracted.

[0024] Furthermore, the two-stage NMS specifically includes:

[0025] For different categories of detected objects, NMS with higher thresholds is performed separately to screen out all targets of each category as much as possible to prevent high-confidence targets from suppressing other categories. For all detection results, NMS with an overall high threshold is performed to prevent the same target from being detected as multiple categories.

[0026] Furthermore, the target trajectory tracking specifically includes:

[0027] The trajectory is initialized for the input gangue and debris categories and coordinate frames, and the initialized trajectories are all uncertain. According to the conveyor belt speed, the time of adjacent frames, and the affine transformation of the actual distance and the image distance, the image position of the target in the next frame is predicted. The predicted value of the previous frame of the trajectory object is associated with the actual detection value of the current frame. The targets detected multiple times in succession are judged as determined trajectories and the ID sequence corresponding to the category is assigned. When the determined trajectory fails to match multiple times in a row, or when the center point of the trajectory target exceeds the image size after motion prediction, the trajectory is deleted.

[0028] Furthermore, the data association specifically includes:

[0029] Through the cascade matching method, the VIT image classification network model is used to extract the features of the image area corresponding to the coordinate frame, calculate the feature similarity for the adjacent frame targets, and generate a cost matrix. The target movement distance is calculated by the conveyor belt speed and the time difference between adjacent frames, and the image center point coordinates of the target in the next frame are predicted by combining the actual distance and the affine transformation matrix of the image distance. The feature cost matrix is ​​updated by the Euclidean distance of the target center point between adjacent frames, and the Hungarian matching algorithm is used to achieve the front and back association of the target. Through the IoU matching method, the overlap of the bounding boxes of the same category of targets in the previous and next two frames is calculated, and the cost matrix is ​​generated, and the data association is performed by the Hungarian matching algorithm.

[0030] Furthermore, the cross-border detection specifically includes:

[0031] For the target in the trajectory, determine whether the line segment formed by the coordinates of the target center point in the previous and next frames intersects with the line segment formed by the pre-set boundary line vertices. If they intersect, the target coordinates and timestamp are stored in the robot arm's grasping matrix, and the robot arm grasps according to the current working state.

[0032] A multi-target tracking system for gangue and debris, comprising:

[0033] The data acquisition module collects the surface image of the conveyor belt through an industrial camera, and obtains the current conveyor belt speed through a belt speed sensor.

[0034] The data preprocessing module preprocesses the images captured by the camera and calculates the moving distance of the targets in adjacent frames in the image coordinate system through motion estimation.

[0035] The target detection module sets the opening and closing of the categories of gangue and debris to be detected through the interface. The preprocessed image is input into the trained target detection model to obtain the categories to be detected and the corresponding coordinate frames.

[0036] The trajectory tracking module predicts the next frame position of the target center point through motion estimation, and realizes the correlation between targets in adjacent frames through cascade matching and IoU matching.

[0037] The coordinate sending module sends the out-of-bounds target coordinates and timestamp through out-of-bounds detection.

[0038] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0039] 1. The multi-target tracking method for gangue and debris of the present invention proposes data enhancement, simulates the different states of gangue and debris in actual situations, increases the sample size of the data set, and improves the robustness of the model; proposes a two-stage NMS to prevent mutual interference between targets of different categories and avoid the same target being detected as different categories; proposes target trajectory tracking to achieve the mutual correlation of targets in adjacent frames; proposes out-of-bounds detection to prevent the same target from being sent to the robotic arm multiple times.

[0040] 2. In target detection, the multi-target tracking method for gangue and debris of the present invention integrates a convolution and self-attention hybrid mechanism in the backbone network and the feature fusion network based on the benchmark model YOLOV5S, and adds a coordinated attention module in the model prediction part to enhance the global feature perception ability of the model and improve the model detection accuracy.

[0041] 3. The multi-target tracking method for gangue and debris of the present invention deploys the model to the gangue and debris sorting system, obtains the conveyor belt image through an industrial camera, realizes multi-target tracking of gangue and debris with the help of the trained model, and determines the position of the target to be grasped by out-of-bounds detection, thereby improving the grasping accuracy of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to provide a further understanding of the present invention and are used together with the description to explain the embodiments of the present invention, but do not constitute a limitation of the present invention. In the accompanying drawings:

[0043] Figure 1 The present invention is a flow chart of the multi-target tracking method for gangue and debris.

[0044] Figure 2 This is a schematic diagram of the structure of the YOLOV5S target detection model with enhanced global feature perception capability of the present invention.

[0045] Figure 3 It is a schematic diagram of the structure of the basic unit components of the target detection model of the present invention.

[0046] Figure 4 Schematic diagram of the structure of the coordinated attention module of the present invention.

[0047] Figure 5 It is a schematic diagram of the structure of the VIT model for image feature extraction of the present invention.

[0048] Figure 6 It is a detection flow chart of the target tracking module of the present invention.

[0049] Figure 7 Schematic diagram of target cross-border detection of the present invention.

[0050] Figure 8The figure is a schematic diagram of the structure of the multi-target tracking system for gangue and debris of the present invention. DETAILED DESCRIPTION

[0051] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0052] The description in this section is only for typical embodiments, and the present invention is not limited to the scope of the embodiments. The combination of different embodiments, the mutual replacement of some technical features in different embodiments, and the mutual replacement of the same or similar prior art means with some technical features in the embodiments are also within the scope of the description and protection of the present invention.

[0053] According to one aspect of the present invention, a multi-target tracking method for coal gangue and debris is provided. Figure 1 The framework flow chart of the method is shown, which specifically includes:

[0054] S1: Obtain the coal gangue and debris image target detection dataset.

[0055] S2: Crop the gangue and debris targets in the target detection dataset to generate an image classification dataset.

[0056] S3: Data enhancement of the gangue and debris image object detection dataset is performed through rotation, image scaling and non-linear gamma.

[0057] In one embodiment, the image is rotated to simulate the inspected object at different angles, and the rotation angle is randomly selected in the interval of (0°, 360°). The image is scaled to simulate the sizes of different inspected objects, and the scaling ratio is randomly selected in the interval of (0.3, 2.5). The results generated by image rotation and scaling need to maintain the original image size, remove pixels that exceed the boundaries of the original image, and set the RGB components that are less than the original image size to 128. The image is nonlinearly gamma-transformed to simulate the effects of coal ash, dust, etc. on the lighting conditions of the inspected object, and the gamma transformation is randomly selected in the interval of (0.1, 3.0). Random cropping is not used to prevent coal from being carried on the local surface of the gangue, resulting in data set contamination.

[0058] S4: Input the samples in the target detection dataset and the data augmentation dataset into the YOLOV5S network model with enhanced global feature perception capability for training to obtain the gangue and debris target detection model.

[0059] In one embodiment, Figure 2 and Figure 3As shown, the YOLOV5S network model with enhanced global feature perception ability adds ACMIX modules to the backbone network and feature fusion network, integrating convolution and self-attention hybrid mechanisms, so that high-level features not only have local features, but also have the ability to perceive global features. The CA coordinated attention module is added to the model prediction network to fully consider the relationship between the feature map channel and the spatial position information, thereby enhancing the model's ability to perceive global features without adding additional computing overhead.

[0060] The ACMIX module splits both convolution and self-attention into two parts. In the convolution part, the traditional convolution with a convolution kernel of k×k is decomposed into k×k separate 1×1 convolutions, and then shifted and summed. In the self-attention part, the projection of Query, Key, and Value is interpreted as multiple 1×1 convolutions, and then the attention weights and aggregation values ​​are calculated.

[0061] like Figure 4 As shown in the figure, the CA coordinated attention module can fully consider the relationship between the feature map channel and the spatial position information, and can be flexibly inserted into the mobile network without adding additional computing overhead. The CA module first performs global average pooling on the feature map output by the CSP2_1 module in both width and height directions, and obtains the feature map in the corresponding direction. The definition is as follows:

[0062]

[0063] in Represents the output feature map of the cth channel at height h. Represents the output feature map of the cth channel with a width of w, and x is a point on the feature map.

[0064] Secondly, the feature maps of the two directions are concatenated, and the dimension is reduced to the original C / r through a shared 1×1 convolution. Then the feature map is normalized and nonlinearity is increased through the h-swish activation function, as shown in Equation 3:

[0065] f=δ(F1([z h , z w ]))

[0066] Where [·,·] represents the fusion operation along the spatial dimension. F1 is a 1×1 convolution, δ represents a nonlinear activation function, here the h-swish function. f represents the intermediate feature map that encodes spatial information in the horizontal and vertical directions.

[0067] It is then decomposed into two separate feature maps f along the spatial dimension h and f w, and use 1×1 convolution to modify the number of feature map channels to the original number of channels, and after the Sigmoid activation function, the attention weights g of the feature map in height and width are obtained respectively. h and the attention weight g in the width direction w . It is defined as follows:

[0068] g h =δ(F h (f h ))

[0069] g w =δ(F w (f w ))

[0070] Where: δ is the sigmoid function, F h and F w Corresponding to the feature map f h and f w 1×1 convolution operation.

[0071] Finally, by multiplying the original feature map with weights, we get the final feature map with attention weights in the width and height directions, which is defined as follows:

[0072]

[0073] Where: y c (i, j) represents the output of c channel with height i and width j.

[0074] S5: Input the samples of the image classification data set into the VIT image classification network for training to obtain the gangue and debris target feature extraction model.

[0075] In one embodiment, Figure 5 As shown in the figure, in the model training stage, the input image is first segmented into serialized image segments, then linearly projected and positionally encoded, features are extracted through the Transformer encoder, and the image classification model training is implemented by the MLP classification head. In the model inference stage, it is mainly used to extract target features, so the extracted image features are output before the MLP classification head to achieve the representation of the target features.

[0076] S6. The real-time image is adjusted to a predetermined resolution and input into the target detection model. After model inference, a two-stage NMS is used to obtain the categories and coordinate frames of the gangue and debris in the image.

[0077] In one embodiment, the prediction results obtained after the input image is inferred by the model need to be filtered by two-stage NMS. First, a higher threshold NMS (the threshold is selected as 0.7) is performed for different categories of detected objects to prevent the high confidence category from suppressing other low confidence categories. Then all detection results are subjected to an overall high threshold NMS (the threshold is selected as 0.9) to prevent the same target from being repeatedly detected as multiple different categories. Among them, in the first stage, the candidate frames of different categories are sorted from high to low according to the confidence scores, and in the second stage, the candidate frames of all categories are sorted from high to low according to the confidence scores. Then select the frame with the highest current confidence, add it to the final output result, calculate the intersection and union ratio of all remaining candidate frames and the currently selected frame, delete all candidate frames whose IoU with the currently selected frame is greater than the set threshold, and repeat the above process until there are no candidate frames of this type.

[0078] S7. Input the categories and coordinate frames of gangue and debris into the target trajectory tracking module, generate the trajectories of gangue and debris through image feature cascade matching and IoU matching, and assign the corresponding categories and category IDs to the determined state trajectories.

[0079] In one embodiment, Figure 6 As shown, the target trajectory tracking module realizes data association by cascade matching and IOU matching.

[0080] In cascade matching, the priority parameter of each survival trajectory is first obtained. The smaller the parameter, the higher the priority. When the trajectory of the determined state matches the detection result of the current frame, the priority parameter is set to 0, otherwise the parameter is increased by 1. For trajectories with high priority, data association is given priority. The association process mainly extracts the target appearance features through the VIT network, and calculates the similarity of the appearance features of the prediction results and the detection results by cosine distance. The cost matrix is ​​generated by combining the Euclidean distance measurement between the target center point and the prediction center point, and finally the Hungarian matching method is used to achieve the mutual association of the targets.

[0081] For the gangue targets that are not matched by cascade matching, the IoU matching strategy is used, and the IoU of the unmatched targets in adjacent frames is used as the cost matrix, combined with Hungarian matching for data association.

[0082] The initialization trajectories are all uncertain, and when the trajectory target is matched three times in a row, the trajectory is set to a certain state. Whenever a false detection occurs, especially when coal is falsely detected as gangue, a target ID is assigned to the false detection target, resulting in a huge difference between the trajectory target ID and the actual number. Therefore, a category ID is assigned to the trajectory in the certain state to ensure the continuity of the counting ID and the accuracy of data statistics.

[0083] S8. For the target in the trajectory, through cross-border detection, determine whether the same target in the previous and next frames crosses the detection boundary line for the first time. Store the coordinates and time of crossing the boundary line into the matrix to be captured, and delete the corresponding trajectory.

[0084] In one embodiment, Figure 7 As shown, the cross-border detection method determines whether the line segments formed by the same target in adjacent frames intersect with the detection boundary line segments, and then determines whether the target crosses the detection boundary line for the first time. The specific method is as follows:

[0085] a. Determine the coordinates of the two vertices A and B of the boundary line, and obtain the Z1 coordinates of the current frame and the Z2 coordinates of the previous frame of the trajectory target;

[0086] b. Determine whether (the outer product of vector Z0Z1 and vector Z0A) × (the outer product of vector Z0Z1 and vector Z0B) is less than 0 to determine whether AB is on both sides of Z0Z1;

[0087] c. Then determine whether (the outer product of vector AB and vector AZ0) × (the outer product of vector AB and vector AZ1) is less than 0 to determine whether Z0Z1 is on both sides of AB;

[0088] d. If both b and c above are true, it is determined that the detection target crosses the detection line.

[0089] like Figure 8 As shown, the present invention also proposes a multi-target tracking system for gangue and debris, comprising:

[0090] The data acquisition module collects the surface image of the conveyor belt through an industrial camera, and obtains the current conveyor belt speed through a belt speed sensor.

[0091] The data preprocessing module preprocesses the images captured by the camera and calculates the moving distance of the targets in adjacent frames in the image coordinate system through motion estimation.

[0092] The target detection module sets the opening and closing of the categories of gangue and debris to be detected through the interface. The preprocessed image is input into the trained target detection model to obtain the categories to be detected and the coordinate frame.

[0093] The trajectory tracking module predicts the next frame position of the target center point through motion estimation, and realizes the correlation between targets in adjacent frames through cascade matching and IoU matching.

[0094] The coordinate sending module sends the out-of-bounds target coordinates and timestamp through out-of-bounds detection.

[0095] Specifically, through out-of-bounds detection, the out-of-bounds target and the current frame timestamp are stored in the robot arm's to-be-grasped matrix, and then the to-be-grasped matrix is ​​sent to the robot arm, and the robot arm's control system determines the optimal grasping process.

[0096] The description and application of the present invention here are illustrative, and it is not intended to limit the scope of the present invention to the above-mentioned embodiments. The effects or advantages involved in the specification may not be reflected in the actual experimental examples due to the uncertainty of specific condition parameters or other factors, and the effects or advantages are not used to limit the scope of the invention. The deformation and change of the embodiments disclosed here are possible, and the replacement of the embodiments and the various equivalent parts are well known to those of ordinary skill in the art. It should be clear to those skilled in the art that the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials and parts without departing from the spirit or essential features of the present invention. Other deformations and changes can be made to the embodiments disclosed here without departing from the scope and spirit of the present invention.

Claims

1. A multi-target tracking method for coal gangue and debris, characterized in that: include: Step S1, obtaining a coal gangue and debris image target detection data set; Step S2, cutting out coal gangue and debris targets from the target detection data set, and generating an image classification data set; Step S3, performing data enhancement on the gangue and debris image target detection dataset by image rotation, scaling and nonlinear gamma; Step S4: input the samples in the target detection data set and the target detection data set after data enhancement into the YOLOV5S network model with enhanced global feature perception capability for training, so as to obtain a coal gangue and debris target detection model; Step S5: Input the samples in the image classification data set into the VIT image classification network model for training to obtain a network model for extracting features of gangue and debris images; Step S6, adjusting the real-time image to a predetermined resolution and inputting it into the gangue and debris target detection model, and after model reasoning, using a two-stage non-maximum suppression NMS to obtain the category and coordinate frame of the gangue and debris in the real-time image; Step S7, inputting the image, category and coordinate frame of the gangue and debris into the gangue and debris image feature extraction network model, obtaining feature data after model inference, tracking the target trajectory through image feature data association, generating the gangue and debris trajectory, and assigning the corresponding category ID to the determined state trajectory; Step S8: For the targets in the determined trajectory of gangue and debris, determine whether the same target in the previous and next frames crosses the detection boundary line for the first time through cross-border detection, store the coordinates of the crossing boundary line and the current timestamp in the robot arm to be grasped matrix, and delete the corresponding trajectory.

2. The multi-target tracking method for gangue and debris according to claim 1, characterized in that: The data enhancement described in S3 includes: The image is randomly rotated to simulate the inspected object at different angles, and the rotation angle is randomly selected in the interval (0°, 360°); The image is randomly scaled to simulate objects of different sizes, and the scaling ratio is randomly selected in the interval (0.3, 2.5); The results of random rotation and random scaling of the image keep the original image size, remove pixels beyond the original image boundary, and set the RGB components that are less than the original image size to 128; The image is subjected to nonlinear gamma transformation to simulate the influence of coal ash, dust, etc. on the illumination conditions of the inspected object, and the gamma transformation takes random values ​​in the interval of (0.1, 3.0).

3. The multi-target tracking method for gangue and debris according to claim 1, characterized in that: The YOLOV5S network model with enhanced global feature perception capability described in S4 includes: In the process of extracting features by downsampling the input image, the CBL module is used to focus on local detail features for low-level features; the ACMIX module is combined with high-level features to extract local features through convolution, and intermediate features through 1x1 convolution. The convolution and self-attention methods are used to reuse and aggregate intermediate features, so that high-level features not only have local features, but also have the ability to perceive global features; For the three effective feature layers extracted after feature fusion, CA coordinated attention modules are added respectively to comprehensively consider the relationship between feature map channels and spatial position information, and enhance the perception ability of global features.

4. The multi-target tracking method for gangue and debris according to claim 1, characterized in that: The VIT image classification network model described in S5 includes: The images in the target detection dataset are divided into serialized image segments, linearly projected and positionally encoded, and the Transformer encoder is used to extract features from the image segments. The MLP classification head is used to train the image classification model.

5. The multi-target tracking method for gangue and debris according to claim 1, characterized in that: The two-stage NMS operation in S6 includes: For different categories of detected objects, NMS with a higher threshold is performed respectively, all targets of each category are screened out, and candidate boxes of different categories are sorted from high to low according to the confidence score, and the higher threshold is 0.7; Perform an overall high-threshold NMS on all detection results, and sort all category candidate boxes from high to low according to the confidence score, where the high threshold is 0.9; Select the coordinate box with the highest current confidence and add it to the final output result. Calculate the intersection and union ratio of all remaining candidate coordinate boxes with the currently selected coordinate box, delete all candidate boxes whose IoU with the currently selected box is greater than the set threshold, and repeat the above process until there are no such candidate boxes.

6. The multi-target tracking method for gangue and debris according to claim 1, characterized in that: The target trajectory tracking in S7 includes: The trajectory is initialized according to the input coal gangue and debris categories and coordinate frames, and the initialized trajectories are all uncertain; Predict the image position of the target in the next frame based on the conveyor belt speed, adjacent frame time and affine transformation of the actual distance and image distance; The predicted value of the previous frame of the trajectory object is associated with the actual detection value of the current frame, and the target detected multiple times in succession is determined as a deterministic trajectory, and an ID sequence corresponding to the category is assigned; when the deterministic trajectory fails to match multiple times in succession, or when the center point of the trajectory target after motion prediction exceeds the image size, the trajectory is deleted.

7. The multi-target tracking method for gangue and debris according to claim 1, characterized in that: The data association in S7 includes: Through cascade matching, the gangue and debris image feature extraction network model is used to extract the features of the image area corresponding to the coordinate frame, calculate the feature similarity for adjacent frame targets, and generate a cost matrix; The target movement distance is calculated by the conveyor belt speed and the time difference between adjacent frames, and the image center coordinates of the next frame are predicted by combining the actual distance with the affine transformation matrix of the image distance. The feature cost matrix is ​​updated by the Euclidean distance of the target center points between adjacent frames, and the front-back correlation of the target is realized by the Hungarian matching algorithm; Through IoU matching, the overlap of the bounding boxes of the same category of targets in the previous and next frames is calculated, and a cost matrix is ​​generated, and the Hungarian matching algorithm is used for data association.

8. The multi-target tracking method for gangue and debris according to claim 1, characterized in that: The cross-border detection in S8 includes: For the target in the trajectory, determine whether the line segment formed by the target center point coordinates of the previous and next two frames intersects with the line segment formed by the pre-set boundary line vertices. If so, store the target coordinates and timestamp in the robot arm's grasping matrix, and the robot arm grasps it according to the current working status.

9. A multi-target tracking system for coal gangue and debris, characterized in that: include: The data acquisition module collects the surface image of the conveyor belt and obtains the conveyor belt speed at the current moment; A data preprocessing module preprocesses the surface image and calculates the moving distance of the target in adjacent frames of the surface image in the image coordinate system through motion estimation; The target detection module sets the opening and closing of the categories to be detected for gangue and debris, inputs the preprocessed surface image into the target detection model, and obtains the categories to be detected and the coordinate frame; The trajectory tracking module predicts the next frame position of the target center point through motion estimation, and associates the targets between adjacent frames through cascade matching and IoU matching; The coordinate sending module sends the out-of-bounds target coordinates and timestamp matrix to be grasped by the robotic arm through out-of-bounds detection.

Images