Coal mining machine roller tracking system and method

Abstract

The invention relates to a coal mining machine roller tracking system and method. The system comprises a target detection module, a multi-feature fusion matching module and a track management module, and the target detection module is used for detecting a roller target based on an input video and extracting appearance features of a roller. The multi-feature fusion matching module fuses the position overlap ratio, the appearance similarity and the motion consistency of the roller target based on the appearance features to judge the matching degree with the historical tracking trajectory of the roller, and the trajectory management module is used for maintaining the tracking trajectory of the complete life cycle of the roller target, the target detection module adopts a YOLOv11 model, and a trunk layer of the YOLOv11 model adopts MobileNetV4. According to the coal mining machine roller tracking system and method, the operation efficiency of coal mining machine roller tracking is improved, the tracking robustness is enhanced, and reliable technical support is provided for intelligent identification and safety monitoring of the operation state of coal mining equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of coal mine safety monitoring; specifically, the present application relates to a shearer drum tracking system and method. BACKGROUND

[0002] With the continuous advancement of intelligent mining in coal mines, as the core production equipment, the precise monitoring and target tracking of the shearer have become an important basis for ensuring the safety production of coal mines and realizing intelligent control. Especially in the shearer drum area, due to its complex environment of high-speed rotation, high dust, strong shielding, etc., traditional manual monitoring methods are inefficient, have large errors, and are difficult to meet the detection needs of high precision and real-time. Therefore, target tracking technology based on computer vision is gradually applied to the operation monitoring of the shearer drum.

[0003] Currently, multi-target tracking algorithms such as DeepSORT and StrongSORT are widely used in the field of target tracking, combined with detectors such as YOLO, which can realize automatic detection and continuous tracking of targets. Among them, StrongSORT performs excellently on public datasets by enhancing the appearance feature extraction and trajectory management mechanism. However, directly applying StrongSORT to the tracking scene of the shearer drum still has problems and deficiencies such as serious feature extraction time consumption, single matching mechanism, and insufficient use of motion information. SUMMARY

[0004] Therefore, the present application provides a shearer drum tracking system and method, thereby solving or at least alleviating one or more of the above problems and other aspects in the prior art.

[0005] In order to achieve the foregoing purposes, the first aspect of the present application provides a shearer drum tracking system, wherein the system comprises a target detection module, a multi-feature fusion matching module and a trajectory management module, the target detection module is used for detecting a drum target and extracting the appearance features of the drum based on an input video, the multi-feature fusion matching module judges the matching degree of the drum target with the historical tracking trajectory of the drum based on the position coincidence degree, the appearance similarity and the motion consistency of the drum target, and the trajectory management module is used for maintaining the tracking trajectory of the complete life cycle of the drum target, wherein the target detection module adopts a YOLOv11 model, and the backbone layer of the YOLOv11 model uses MobileNetV4.

[0006] In the system as described above, optionally, the structural framework of the YOLOv11 of the target detection module comprises an input layer, a backbone layer, a neck layer and a head layer, the input layer is used for raw video input, the backbone layer takes the MobileNetV4 as a core feature extraction network, the neck layer comprises a plurality of C3k2 modules, convolutional layers and up-sampling modules, and the head layer comprises a plurality of detection modules.

[0007] In the system as described above, optionally, the multi-feature fusion matching module extracts the appearance features of the drum through the MobileNetV4, predicts the motion state of the drum target by using Kalman filtering, comprehensively calculates the position coincidence degree, appearance similarity and motion consistency of the drum and performs fusion by dynamic weighting, and completes the optimal matching of the detection frame of the drum target and the tracker of the drum target by using the Hungarian algorithm.

[0008] In the system as described above, optionally, the weight of the motion consistency is increased when the drum target is occluded.

[0009] To achieve the foregoing object, a second aspect of the present application provides a coal cutter drum tracking method based on the coal cutter drum tracking system according to any one of the foregoing first aspects.

[0010] In the method as described above, optionally, the method comprises the following steps: Step I, start, and then proceed to step II; Step II, collect coal cutter drum pictures as a training set and a verification set, and then proceed to step III; Step III, train the YOLOv11 model, and then proceed to step IV; Step IV, input a coal cutter drum video, and then proceed to step V; Step V, detect the coal cutter drum by using the trained YOLOv11 model, and then proceed to step VI; Step VI, optimize the detection result according to a confidence threshold, and then proceed to step VII; Step VII, predict the position of the drum according to Kalman filtering, and then proceed to step VIII; Step VIII, determine whether the drum is detected in the current frame, yes, proceed to step IX, or no, proceed to step XVIII; Step IX, extract deep appearance features of the drum, and then proceed to step X; Step X, determine a final association score value based on appearance feature cosine similarity, detection frame and predicted frame IoU and motion trajectory consistency, and then proceed to step XI; Step XI, match using Hungarian algorithm, then proceed to Step XII; Step XII, determine if match is successful, if yes, proceed to Step XIX, otherwise proceed to Step XIII; Step XIII, determine if bounding box is matched, if yes, proceed to Step XIX, otherwise proceed to Step XIV; Step XIV, track initialization, then proceed to Step XV; Step XV, determine if trajectory tracker is matched, if yes, proceed to Step XIX, otherwise proceed to Step XVI; Step XVI, determine if loss count exceeds m frames, if yes, proceed to Step XVII, otherwise proceed to Step XX; Step XVII, delete tracker, then proceed to Step V; Step XVIII, determine if no detection for n consecutive frames, if yes, proceed to Step XVII, otherwise proceed to Step XX; Step XIX, update drum position, then proceed to Step XXI; Step XX, output predicted position, then proceed to Step XXI; Step XXI, output updated position, then proceed to Step XXII; Step XXII, end.

[0011] In the method as described above, optionally, m = 20, n = 5 In the method as described above, optionally, wherein: In the step II, the training set and the validation set contain drum images under different light and shielding conditions, and the drum is labeled by using a labelme labeling tool; and / or In the step III, the training period of the YOLOv11 model is set to 150 epochs, the batch size is 16, the input image size is 640, a pre-trained model is used for fine-tuning, the optimizer is Adam, the initial learning rate lr0 is set to 0.01, the terminal learning rate is attenuated through lrf = 0.01, the optimizer with a momentum of 0.937 is used, and weight decay is introduced to prevent overfitting; and / or In the step V, the original bounding box information is output: , , w, h, conf, class], wherein is the center coordinate of the box, conf is the confidence, w is the width of the detection box, h is the height of the detection box, and class is the class label of the target in the detection box; and / or In the step VI, the bounding boxes with the confidence lower than the confidence threshold are filtered out, non-maximum suppression is performed based on the IoU threshold, and overlapping boxes are eliminated to obtain a list of valid bounding boxes detections = where IoU is the intersection over union, defined as follows: where A and B are the bounding box A and the bounding box B, respectively, represents the intersection of the two boxes, and if the two boxes have no overlap, the intersection area is 0, represents the union of the two boxes, and for each bounding box , the image in the cropped box is cropped and input into the MobileNetV4, and a 128-dimensional feature vector f(i) is extracted from each cropped image; and / or In the step VII, the moving speed of the roller is defined as: wherein, and are the center point coordinates of the roller at the current time t, and are the coordinates of the center point of the roller at t-1 time, is the adjacent frame time interval, and the state vector of the roller is defined as: The state transition matrix F is defined as follows: According to Kalman filtering, the predicted position of the roller is: obtained: wherein, is the predicted state vector of the roller at t+1 time, and are the center point coordinates of the predicted roller at t+1 time, and the length and width of the predicted box are the same as those at t time; and / or In the step X, the detection box and the prediction box are used to calculate: wherein, represents the detection box, represents the prediction box, appearance cosine similarity The appearance feature vector is defined as follows by the following calculation: wherein, is the detection image of the current frame, is the feature extraction model MobileNetV4, is the appearance feature vector of the extracted target, the appearance feature similarity of the target between the current frame i and the last frame i-1 is calculated: The motion consistency is calculated by calculating the Mahalanobis distance to evaluate the matching degree of the current detection target and the existing trajectory: wherein, z is the detected new position of the roller, H is the observation matrix, S is the measurement noise covariance matrix, which is related to the camera resolution and image noise, and represent the detection error variance in the X and Y directions, which can be obtained by statistical data of historical measurement data or set as an empirical value, and a set of target boxes are detected in the current frame , which need to be matched one by one with the target boxes predicted by the last frame tracker to construct a comprehensive cost matrix , wherein each element represents the mismatch cost of the detection box and the predicted box , and each cost element comprehensively considers three kinds of similarity information: position IoU, appearance feature, and motion consistency: The position cost is: The appearance cost is: The motion consistency cost is: , is a adjustable normalization parameter, defines the average stability index of each similarity term, and then the weight is dynamically calculated according to the average stability index: IoU average confidence average value of appearance similarity average value of Mahalanobis distance Wherein, N represents the total number of the combined pairs of detection frame and prediction frame to be matched in the current frame, the normalization is integrated, and a dynamically updated weight is generated: Then the final cost matrix is ; and / or In step XI, a Hungarian algorithm is applied to solve the minimum cost matching, wherein Score= A matching threshold thresh is set to determine whether the current detection result and the historical trajectory are successfully matched, and the specific determination rule is as follows: if Score ≤ thresh, it is considered that the detection frame and the tracking trajectory are successfully matched, the position of the current detection frame is used to update the corresponding trajectory state, the latest spatial coordinates of the drum are obtained, and if Score> thresh, it is considered that the matching cost between the detection result and the existing trajectory is too large, and the matching fails.

[0012] In the method as described above, optionally, the confidence threshold is 0.5, and the IoU threshold is 0.6.

[0013] In the method as described above, optionally, in the step XX, a Kalman filter is started to perform motion prediction on the position of the drum target.

[0014] The coal winning machine drum tracking system of the present application adopts a lightweight feature extraction network MobileNetV4 in the target matching stage, significantly reduces redundant calculation, solves the problem of time-consuming feature extraction, and improves the real-time performance and deployment efficiency of the overall tracking system.

[0015] The present application further provides a coal winning machine drum tracking method based on the coal winning machine drum tracking system of the present application, so the method also has the above advantages. BRIEF DESCRIPTION OF DRAWINGS

[0016] The disclosure of the present application will be more apparent with reference to the accompanying drawings. It should be understood that these drawings are only for illustrative purposes, and are not intended to limit the scope of protection of the present application. In the drawings: Figure 1 is a schematic block diagram of an embodiment of the coal winning machine drum tracking system of the present application; Figure 2 is a structural schematic diagram of a target detection module of an embodiment of the coal winning machine drum tracking system of the present application; Figure 3A model training result diagram of a target detection module of an embodiment of the coal mining machine drum tracking system of the present application; Figure 4 A drum tracking visualization diagram of a video frame 130 of an embodiment of the coal mining machine drum tracking system of the present application; Figure 5 A drum tracking visualization diagram of a video frame 144 of an embodiment of the coal mining machine drum tracking system of the present application; Figure 6 A drum tracking visualization diagram of a video frame 160 of an embodiment of the coal mining machine drum tracking system of the present application; Figure 7 A drum tracking visualization diagram of a video frame 176 of an embodiment of the coal mining machine drum tracking system of the present application; Figure 8 A flowchart of an embodiment of the coal mining machine drum tracking method of the present application. DETAILED DESCRIPTION

[0017] The structure, composition, features and advantages of the coal mining machine drum tracking system and method of the present application will be described below in an exemplary manner with reference to the accompanying drawings and specific embodiments, however, all the descriptions shall not be used to form any limitation on the present application.

[0018] In addition, for any single technical feature described or implied in the embodiments mentioned herein, or any single technical feature shown or implied in the drawings, the present application still allows any combination or deletion to be continued between these technical features (or their equivalents) without any technical obstacles, so it should be considered that more embodiments according to the present application are also within the scope of the description herein.

[0019] Figure 1 A schematic block diagram of an embodiment of the coal mining machine drum tracking system of the present application.

[0020] As shown in the figure, in this embodiment, the system includes a target detection module, a multi-feature fusion matching module and a trajectory management module. Among them, the target detection module detects the coal mining machine drum using YOLOv11 and transmits it to the multi-feature fusion matching module, and outputs the target position and appearance features to the multi-feature fusion matching module; the multi-feature fusion matching module fuses the target appearance features (extracted by MobileNetV4), the position overlap degree IoU and the motion trajectory consistency; the trajectory management module constructs a multi-feature cost matrix according to the three features fused by the multi-feature fusion matching module, and then performs cascade matching.

[0021] Specifically, the target detection module serves as the front end of the system, adopts a lightweight YOLOv11 model to perform real-time analysis on the input video frames, eliminates low-quality detection through a confidence threshold, eliminates redundant boxes through non-maximum suppression, and outputs a high-precision detection box of the shearer drum, which frames the accurate position of the shearer drum in the input video frame. Optionally, the confidence threshold is 0.5, and the parameter IoU threshold in the non-maximum suppression is 0.6. The YOLOv11 is an iterative version of the Ultralytics YOLO series real-time target detector.

[0022] Further, the target detection module is optimized for the special underground environment, and uses MobileNetV4 as the backbone network of YOLOv11, effectively overcoming the detection challenges caused by dust interference and changes in illumination.

[0023] Further, the target detection module uses the MobileNetV4 backbone network to extract appearance features.

[0024] Specifically, the multi-feature fusion matching module constitutes the core decision-making layer of the system, and innovatively integrates multiple feature information of the target. The MobileNetV4 backbone network of the target detection module extracts appearance features, and simultaneously uses Kalman filtering to predict the motion state of the target, and then comprehensively calculates three key indicators of position coincidence degree, appearance similarity and motion consistency.

[0025] Further, the multi-feature fusion matching module uses a dynamic weighting fusion strategy to enable the system to adaptively balance the importance of different features, which can include increasing the weight of the motion feature when the occlusion is severe. The Hungarian algorithm is used to finally complete the optimal matching of the detection box and the tracker. This multi-dimensional decision mechanism significantly improves the accuracy of matching. The tracker is a dynamic data management unit that is continuously bound to a specific shearer drum target.

[0026] Specifically, the trajectory management module is responsible for maintaining the complete life cycle of the tracked target. For the matched tracker, the module updates its motion state and appearance features in a timely manner; for a newly appearing detection target, a cautious initialization strategy is adopted, and a new tracker is established only after continuous multiple frames are confirmed; for a temporarily lost target, short-time tracking is maintained through motion prediction.

[0027] Further, the trajectory management module also includes a perfect tracker cleaning mechanism to ensure effective use of system resources. And it is specially optimized for the special motion characteristics of the shearer drum, including expanding the state vector to better describe the rotational motion, adjusting the parameters to cope with the harsh underground environment, and finally achieving stable and reliable real-time tracking performance.

[0028] Figure 2A structural diagram of a target detection module of an embodiment of a coal cutter drum tracking system of the present application.

[0029] As shown in the figure, in this embodiment, the target detection module includes an input layer, a backbone layer, a neck layer and a head layer. The input layer Input represents the original image input; the backbone layer adopts MobileNetV4 for feature extraction, and sends the extracted multi-level features to the neck layer; the neck layer includes multiple C3k2 modules, convolution layers (Conv), up-sampling modules (Upsample) and multiple connection operations (C) for feature fusion and enhancement; the head layer includes multiple detection modules (Detect) that receive the multi-scale fusion feature maps output by the neck layer and perform target detection at different scales respectively.

[0030] The target detection module of this embodiment adopts an optimized YOLOv11 model, and its structure includes an input layer, a backbone layer, a neck layer and a head layer. Among them, C3k2 is a feature extraction module in YOLOv11.

[0031] Specifically, the input layer inputs the original image, which will be subjected to multi-scale feature extraction and target detection.

[0032] Specifically, the backbone layer adopts MobileNetV4 as the core feature extraction network. MobileNetV4 is the fourth generation model of the MobileNet series of deep learning network models, which has a general and efficient architecture design suitable for mobile devices. It introduces a universal inverse bottleneck (UIB) module, which combines an inverse bottleneck, ConvNeXt, a feedforward network (FFN) and a new enhanced depth convolution, forming a unified and flexible structure. Combined with a mobile quick attention module (Mobile MQA) designed specifically for mobile accelerators and an optimized neural architecture search (NAS) strategy, it significantly reduces the parameter quantity and computational complexity while maintaining strong feature expression ability. The multi-level features extracted by the backbone layer are then sent to the neck layer.

[0033] Specifically, the neck layer mainly undertakes feature fusion and enhancement functions. It receives the multi-level features extracted by the backbone layer and performs further multi-scale fusion. The layer includes multiple C3k2 modules, convolution layers (Conv), up-sampling modules (Upsample), and connection operations (C). The C3k2 module realizes efficient multi-scale feature extraction and lightweight design through parallel convolution branches and flexible parameter configuration, improves the calculation speed and accuracy, and is particularly suitable for mobile deployment; the convolution layer (Conv) is used to adjust the channel or enhance the features of the fused feature map, to ensure that the features transmitted to the downstream detection head have stronger discrimination ability; the up-sampling module (Upsample) is used to restore the low-resolution deep feature map to a higher resolution, so as to be fused with the shallow layer features, realize cross-layer feature alignment and multi-scale perception; the connection operation (C) represents the splicing or fusion between feature maps, which is used to enhance the expression of multi-source information, and the connection operation can be optionally selected as Concat operation. The neck layer outputs multi-scale fusion feature maps to the head layer.

[0034] Specifically, the head layer is composed of multiple detection modules (Detect), which are respectively used for target detection at different scales to meet the identification needs from small targets to large targets. These detection heads receive the multi-scale fusion feature maps output by the neck layer and are respectively responsible for predicting the position of the bounding box, the class probability, and other information.

[0035] Figure 3 A model training result graph of the target detection module of an embodiment of the coal mining machine drum tracking system of the present application.

[0036] In this embodiment, the target detection module uses an optimized YOLOv11 model, which has the same structure as the model used in the embodiment shown in Figure 2 and will not be described here again.

[0037] The figure shows the process of detecting and training the coal mining machine drum by the optimized YOLOv11 model, which shows the changes of the training and validation indicators of the target detection model. They include the bounding box regression loss (box_loss), the classification loss (cls_loss), and the distribution focal loss (dfl_loss) in the training (train) and validation (val) processes, as well as the evaluation indicators of precision, recall, mAP50, and mAP50-95.

[0038] As shown in the figure, during the model training and validation process, the various losses (box_loss, cls_loss, dfl_loss) steadily decrease, and the precision, recall, and mAP50 improve to more than 0.8, indicating that the model has effectively learned.

[0039] Figure 4 、 Figure 5 、 Figure 6 and Figure 7 are the roller tracking visualization diagrams of the 130th frame, the 144th frame, the 160th frame and the 176th frame of a video of an embodiment of the coal mining machine roller tracking system of the present application.

[0040] The green roller detection box is shown in the figure; the pentagram in the detection box represents the center position of the roller; the roller identification DRUM 1 is labeled above the detection box, i.e., the roller ID identification is 1; and the confidence (Conf) is labeled above the detection box, the detection box confidence in the 130th frame and the 144th frame is 0.92, and the detection box confidence in the 160th frame and the 176th frame is 0.91; the serial number of the current frame (Frame) is labeled in the upper right corner of the figure. As shown in the figure, the coal mining machine roller tracking system of the embodiment can continuously track the position of the coal mining machine roller from the video frames, including the center position, size and range of the roller in the video frames, and the detection box has a high confidence, indicating that the system can accurately track the roller.

[0041] Figure 8 is a flowchart of an embodiment of the coal mining machine roller tracking method of the present application.

[0042] As shown in the figure, in this embodiment, the method includes the following steps: Step I, start, and then proceed to step II; Step II, collect coal mining machine roller pictures as a training set and a verification set, and then proceed to step III; Step III, train with the improved YOLOv11 model, and then proceed to step IV; Step IV, input the coal mining machine roller video, and then proceed to step V; Step V, use the trained model to detect the coal mining machine roller, and then proceed to step VI; Step VI, judge the detection result according to the confidence threshold, and then proceed to step VII; Step VII, predict the position of the roller according to the Kalman filter, and then proceed to step VIII; Step VIII, judge whether the roller is detected in the current frame, yes to proceed to step IX, or no to proceed to step XVIII; Step IX, extract the depth appearance features of the roller, and then proceed to step X; Step X, determine the final association score value based on the appearance feature cosine similarity, the detection box and the predicted box IoU, and the motion trajectory consistency, and then proceed to step XI; Step XI, match using the Hungarian algorithm, and then proceed to step XII; Step XII, determine if the match is successful, yes proceed to step XIX, otherwise proceed to step XIII; Step XIII, determine if the bounding box matches, yes proceed to step XIX, otherwise proceed to step XIV; Step XIV, track initialization, then proceed to step XV; Step XV, determine if the trajectory tracker matches, yes proceed to step XIX, otherwise proceed to step XVI; Step XVI, determine if the loss count exceeds 20 frames, yes proceed to step XVII, otherwise proceed to step XX; Step XVII, delete the tracker, then proceed to step V; Step XVIII, determine if there have been 5 consecutive frames without detection, yes proceed to step XVII, otherwise proceed to step XX; Step XIX, update the drum position, then proceed to step XXI; Step XX, output the predicted position, then proceed to step XXI; Step XXI, output the updated position, then proceed to step XXII; Step XXII, end.

[0043] Specifically, in step II, a coal mining machine drum image dataset is collected, which contains drum images under different light and occlusion conditions, and the drum is labeled with a labeling tool. Optionally, the labeling tool can be labelme.

[0044] Specifically, in step III, the improved YOLOv11 model is trained on the aforementioned dataset, and the improved YOLOv11 model uses a lighter MobileNetV4 as the backbone feature extraction network. In the experiment, the target detection module selects appropriate training period, batch size and input image size, uses a pre-trained model for fine-tuning, sets an appropriate initial learning rate lr0 and decay factor lrf for the Adam (Adaptive Moment Estimation, an adaptive learning rate optimization algorithm) optimizer, so that the end learning rate is finally decayed to lr0 * lrf through lrf; an appropriate momentum optimizer is used, and weight decay is introduced to prevent overfitting. Optionally, the training period is set to 150 epochs (an epoch represents a complete traversal of the training dataset), the batch size is 16, the input image size is 640, the initial learning rate lr0 is 0.01, the decay factor lrf is 0.01, the optimizer momentum is 0.937, and the weight decay is 0.0005.

[0045] Specifically, in step IV, the image input size can be 640x640.

[0046] Specifically, in step V, the trained YOLOv11 model is used to detect the shearer drum video, and the original detection box information is output: , , w, h, conf, class], wherein is the box center coordinate, w is the width of the detection box, h is the height of the detection box, conf is the confidence, and class is the class label of the target in the detection box.

[0047] Specifically, in step VI, the detection boxes with a confidence lower than the confidence threshold are filtered out, non-maximum suppression (NMS) is performed based on the IoU threshold, and overlapping boxes are eliminated to obtain the valid detection box list detections = . Optionally, the confidence threshold is 0.5, and the IoU threshold is 0.6. Moreover, the IoU is defined as follows: wherein A and B are the boxes A and B, respectively, the intersection of the two boxes is the intersection area, and if the two boxes have no overlap, the intersection area is 0, and the union of the two boxes is the union area.

[0048] Specifically, in step VII, the moving speed of the drum is defined as: wherein and are the center point coordinates of the drum at the current t time, and are the coordinates of the center point of the drum at t-1 time, and ∆t is the adjacent frame time interval. Then the state vector of the drum is defined as: The state transition matrix F is defined as follows: According to Kalman filtering, the predicted position of the drum is: Thus, wherein is the predicted state vector of the drum at t+1 time, and For the center point coordinates of the predicted drum at time t+1, the length and width of the prediction box are the same as those at time t.

[0049] Specifically, in step IX, for each detection box , the image within the box is cropped, and the shared MobileNetV4 backbone network is inputted, and a feature vector f(i) is extracted from each cropped image. Optionally, the feature vector has a dimension of 128.

[0050] Specifically, in step X, the IoU of the detection box and the prediction box is calculated as follows: wherein, represents the detection box, represents the prediction box.

[0051] Specifically, in step X, the appearance feature cosine similarity is calculated as follows: the appearance feature vector is defined as: wherein, I(i) is the detection image of the current frame, is the feature extraction model MobileNetV4, and f(i) is the appearance feature vector of the extracted target, and the appearance feature similarity of the target between the current frame i and the previous frame i-1 is calculated as follows: The cosine similarity has a value range of [-1, 1], and the closer the value is to 1, the higher the matching degree.

[0052] Specifically, in step X, the motion trajectory consistency is calculated as follows: the Mahalanobis distance is calculated to evaluate the matching degree of the current detection target and the existing trajectory: wherein, z is the detected new position of the drum, H is an observation matrix, and S is a measurement noise covariance matrix, which is related to the camera resolution and image noise, and represent the detection error variances in the X and Y directions. Optionally, and can be obtained through historical measurement data statistics or set as empirical values.

[0053] Specifically, in step X, the final association score is embodied by a cost matrix, which is calculated by: a set of bounding boxes are detected in the current frame , which need to be matched one by one with the bounding boxes predicted by the tracker in the last frame , to construct a comprehensive cost matrix , where each element represents the cost of mismatch between the detection box and the predicted box , and the smaller the cost is, the higher the matching possibility is. Each cost element comprehensively considers three kinds of similarity information: position IoU, appearance feature, and motion consistency. The position cost is calculated by Similarly, the appearance cost is: Similarly, the motion consistency cost is: where τ is a tunable normalization parameter The larger the above position cost, appearance cost, and motion consistency cost are, the more mismatched they are. The average stability indicators IoU average confidence , average value of appearance similarity , and average value of Mahalanobis distance of each similarity term are defined, and then the weights are dynamically calculated according to the indicators: where N represents the total number of combinations of detection boxes and predicted boxes to be matched in the current frame. The weights are normalized and generated dynamically: The final cost matrix is calculated as Specifically, in step XI, the Hungarian algorithm is applied to solve the minimum cost matching. In order to realize stable target tracking of the coal mining machine roller, in each frame of image, the current detection result is first matched with the historical tracking trajectory. The matching is based on comprehensive consideration of the spatial position, appearance feature, and motion continuity between targets, so as to calculate a comprehensive matching cost score Score: Score ​Specifically, in step XII, a matching threshold thresh is set to determine whether the current detection result matches the historical trajectory. The specific determination rule is as follows: if Score<=thresh, it is considered that the detection frame matches the tracking trajectory, the position of the current detection frame is used to update the corresponding trajectory state, and the latest spatial coordinates of the drum are obtained; if Score>thresh, it is considered that the matching cost of the detection result and the existing trajectory is too large, and the matching fails.

[0054] Specifically, in step XIII, for the detection frame in the current frame that is not matched by any tracker, that is, the new target detected by the target detector, the following rules are used for processing: if the detection frame can satisfy certain appearance or motion prediction similarity with a certain unsuccessful matching tracker, it is still considered as potential matching, and the tracker state is updated; if the detection frame does not match all existing trackers, and its detection confidence exceeds a certain threshold, it is considered as a new target, step XIV is entered, a new tracker is created and its state is initialized, and is used for subsequent target tracking.

[0055] Specifically, in step XVI, for each unsuccessful matching tracker, it is explained that the target may be temporarily disappeared due to reasons such as occlusion, illumination change, target deformation or detector missing. At this time, a two-level processing strategy of prediction tracking and deleting the tracker is adopted. The prediction tracking strategy includes: if the lost frame number of the trajectory does not exceed 20 frames, a Kalman filter is enabled to perform motion prediction on the position of the target, and the prediction result is used as the output drum position of the current frame to ensure tracking continuity. The delete tracker strategy includes: if the cumulative lost frame number of the trajectory exceeds 20 frames, it is considered that the target has permanently left the field of view, the current tracker is invalid, and it needs to be deleted from the system to release the computing resources and maintain the system stability.

[0056] The tracking matching process realized by steps XI, XII, XIII and XVI ensures that the coal mining machine drum tracking method still has strong robustness and continuity in the case of detection result fluctuation or short-time target missing. Through the dynamic threshold matching strategy, the detection and tracking joint update mechanism, the Kalman filter prediction compensation, and the redundant tracker timely cleaning mechanism, stable and efficient tracking of the coal mining machine drum can be realized, and a stable data basis is provided for subsequent attitude recognition, state evaluation and other tasks.

[0057] Some embodiments of the coal mining machine drum tracking system and method of the present application can have one or more of the following beneficial effects compared with the prior art: (1) Improve operation efficiency: In the target matching stage, the method uses the lightweight feature extraction network MobileNetV4 shared with the YOLOv11 detector, significantly reduces redundant calculation, solves the problem of time-consuming feature extraction, and improves the real-time performance and deployment efficiency of the whole tracking system; (2) Enhance tracking robustness: The position coincidence degree, appearance feature similarity and motion trajectory consistency of the target are fused for joint matching decision, which significantly improves the accuracy and stability of the target tracking of the shearer drum under complex working conditions such as occlusion, shaking and low light, and reduces the problem of mis-matching and ID switching.

[0058] The technical scope of the present application is not limited to the above description, and those skilled in the art can make various modifications and changes to the above embodiments without departing from the technical idea of the present application, and these modifications and changes should be within the scope of the present application.

Claims

1. A shearer drum tracking system characterised in that, The system comprises a target detection module, a multi-feature fusion matching module and a track management module, the target detection module is used for detecting a drum target based on an input video and extracting appearance features of the drum, the multi-feature fusion matching module judges matching degrees of the drum target with historical tracking tracks of the drum based on position coincidence degrees, appearance similarities and motion consistencies of the drum target fused based on the appearance features, and the track management module is used for maintaining tracking tracks of a complete life cycle of the drum target, The target detection module adopts a YOLOv11 model, and a backbone layer of the YOLOv11 model uses MobileNetV4.

2. The shearer drum tracking system as claimed in claim 1, wherein, The structural framework of YOLOv11 of the target detection module comprises an input layer, a backbone layer, a neck layer and a head layer, the input layer is used for input of an original video, the backbone layer takes the MobileNetV4 as a core feature extraction network, the neck layer comprises a plurality of C3k2 modules, convolution layers and up-sampling modules, and the head layer comprises a plurality of detection modules.

3. The shearer drum tracking system as claimed in claim 1, wherein, The multi-feature fusion matching module extracts appearance features of the drum through the MobileNetV4, predicts a motion state of the drum target by using Kalman filtering, comprehensively calculates position coincidence degrees, appearance similarities and motion consistencies of the drum and fuses them by dynamic weighting, and completes optimal matching of a detection frame of the drum target and a tracker of the drum target by using a Hungarian algorithm.

4. The shearer drum tracking system as claimed in claim 1, wherein, When the drum target is occluded, the weight of the motion consistency is increased.

5. A coal cutter drum tracking method based on the coal cutter drum tracking system according to any one of claims 1 to 4.

6. The shearer drum tracking method as claimed in claim 5, wherein, The method comprises the following steps: Step I, start, and then proceed to step II; Step II, collect coal cutter drum pictures as a training set and a verification set, and then proceed to step III; Step III, train the YOLOv11 model, and then proceed to step IV; Step IV, input a coal cutter drum video, and then proceed to step V; Step V, detect a coal cutter drum by using the trained YOLOv11 model, and then proceed to step VI; Step VI, optimize the detection result according to a confidence threshold, and then proceed to step VII; Step VII, predict a position of the drum according to Kalman filtering, and then proceed to step VIII; Step VIII, judge whether the drum is detected in a current frame, yes, proceed to step IX, and no, proceed to step XVIII; Step IX, extract deep appearance features of the drum, and then proceed to step X; Step X, determine a final correlation score value based on appearance feature cosine similarity, detection frame and predicted frame IoU and motion track consistency, and then proceed to step XI; Step XI, match by using a Hungarian algorithm, and then proceed to step XII; Step XII, judge whether the matching is successful, yes, proceed to step XIX, and no, proceed to step XIII; Step XIII, judge whether the detection frame is matched, yes, proceed to step XIX, and no, proceed to step XIV; Step XIV, track initialization, and then proceed to step XV; Step XV, determining whether the trajectory tracker matches, if yes, proceeding to step XIX, otherwise proceeding to step XVI; Step XVI, determining whether the loss count exceeds m frames, if yes, proceeding to step XVII, otherwise proceeding to step XX; Step XVII, deleting the tracker, and then proceeding to step V; Step XVIII, determining whether no detection is made for n consecutive frames, if yes, proceeding to step XVII, otherwise proceeding to step XX; Step XIX, updating the drum position, and then proceeding to step XXI; Step XX, outputting the predicted position, and then proceeding to step XXI; Step XXI, outputting the updated position, and then proceeding to step XXII; Step XXII, ending.

7. A shearer drum tracking method as claimed in claim 6 wherein, m = 20, n = 5.

8. The shearer drum tracking method of claim 6, characterized in that: in the step II, the training set and the validation set contain drum images under different light and occlusion conditions, and the drum is labeled by using a labelme labeling tool; and / or in the step III, the training period of the YOLOv11 model is set to 150 epochs, the batch size is 16, the input image size is 640, a pre-trained model is used for fine-tuning, the optimizer is Adam, the initial learning rate lr0 is set to 0.01, the end learning rate is attenuated by a decay factor lrf = 0.01, the optimizer with a momentum of 0.937 is used, and weight decay is introduced to prevent overfitting; and / or In the step V, output the original detection frame information: , , w, h, conf, class], wherein is the frame center coordinate, conf is the confidence, w is the width of the detection frame, h is the height of the detection frame, and class is the class label of the target in the detection frame; and / or In the step VI, the bounding boxes with the confidence lower than the confidence threshold are filtered out, non-maximum suppression is performed based on the IoU threshold, overlapping boxes are eliminated, and a list of valid bounding boxes detections is obtained And the IoU is defined as follows: wherein A and B are frame A and frame B respectively, represents the intersection of two frames, and if the two frames have no overlap, the intersection area is 0, represents the union of two frames, and for each detection frame , crop the image in the frame, input the MobileNetV4, and extract a 128-dimensional feature vector f(i) from each cropped image; and / or in the step VII, the moving speed of the drum is defined as: where, and is the coordinate of the center point of the drum at the current time t, and is the coordinate of the center point of the drum at time t-1, is the time interval of adjacent frames, then the state vector of the drum is defined as: The state transition matrix F is defined as follows: According to Kalman filtering, the predicted position of the drum is obtained as: Obtained: wherein, is the predicted state vector of the drum at time t+1, and is the predicted center point coordinate of the drum at time t+1, the length and width size of the prediction box are the same as those at time t; and / or In said step X, The calculation is performed with detection and prediction boxes: wherein, represents a detection box, represents a prediction box, appearance cosine similarity is defined as follows: appearance feature vector wherein, is a detection image of the current frame, is a feature extraction model MobileNetV4, is the appearance feature vector of the extracted target, the appearance feature similarity of the target between the current frame i and the last frame i-1 is calculated. Motion consistency is calculated by calculating the Mahalanobis distance to evaluate the matching degree of the current detected target and the existing trajectory: where z is the detected new position of the drum, H is the observation matrix, S is the measurement noise covariance matrix, which is related to the camera resolution, image noise, and represent the variance of detection error in X and Y direction, which can be obtained by historical measurement data statistics or set as empirical values, and a set of target boxes are detected in the current frame , which need to be matched with the target boxes predicted by the tracker in the last frame , to build a comprehensive cost matrix , where each element represents the mismatch cost of the detection box and the predicted box , and each cost element comprehensively considers three kinds of similarity information: position IoU, appearance feature, and motion consistency: The position cost is: The appearance cost is: The motion consistency cost is , is a tunable normalization parameter, The average stability index of each similarity term is defined, and the weight is dynamically calculated accordingly: For IoU average confidence, appearance similarity average value, is the average value of the Mahalanobis distance, where N represents the total number of the combined pairs of the detection frame to be matched and the prediction frame in the current frame, and the dynamic updated weight is generated after comprehensive normalization: The final cost matrix is ; and / or In step XI, the Hungarian algorithm is applied to solve the minimum cost matching, wherein Score= The matching threshold thresh is set to determine whether the current detection result and the historical trajectory match successfully, and the specific determination rule is as follows: if Score ≤ thresh, it is considered that the detection frame and the tracking trajectory match successfully, the position of the current detection frame is used to update the corresponding trajectory state to obtain the latest spatial coordinates of the drum, and if Score > thresh, it is considered that the matching cost between the detection result and the existing trajectory is too large, and the matching fails.

9. The shearer drum tracking method as claimed in claim 8, wherein, The confidence threshold is 0.5, and the IoU threshold is 0.

6.

10. The shearer drum tracking method as claimed in claim 7, wherein, In the step XX, the Kalman filter is started to predict the motion of the drum target.

Images