Identification method of coal mine personnel chasing overhead passenger devices based on computer vision

Abstract

The present invention belongs to the field of computer vision technology, and discloses a method for identifying coal mine personnel chasing overhead rides based on computer vision. The specific technical scheme is: obtaining a real-time video stream of a monitoring point of a scene of riding an overhead ride, intercepting multiple videos of pedestrians chasing overhead rides, intercepting the videos frame by frame and obtaining a data set; performing data enhancement preprocessing and data labeling on the acquired data respectively; using an improved algorithm to identify personnel and overhead rides; step four, realizing multi-camera tracking based on a tracking algorithm; step five, calculating motion vectors and distances between tracking targets; step six, determining whether there is a chasing behavior; the present invention solves the problems of waste of human resources and blind spot monitoring caused by relying on manual inspection and monitoring in the past, and the illegal operations and safety hazards of coal mine workers in the production process are promptly checked, thereby improving the working efficiency of coal mines and reducing the risk of accidents caused by coal mine personnel riding overhead rides in an irregular manner.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a method for identifying coal mine personnel chasing an overhead passenger device based on computer vision. Background Art

[0002] The traditional coal mine production process often relies on manual inspection and monitoring, which leads to the problems of human resource waste and blind spot monitoring. In addition, the illegal operations and safety hazards of coal miners in the production process cannot be checked in time, which will reduce the efficiency of coal mines and increase the safety risk factor. With the implementation of artificial intelligence technology, intelligent management of the coal mining industry has also become a trend. Based on video images, by deploying AI algorithm models, intelligent recognition and analysis of underground coal mine monitoring scenes can be achieved, and abnormal events, illegal operations, safety hazards, etc. in the safe production process of coal mines can be monitored and warned in real time.

[0003] An overhead passenger device is a device used for underground transportation in mines. It consists of a wire rope, a pulley, a motor, etc., and can run up and down in an inclined shaft or a vertical shaft. Through the on-site practical application of the overhead passenger device in mining enterprises, it can not only effectively improve and increase the efficiency of mine production in my country, but also effectively reduce the work intensity of mine workers. However, in the actual application scenarios of the overhead passenger device, there are often a series of accidents caused by the irregular use of this device. Among them, coal mine personnel chasing the overhead passenger device will cause personnel to fall, and the device will have a large deflection when getting on the car, causing the wire rope to fall off the rope wheel or the rope gripper to get stuck on the rope wheel and cause a safety accident. Therefore, detecting this irregular behavior and pushing the alarm results in a timely manner can effectively reduce such safety problems. Summary of the invention

[0004] In order to solve the technical problems existing in the prior art, the present invention provides a method for identifying coal mine personnel chasing overhead passenger devices based on computer vision, which mainly obtains the real-time video stream of the monitoring point of the overhead passenger device scene, extracts frames from the video, and pre-processes and annotates the intercepted data set, and then adopts the YOLOv8-MCW improved algorithm to detect the miners and the single overhead passenger device targets in the video, and adopts the target tracking technology to track the detected targets. Considering the problem that the target identity is easily lost and switched due to mutual occlusion between targets during the tracking process, the homography matrix of two perspectives is used to realize the splicing of images from different perspectives, and the movement speed and direction of the target are judged based on the tracking results, and the calculated distance between the two targets is combined to finally judge whether the person has the behavior of chasing the overhead passenger device.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is: a method for identifying coal mine personnel chasing overhead passenger devices based on computer vision, and the specific steps are as follows:

[0006] S1. Image data acquisition: Use existing cameras in actual coal mine application scenarios to obtain real-time video streams from monitoring points of aerial ride scenes, capture multiple videos of pedestrians chasing aerial rides, and then filter the captured videos and capture them frame by frame to obtain a data set for subsequent processing.

[0007] S2. Dataset preparation: Perform data enhancement preprocessing and data labeling on the acquired data.

[0008] Furthermore, step S2 comprises the following steps:

[0009] S21. Data enhancement: The method of transforming and expanding the data collected in the above step S1 to generate more and richer samples can improve the diversity of training samples, reduce the model's dependence on specific data distribution, and enhance the generalization ability and robustness of the model.

[0010] S22. Data labeling: Use LabelImg software to label pedestrians and overhead riding devices on the images after data enhancement preprocessing. The pedestrian category is named "person" and the overhead riding device category is named "man_riding_device". After labeling, a txt file with location coordinates and category name will be generated; the labeled data set is divided into training set, test set and validation set in a ratio of 7:2:1.

[0011] S3. Model training: This method uses the improved YOLOv8-MCW algorithm to realize the recognition of personnel and overhead passenger devices. Although the improved YOLOv8 algorithm has made significant progress in detection accuracy and speed, there is still room for improvement in specific application scenarios. Considering that the present invention performs target detection in places where overhead passenger devices are used in wellheads and tunnels, it is necessary to further optimize the YOLOv8 algorithm to improve the overall performance of target detection.

[0012] The improvement of YOLOv8 algorithm includes the following three core points: First, MobileNetV3-Large is used to replace the backbone structure of the traditional yolov8 network to ensure that the model is lightweight and easy to deploy; secondly, the channel attention module is introduced. Since the camera is installed far away from the detection target, the detection target is small in the whole picture and the underground target is blurred. These small targets are often easily missed by the detection network. Therefore, the attention mechanism is added to improve the model's detection effect on the distant overhead equipment and personnel targets; in order to further optimize the algorithm model and improve the accuracy of target detection and positioning, WIoU v3 is selected as the bounding box regression loss function. Through the above optimization strategies, the efficient and accurate detection of personnel and overhead equipment targets at the wellhead and underground is finally achieved, and the improved algorithm is named YOLOv8-MCW.

[0013] Furthermore, step S3 comprises the following steps:

[0014] S31, MobileNetV3-Large replaces the traditional yolov8 network backbone structure, which ensures the model is lightweight while almost losing the model accuracy. MobileNetV3 introduces 5×5 depth convolution to replace some 3×3 depth convolutions, and introduces Squeeze-and-excitation (SE) module and h-swish (HS) activation function to improve model accuracy. The yolov8-M model is concise, with excellent performance and speed.

[0015] S32. Adding an attention mechanism module: The present invention introduces a coordinate attention mechanism (CA) into the neck network, embeds the precise positioning signal into the channel attention, and divides it into feature codes in two different directions, one of which is used to retain short position signals, and the other is used to capture long-range dependencies. The feature codes in the two directions are respectively encoded to form a feature image containing a pair of target perception and positioning signals, which can enhance the accurate positioning and target recognition capabilities of the feature map.

[0016] S33, optimize the loss function: The traditional Intersection over Union (IoU) only considers the overlapping part of the predicted box and the real box, without considering the area between the two, which may lead to deviations in the evaluation results. Considering factors such as aspect ratio, centroid distance and overlapping area, as well as reducing the arithmetic calculation power consumption caused by inverse trigonometric functions. The excellent dynamic non-monotonic focusing mechanism of WIoUv3 is more suitable for the task of blurry small target detection. The present invention selects WIoUv3 as the loss function of YOLOv8-CW. The specific calculation process of WIoU v3 loss function is as follows.

[0017] LWIoUv1 The bounding box loss function is defined as follows:

[0018] L IoU =1-IoU

[0019]

[0020] L WIoUv1 =R WIoU L IoU

[0021] In the formula, IoU is used to measure the overlap between the predicted bounding box and the true bounding box, L IoU is the bounding box loss function, c h and c w Represent the height and width of the minimum enclosed box formed by the predicted box and the true box, respectively. and represents the centroid coordinates of the real box, and Represents the centroid coordinates of the predicted box.

[0022] In L WIoUv1 L is defined based on multiplying the gradient gain r WIoUv3 The bounding box loss function is shown below:

[0023]

[0024] Where r is the non-monotonic focus factor, L WIoUv1 is the bounding box loss function, is the monotone focus coefficient, is the average bounding box loss value set, the ratio of the two is β, and the sample quality can be evaluated by β. Both δ and α are hyperparameters manually set to adapt to different models.

[0025] WIoU v3 uses a dynamic non-monotonic mechanism to evaluate the quality of anchor frames, which makes the model pay more attention to anchor frames of ordinary quality and improves the model's ability to locate objects. For target detection tasks in complex underground scenes, due to the fuzzy targets and small target sizes, detection is difficult. The WIoU v3 loss function can dynamically optimize the loss weights of small targets to improve the detection performance of YOLOv8-MCW.

[0026] S34, model detection effect and performance evaluation: In order to test the improved model detection performance of the present invention, the present invention uses accuracy, recall rate, mAP@0.5, model parameter scale, model calculation amount (floating point operations, FLOPs), model training time and detection speed (unit: frame / s) as evaluation indicators. The following parameters are used in the formula of the above evaluation indicators: TP (predicted as a positive sample, which is actually a positive sample), FP (predicted as a positive sample, which is actually a negative sample) and FN (predicted as a negative sample, which is actually a positive sample).

[0027] S4. Based on the DeepSort tracking algorithm, multi-camera tracking is implemented. When tracking a target, mutual occlusion between targets and changes in light in the environment may lead to tracking failure, which in turn affects the subsequent determination of the chasing relationship between targets. Therefore, it is particularly necessary to effectively use multi-view information to improve the accuracy and stability of target tracking.

[0028] S41, calculate the homography matrix corresponding to the cameras with different viewing angles, use the homography matrix to map the image space with overlapping areas, and then process the occlusion to complete the target matching. The homography matrix can establish the correspondence between different images, so that the model can infer the information in one image from another image.

[0029] Specifically, the shift algorithm is used to extract key points from the overlapping areas between two or more different perspectives, and the spatial connection between different perspectives is derived by using the relationship between the image space and the world coordinate system, and then the homography matrix between different perspectives is obtained.

[0030] S42, DeepSort tracking algorithm is a multi-target tracking algorithm, which uses Kalman filter algorithm to infer the motion at the next moment based on the motion at the current moment, combines the motion information and appearance information, and uses Hungarian algorithm to match the detection frame and the prediction frame. In each frame, the algorithm first uses the deep learning model to predict all possible targets, then performs data association based on the appearance information, and finally uses Kalman filter to update the trajectory information.

[0031] The Kalman filter algorithm is a recursive filtering algorithm that continuously improves the estimate of the state by updating the observation value.

[0032] The Hungarian algorithm is an optimization algorithm for solving the assignment problem. It is based on the maximum matching and linear programming methods in graph theory and is used to find the best task allocation method in a given cost matrix. Its purpose is to match multiple targets between frames in multi-target tracking, including the appearance of new targets, the disappearance of old targets, and the matching of target IDs between the previous frame and the current frame.

[0033] The algorithm shows excellent performance in complex scenes such as dense pedestrians and occlusions, and can accurately track the movement trajectory of pedestrians, providing strong support for real-time acquisition of moving pedestrian information. At the same time, the excellent performance of multi-camera tracking based on the DeepSort tracking algorithm also proves that the combination of the two can achieve more efficient and accurate target tracking.

[0034] S43, calculate the homography matrix between videos, stitch the multi-view videos together, and use them as input to predict data using Kalman filtering.

[0035] The target is then determined by combining the homography matrix and the Hungarian algorithm, thereby achieving target tracking between multiple cameras.

[0036] S5. Calculate the motion vector and distance between the tracked targets: Based on the tracking results, obtain the center point position of the object in each frame. In each pair of adjacent frames, calculate the object's motion direction and speed and the Euclidean distance between the two points according to the change in the center point position to obtain the object's motion information.

[0037] S6. The specific steps of pursuit judgment are as follows: 1) If the speed of object A is greater than that of object B, and this situation is continuous, it is considered that pursuit may occur; 2) Calculate the moving direction vectors of the two objects. If the angles of the two vectors are very small (that is, the two direction vectors are close to parallel, and an angle threshold can be set, such as 30 degrees), it means that object A is moving towards object B. 3) If the distance between object A and object B decreases over time, this further confirms the existence of pursuit behavior.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. Aiming at the problems of fuzzy data and small targets in materials collected from special scenes in coal mines, the present invention adds a CA attention mechanism to better complete feature extraction, and solves the problems of ambient light interference and diverse and complex targets, thereby achieving an increase in detection accuracy.

[0040] 2. The present invention improves the accuracy and stability of target tracking. Through multi-camera tracking, it can overcome problems such as perspective changes and occlusions in a multi-camera environment, improve the accuracy and robustness of target tracking, and effectively utilize multi-perspective information to improve the accuracy and stability of target tracking. It can better cope with challenges such as occlusion and light changes, and provide strong support for the tracking of real-time miners and overhead passenger devices.

[0041] 3. The method for identifying coal mine personnel chasing overhead riding devices in the present invention solves the problems of waste of human resources and blind spot monitoring caused by previous reliance on manual inspection and monitoring. The illegal operations and safety hazards of coal mine workers in the production process are promptly checked, which improves the work efficiency of coal mines and reduces the risk of accidents caused by coal mine personnel riding overhead riding devices in an irregular manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention is a schematic diagram of the overall process of the method for judging whether a person is chasing an overhead passenger device.

[0043] Figure 2 This is a schematic diagram of the LabelImg software image annotation working interface.

[0044] Figure 3 This is a schematic diagram of the network structure of the improved algorithm YOLOV8-MCW of the present invention.

[0045] Figure 4 This is the structural diagram of the CA attention mechanism model.

[0046] Figure 5 Target instance detected by the YOLOV8-MCW algorithm Figure 1 .

[0047] Figure 6 Target instance detected by the YOLOV8-MCW algorithm Figure 2 .

[0048] Figure 7 Schematic diagram of the Deepsort algorithm multi-camera tracking process.

[0049] Figure 8 This is an example diagram of detection and tracking in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0050] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] like Figure 1As shown, the recognition method of coal mine personnel chasing overhead passenger devices based on computer vision, the specific steps are as follows:

[0052] S1. Image data acquisition: Use existing cameras in actual coal mine application scenarios to obtain real-time video streams from monitoring points of aerial ride scenes, capture multiple videos of pedestrians chasing aerial rides, and then filter the captured videos to delete videos with blurred targets. Next, use PotPlayer software to capture the videos frame by frame to obtain 1,500 image data required for model detection for subsequent data processing.

[0053] S2. Dataset preparation: Perform data enhancement preprocessing and data labeling on the acquired data.

[0054] S21. Data augmentation: Data augmentation refers to the method of using limited data to transform and expand in order to generate more and richer samples. It is an effective means to improve the diversity of training samples, reduce the model’s dependence on specific data distribution, and enhance the model’s generalization ability and robustness.

[0055] In order to make up for the deficiency of the data set, the present invention adopts data enhancement technology to expand the original training set. In this process, the imgaug framework (image enhancement library) is selected as the implementation tool. The imgaug framework allows the introduction of multiple image transformations, which can be randomly applied during the training process to generate image samples with diversity. In the data enhancement process, a series of transformation strategies are adopted, such as random rotation, horizontal flipping, vertical flipping, random scaling, brightness and contrast adjustment, etc. By randomly applying these transformations during training, a large number of images of different angles and scales can be generated, so that the image diversity in the real scene can be better simulated. For example, the present invention performs data enhancement by flipping the image horizontally by 90°, rotating it clockwise by 15°, adding 5% noise, 25% grayscale and 40% brightness, etc., so as to increase the diversity of the training data set and improve the robustness of the model.

[0056] S22. Data labeling: Use LabelImg software (labeling tool) to label pedestrians and overhead passenger devices on the images after data enhancement preprocessing. Figure 2 For example, this is the working interface of the images of people and overhead riding devices when using LabelImg software to annotate. The pedestrian category is named "person" and the overhead riding device can be named "man_riding_device". After annotation, a txt file with location coordinates and category name will be generated.

[0057] In the specific operation process, drag the mouse in the labelImg tool to get a labeling box, and then label and classify the framed area, and name it "person" and "man_riding_device" respectively during data annotation. In order to ensure the accuracy of image annotation in the dataset, the annotation standard is uniformly set to the minimum rectangle that can contain the target position of the image before annotation.

[0058] The labeled data is divided into training set, test set and validation set in a ratio of 7:2:1. The training set contains 1050 images, the test set contains 300 images, and the validation set contains 150 images, and the data set is in txt format.

[0059] S3. Model training: The present invention uses an improved YOLOv8 algorithm (target detection algorithm) to realize the recognition of personnel and overhead passenger devices. Although the YOLOv8 algorithm has made significant progress in detection accuracy and speed, there is still room for improvement in specific application scenarios. Considering that the present invention performs target detection in places where overhead passenger devices are used in wellheads and tunnels, it is necessary to further optimize the YOLOv8 algorithm to improve the overall performance of target detection.

[0060] Regarding the improvement of the YOLOv8 algorithm, there are mainly three core points: First, MobileNetV3-Large is used to replace the backbone structure of the traditional yolov8 network to ensure that the model is lightweight and easy to deploy; secondly, the channel attention (CA) module is introduced. Since the camera is installed far away from the detection target, the detection target is small in the entire picture and the underground target is blurred. These small targets are often easily missed by the detection network. Therefore, the attention mechanism is added to improve the model's detection effect on distant overhead passenger devices and personnel targets; then, in order to further optimize the algorithm model and improve the accuracy of target detection and positioning, WIoU v3 is selected as the bounding box regression loss function. Through the above optimization strategies, efficient and accurate detection of personnel and overhead passenger device targets at the wellhead and underground can be achieved. At the same time, the present invention names the improved algorithm YOLOv8-MCW. The network structure diagram of the improved algorithm YOLOV8-MCW is shown in the figure below. Figure 3 shown.

[0061] Furthermore, step S3 comprises the following steps:

[0062] S31, MobileNetV3-Large (a version of the third generation of the MobileNet series of lightweight network models) was used to replace the backbone structure of the traditional yolov8 network, ensuring the lightweight of the model without losing the accuracy of the model. MobileNetV3 (lightweight network model) introduced 5×5 depth convolution to replace some 3×3 depth convolutions, introduced Squeeze-and-excitation (SE) modules and h-swish (HS) activation functions to improve model accuracy, yolov8-M (the abbreviation of the improved yolov8 algorithm after changing the yolov8 backbone network to MobileNetV3-Large) model is concise, with excellent performance and speed. The model detection effect after replacing the backbone network is shown in Table 1 below:

[0063] Table 1 Comparison of model detection results after replacing the backbone network

[0064]

[0065] Compared with the original YOLOv8 network, the improved network model has a detection speed of 90.29FPS, with a 0.2% decrease in accuracy, a 3.6% decrease in recall, and a 0.4% decrease in average precision. The amount of calculation is reduced by 11.4G compared to the original YOLOv8 model, and finally, the model size is reduced by 8.4MB. The relevant data in Table 1 proves that replacing the backbone network of YOLOv8 with MobileNetV3 is efficient and lightweight.

[0066] S32. Adding attention mechanism module: The present invention introduces the coordinate attention mechanism (CA) in the neck network, embeds the precise positioning signal in the channel attention, and divides it into two feature codes in different directions, one of which is used to retain short position signals, and the other is used to capture long-range dependencies. The model structure diagram of the CA attention mechanism is as follows: Figure 4 As shown. Encoding them separately to form a feature image containing a pair of target perception and positioning signals can enhance the accurate positioning and target recognition capabilities of the feature map. The detection effect of adding the attention mechanism model is shown in Table 2 below:

[0067] Table 2 Comparison of detection results of the model with added CA attention mechanism

[0068]

[0069] As described in step S21, after the backbone network is replaced with MobileNetV3, although the detection speed and model size are optimized, the accuracy, recall rate and average precision are reduced. The results in Table 2 show that adding the CA attention mechanism can effectively compensate for the loss of some indicators caused by replacing the backbone of the YOLOv8 network with MobileNetV3.

[0070] S33, optimize the loss function: the traditional Intersection over Union (IoU measurement standard) only considers the overlapping part of the predicted box and the real box, and does not consider the area between the two, which may lead to deviations in the evaluation results. Considering factors such as aspect ratio, centroid distance and overlapping area, as well as reducing the arithmetic calculation power consumption brought by inverse trigonometric functions, and the excellent dynamic non-monotonic focusing mechanism of WIoU v3 is more suitable for the task of blurry small target detection, the present invention selects WIoU v3 as the loss function of YOLOv8-MC (the abbreviation of the improved yolov8 algorithm after changing the yolov8 backbone network to MobileNetV3-Large and adding the attention module), and the specific calculation process of the WIoU v3 loss function is as follows.

[0071] L WIoUv1 The bounding box loss function is defined as follows:

[0072] L IoU =1-IoU

[0073]

[0074] L WIoUv1 =R WIoU L IoU

[0075] In the formula, IoU is used to measure the overlap between the predicted bounding box and the true bounding box, L IoU is the bounding box loss function, c h and c w Represent the height and width of the minimum enclosed box formed by the predicted box and the true box, respectively. and represents the centroid coordinates of the real box, and Represents the centroid coordinates of the predicted box.

[0076] In L WIoUv1 L is defined based on multiplying the gradient gain r WIoUv3 The bounding box loss function is shown below:

[0077]

[0078] Where r is the non-monotonic focus factor, L WIoUv1 is the bounding box loss function, is the monotone focus coefficient, is the average bounding box loss value set, the ratio of the two is β, and the sample quality can be evaluated by β. Both δ and α are hyperparameters manually set to adapt to different models, and are set to 1.9 and 3 in the present invention.

[0079] WIoU v3 uses a dynamic non-monotonic mechanism to evaluate the quality of anchor frames, so that the model pays more attention to anchor frames of ordinary quality, improving the model's ability to locate objects. For target detection tasks in complex underground scenes, due to the fuzzy targets and small target sizes, detection is difficult. The WIoU v3 loss function can dynamically optimize the loss weights of small targets to improve the detection performance of YOLOv8-MCW. The effects of different loss functions on model detection are shown in Table 3 below:

[0080] Table 3 Comparison of detection results of yolov8-MC with different loss functions

[0081]

[0082] According to the data in Table 3, it can be clearly observed that the YOLOv8-MCW improved model proposed in this invention has achieved obvious advantages in the detection task of personnel and overhead passenger devices in coal mines. Figure 5 and Figure 6 As shown, the detected category and confidence are indicated in the upper left corner. Compared with the traditional YOLOv8 target detection algorithm model, the improved model of the present invention shows better performance in detecting personnel in coal mines and overhead passenger equipment targets.

[0083] S34, model detection effect and performance evaluation: In order to test the detection performance of the improved model of the present invention, the present invention uses accuracy, recall rate, mAP@0.5, model parameter scale, model calculation amount (floating point operations, FLOPs), model training time and detection speed (unit: frame / s) as evaluation indicators. The following parameters are used in the formula of the above evaluation indicators: TP (predicted as a positive sample, which is actually a positive sample), FP (predicted as a positive sample, which is actually a negative sample) and FN (predicted as a negative sample, which is actually a positive sample).

[0084] Precision is the ratio of the number of positive samples predicted by the model to the number of all detected samples, and is calculated as follows:

[0085]

[0086] In the formula, TP represents the number of positive samples predicted by the model, and FP represents the number of negative samples predicted by the model.

[0087] Recall is the ratio of the number of positive samples correctly predicted by the model to the number of positive samples that actually occurred. The recall calculation is as follows:

[0088]

[0089] In the formula, TP represents the number of positive samples predicted by the model, and FN represents the number of positive samples predicted incorrectly by the model.

[0090] The average precision (AP) is the area under the precision and recall curves. The mean average precision (mAP@0.5) is the result obtained by the weighted average of the AP values ​​of all sample categories. It is used to evaluate the detection performance of the model in all categories. The threshold of the intersection ratio of the predicted box and the true box is set to 0.5. The formula is as follows:

[0091]

[0092] Where N is the sample category, AP represents the area under the precision and recall curves, and @0.5 means that the IoU threshold used when calculating the area under the precision and recall curves is 0.5.

[0093] S4. Implement multi-camera tracking based on DeepSort tracking algorithm. When tracking a target, mutual occlusion between targets and changes in light in the environment may lead to tracking failure, which in turn affects the subsequent determination of the pursuit relationship between targets. Therefore, it is particularly necessary to effectively use multi-view information to improve the accuracy and stability of target tracking. The schematic diagram of the multi-camera tracking process based on the Deepsort algorithm is shown in Figure 7 shown.

[0094] S41, calculate the homography matrix corresponding to the cameras with different viewing angles, use the homography matrix to map the image space with overlapping areas, and then process the occlusion to complete the target matching. The homography matrix can establish the correspondence between different images, so that the model can infer the information in one image from another image.

[0095] Specifically, the shift algorithm is used to extract key points from the overlapping areas between two or more different perspectives. The spatial connection between different perspectives is derived by using the relationship between the image space and the world coordinate system, and then the homography matrix between different perspectives is obtained.

[0096] S42, DeepSort tracking algorithm, DeepSort is a multi-target tracking algorithm that uses the Kalman filter algorithm to infer the next moment's motion based on the current moment's motion, combines the motion information and appearance information, and uses the Hungarian algorithm to match the detection frame and the prediction frame. In each frame, the algorithm first uses a deep learning model to predict all possible targets, then performs data association based on appearance information, and finally uses the Kalman filter to update the trajectory information.

[0097] The Kalman filter algorithm is a recursive filtering algorithm that continuously improves the estimate of the state by updating the observation value.

[0098] About the Hungarian algorithm, the Hungarian algorithm is an optimization algorithm for solving the assignment problem. Its principle is based on the maximum matching and linear programming methods in graph theory, and is used to find the best task allocation method in a given cost matrix. The purpose is to match multiple targets between frames in multi-target tracking, including the appearance of new targets, the disappearance of old targets, and the target ID matching between the previous frame and the current frame.

[0099] The algorithm shows excellent performance in complex scenes such as dense pedestrians and occlusions, and can accurately track the movement trajectory of pedestrians, providing strong support for real-time acquisition of moving pedestrian information. At the same time, the excellent performance of multi-camera tracking based on the DeepSort tracking algorithm also proves that the combination of the two can achieve more efficient and accurate target tracking. Figure 8 An example diagram of a person and an overhead passenger device detected by using the target detection algorithm and tracking algorithm adopted by the present invention is shown, and the tracked target category and category ID are marked in the upper left corner.

[0100] S43, to achieve multi-camera target tracking, calculate the homography matrix between videos, stitch the multi-view videos together, and use them as input to predict data using Kalman filtering.

[0101] Combined with the homography matrix and Hungarian algorithm, the target is determined, thus achieving target tracking between multiple cameras. The overall process of multi-camera target tracking is as follows: Figure 4 shown.

[0102] S5. Calculate the motion vector of the tracked target and the distance between the targets, obtain the center point position of the object in each frame based on the tracking result, and in each pair of adjacent frames, calculate the object's motion direction and speed and the Euclidean distance between the two points according to the change of the center point position, so as to obtain the object's motion information.

[0103] Furthermore, step S5 comprises the following steps:

[0104] S51. Calculate the Euclidean distance between the two targets according to the detected coordinate information. The calculation formula is as follows:

[0105]

[0106]

[0107]

[0108] In the formula, (x min1 ,y min1 ,x max1 ,y max1 ) and (x min2 ,y min2 ,x max2 ,y max2 ) are the coordinates of two different targets, (x center1 ,y center1 ) and (x center2 ,y center2 ) are the center coordinates of the two targets, and d is the Euclidean distance between the targets.

[0109] S52, based on the tracking results, respectively calculate the movement direction of the same target in n consecutive frames, and calculate the direction angle of two types of targets, pedestrians (target A) and overhead passenger devices (target B), and the calculation formula is as follows:

[0110]

[0111]

[0112] In the formula, x1 and y1 are the horizontal and vertical coordinates of target A in the current frame, x′1 and y′1 are the horizontal and vertical coordinates of target A in the next frame, x2 and y2 are the horizontal and vertical coordinates of target B in the current frame, and x′2 and y′2 are the horizontal and vertical coordinates of target B in the next frame. They represent the vectors of the two targets respectively, and θ is the angle between the two targets.

[0113] S53, the relative motion speed between targets is calculated. According to the Euclidean distance result obtained in step S61, the motion time is calculated by the frame rate and the number of frames counted, and the motion speed is finally obtained. The calculation formula is as follows:

[0114]

[0115] Where frame_num is the set number of frames, fps is the video frame rate, and d is the distance the target moves within the set number of frames.

[0116] S6, pursuit judgment: 1) According to the relative movement speed between the targets calculated in S63, if the speed of object A is greater than that of object B, and this situation is continuous, it is considered that pursuit may occur; 2) According to the angle between the movement directions of the two objects calculated in S62, if the angle between the movement directions in consecutive frames is less than the set threshold (the threshold of the present invention is set to 20°), it means that object A is moving towards object B; 3) If the distance between object A and object B decreases over time, this further confirms the existence of pursuit behavior.

[0117] The above description is only a preferred embodiment of the invention and is not intended to limit the invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the invention should be included in the scope of the invention.

Claims

1. A method for identifying coal mine personnel chasing overhead passenger devices based on computer vision, characterized in that: The specific steps are: Step 1: Use existing cameras in actual coal mine application scenarios to obtain real-time video streams from monitoring points of the aerial ride scene, intercept multiple segments of videos of pedestrians chasing the aerial ride, and then filter the intercepted videos, intercept the videos frame by frame, and obtain a data set; Step 2: Perform data enhancement preprocessing and data labeling on the data obtained in step 1; Step 3: Use the YOLOv8-MCW improved algorithm to identify people and overhead passenger devices; The YOLOv8-MCW improved algorithm specifically includes: using MobileNetV3-Large as the backbone structure, introducing 5×5 deep convolution, Squeeze-and-excitation module and h-swish activation function; The coordinate attention mechanism is introduced into the neck network and the positioning signal is embedded in the channel attention. Two feature codes with different directions are divided, one of which is used to retain short position signals and the other is used to capture long-range dependencies. They are encoded separately to form a feature image containing a pair of target perception and positioning signals. Select WIoUv3 as the loss function of YOLOv8-MCW. The specific calculation process of WIoUv3 loss function is as follows: L WIoUv1 The bounding box loss function is defined as follows: L IoU =1-IoU L WIoUv1 =R WIoU L IoU Where IoU is used to measure the overlap between the predicted bounding box and the true bounding box, L IoU is the bounding box loss function, c h Indicates the height of the minimum enclosing box formed by the predicted box and the true box, c w Indicates the width of the minimum enclosing box formed by the predicted box and the true box. and represents the centroid coordinates of the real box, and Represents the centroid coordinates of the prediction box; In L WIoUv1 Define L based on multiplying the gradient gain r WIoUv3 The bounding box loss function is shown below: Where r is the non-monotonic focus factor, L WIoUv1 is the bounding box loss function, is the monotone focus coefficient, is the average bounding box loss value set, the ratio of the two is β, and the sample quality can be evaluated by β. Both δ and α are hyperparameters manually set to adapt to different models; Use precision, recall, mAP@0.5, model parameter size, model computation, model training time, and detection speed as evaluation indicators; Step 4: Realize target tracking between multiple cameras based on DeepSort tracking algorithm; Step 5: Calculate the motion vector and distance between the tracking targets; Step 6: Determine whether there is any pursuit behavior.

2. The method for identifying coal mine personnel chasing overhead passenger devices based on computer vision according to claim 1, characterized in that: In step 2, the specific steps are as follows: Step S21, data enhancement preprocessing: transform and expand the data collected in step 1 to generate new samples; Step S22, data labeling: label pedestrians on the image after data enhancement preprocessing and name the category as person, label overhead riding devices on the image after data enhancement preprocessing and name the category as man_riding_device. After labeling, a txt file with position coordinates and category name will be generated. The labeled data set is divided into training set, test set and validation set in a ratio of 7:2:

1.

3. The method for identifying coal mine personnel chasing overhead passenger devices based on computer vision according to claim 2, characterized in that: In step 4, the specific steps are as follows: Step S41, calculating the homography matrix corresponding to the cameras with different viewing angles, using the homography matrix to map the image space with overlapping areas, processing the occlusion and completing the target matching; Step S42: Use the Kalman filter algorithm to infer the amount of motion at the next moment based on the amount of motion at the current moment, combine the motion amount information and the appearance information, and use the Hungarian algorithm to match the detection frame and the prediction frame; Step S43, calculate the homography matrix between videos, stitch the multi-view videos, and use the stitched videos as input, use Kalman filtering to predict data, and then combine the homography matrix and Hungarian algorithm to determine the target, so as to achieve target tracking between multiple cameras.

4. The method for identifying coal mine personnel chasing overhead passenger devices based on computer vision according to claim 3, characterized in that: In step five, the specific calculation steps are as follows: based on the tracking results, the center point position of the object in each frame is obtained; in each pair of adjacent frames, the object's movement direction, speed, and Euclidean distance between tracking targets are calculated according to the change in the center point position to obtain the object's movement information.

5. The method for identifying coal mine personnel chasing overhead passenger devices based on computer vision according to claim 4, characterized in that: In step six, the specific determination method is: 1) If the speed of object A is greater than the speed of object B, and this situation is continuous, it is considered that pursuit may occur; 2) Calculate the moving direction vectors of object A and object B. If the angle between the two vectors is less than the set value, it is determined that object A is moving towards object B; 3) If the distance between object A and object B decreases over time, it is determined that pursuit behavior exists.

Images