Personnel mine car climbing detection method based on underground environment

Abstract

The invention belongs to the technical field of coal mine safety engineering and artificial intelligence, and discloses a mine car climbing personnel detection method based on underground environment, and the specific technical scheme is as follows: collecting climbing actions of an underground mine car and mine car workers; labeling the data set; disorganizing the sequence of the data sets and dividing the data sets into a training set, a verification set and a test set; a figure posture detection model YOLO-Pose is established; according to the reasoning result of the improved YOLO-Pose model based on YOLOv5, whether the detected worker has the action behavior of climbing the mine car or not in the video detection range is judged, and if yes, an alarm signal is output; according to the invention, the positions and actions of the personnel can be accurately detected in real time, and the personnel climbing the mine car in the underground environment can be effectively and timely identified, so that the occurrence rate of accidents is effectively reduced, and the safety of underground workers is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the fields of coal mine safety engineering technology and artificial intelligence technology, and specifically relates to a method for detecting personnel climbing a mine car in an underground environment based on an improved YOLOv5. Background Art

[0002] As a country with abundant mineral resources, China has a large number of mineral producing areas. In all mining areas, transportation is an extremely core issue, and safety is the top priority in underground mining transportation. The underground working environment is extremely complex, with many potential dangers, such as severe dust, obstructed vision, and changing terrain. Mine cars are special vehicles for transporting coal, gangue, equipment, equipment, and personnel on mine rails. They are mainly used for rail transportation in mine shafts, shafts, and on the ground. According to their operating functions, they are often divided into trucks, people cars, and material cars. In order to adapt to the narrow conditions of underground tunnels, mine cars are generally pulled by locomotives or winches. They are the largest and most widely used transportation equipment in coal mines.

[0003] For underground mine cars that transport workers, if the workers violate safety regulations and climb the mine cars during the transportation of the mine cars, the following risks may exist: 1. Risk of unbalanced mine car falling: Workers climbing the mine cars at will may cause people to fall or lose balance, fall from the mine cars and cause injuries; 2. Risk of electrical electric shock: Underground mine cars are driven by electrical appliances. When climbing, workers may touch live parts and cause electric shock accidents; 3. Risk of mechanical injury: When climbing the mine car, because the mine car will not slow down and the underground lighting is dim, workers may be pinched or collided with mechanical parts such as wheels and transmission devices and cause injuries; 4. Collapse risk: If the climbed mine car has a load limit or has not been inspected and maintained, the mine car may collapse or tip over, causing injuries to the workers; 5. Risk of accidental start-up: When climbing the mine car, if the start button or control lever is accidentally touched, the vehicle may start accidentally and cause injuries.

[0004] Therefore, in order to avoid these risks, supervision of underground mine car operations should be strengthened to ensure the safe operation of underground mine cars and the safe operation of underground workers. Using intelligent AI to monitor workers climbing mine cars can promptly detect violations, alert workers and remind managers to take immediate action to prevent violations, reduce accidents, and effectively protect the lives of workers. At the same time, AI monitoring and alarms can remind and record workers to comply with operating procedures and safety standards around the clock, form good work habits, and standardize work behaviors, thereby reducing accident costs such as casualties and equipment damage. However, the current mine car monitoring cannot provide timely and accurate feedback on alarm information, and cannot provide effective data support for the safe operation of mine cars and the safe operation of underground workers. Summary of the invention

[0005] In order to solve the technical problems existing in the prior art, the present invention collects and produces a posture data set of underground workers climbing mine cars based on the YOLOv5 detection model, and then combines the image preprocessing algorithm in the model training stage to increase the ability to capture different local information; after training, the model outputs the posture of the staff during detection, and uses post-processing to determine whether the staff has climbing mine cars. If so, an alarm message is sent to prompt, providing a monitoring solution for the safe operation of underground mine cars and the standardized and safe operation of underground workers.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is: a method for detecting personnel climbing a mine car based on an underground environment, the specific steps are:

[0007] Step 1. Dataset collection: Collect the climbing actions of underground mine cars and mine car workers. The collection scenes include monitoring images of different working time periods, different mine car working scenes, and different mine cars and workers climbing mine cars from multiple angles. The collected data should cover as many abnormal situations as possible, including but not limited to abnormal situations in the working area of ​​the monitoring screen, emergencies, and possible screen occlusions to enhance the robustness of the model. Ensure that the collected images can cover the actions of workers climbing mine cars under different types of mine cars and different lighting, different working scenes, and different angles to ensure the diversity and representativeness of the materials so that subsequent model training can learn more features. Use reliable storage media and data management systems to ensure that image data and annotation data can be stored for a long time and easily accessed. At the same time, pay attention to the appropriate processing and protection of the collected sensitive information of the staff to prevent privacy data leakage.

[0008] Step 2, dataset annotation: The dataset annotation should ensure the consistency, accuracy, and completeness of the annotation, and avoid annotation ambiguity. The annotation of the character pose dataset should follow the human target annotation format in the COCO Keypoint detection dataset, that is, the 17 points should be annotated in sequence according to the order of the COCO dataset. All climbing actions that can be judged by the naked eye are annotated, and appropriate targets are discarded for situations that are difficult to distinguish with the naked eye, ensuring that all features or events that need to be annotated are correctly annotated. According to the detection requirements, appropriate fine-grained annotation is performed, and multiple objects in complex scenes are finely annotated to ensure that the annotated data covers a variety of different scenarios and possible working situations to enhance the representativeness of the data.

[0009] Step 3: Dataset splitting and training image preprocessing: Shuffle the dataset and divide it into training set, validation set, and test set in a ratio of 7:1:2. Make full use of all data and reduce the risk of overfitting to more accurately estimate the performance of the model.

[0010] Scaling, image color space adjustment, and Mosaic image data augmentation are used to increase data diversity, enhance model robustness, eliminate the effects of randomness, and make the selection results more universal and more robust in practical applications.

[0011] Step 4. Character pose detection model YOLO-Pose: YOLO-Pose is a pose estimation based on the YOLOv5 target detection framework. It uses end-to-end joint detection training with the output of the heatmap without intermediate detection, and optimizes the target key point similarity (OKS) metric itself. It jointly detects the bounding box (with a person in the box) and its corresponding two-dimensional pose in a single forward pass calculation. At the same time, YOLO-Pose does not require post-processing of the bottom-up method to group the detected key points into a skeleton, because each bounding box has an associated pose, so the key points themselves have inherent grouping. And YOLO-Pose is different from the top-down method because all people's poses are localized, so multiple forward propagation calculations can be cancelled.

[0012] Step 5: Improved YOLOv5 model and post-processing: Introduce Transformer encoding blocks in the original YOLOv5 framework to replace some convolutional blocks in YOLOv5. Each Transformer encoding block contains two sub-layers, and residual connections are used between each sub-layer to increase the ability to capture different local information. Finally, in the detection stage, according to the reasoning results of the improved YOLO-Pose model based on YOLOv5, it is determined whether the staff detected within the video detection range have any action behavior of climbing the mine car. If so, an alarm signal is output.

[0013] Compared with the prior art, the present invention has the following specific beneficial effects:

[0014] 1. The method for detecting people climbing mine cars based on the underground environment proposed by the present invention is an efficient target detection algorithm, which can accurately and in real time detect the position and movement of people, effectively and timely identify people climbing mine cars in the underground environment, thereby effectively reducing the incidence of accidents and ensuring the safety of underground workers. In summary, the present invention can realize the automated detection of people climbing mine cars in the underground environment, reduce manual intervention, improve work efficiency and work environment safety, and has the advantages of safety, strong stability, and high efficiency.

[0015] Second, the present invention produces a high-quality human posture detection data set for the working status of underground workers, and uses the YOLO-Pose human posture detection framework based on the mainstream YOLOv5 detection model. On this basis, the TPH attention optimization module is introduced to replace some convolution blocks and CSP bottleneck blocks in YOLOv5, increasing the model's ability to capture different local information and mine feature representations. Finally, multi-level judgment is introduced to process and judge the results obtained after reasoning, realizing the detection of personnel climbing mine cars in various scenarios underground, and using the judged alarm signal as the output of the algorithm, providing an accurate and effective monitoring solution for the safety production supervision of the underground working environment.

[0016] 3. The present invention uses AI technology to monitor underground personnel climbing mine cars. It can monitor the behavior of workers climbing mine cars in underground environments around the clock, promptly detect and avoid improper operations or accidents, reduce the probability of underground production safety accidents, and thus ensure the safety of workers. At the same time, the monitoring method can also promptly detect potential safety hazards, ensure the continuous operation and efficiency of the production line, reduce the probability of accidents, reduce personal injury and property losses caused by accidents, and improve the efficiency and safety of underground operations. Therefore, the social benefits obtained by the detection method of personnel climbing mine cars based on the underground environment are outstanding and worthy of promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Annotated example image for COCO Keypoint Detection.

[0018] Figure 2 This is a flow chart of the method for detecting people climbing mine cars based on improved YOLOv5.

[0019] Figure 3 This is a structural diagram of the YOLOv5 backbone model.

[0020] Figure 4 Schematic diagram of two CSP structures of YOLOv5. Figure 4 (a) is a schematic diagram of the CSP1_X structure; Figure 4 (b) is a schematic diagram of the CSP2_X structure.

[0021] Figure 5 It is a schematic diagram of the structure of the TPH block.

[0022] Figure 6 Flow chart of post-detection processing judgment. DETAILED DESCRIPTION

[0023] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] like Figure 1 As shown, a method for detecting personnel climbing a mine car based on an underground environment, the specific steps are as follows:

[0025] Step 1: Dataset collection: Download the surveillance camera history records containing underground mine cars, and randomly extract frames to save images of underground mine cars and surrounding workers working in different time periods. Select images of workers climbing mine cars and working normally from multiple angles and scenes, and filter and save images. The saved images include images collected in different working time periods, different lighting, different working scenes, and some occasional situations, to ensure that the collected images can cover as many possible underground mine cars and workers working conditions as possible, and ensure the diversity and representativeness of the materials so that subsequent deep learning can learn more features.

[0026] Step 2: Dataset annotation: Use the labelme dataset annotation tool to annotate the poses of people in the collected images. Follow the COCO pose annotation rules and annotate 17 key points. The labels and corresponding numbers are shown in Table 1.

[0027] Table 1 Human body posture annotation labels and their corresponding annotation numbers

[0028] Tag Name Marking number Tag Name Marking number nose 1 left wrist 10 Left Eye 2 right wrist 11 Right eye 3 Left hip 12 left ear 4 Right hip 13 Right ear 5 Left knee 14 Left shoulder 6 right knee 15 Right shoulder 7 Left ankle 16 Left Arm 8 Right ankle 17 Right Arm 9

[0029] After creating the control points in labelme, the corresponding target person object is labeled using the target box. The labeling example is shown in the figure below: Figure 1 As shown. The annotation label is person. If the target object only shows half of the body or the body is blocked by less than 17 key points, the key points that cannot be annotated are skipped. When annotating people, only clean targets that can be distinguished by the human eye are annotated, and blurred targets that are difficult to distinguish are discarded. Ensure that the target box is not on the image boundary to prevent out-of-bounds errors during data loading or data expansion, and discard target boxes with extreme aspect ratios. Finally, convert the JSON file annotated in the COCO dataset format into the txt file required by the YOLO dataset.

[0030] Step 3: Dataset splitting and training image preprocessing: Bind the image files and annotation files, shuffle their order, divide the dataset into 7:1:2 ratios for training set: validation set: test set, and save the divided data and labels in designated folders respectively.

[0031] In the training phase, the training data is preprocessed and the data is enhanced online using image scaling, image color space adjustment, mosaic enhancement, and image Mosaic enhancement. The Mosaic image data enhancement algorithm combines multiple images into one image at a certain ratio, allowing the model to identify targets in a smaller range. First, the coordinates of the image splicing reference point (x c ,y c ), randomly select four pictures. The four pictures are based on the reference point (x c ,y c ), after geometric scaling and resizing, they are placed at the upper left, upper right, lower left, and lower right positions of the original image. According to the resizing method of each image, the large image is spliced ​​according to the specified horizontal and vertical coordinates, and the mapping relationship is mapped to the image label to process the detection frame coordinates that exceed the boundary. Usually, the Mosaic image data enhancement algorithm uses four images for splicing.

[0032] Step 4: Build the character posture detection model YOLO-Pose:

[0033] Step S41, YOLO-Pose backbone: The YOLOv5 backbone model is mainly divided into three parts: BackBone, Neck and Head. YOLO-Pose uses CSP-darknet53 as the Backbone, and uses PANet to fuse features of different scales from the Backbone, followed by 4 heads of different scales. Finally, there are 2 Decoupled Heads for predicting boxes and keypoints. The target box detection head (Box head) predicts the bounding box, target score and confidence score of each Anchor. There are 3 Anchors of different shapes corresponding to each grid position, and the specific structure is as follows Figure 2 shown.

[0034] For human pose detection, it can be regarded as a single-class person recognition problem. Each human target has 17 related key points. For each key point, the key point detection head is used to identify the bounding box, target score and confidence score again. Therefore, for a single target person, the 17 key points associated with a detected target box have a total of 51 elements, that is, for a predicted anchor, the key point detection head predicts 51 elements, and the target box detection head predicts 6 elements. For an anchor with n key points, the overall prediction vector P v Defined as:

[0035]

[0036] Among them, C x is the x coordinate of the center point of the marking box, Cy is the y coordinate of the center point of the marker, W is the width of the marker frame, and H is the height of the marker frame. is the x coordinate of the nth key point, is the y coordinate of the nth key point, is the confidence of the nth key point. The key point confidence is trained based on the visibility flag of the key point. If a key point is visible or occluded, the confidence is set to 1; otherwise, if the key point is outside the field of view, the confidence is set to 0.

[0037] Step S42, loss function: YOLO-Pose uses CIoU Loss for bounding box supervision. For the Ground Truth bounding box matched by the kth Anchor at position and scale s, the loss function is defined as follows:

[0038]

[0039] in, is the predicted box of the kth Anchor at position (i, j) and scale.

[0040] OKS is different from the standard IoU loss. The gradient of IoU loss will disappear in the case of non-overlap, while OKS loss will never store the entire pose information for each bounding box. Therefore, if a GT bounding box matches the Anchor in position and scale s, YOLO-Pose will predict the key points relative to the center of the Anchor, and calculate OKS for each key point separately, and then sum them up to give the final OKS loss or key point IOU loss, calculated as follows:

[0041]

[0042] Among them, d n is the Euclidean distance between the predicted position and the actual position of the nth key point, k n is the special weight of the key point, s is the target field of view, δ(v n ) is the flag bit of whether the nth key point is visible. For each key point, the model learns a confidence parameter, which represents whether the key point exists in the target person. Here, the visibility flag of the nth key point δ(v n ) is used as GT, and the overall loss function of the model is as follows:

[0043]

[0044] Among them, L is the loss function, and the hyperparameter λ is used to balance the loss.

[0045] Step 5: Improved YOLOv5 model and post-processing:

[0046] Step S51, Transformer Prediction Heads (TPH) optimization.

[0047] The CSP structure of YOLOv5s divides the original input into two branches, performs convolution operations on each branch to halve the number of channels, and then performs Bottleneck*N operations on one of the branches. The two branches are then combined to make the input and output sizes of Bottleneck CSP the same, allowing the model to learn more features. There are two designs of CSP in YOLOv5, namely CSP1_X structure and CSP2_X structure. The schematic diagrams of the two CSP structures are shown in the figure below. Figure 3 shown.

[0048] CSP1_X divides the input into two branches. One branch first passes through CBS, then passes through N residual units (Resunit), i.e. Bottleneck*N, and then performs another convolution. The other branch directly performs convolution. After that, the two branches are spliced. The spliced ​​result is processed by normal distribution (BN), then processed by activation function, and finally CBS calculation is performed. CSP1_X is applied to the backbone network to avoid the gradient disappearance caused by deepening, so that more fine-grained features can be extracted without worrying about network degradation. Compared with CSP1_X, CSP2_X replaces the residual unit (Resunit) with 2*X CBS modules, which is mainly used in Neck networks.

[0049] TPH replaces some convolutional blocks and CSP bottleneck blocks in YOLOv5 with Transformer encoder blocks. Its structure is as follows Figure 4 As shown in the figure, each Transformer encoder block contains two sublayers, the first sublayer is a multi-head attention layer, and the second sublayer is a fully connected layer (MLP). Residual connections are used between each sublayer. In addition to increasing the ability to capture different local information, the Transformer encoder block of THP also uses the self-attention mechanism to explore the potential of feature representation.

[0050] Step S52, post-processing of underground personnel climbing posture detection: post-detection processing and judgment flow chart is as follows Figure 5As shown, in the detection stage, after the model infers and obtains the inference target, it outputs the detected working posture judgment of the underground staff. If the staff is detected to have a climbing posture, it is judged again whether the mine car target frame is detected in the picture. If the staff climbing posture is detected, but the mine car target is not detected in the picture, it means that the staff made a climbing action but did not climb the mine car, so the alarm condition is not met; if the mine car target is detected in the picture, the distance between the mine car target and the target frame of the person who detected the climbing action is calculated again. If the distance between the two target frames is less than the threshold or the two target frames overlap, it means that the staff is climbing the frame car. At this time, the alarm condition is met and the alarm signal is output; if the distance between the mine car target frame and the person target frame is far beyond the threshold, it is not judged that the staff is climbing the mine car. At this time, the alarm condition is not met and the alarm signal is not output.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A method for detecting people climbing mine cars based on underground environment, It is characterized in that The specific steps are as follows: Step 1: Dataset collection: Collect climbing movements of underground mine cars and mine car workers; Step 2: Dataset annotation: According to the human key point annotation specification of the COCO dataset, the 17 key points of the human images in the dataset are annotated in order; Step 3: Dataset splitting and training image preprocessing: Shuffle the data set and divide it into training set, validation set, and test set in a ratio of 7:1:2; use scaling, image color space adjustment, and Mosaic image data enhancement to process the data set; Step 4: Establish the human pose detection model YOLO-Pose: Based on the pose estimation of the YOLOv5 target detection framework, use end-to-end joint detection training without heat map output, optimize the target key point similarity metric itself, and jointly detect multiple bounding boxes containing people and the corresponding two-dimensional poses in one forward pass calculation; Step 5. Improve the YOLOv5 model and post-processing: Introduce the Transformer encoder block in the YOLOv5 framework. Each Transformer encoder block contains two sub-layers, and residual connections are used between each sub-layer. In the detection stage, according to the reasoning results of the YOLO-Pose model, it is determined whether the staff detected within the video detection range have any action behavior of climbing the mine car. If so, an alarm signal is output.

2. According to claim 1, a method for detecting personnel climbing a mine car based on an underground environment, It is characterized in that In step one, the specific steps are: the acquisition scenes include monitoring images of different working time periods, different mine car working scenes, different mine cars and multi-angle images of workers climbing mine cars. The acquired images cover different types of mine cars, different lighting, different working scenes, and the actions of workers climbing mine cars under different angles.

3. According to claim 2, a method for detecting personnel climbing a mine car based on an underground environment, It is characterized in that In step 2, all climbing movements that can be identified by the naked eye are marked, and those that are difficult to distinguish by the naked eye are discarded, and fine-grained marking is performed according to detection requirements.

4. According to the method for detecting personnel climbing a mine car based on an underground environment as described in claim 1, It is characterized in that In step three, scaling, image color space adjustment, and Mosaic image data augmentation are used to increase the diversity of the data in the dataset.

5. According to the method for detecting personnel climbing a mine car based on an underground environment as described in claim 1, It is characterized in that In step 4, each keypoint uses the detection head to identify the bounding box, object score and confidence score again. For an anchor with n keypoints, the overall prediction vector P v Defined as: Among them, C x is the x coordinate of the center point of the marking box, C y is the y coordinate of the center point of the marking box, W is the width of the marking box, and H is the height of the marking box. is the x coordinate of the nth key point, is the y coordinate of the nth key point, is the confidence of the nth key point; box conf is the confidence of the detection box, class conf is the classification confidence; If a keypoint is visible or occluded, the confidence is set to 1; otherwise, if the keypoint is out of the field of view, the confidence is set to 0.

6. A method for detecting a person climbing a mine car based on an underground environment according to claim 5, It is characterized in that In step 4, the loss function is arranged. The specific loss function definition is shown in the following formula: in, is the predicted box of the location (i, j) and scale of the kth anchor point; is the true value of the position (i, j) and scale of the kth anchor point; The final OKS loss is as follows: Among them, d n is the Euclidean distance between the predicted position and the actual position of the nth key point, k n is the special weight of the key point, s is the target field of view, δ(v n ) is the flag bit indicating whether the nth key point is visible; The visibility flag δ(v n ) is used as the true label, and the overall loss function of the model is as follows: Among them, L represents the loss function, and the hyperparameter λ is used to balance the loss.

Images