Detection method for underground roof bolt support process based on skeleton characteristics

Abstract

The invention belongs to the technical field of video image processing, and particularly relates to a method for detecting an underground roof bolt support process based on skeleton characteristics, and the specific technical scheme is as follows: firstly, selecting a proper camera or sensor to obtain video information of human motion, recording video clips containing different actions through acquisition equipment, and recording the video clips; the method comprises the following steps: firstly, training a video frame, then detecting and extracting human body key points in the video frame by using a related human body posture estimation framework, and finally, reasoning by inputting an extracted skeleton sequence into a trained model by adopting an improved sequential network TGA-GRU to obtain an action classification result. According to the method, the situation that due to improper operation, the roof support is not firm, the roof fall occurs, and casualties are caused is prevented, and the proposed human body posture key points are combined with the TGA-GRU sequential network model, so that good stability and robustness are achieved in the aspect of process action recognition of dynamic classification.

Description

Technical Field

[0001] The invention belongs to the technical field of video image processing, and in particular relates to a method for detecting an underground roof anchor support process based on skeleton features. Background Art

[0002] Underground support anchors play an important role in underground support operations. They provide a good guarantee for the safety and stability of the underground working environment by fixing support, reinforcing geology, reducing ground stress, etc. However, improper operation by underground workers during anchor support may lead to a series of serious consequences. For example, due to improper construction procedures, the anchors may not be effectively anchored in the rock, causing the anchors to be pulled out or broken when subjected to tension or shear force, and thus lose the support effect; improper procedures may lead to unstable rock formations and the risk of local collapse.

[0003] Traditional action recognition methods based on image features alone are limited when faced with actual scenes where the background color or environmental factors lead to poor quality of RGB image models. External factors such as the brightness of the environment, occlusion by other objects, and self-occlusion also affect the robustness of the action recognition model. The skeleton recognition method successfully avoids the interference of background information because it can represent human motion information through the movement of key points of the skeleton in space and time dimensions. Summary of the invention

[0004] In order to monitor and warn in real time the problems of improper procedures of underground coal mine personnel when performing roof anchor support and reduce the safety risks of coal mines, the present invention provides a detection method for underground roof anchor support process based on skeleton features, which records video clips containing different actions through acquisition equipment, extracts skeleton information of human action behaviors, and obtains the result of action classification by inputting the extracted skeleton sequence into the trained temporal network model for inference.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is: a detection method for underground roof bolt support process based on skeleton features, the specific steps are as follows:

[0006] Step S1, image data acquisition, installing a suitable camera on the underground excavation working face to obtain video information of human body movement, and recording video clips containing different movements through acquisition equipment to obtain a data set for subsequent processing.

[0007] Step S2, data set preparation, according to the data obtained in the above step S1, the data set is prepared, corresponding folders are established for different actions, 50 videos are recorded for each action, and subfolders containing 50 videos are generated under the corresponding folders. The video is converted into frame images using OpenCV (computer vision library) to ensure that each video folder is a frame image of continuous action. Seven process actions based on skeleton features are established, and the specific process actions are: knocking the top, installing temporary support nets, installing drill rods, drilling with anchor bolters, pushing anchors, pushing anchors, and stirring anchors.

[0008] Step S3, defining skeleton key points, because the position changes of the key points of the upper body and head of the human body are mainly the key points of the underground support, and the key points of the face and the lower body are almost not involved in the occurrence of the action during the support process, so there is no need for detection. Therefore, the present invention uses 12 key points to detect the personnel support action, namely: head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, chest, back, left hip, right hip.

[0009] Step S4, data labeling and division, using OpenCV and Pillow (both are computer vision libraries) to rotate, crop and add noise to the preprocessed data set to obtain the preprocessed expanded data set, and then the data set is labeled. The labeling work of the skeleton key point data set is completed by the labeling software Labelme, and the labeled data set is divided into a training set, a validation set and a test set according to a ratio of 7:2:1 to ensure that the training, validation and testing processes of the model are carried out reasonably;

[0010] Step S5: training a key point detection model based on skeleton data. For the data set divided in step S4, the OpenPose model is used to train and extract human skeleton data.

[0011] Furthermore, step S5 comprises the following steps:

[0012] Step S51, data preprocessing, the input image is preprocessed, including resizing, normalization, etc., to adapt to the input requirements of the network.

[0013] Step S52, model training. The present invention adopts the OpenPose model to perform model training for detecting key point information of the human body. OpenPose is a bottom-up key point detection system. The skeleton-based behavior recognition is robust to environmental and illumination changes. It can capture human key points, skeleton structure and posture information from images or videos. In the end-to-end training process, the model realizes synchronous learning of feature extraction and key point detection, and uses standard convolution technology to extract underlying features, thereby achieving a more efficient effect, while having high precision and real-time performance.

[0014] Specifically, the OpenPose network uses the VGG-19 network to parse the underlying features of the input image at the initial stage. These features become the key cornerstone of subsequent processing, and then use the dual parallel execution CNN structure to obtain these underlying features. OpenPose uses a partial affinity domain algorithm to connect the extracted key limb parts to construct a complete human skeleton structure. Finally, the Hungarian algorithm is used to reprocess the output data of the two convolutional networks, connect the identified key limb parts, and thus construct a complete and detailed human skeleton structure.

[0015] Regarding the VGG-19 network (Visual Geometry Group Network 19), the first 10 layers of the network extract the input features of the video, obtain the function graph, and then send it to a two-branch multi-level CNN for prediction. The key prediction element of Branch 1 is the confidence map (Pant Confidence Maps, PCM), which assigns different weights to each pixel in the candidate area to obtain the corresponding confidence value. Branch 2 outputs the joint affinity field (Part Afinity Fields, PAF), and the task is to infer the connection between various parts of the body. By analyzing the relationship between the confidence map and PAF, the bipartite graph matching method is used to determine the posture of the human body. For two key points with physical connections, the midpoint of the limb area where the connection relationship is constructed is marked as the unit vector corresponding to the connection direction, and the formula is expressed as:

[0016]

[0017] In the formula, The cth group of limb vectors representing the kth person’s body, described by the affinity vector at pixel p, x j1,k With x j2,k Represents the real coordinate value of the key point, and v is the unit vector describing the affinity between the two key points.

[0018] About Partial Afinity Fields (PAFs) and the Hungarian algorithm, the Partial Affinity Fields algorithm is an algorithm for human pose estimation, which is used to connect the detected human key points in the image to form a complete human pose. The main purpose of this algorithm is to calculate the confidence between candidate key points to determine whether they should be connected. The Hungarian algorithm model has the ability to identify multiple possible connection modes between key parts of the human body, and can efficiently combine different action entities, thereby screening out the limb combination with the highest overall confidence.

[0019] Step S53: In order to test the detection performance of the model of the present invention, precision, recall rate, mAP@0.5, and F1 score are used as evaluation indicators.

[0020] Precision is the ratio of the number of positive samples predicted by the model to the number of all detected samples, and is calculated as follows:

[0021]

[0022] In the formula, TP represents the number of positive samples predicted by the model, and FP represents the number of negative samples predicted by the model.

[0023] Recall is the ratio of the number of positive samples correctly predicted by the model to the number of positive samples that actually occurred. The recall calculation is as follows:

[0024]

[0025] In the formula, TP represents the number of positive samples predicted by the model, and FN represents the number of positive samples predicted incorrectly by the model.

[0026] The average precision (AP) is the area under the precision and recall curves. The mean average precision (mAP@0.5) is the result obtained by the weighted average of the AP values ​​of all sample categories. It is used to evaluate the detection performance of the model in all categories. The threshold of the intersection ratio of the predicted box and the true box is set to 0.5. The formula is as follows:

[0027]

[0028] Where N is the sample category, AP represents the area under the precision and recall curves, and @0.5 means that the IoU threshold used when calculating the area under the precision and recall curves is 0.5.

[0029] The F1 score, also known as the balanced F score, is the harmonic mean of precision and recall. Its maximum value is 1 and its minimum value is 0. The larger the value, the better the model performance. The formula is as follows:

[0030]

[0031] Where Precision and Recall represent the model accuracy and recall rate respectively, TP represents the number of positive samples predicted by the model, FP represents the number of negative samples predicted by the model, and FN represents the number of negative samples predicted by the model.

[0032] Step S6: According to the above step S5, the position information of the key parts is located, the feature vector of the key parts is extracted, and the obtained feature vector is input into the action classifier to classify the human body action. For each process action, the skeleton key point trajectory in the time sequence is recorded, that is, the position of each key point changes with time within a period of time.

[0033] Furthermore, step S6 comprises the following steps:

[0034] Step S61, propose an improved model TGA-GRU based on skeleton sequence, based on the human joint key point information extracted from the image in step S5 above, combined with TGA-GRU can help capture the dynamic changes of the body movements of the characters in the video, and improve the ability to identify and track the key points of the human body in continuous image frames. TGA-GRU uses the target-guided attention mechanism to focus on the key parts of the human body (whether the target-guided attention mechanism is specifically described) and identify human body movements;

[0035] The target-guided attention mechanism (TGA) is to use the information on the target language side to guide the distribution of attention. Specifically, when the decoder generates output at each time step, the TGA mechanism calculates an attention weight vector based on the current target language side information. Through the target-guided attention mechanism, the model can better capture relevant information and improve the performance of the neural network. By introducing the TGA mechanism in the decoder, the model can focus on key parts related to human body movements and improve recognition accuracy.

[0036] Step S62, define the TGA-GRU network structure, set the feature dimension of the data to 5, the hidden unit dimension to 100, then pass through the fully connected layer with a dimension of 5, and finally identify 5 human body actions for the Softmax layer and the classification layer, use Adam to optimize the hyperparameters, the initial learning rate is 0.01, and the decline cycle is 20.

[0037] Step S63, calculation of the TGA-GRU model. The TGA-GRU model is a recurrent neural network model based on a target-guided attention mechanism. Its mechanism is based on the LSTM model and adds an attention mechanism, which can better handle the modeling and prediction problems of sequence data. The calculation process of the TGA-GRU model is as follows:

[0038] Step S63, the calculation process of the TGA-GRU model is as follows:

[0039] First, assume that the input sequence is x = {x1, x2, ..., x T}, the output states of GRU are h1,h2,…,h T Specifically, GRU can update the gate z t and reset gate r t Generate the current time unit control state h t The update gate determines the extent to which the cell updates its activation or content. The reset gate effectively makes the cell behave as if it is reading the first signal of the input sequence, allowing previously computed states to be forgotten.

[0040] Secondly, GRU transforms h tAs the input of the attention mechanism, we get the attention weight α t :

[0041]

[0042] In the formula, represents the weight vector, W α represents the weight matrix, h t is the output state of the GRU unit, b α is the bias vector;

[0043] Then, use α t Perform weighted summation on all time steps of the input sequence to obtain the weighted vector z t :

[0044]

[0045] In the formula, α t is the calculated attention weight, h t is the output state of the GRU unit;

[0046] Finally, z t With h t Connect and input a fully connected layer to get the final output:

[0047] y t =Softmax(W y [z t ;h t ]+b y )

[0048] In the formula, [z t ;h t ] is z t and h t The splicing of W y and b y are the parameters of the fully connected layer, which are the weight matrix and the bias vector respectively.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. Improved model accuracy: The present invention adopts a target-guided attention mechanism to focus attention on key parts of the human body. By focusing on the target area, the model can reduce the impact of noise from the background or other interference sources, improve overall robustness, and improve model accuracy. Under target guidance, the model can extract more fine-grained features and more accurately identify human movements.

[0051] 2. Real-time performance: The present invention adopts an improved model TGA-GRU for action classification and recognition. Compared with the traditional LSTM (long short-term memory network), the improved method has fewer parameters and computational overhead. It can improve computational efficiency while maintaining good performance, making the TGA-GRU model more efficient in processing large-scale data and suitable for deployment on embedded systems and mobile devices.

[0052] 3. Improved algorithm robustness: The present invention identifies underground process actions based on the skeleton recognition method. This method can represent human motion information through the movement of skeleton key points in space and time dimensions, successfully avoiding the interference of background information and minimizing the impact of background information on the recognition results. For poor lighting conditions and complex backgrounds underground, the algorithm proposed by the present invention can adapt to action recognition in such complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a workflow flowchart of the skeleton feature-based action recognition of the present invention.

[0054] Figure 2 This is the architecture diagram of the action classifier based on TGA-GRU.

[0055] Figure 3 This is the schematic diagram of the TGA-GRU model.

[0056] Figure 4 This is an example diagram of the action of installing a drill rod during the support process.

[0057] Figure 5 This is an example diagram of the drilling action in the support process.

[0058] Figure 6 This is an example diagram of the action of pushing the anchor agent during the support process.

[0059] Figure 7 This is an example diagram of the action of knocking on the top before the support process.

[0060] Figure 8 This is the OpenPose network structure diagram.

[0061] Fig. 9 This is the VGG-19 network structure diagram. DETAILED DESCRIPTION

[0062] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] like Figure 1 As shown, a detection method for underground roof bolt support process based on skeleton features, the specific steps are as follows:

[0064] Step S1, image data acquisition, select a suitable camera or sensor to obtain video information of human body movement, record video clips containing different movements through the acquisition device, and obtain a data set for subsequent processing.

[0065] Step S2, data set preparation, data set preparation is performed according to the image data obtained in the above step S1, 7 corresponding folders are established for 7 actions, 50 videos are recorded for each action, and a folder containing 50 videos is generated accordingly. Each video generates dynamic actions from 25 frames of continuous pictures, and 25 numpy files are generated accordingly. As shown in Table 1, 7 process actions based on skeleton features are established. Different actions are simulated according to this table, and encoded according to the order of action category collection, which are: knocking on the top, installing temporary support nets, installing drill rods, drilling with anchor bolters, pushing anchors, pushing anchors, and stirring anchors. The specific working process is as follows: Figure 4-7 shown.

[0066] Table 1 Process action simulation table based on skeleton features

[0067]

[0068] Step S3, defining skeleton key points. Since the position changes of the key points of the upper body and head of the human body are mainly the key points during underground support, and the key points of the face and lower body are almost not involved in the occurrence of actions during the support process, there is no need for detection. Therefore, the present invention uses the following 12 key points to detect personnel support actions, and the specific distribution is shown in Table 2 below.

[0069] Table 2 Key point distribution table

[0070]

[0071] Step S4, data labeling and division. The labeling work of the skeleton key point data set of the present invention is mainly completed by the labeling software Labelme. A total of 3,000 pictures are labeled. Considering that the number of samples can directly determine the robustness and generalization of the model, the data set is increased to 10,000 pictures and data division is performed by data expansion methods such as rotation, cropping, and adding noise in the later stage. The training data accounts for 70% (7,000 pictures) of the entire data set, the verification set accounts for 20% (2,000 pictures) of the entire data set, and the test set accounts for 10% (1,000 pictures) of the entire data set.

[0072] Specifically, each corresponding numpy file generated in step S2 contains 24 key point coordinate data of the body, and a single action generates 600 key point coordinate data. For example, the partial skeleton sequence data of the "knock on the side and ask for the top" action in the support action is shown in Table 3. Each data has 25 columns, the first column is the label, and the next 24 columns are the coordinates of 12 key points.

[0073] Table 3 Partial skeleton sequence data of knocking, asking and top action

[0074]

[0075] Step S5: training a key point detection model based on skeleton data. For the data set divided in step S4, the OpenPose model is used to train and extract human skeleton data.

[0076] Furthermore, step S5 comprises the following steps:

[0077] Step S51, data preprocessing, the input image is preprocessed, including resizing, normalization, etc., to adapt to the input requirements of the network.

[0078] Step S52, model training. The present invention uses the OpenPose model to perform model training for detecting key point information of the human body. OpenPose is a bottom-up key point detection system. The skeleton-based behavior recognition is robust to environmental and illumination changes. It can capture human key points, skeleton structure and posture information from images or videos. In the end-to-end training process, the model realizes the synchronous learning of feature extraction and key point detection, and uses standard convolution technology to extract underlying features, thereby achieving a more efficient effect, and having high precision and real-time performance. Its network structure is as follows: Figure 8 shown.

[0079] Specifically, the OpenPose network uses the VGG-19 network to parse the underlying features of the input image at the initial stage. These features become the key cornerstone of subsequent processing, and then use the dual parallel execution CNN structure to obtain these underlying features. OpenPose uses a partial affinity domain algorithm to connect the extracted key limb parts to construct a complete human skeleton structure. Finally, the Hungarian algorithm is used to reprocess the output data of the two convolutional networks, connect the identified key limb parts, and thus construct a complete and detailed human skeleton structure.

[0080] Regarding the VGG-19 network (Visual Geometry GroupNetwork 19), its network structure is as follows Fig. 9As shown. The first 10 layers of the network extract the input features of the video, obtain the function graph, and then send it to the 2-branch multi-level CNN for prediction. The key prediction element of branch 1 is the confidence map (Pant Confidence Maps, PCM), which obtains the corresponding confidence value by assigning different weights to each pixel in the candidate area. Branch 2 outputs the joint affinity field (PartAfinityFields, PAF), and the task is to infer the connection between various parts of the body. By analyzing the relationship between the confidence map and PAF, the bipartite graph matching method is used to determine the posture of the human body. For two key points with physical connection, the midpoint of the limb area where the connection relationship is constructed is marked as the unit vector corresponding to the connection direction. The formula is expressed as:

[0081]

[0082] In the formula, The cth group of limb vectors representing the kth person’s body, described by the affinity vector at pixel p, x j1,k With x j2,k Represents the real coordinate value of the key point, and v is the unit vector describing the affinity between the two key points.

[0083] About Partial Afinity Fields (PAFs) and the Hungarian algorithm, the Partial Affinity Fields algorithm is an algorithm for human pose estimation, which is used to connect the detected human key points in the image to form a complete human pose. The main purpose of this algorithm is to calculate the confidence between candidate key points to determine whether they should be connected. The Hungarian algorithm model has the ability to identify multiple possible connection modes between key parts of the human body, and can efficiently combine different action entities, thereby screening out the limb combination with the highest overall confidence.

[0084] Step S53: In order to test the detection performance of the model of the present invention, precision, recall rate, mAP@0.5, and F1 score are used as evaluation indicators.

[0085] Precision is the ratio of the number of positive samples predicted by the model to the number of all detected samples, and is calculated as follows:

[0086]

[0087] In the formula, TP represents the number of positive samples predicted by the model, and FP represents the number of negative samples predicted by the model.

[0088] Recall is the ratio of the number of positive samples correctly predicted by the model to the number of positive samples that actually occurred. The recall calculation is as follows:

[0089]

[0090] In the formula, TP represents the number of positive samples predicted by the model, and FN represents the number of positive samples predicted incorrectly by the model.

[0091] The average precision (AP) is the area under the precision and recall curves. The mean average precision (mAP@0.5) is the result obtained by the weighted average of the AP values ​​of all sample categories. It is used to evaluate the detection performance of the model in all categories. The threshold of the intersection ratio of the predicted box and the true box is set to 0.5. The formula is as follows:

[0092]

[0093] Where N is the sample category, AP represents the area under the precision and recall curves, and @0.5 means that the IoU threshold used when calculating the area under the precision and recall curves is 0.5.

[0094] The F1 score, also known as the balanced F score, is the harmonic mean of precision and recall. Its maximum value is 1 and its minimum value is 0. The larger the value, the better the model performance. The formula is as follows:

[0095]

[0096] Where Precision and Recall represent the model accuracy and recall rate respectively, TP represents the number of positive samples predicted by the model, FP represents the number of negative samples predicted by the model, and FN represents the number of negative samples predicted by the model.

[0097] Step S6, locate the position information of the key parts according to the above step S5, extract the feature vector of the key parts, and input the obtained feature vector into the action classifier to classify the human body action. For each process action, record its skeleton key point trajectory in time sequence, that is, the change of the position of each key point over time within a period of time. In view of the problems of long training time, too many parameters, and easy gradient disappearance of LSTM model, the present invention proposes an improved model TGA-GRU for human body action recognition. The action classifier architecture based on TGA-GRU and the principle diagram of TGA-GRU model are shown in the figure. Figure 2 , Figure 3 shown.

[0098] GRU network is an effective algorithm for learning, classifying and predicting time series signals. The basic architecture of GRU is the same as LSTM, except that the update of the hidden layer is replaced by a specially constructed storage unit. The GRU unit accepts an input sequence x = {x1, x2, ..., x T}, where x t is the feature vector of the tth time step, and calculates the output sequence y = {y1, y2, ..., yt}. GRU can update the gate z tand reset gate r t Generate the current time unit control state h t The update gate determines the extent to which the cell updates its activation or content. The reset gate effectively makes the cell behave as if it is reading the first signal of the input sequence, allowing previously computed states to be forgotten.

[0099] Furthermore, step S5 comprises the following steps:

[0100] Step S61, GRU (temporal feature modeling method) is improved. In order to improve the real-time performance of detection and reduce the computing resource usage, an improved model TGA-GRU based on skeleton sequence is proposed. The target-guided attention mechanism can focus on the key parts of the human body, so as to more accurately identify human body movements. The model principle is as follows Figure 1 shown.

[0101] Regarding the GRU model, GRU is an improved model of LSTM. Both use the gate mechanism to retain important time series features, but GRU implements the same process in a simpler way. It can also effectively solve the gradient disappearance problem in RNN, and can reduce training parameters and improve prediction speed while maintaining training effects.

[0102] The target-guided attention mechanism (TGA) is to use the information on the target language side to guide the distribution of attention. Specifically, when the decoder generates output at each time step, the TGA mechanism calculates an attention weight vector based on the current target language side information. Through the target-guided attention mechanism, the model can better capture relevant information and improve the performance of the neural network. By introducing the TGA mechanism in the decoder, the model can focus on key parts related to human body movements and improve recognition accuracy.

[0103] GRU (time series feature modeling method) is an improved model of LSTM. Both use the gate mechanism to retain important time series features, but GRU implements the same process in a simpler way. It can also effectively solve the gradient disappearance problem in RNN, and can reduce training parameters and improve prediction speed while maintaining training effects.

[0104] Step S62, define the TGA-GRU network structure, set the feature dimension of the data to 5, the hidden unit dimension to 100, then pass through the fully connected layer with a dimension of 5, and finally identify 5 human body movements for the Softmax layer and the classification layer, use Adam to optimize the hyperparameters, the initial learning rate is 0.01, and the decline cycle is 20.

[0105] Step S63, calculation of the TGA-GRU model. The TGA-GRU model is a recurrent neural network model based on a target-guided attention mechanism. Its mechanism is based on the LSTM model and adds an attention mechanism, which can better handle the modeling and prediction problems of sequence data. The calculation process of the TGA-GRU model is as follows:

[0106] Step S63, the calculation process of the TGA-GRU model is as follows:

[0107] First, assume that the input sequence is x = {x1, x2, ..., x T}, the output states of GRU are h1,h2,…,h T Specifically, GRU can update the gate z t and reset gate r t Generate the current time unit control state h t The update gate determines the extent to which the cell updates its activation or content. The reset gate effectively makes the cell behave as if it is reading the first signal of the input sequence, allowing it to forget the previously calculated state.

[0108] Secondly, GRU transforms h t As the input of the attention mechanism, we get the attention weight α t :

[0109]

[0110] In the formula, represents the weight vector, W α represents the weight matrix, h t is the output state of the GRU unit, b α is the bias vector;

[0111] Then, use α t Perform weighted summation on all time steps of the input sequence to obtain the weighted vector z t :

[0112]

[0113] In the formula, α t is the calculated attention weight, h t is the output state of the GRU unit;

[0114] Finally, z t With h t Connect and input a fully connected layer to get the final output:

[0115] y t =Softmax(W y [z t ;h t]+b y )

[0116] In the formula, [z t ;h t ] is z t and h t The splicing of W y and b y are the parameters of the fully connected layer, which are the weight matrix and the bias vector respectively.

[0117] After acquiring the image through video information, it is input into the human posture estimation model to extract the skeleton sequence data, thereby obtaining the corresponding skeleton feature vectors, and then these feature vectors are input into the GRU network for time series modeling. The last hidden state Hn is selected to represent the high-level features of a certain skeleton sequence. W and U in the network represent weights. Due to the parameter sharing mechanism, each neuron has the same weight and bias parameters. Finally, after the GRU network time series modeling, a high-level feature Hn of the skeleton sequence is obtained, and the feature is input into the Softmax classifier for category decision. The process structure diagram is shown below. Figure 2 shown.

[0118] Step S7, action classification result analysis, for the process action recognition task based on skeleton features, three models with different structures were trained, namely LSTM, GRU and TGA-GRU, and the main focus was on the accuracy of the three models. The changes in the accuracy of different network models with the number of training rounds are shown in Table 4.

[0119] Table 4 Network model accuracy with different training rounds

[0120]

[0121]

[0122] From the training results in the table above, we can see that the LSTM network reaches an accuracy of 91.66% after 300 iterations, and the GRU network reaches an accuracy of 94.26% after 150 iterations and tends to be stable. The TGA-GRU network reaches an accuracy of 94.70% after 150 iterations and tends to be stable. The final experimental results show that the designed TGA-GRU network achieves the highest accuracy in the process action recognition task based on skeleton features, and achieves the best results in the dataset containing 7 types of process actions.

[0123] The present invention proposes a method for detecting the underground roof bolt support process based on skeleton features, using 0penPose to extract human key points from video frames, and then using TGA-GRU to classify them into one of seven action categories. The present invention provides a feasible method for detecting the underground bolt support process, which can efficiently and quickly detect the support action, has feasibility and effectiveness, and can prevent the roof collapse and casualties caused by improper operation of the roof support in practical applications, which has certain practical significance.

[0124] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A method for detecting underground roof bolt support process based on skeleton features, characterized in that: The specific steps are as follows: Step S1, image data acquisition: a camera is installed on the underground excavation working face to obtain video information of human body movement, and video clips containing different movements are recorded and saved through a collection device; Step S2, data set preparation: according to the data obtained in the above step S1, a data set is prepared, corresponding folders are established for different actions, 50 videos are recorded for each action, and a subfolder containing 50 videos is generated under the corresponding folder. The video is converted into a frame image using OpenCV to ensure that each video folder is a frame image of a continuous action, and 7 process actions based on skeleton features are established. The specific process actions are: knocking the top, installing a temporary support net, installing a drill rod, drilling with an anchor bolter, pushing an anchor, pushing an anchor, and stirring the anchor; Step S3, defining skeleton key points: 12 key points are used to detect personnel support actions, and the specific key points are: head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, chest, back, left hip, and right hip; Step S4, data labeling and division: OpenCV and Pillow are used to rotate, crop and add noise to the preprocessed data set to obtain the preprocessed expanded data set, and then the data set is labeled. The labeling of the skeleton key point data set is completed by the labeling software Labelme, and the labeled data set is divided into a training set, a validation set and a test set according to a ratio of 7:2:1; Step S5, training a key point detection model based on skeleton data: using the OpenPose model to train the data set divided in step S4 and extract human skeleton data; Step S6: locate the position information of the key parts according to the above step S5, extract the feature vectors of the key parts, and input the obtained feature vectors into the action classifier to classify the human body movements and record the skeleton key point trajectory of the process action in time sequence.

2. A method for detecting underground roof bolt support process based on skeleton features according to claim 1, characterized in that: The step S5 comprises the following steps: Step S51, preprocessing the input image, the preprocessing includes adjusting the image size and image normalization, adjusting the image size is to use the calculated scaling factor to proportionally adjust the image width and height to adapt to the input requirements of the network, and image normalization is to adjust the image pixel values ​​by using the cv2.normalize() function to scale all pixel values ​​to a specified range; Step S52: model training, using the OpenPose model to perform model training for detecting key point information of the human body. In the end-to-end training process, the model realizes synchronous learning of feature extraction and key point detection. Step S53, using precision, recall, mAP@0.5, and F1 score as evaluation indicators to test the performance of the detection model; Precision is the ratio of the number of positive samples predicted by the model to the number of all detected samples, and is calculated as follows: In the formula, TP represents the number of positive samples predicted by the model, and FP represents the number of negative samples predicted by the model; Recall is the ratio of the number of positive samples correctly predicted by the model to the number of positive samples that actually occurred. The recall calculation is as follows: In the formula, TP represents the number of positive samples predicted by the model, and FN represents the number of positive samples predicted incorrectly by the model; The average precision AP is the area under the precision and recall curve. The average precision mean mAP@0.5 is the result obtained by the weighted average of the AP values ​​of all sample categories. It is used to evaluate the detection performance of the model in all categories. The threshold of the intersection ratio of the predicted box and the true box is set to 0.

5. The formula is as follows: Where N is the sample category, AP represents the area under the precision and recall curves, and @0.5 means that the IoU threshold used to calculate the area under the precision and recall curves is 0.5; The F1 score is the balanced F score, which is the harmonic mean of precision and recall. The maximum value of the F1 score is 1, and the minimum value of the F1 score is 0. The formula is as follows: In the formula, Precision represents the model accuracy, Recall represents the recall rate, TP represents the number of positive samples predicted by the model, FP represents the number of negative samples predicted by the model, and FN represents the number of negative samples predicted by the model.

3. A method for detecting underground roof bolt support process based on skeleton features according to claim 2, characterized in that: In step S52, the OpenPose network uses the VGG-19 network to parse the underlying features of the input image at the initial stage, and uses the dual parallel execution CNN structure to obtain the underlying features. OpenPose uses a partial affinity domain algorithm to connect the extracted key limb parts, and applies the Hungarian algorithm to reprocess the output data of the two convolutional networks, connect the identified key limb parts, and construct the human skeleton structure; The first 10 layers of the VGG-19 network extract the input features of the video to obtain a function graph, which is then sent to a two-branch multi-level CNN for prediction. The key prediction element of branch 1 is the confidence map, which assigns different weights to each pixel in the candidate area to obtain the corresponding confidence value. Branch 2 outputs the joint affinity domain. By analyzing the relationship between the confidence map and the affinity domain, the bipartite graph matching method is used to determine the posture of the human body. For two key points with physical connections, the midpoint of the limb area where the connection relationship is constructed is marked as the unit vector of the corresponding connection direction. The formula is expressed as: In the formula, The cth group of limb vectors representing the kth person’s body, described by the affinity vector at pixel p, x j1,k With x j2,k Represents the real coordinate value of the key point, and v is the unit vector describing the affinity between the two key points.

4. A method for detecting underground roof bolt support process based on skeleton features according to claim 3, characterized in that: The step S6 comprises the following steps: Step S61, proposing an improved model TGA-GRU based on skeleton sequence, based on the human joint key point information extracted from the image in the above step S5, combined with TGA-GRU to capture the dynamic changes of the body movements of the characters in the video, TGA-GRU uses the target guided attention mechanism to focus on the key parts of the human body and identify the human body movements; When the decoder generates output at each time step, the TGA mechanism calculates an attention weight vector based on the current target language information. By introducing the TGA mechanism in the decoder, the model focuses on the key parts related to human body movements. Step S62, define the TGA-GRU network structure, set the feature dimension of the data to 5, the hidden unit dimension to 100, identify 5 human body actions for the Softmax layer and the classification layer through a fully connected layer with a dimension of 5, use Adam to optimize the hyperparameters, the initial learning rate is 0.01, and the decline cycle is 20; Step S63, the calculation process of the TGA-GRU model is as follows: First, assume that the input sequence is x = {x1, x2, ..., x T }, the output states of GRU are h1,h2,…,h T , GRU updates the gate z t and reset gate r t Generate the current time unit control state h t ,The update gate determines the extent to which the unit updates its activations; Secondly, GRU transforms h t As the input of the attention mechanism, we get the attention weight α t : In the formula, represents the weight vector, W α represents the weight matrix, h t is the output state of the GRU unit, b α is the bias vector; Then, use α t Perform weighted summation on all time steps of the input sequence to obtain the weighted vector z t : In the formula, α t is the calculated attention weight, h t is the output state of the GRU unit; z t With h t Connect and input a fully connected layer to get the final output: y t =Softmax(W y [z t h t ]+b y ) In the formula, [z t ;h t ] is z t and h t The splicing of W y is the weight matrix of the fully connected layer, b y is the bias vector of the fully connected layer.

Images