A coal mine underground human action recognition method suitable for edge terminal

Abstract

The application discloses a coal mine underground human action recognition method suitable for an edge terminal and belongs to the field of computer image recognition. The method is characterized in that a lightweight action recognition model suitable for low illumination is designed by using coal mine underground human action video data, and action recognition is realized on an edge terminal. First, a data acquisition and preprocessing module is used to construct a coal mine underground personnel action data set. Then, an action classification module is used to realize training of a mass parameter network model under an action video feature classification task. Next, a situation network pruning module is used to realize a preliminary lightweight model suitable for low-illumination video data. Finally, a teaching model distillation module is used to further lighten the preliminary lightweight model, and the final lightweight model is applied to the edge terminal. The edge terminal adopts a pruning technology with recovery, reduces the interference of low illumination on compression, adopts coding technology, optimizes storage, and realizes underground edge terminal deployment of a complex model under the premise of ensuring recognition rate.

Description

Technical Field

[0001] This invention relates to a method for recognizing human movements in coal mines, applicable to edge terminals, and particularly applicable to the field of computer image recognition technology. Background Technology

[0002] Standardizing the operations of production personnel can significantly reduce the probability of accidents. Traditional production supervision relies on human supervision, but during long production operations, it is difficult for people to maintain their attention on one point for an extended period. This method of supervision is inefficient and prone to oversight. Even with cameras transmitting video from underground mines to a monitoring room on the surface, manual monitoring and alerts are still required. This method of supervision also has shortcomings: First, if production personnel engage in dangerous behavior, the video image information, after transmission to the monitoring room, the warning issued by the monitoring personnel, and the subsequent transmission of the warning signal back to the mine, can lead to communication delays and signal loss, preventing production personnel from receiving the warning information in a timely manner and thus hindering their ability to follow procedures and prevent accidents from occurring. Second, with centralized monitoring, it is difficult for monitoring personnel to simultaneously monitor multiple monitors, and accidents often occur in the blink of an eye due to momentary negligence.

[0003] Action recognition in coal mines can essentially be seen as an extension of the field of computer image recognition. It involves collecting data, extracting features, and then determining the action category based on those features. In computer image recognition, well-trained deep neural networks can recognize various human actions and exhibit excellent performance. Currently, action recognition using deep neural networks largely relies on convolutional neural networks (CNNs) and recurrent neural networks (RNNs) or their variants. However, CNNs suffer from information loss during pooling, leading to a reduced connection between local and global data. RNNs improve the connection between local and global data to some extent but cannot perform parallel computation, resulting in significantly reduced performance. Existing action recognition models rarely consider low-light application scenarios and have a large number of parameters, requiring deployment on high-performance computers. Unlike common application scenarios, the underground environment of coal mines is dark, making it difficult to deploy high-performance computers. Furthermore, signal transmission is easily interfered with, and a large number of monitoring nodes are required, making it impractical to transmit large-scale underground coal mine video to the surface in real time.

[0004] Existing technologies of the same type still have the following key limitations: First, the judgment models that can be used in existing technologies are huge in size, have high requirements for computing and storage resources, and are difficult to deploy in lightweight terminals; due to the poor lighting conditions in coal mines, conventional pruning methods are prone to accidentally pruning key neurons and weights; existing deep neural network models cannot take into account both parallel computing and attention to local and global information. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for human motion recognition in coal mines, suitable for edge terminals. This method solves the problems of real-time detection limitations, high computational load, and unstable transmission in current coal mine motion recognition technologies. Based on lightweight deep learning video motion recognition technology, this invention utilizes computer vision technology and the computing power of lightweight edge computing terminals to perform real-time analysis of videos collected in coal mines. It identifies the actions of production workers and compares these actions with a pre-designed safety production standard operating procedure action library to determine whether the workers' operations comply with regulations.

[0006] To achieve the above technical objectives, the present invention provides a method for recognizing human actions in coal mines that is suitable for edge terminals. In response to the low-light environment in coal mines, a lightweight action recognition model is constructed that can recognize human actions in low-light environments, and human actions can be recognized using edge terminals.

[0007] The specific steps are as follows:

[0008] Step 1: Use underground cameras to collect videos of mine operations, including those of workers, under both well-lit and low-light conditions. The videos contain human movements of the workers during their operations. The collected video data is preprocessed by classifying the data according to the work scenario, removing videos that do not contain any movements, and generating videos based on the principle that each complete production operation action forms a video data set. Text description tags are added to the corresponding production operation actions to obtain a coal mine underground action image sequence dataset. The action image sequence dataset is then divided into a training set and a test set.

[0009] Step 2: Train the Vision Transformer network using the coal mine underground action image sequence dataset, so that after training, the Vision Transformer network can accurately output text descriptions of the underground personnel's production operations based on the input video. The Vision Transformer network is a Vision Transformer network with an improved loss function, including an embedding layer and an encoder layer.

[0010] Step 3: Perform context pruning on the trained Vision Transformer network to make it lightweight, so that the lightweight Vision Transformer network can adapt to video data shot in low-light conditions in coal mines. The context pruning includes neuron pruning and weight pruning.

[0011] Step 4: Distill the teaching model of the lightweight Vision Transformer network to obtain the final lightweight model after concentration;

[0012] Step 5: Set the final lightweight model in the edge terminal in the coal mine, and use the edge terminal to recognize real-time underground coal mine production operation video data; the edge terminal converts the video data to be recognized into image sequence action recognition, and outputs text descriptions of the production operation actions contained in the video.

[0013] Furthermore, for video images with unclear edge information of personnel and equipment obtained in step 1, spatial domain image enhancement technology is used to sharpen them, and for video images with excessive noise, spatial domain image enhancement technology is used to smooth them. Then, convolution operation is used on the images cropped into frame sequences to cut each frame into 3×3 blocks, a total of 9 small images, forming an image sequence. The obtained coal mine underground action image sequence dataset is divided into training set and test set in an 8:2 ratio.

[0014] Furthermore, in step 1, collecting and constructing a coal mine underground action image sequence dataset requires preprocessing the video data and converting it into image sequence data. The process for creating the action image sequence dataset is as follows:

[0015] Step 1A: Trim the preprocessed video according to the standard of t frames per second, so that it becomes a frame sequence, represented as V={v1,v2,…,v i ,…,v T}; where T is the total number of frames in the video, v i This represents the i-th frame of the video; each frame in the frame sequence can be represented as: Where (H,W) are the height and width of the original image, and C is the number of channels;

[0016] Step 1B: Transfer frame image v i Cut into S = (H × W) / P 2 Small images are divided into blocks, forming an image block sequence F. i ={f i1 ,f i2 ,…,f ij ,…,f iS}, where each small image is represented as f iS Let S represent the S-th small image block in the i-th frame, and P be the side length of each small image block, thus obtaining the result from the image sequence F. i The image sequence dataset consists of: X = {F1, F2, ..., F...} i ,…,F n};

[0017] The method for dividing the training set and the test set is as follows: the obtained coal mine underground action image sequence dataset is divided into a training set and a test set according to the ratio of λ and (1-λ).

[0018] Next, the test set is input into the pre-trained Vision Transformer neural network model, and the action recognition accuracy of the Vision Transformer neural network model on the test set is calculated and used as the evaluation metric of the Vision Transformer neural network model. Based on the selected evaluation metric and the output results of the model, it is determined whether the expected results are achieved. If the expected results are achieved, proceed to step 3. If the expected results are not achieved, the parameters of the Vision Transformer neural network model and the size of the input sequence need to be adjusted. After adjusting the parameters, step 2 is repeated.

[0019] Furthermore, step 2 involves training a Vision Transformer network using a dataset of motion image sequences from underground coal mines. The specific process is as follows:

[0020] The Vision Transformer neural network model structure includes: an embedding layer and an encoder layer;

[0021] The input to the embedding layer is x p ∈R S×(P×P×C) , where x p S is the vector value encoded from the input image, i.e., the embedding block, and S is the total number of embedding blocks; a linear mapping is performed on the input vector, and the result of the mapping is... Let represent the i-th embedding block, S be the total number of embedding blocks, and E be the fully connected layer; a learnable classification vector x is added before the matrix after the embedding blocks are mapped. class As a category identifier, the matrix after adding it is: Then add location information E pos ∈R (S+1 ) *D The output is a vector that combines the category identifier matrix and location information:

[0022]

[0023] The encoder layer consists of a normalized operation layer (LN), a multi-head self-attention layer (MSA), and a multilayer perceptron (MLP). The input and output of the multi-head self-attention layer are:

[0024] Z′ l =MSA(LN(Z) l-1 ))+Z l-1

[0025] Where MSA is the multi-head self-attention layer, LN is the normalization operation layer, and Z' is the Z' layer. l Z is the output of the multi-head self-attention layer. l-1The input to the multi-head self-attention layer is used as the input to the multilayer perceptron (MLP), and the output of the multi-head self-attention layer is used as the input to the MLP. The input-output representation of the MLP is as follows:

[0026] z L =MLP(LN(Z′) l ))+Z′ l

[0027] Finally, the prediction results are output:

[0028]

[0029] The specific training process is as follows:

[0030] Step 2A: Divide the obtained coal mine underground action image sequence data into a training set and a test set according to the ratio of λ and (1-λ), where λ<1. The image sequence dataset is represented as: X={F1,F2,…,F…} i ,…,F n}, X is composed of image sequence F i composition;

[0031] Step 2B: Define the loss function Loss:

[0032]

[0033] Step 2C: Use the cross-entropy loss function to represent the difference between the probability of the recognized action type and the true action type, where N is the number of action types, p ic y is the probability that sample i belongs to class c. ic It is a sign function. If the true class of sample i is the same as class c, it is set to 1; otherwise, it is set to 0. During training, the weights and biases of each neuron are continuously updated until the number of training iterations reaches the preset value or the accuracy of the model prediction reaches the preset value. After multiple training iterations, the network model parameters with the highest accuracy can be selected and saved. At this time, the coal mine underground action features extracted by the model are the output of the last layer of the multilayer perceptron.

[0034] Furthermore, the neuron pruning process involves evaluating the importance of neurons and weights in the Vision Transformer neural network model obtained in step 2, determining the set of Vision Transformer neurons that are less correlated with illumination levels, and then lightweighting the process as follows:

[0035] Step 3A: Obtain the set of non-essential neurons under sufficient lighting conditions: Input the dataset of action image sequences collected under sufficient lighting conditions into the neural network model and train it M1 times to obtain the neural network model under sufficient lighting conditions; set variables It is a neuron n i The number of times the output value is non-zero, set Given a threshold of U, calculate the neuron n under sufficient illumination conditions. i The number of times the output is non-zero. If neuron n i The number of times the output is non-zero Therefore, it can be assumed that the neuron has little effect under sufficient lighting conditions, and thus this neuron is grouped into set N. w In, that is:

[0036] n i ∈N w

[0037] Where, n i For the neurons in the human motion recognition model in underground coal mines, the set N w A collection of neurons with low importance levels under sufficient light conditions;

[0038] Step 3B: Obtain the set N of neurons with low importance levels under low illumination conditions. b A dataset of motion image sequences from underground coal mines, collected under low-light conditions and of the same sample size, is input into a neural network model and trained M2 times to obtain a neural network model under low-light conditions; the number of neurons n under low-light conditions is calculated. i The number of times the output is non-zero. If neuron n i The number of times the output is non-zero Therefore, it can be assumed that the neuron has little effect under low illumination conditions, and thus this neuron is grouped into set N. b In, that is:

[0039] n i ∈N b

[0040] Among them, set N b This is a collection of neurons with low importance levels under low illumination conditions.

[0041] Step 3C: Identify the set of neurons with weak correlation to light intensity: Select the set N of neurons with low importance levels under sufficient light conditions. w The set N of neurons with low importance under low illumination conditions b By taking the intersection, we obtain a set of neurons that are less correlated with the light intensity. This operation can eliminate the influence of some neurons becoming "pseudo-low importance neurons" due to low light intensity, thus obtaining a set N of neurons with low importance level.

[0042] Step 3D: Delete neurons in the network that belong to set N;

[0043] Step 3E: Recovering some neurons: To prevent a significant drop in accuracy due to deleting too many neurons at once, a strategy of retaining the top K percent is adopted; the specific steps are as follows:

[0044] Step 3E-1, calculate neuron n under the combined condition. i The number of times the output is non-zero.

[0045]

[0046] Step 3E-2: Transfer neuron n i ∈N Arrange the neurons from largest to smallest and restore the first K% to the model.

[0047] Furthermore, weight pruning is used to evaluate the importance of the weights in the pruned network, and after determining the set of weights with weak correlation to illumination levels, the Vision Transformer neural network model is further lightweighted. The specific method is as follows:

[0048] Step 3F-1: Using the same dataset of human motion images from underground coal mines, train a Vision Transformer neural network with pruned neurons. During the training process, if a weight remains at or near zero, it is considered to have little importance and can be removed. That is, set a threshold H for the number of times a weight is non-zero, and calculate the weight p. i The number of times it is non-zero If the weight p i The number of times the output is non-zero Therefore, the effect of this weight can be considered very small, and this weight can be attributed to set P, as shown in the following expression:

[0049] p i ∈P

[0050] Where, p i Let P be the set of weights in the human motion recognition model for underground coal mines, where P is the set of weights with low importance levels.

[0051] Step 3F-2: Encoding and storing weight values: Since the Vision Transformer neural network contains a large proportion of similar weights, storing the same weight value multiple times would waste storage space. Therefore, the weight value p can be encoded:

[0052] m0=Γ(p)

[0053] Here, m0 represents the encoded weights, signifying a series of similar weight values. These similar weights are fixed to a single value after encoding, significantly reducing model storage space. When reading the weight values, they are then restored from the file containing the encoded weight values.

[0054] p = Γ -1 (m0)

[0055] Among them, Γ -1 (·) is the inverse function of Γ(·), used to recover similar weight values;

[0056] Step 3F-3: Fine-tuning to restore model accuracy and obtain the pruned Vision Transformer neural network model: After removing some neurons and weights, the accuracy of the Vision Transformer neural network model decreases. At this point, the pruned Vision Transformer neural network model is retrained using the training set of the coal mine underground action image sequence dataset. The model's accuracy is restored through fine-tuning. The storage space occupied by the model, the computational cost, and the accuracy of the model after fine-tuning are calculated. These three factors are used as evaluation indicators of the model compression effect. If these three indicators do not meet the expected results, the network pruning operation is repeated until all three indicators meet the expected results. The pruned Vision Transformer neural network model that meets the three indicators is output.

[0057] Furthermore, the knowledge distillation module works as follows: Based on a conventional teaching network model, a Vision Transformer neural network model that meets three criteria after pruning is set as the "teacher" model. A "student" network model is designed based on the teacher model, directly learning from the teacher model without needing the original coal mine underground action image sequence dataset. The student network model adopts a "teacher model output - encoder - student model output" structure. To make the output distribution of the teacher model easier for the student model to learn, a temperature value T is set in the SoftMax layer of both the teacher and student models to amplify the loss value corresponding to the probability values ​​of action classifications other than top 1 when calculating the loss function.

[0058] The specific steps of the distillation process are as follows:

[0059] Step 4A: Input the coal mine underground action image sequence dataset from Step 1 into the pruned teacher network and student network respectively, and obtain the logit output by the teacher network and student network;

[0060] Step 4B: Encoding and storing similar logits: For both the teacher and student networks, when training data is input, both networks output a series of logits. To further reduce storage consumption, an encoder is introduced:

[0061] θ i =Θ(logit) i ,I i )

[0062] Each logit and its corresponding index I are encoded and stored, where logit i It is the i-th logit output by the network, I i It is logit i The corresponding index; the encoded logit saves storage space and can be decoded when needed:

[0063] logirt i ,I i =Θ -1 (θ i )

[0064] Step 4C: Modify the classification function definition of the SoftMax layer in the teacher network and student network, and set the temperature value T. Each input of the SoftMax classifier is divided by the temperature value T. The classification function is:

[0065]

[0066] Step 4D: Calculate the distillation loss function: Decode the teacher logit file encoded and saved in Step 4B, input it into the teacher network at temperature T, and obtain the soft label; set T of the student network to be consistent with T of the teacher network, and decode the student logit file and input it into the student network to obtain the output after passing through the student network, which is called the soft prediction. Calculate the loss function using the soft label and the soft prediction, which is called the distillation loss function L1:

[0067] L1(p,q)=-∑ x (p(x)logq(x)+(1-p(x))log(1-q(x)))

[0068] Where p(x) is the output probability of the teacher network for each possibility, q(x) is the output probability of the student network for each possibility, and the loss function L1 represents the error of the similarity between the prediction results of the student model and the teacher model. The smaller the value, the more similar the prediction results of the student model and the teacher model are.

[0069] Step 4E: Calculate the hard loss function: Set the temperature T=1 for the student network's Softmax function, and decode the student logit file again and input it into the student network. At this time, the softmax output of the student network is called the hard prediction. Calculate the loss function by combining the hard prediction with the hard label; this is called the hard loss function L2.

[0070] L2(m,n)=-∑ x (m(x)logn(x)+(1-m(x))log(1-n(x)))

[0071] Where m(x) is the probability of the hard label, and n(x) is the output probability of the student network for each possibility when T=1. Since the hard label has only one value of 1 and all other values ​​are 0, L2 can be simplified to:

[0072] L2(m,n)=-logn(x)

[0073] The loss function L2 represents the error in the similarity between the student model's prediction and the actual result. The smaller the value, the more similar the student model's prediction and the actual result are.

[0074] Step 4F: Calculate the final loss function: Summate the distillation loss function L1 with the hard loss function L2 to obtain the final loss function L of the student network.

[0075] L=αL1+(1-α)L2

[0076] Here, α is the weight of the loss function L1. The student model is trained using the loss function L to obtain the final lightweight model.

[0077] A human motion recognition system for underground coal mines suitable for edge terminals includes a sequentially arranged motion classification module, a context pruning module, and a teaching model distillation module.

[0078] The action classification module is used to annotate text descriptions for each action of the workers in the production operation videos captured by the underground camera;

[0079] The context pruning module is a lightweight action classification module designed for the specific application scenario of underground coal mines. This lightweight action classification module can adapt to video data shot in low-light conditions in underground coal mines.

[0080] The teaching model distillation module processes the lightweight action classification module to obtain the final lightweight model.

[0081] A lightweight edge terminal installed underground, on which a computer program is stored, which, when executed by the edge terminal, implements the underground human motion recognition method in coal mines as described in any one of claims 1-8.

[0082] Beneficial Effects: By employing the above-mentioned scheme, this invention utilizes the idea of ​​training a neural network once and deploying it multiple times. The trained neural network is compressed and deployed to the edge computing terminal underground in the coal mine. This allows the collected video information to be recognized directly at the edge computing terminal without needing to be transmitted to the ground for recognition, greatly ensuring the real-time performance and stability of action recognition. Because the model undergoes network pruning and knowledge distillation operations, the existing model, compared to the original VisionTransformer neural network model, has reduced its storage space and computational cost to a certain extent, significantly alleviating the pressure on the edge computing terminal.

[0083] The Vision Transformer neural network model is a type of neural network model with global observation capabilities and supports parallel computation. Global observation capabilities are crucial in the field of action recognition. Using the Vision Transformer neural network model for action recognition can improve the accuracy of action recognition. Simultaneously, the parallel computation supported by the Vision Transformer neural network model significantly accelerates network training and computation, fully utilizing the computing resources of edge computing terminals. Therefore, this invention employs the Vision Transformer neural network model for action recognition. After distilling and pruning the action recognition model, the storage space occupied by the model is reduced, and the computing resources required for computation are also lowered, allowing for deployment on lightweight edge computing terminals.

[0084] This invention solves the problems of inability to detect movements in coal mines in real time, large computational load, and unstable transmission, thus achieving the objective of the invention.

[0085] Advantages: This invention has the advantages of reducing network transmission loss, improving the real-time performance of behavior recognition, and reducing the resource consumption of edge computing terminals.

[0086] Compared with existing technologies, this invention fully considers the unstable signal transmission and susceptibility to interference leading to data loss in underground coal mines. It employs a pruning compression method with recovery to reduce the interference of low illumination on compression; it uses a distillation technique to encode and store teaching model parameters, reducing storage space; and it provides an effective technical means for deploying complex models on underground edge terminals while ensuring recognition accuracy. This allows the acquired video image information to be recognized directly on the edge computing terminal without needing to be transmitted to the ground for further recognition, greatly ensuring the real-time performance and stability of action recognition. Furthermore, because the Vision Transformer neural network model of this invention undergoes multiple compression processes, it minimizes the number of parameters in the neural network model while maintaining accuracy, reducing the computational overhead to a certain extent. This enables good action recognition results even when deployed on edge computing terminals with average performance. Attached image description:

[0087] Figure 1 This is a flowchart illustrating the human motion recognition method for coal mines applicable to edge terminals according to the present invention.

[0088] Figure 2 This is a schematic diagram of network pruning operations used in an embodiment of the present invention.

[0089] Figure 3 This is a schematic diagram of network distillation used in an embodiment of the present invention. Detailed Implementation

[0090] The specific embodiments of the present invention will be further described with reference to the accompanying drawings:

[0091] like Figure 1 As shown, the present invention provides a method for human motion recognition in coal mines applicable to edge terminals, and the specific steps are as follows:

[0092] Step 1: Use underground cameras to collect videos of mine operations, including those of workers, under both well-lit and low-light conditions. The videos contain human movements of the workers during their operations. The collected video data is preprocessed by classifying the data according to the work scenario, removing videos that do not contain any movements, and generating videos based on the principle that each complete production operation action forms a video data set. Text description tags are added to the corresponding production operation actions to obtain a coal mine underground action image sequence dataset. The action image sequence dataset is then divided into a training set and a test set.

[0093] Step 2: Train the Vision Transformer network using the coal mine underground action image sequence dataset, so that after training, the Vision Transformer network can accurately output text descriptions of the underground personnel's production operations based on the input video. The Vision Transformer network is a Vision Transformer network with an improved loss function, including an embedding layer and an encoder layer.

[0094] Step 3: Perform context pruning on the trained Vision Transformer network to make it lightweight, so that the lightweight Vision Transformer network can adapt to video data shot in low-light conditions in coal mines. The context pruning includes neuron pruning and weight pruning.

[0095] Step 4: Distill the teaching model of the lightweight Vision Transformer network to obtain the final lightweight model after concentration;

[0096] Step 5: Set the final lightweight model in the edge terminal in the coal mine, and use the edge terminal to recognize real-time underground coal mine production operation video data; the edge terminal converts the video data to be recognized into image sequence action recognition, and outputs text descriptions of the production operation actions contained in the video.

[0097] In step 1, the specific process of collecting and constructing a coal mine underground action image sequence dataset, preprocessing the video data, and converting the video data into an image sequence is as follows:

[0098] The method for collecting video data of underground coal mine production operations is as follows: a standard dustproof and explosion-proof underground coal mine camera is used and installed in a place that can capture personnel and equipment in the production operation space to collect video data of various production operation actions, and at the same time, video data of some production operation actions under sufficient lighting conditions is also collected.

[0099] The video data preprocessing method is as follows: the acquired underground coal mine video data is classified according to the operation scenario, and the video data is filtered to discard video information that does not contain any action. Each complete action forms a video data and a corresponding action tag. After the above initial screening, spatial domain image enhancement technology is used to sharpen images with unclear edge information of personnel and equipment, and to smooth video information with too much noise, so as to obtain enhanced video data.

[0100] The process of creating the motion image sequence dataset is as follows:

[0101] Step 1A: Trim the preprocessed video according to the standard of t frames per second, so that it becomes a frame sequence, represented as V={v1,v2,…,v i ,…,v T}; where T is the total number of frames in the video, v i This represents the i-th frame of the video; each frame in the frame sequence can be represented as: Where (H,W) are the height and width of the original image, and C is the number of channels;

[0102] Step 1B: Transfer frame image v i Cut into S = (H × W) / P 2 Small images are divided into blocks, forming an image block sequence F. i ={f i1 ,f i2 ,…,f ij ,…,f iS}, where each small image is represented as f iS Let S represent the S-th small image block in the i-th frame, and P be the side length of each small image block, thus obtaining an image sequence dataset consisting of image sequences Fi: X = {F1, F2, ..., F...} i ,…,F n};

[0103] The method for dividing the training set and the test set is as follows: the obtained coal mine underground action image sequence dataset is divided into a training set and a test set according to the ratio of λ and (1-λ).

[0104] In step 2, training the massive parameter network model requires inputting the image sequence dataset into the action classification module composed of the massive parameter model. The specific process is as follows:

[0105] The massive parameter model mentioned above adopts the Vision Transformer neural network model, whose structure includes: an embedding layer and an encoder layer;

[0106] The input to the embedding layer is x. p ∈R S×(P×P×C) , where x p S is the vector value encoded from the input image, i.e., the embedding block, and S is the total number of embedding blocks; a linear mapping is performed on the input vector, and the result of the mapping is... Let represent the i-th embedding block, S be the total number of embedding blocks, and E be the fully connected layer; a learnable classification vector x is added before the matrix after the embedding blocks are mapped. class As a category identifier, the matrix after adding it is: Then add location information E pos ∈R (S+1)*DThe output is a vector that combines the category identifier matrix and location information:

[0107]

[0108] The encoder layer consists of a normalized operation layer (LN), a multi-head self-attention layer (MSA), and a multilayer perceptron (MLP), wherein the input and output of the multi-head self-attention layer are:

[0109] Z′ l =MSA(LN(Z) l-1 ))+Z l-1

[0110] Where MSA is the multi-head self-attention layer, LN is the normalization operation layer, and Z' is the Z' layer. l Z is the output of the multi-head self-attention layer. l-1 The input to the multi-head self-attention layer is used as the input to the multilayer perceptron (MLP), and the output of the multi-head self-attention layer is used as the input to the MLP. The input-output representation of the MLP is as follows:

[0111] z L =MLP(LN(Z′) l ))+Z′ l

[0112] Finally, the prediction results are output:

[0113]

[0114] The specific training process of the massive parameter network model is as follows:

[0115] Step 2A: Divide the obtained coal mine underground action image sequence data into a training set and a test set according to the ratio of λ and (1-λ), where λ<1. The image sequence dataset is represented as: X={F1,F2,…,F…} i ,…,F n}, X is composed of image sequence F i composition;

[0116] Step 2B: Define the loss function Loss:

[0117]

[0118] Step 2C: Use the cross-entropy loss function to represent the difference between the probability of the recognized action type and the true action type, where N is the number of action types, p ic y is the probability that sample i belongs to class c. icIt is a sign function. If the true class of sample i is the same as class c, it is set to 1; otherwise, it is set to 0. During training, the weights and biases of each neuron are continuously updated until the number of training iterations reaches the preset value or the accuracy of the model prediction reaches the preset value. After multiple training iterations, the network model parameters with the highest accuracy can be selected and saved. At this time, the coal mine underground action features extracted by the model are the output of the last layer of the multilayer perceptron.

[0119] like Figure 2 As shown, in step 3, to implement a massive parameter network model suitable for low-light video data, the action recognition model needs to be processed through a context pruning module to obtain a preliminary lightweight model; the context pruning module includes neuron pruning and weight pruning.

[0120] The specific process of neuron pruning is as follows:

[0121] This involves evaluating the importance of neurons and weights in the Vision Transformer neural network model obtained in step 2, determining the set of neurons that are less correlated with illumination levels, and then lightweighting the massive parameter model. The specific method is as follows:

[0122] Step 3A: Obtain the set of non-essential neurons under sufficient lighting conditions: Input the dataset of action image sequences collected under sufficient lighting conditions into the neural network model and train it M1 times to obtain the neural network model under sufficient lighting conditions; set variables It is a neuron n i The number of times the output value is non-zero, set Given a threshold of U, calculate the neuron n under sufficient illumination conditions. i The number of times the output is non-zero. If neuron n i The number of times the output is non-zero Therefore, it can be assumed that the neuron has little effect under sufficient lighting conditions, and thus this neuron is grouped into set N. w In, that is:

[0123] n i ∈N w

[0124] Where, n i For the neurons in the human motion recognition model in underground coal mines, the set N w A collection of neurons with low importance levels under sufficient light conditions;

[0125] Step 3B: Obtain the set N of neurons with low importance levels under low illumination conditions. bA dataset of motion image sequences from underground coal mines, collected under low-light conditions and of the same sample size, is input into a neural network model and trained M2 times to obtain a neural network model under low-light conditions; the number of neurons n under low-light conditions is calculated. i The number of times the output is non-zero. If neuron n i The number of times the output is non-zero Therefore, it can be assumed that the neuron has little effect under low illumination conditions, and thus this neuron is grouped into set N. b In, that is:

[0126] n i ∈N b

[0127] Among them, set N b This is a collection of neurons with low importance levels under low illumination conditions.

[0128] Step 3C: Identify the set of neurons with weak correlation to light intensity: Select the set N of neurons with low importance levels under sufficient light conditions. w The set N of neurons with low importance under low illumination conditions b By taking the intersection, we obtain a set of neurons that are less correlated with the light intensity. This operation can eliminate the influence of some neurons becoming "pseudo-low importance neurons" due to low light intensity, thus obtaining a set N of neurons with low importance level.

[0129] Step 3D: Delete neurons in the network that belong to set N;

[0130] Step 3E: Recovering some neurons: To prevent a significant drop in accuracy due to deleting too many neurons at once, a strategy of retaining the top K percent is adopted; the specific steps are as follows:

[0131] Step 3E-1: Calculate neuron n under the combined condition i The number of times the output is non-zero.

[0132]

[0133] Step 3E-2: Transfer neuron n i ∈N Arrange the neurons from largest to smallest, and restore the first K% of neurons into the model;

[0134] Step 3F: Weight pruning involves evaluating the importance of weights in the pruned network, identifying a set of weights less correlated with illumination levels, and then further lightweighting the massive parameter network model. The specific method is as follows:

[0135] Step 3F-1: Using the same dataset of human motion images from underground coal mines, train the network with pruned neurons. During the training process, if a certain weight is consistently zero or close to zero, it is considered to have very little importance and can be removed. That is, set a threshold H for the number of times a weight is non-zero, and calculate the weight p. i The number of times it is non-zero If the weight p i The number of times the output is non-zero Therefore, the effect of this weight can be considered very small, and this weight can be attributed to set P, as shown in the following expression:

[0136] p i ∈P

[0137] Where, p i Let P be the set of weights in the human motion recognition model for underground coal mines, where P is the set of weights with low importance levels.

[0138] Step 3F-2: Encoding and storing weight values: Since the Vision Transformer neural network contains a large proportion of similar weights, storing the same weight value multiple times would waste storage space. Therefore, the weight value p can be encoded:

[0139] m0=Γ(p)

[0140] Here, m0 represents the encoded weights, signifying a series of similar weight values. These similar weights are fixed to a single value after encoding, significantly reducing model storage space. When reading the weight values, they are then restored from the file containing the encoded weight values.

[0141] p = Γ -1 (m0)

[0142] Among them, Γ -1 (·) is the inverse function of Γ(·), used to recover similar weight values;

[0143] Step 3F-3: Fine-tuning to restore model accuracy and obtain the pruned Vision Transformer neural network model: After removing some neurons and weights, the accuracy of the Vision Transformer neural network model may decrease. At this time, it is necessary to retrain the network using the training set of the coal mine underground action image sequence dataset obtained in Step 1. The accuracy of the model is restored by fine-tuning. The storage space occupied by the model, the computational cost of the model, and the accuracy of the model after fine-tuning are calculated. These three factors are used as evaluation indicators of the model compression effect. If these three indicators do not achieve the expected results, the network pruning operation is repeated until all three indicators achieve the expected results. Thus, the pruned Vision Transformer neural network model can be obtained.

[0144] like Figure 3 As shown, in step 4, the preliminary lightweight model is further processed and applied through the teaching model distillation module. The specific process is as follows:

[0145] The knowledge distillation method is as follows: Based on the conventional teaching network model, a Vision Transformer neural network model that meets three criteria after pruning is set as the "teacher" model, and a "student" network model is designed based on the teacher model to directly learn from the teacher model without going through the original coal mine underground action image sequence dataset; the student network model adopts a "teacher model output - encoder - student model output" structure; in order to make the output distribution of the teacher model easier for the student model to learn, a temperature value T is set in the SoftMax layer of the teacher model and the student model respectively, so as to amplify the loss value corresponding to the probability value of other action classifications except top1 when calculating the loss function;

[0146] The specific steps of the distillation process are as follows:

[0147] Step 4A: Input the coal mine underground action image sequence dataset from Step 1 into the pruned teacher network and student network respectively, and obtain the logit output by the teacher network and student network;

[0148] Step 4B: Encoding and storing similar logits: For both the teacher and student networks, when training data is input, both networks output a series of logits. To further reduce storage consumption, an encoder is introduced:

[0149] θ i =Θ(logit) i ,I i )

[0150] Each logit and its corresponding index I are encoded and stored, where logit i It is the i-th logit output by the network, I i It is logit i The corresponding index; the encoded logit saves storage space and can be decoded when needed:

[0151] logirt i ,I i =Θ -1 (θ i )

[0152] Step 4C: Modify the classification function definition of the SoftMax layer in the teacher network and student network, and set the temperature value T. Each input of the SoftMax classifier is divided by the temperature value T. The classification function is:

[0153]

[0154] Step 4D: Calculate the distillation loss function: Decode the teacher logit file encoded and saved in Step 4B, input it into the teacher network at temperature T, and obtain the soft label; set T of the student network to be consistent with T of the teacher network, and decode the student logit file and input it into the student network to obtain the output after passing through the student network, which is called the soft prediction. Calculate the loss function using the soft label and the soft prediction, which is called the distillation loss function L1:

[0155] L1(p,q)=-∑ x (p(x)logq(x)+(1-p(x))log(1-q(x)))

[0156] Where p(x) is the output probability of the teacher network for each possibility, q(x) is the output probability of the student network for each possibility, and the loss function L1 represents the error of the similarity between the prediction results of the student model and the teacher model. The smaller the value, the more similar the prediction results of the student model and the teacher model are.

[0157] Step 4E: Calculate the hard loss function: Set the temperature T=1 for the student network's Softmax function, and decode the student logit file again and input it into the student network. At this time, the softmax output of the student network is called the hard prediction. Calculate the loss function by combining the hard prediction with the hard label; this is called the hard loss function L2.

[0158] L2(m,n)=-∑ x (m(x)logn(x)+(1-m(x))log(1-n(x)))

[0159] Where m(x) is the probability of the hard label, and n(x) is the output probability of the student network for each possibility when T=1. Since the hard label has only one value of 1 and all other values ​​are 0, L2 can be simplified to:

[0160] L2(m,n)=-logn(x)

[0161] The loss function L2 represents the error in the similarity between the student model's prediction and the actual result. The smaller the value, the more similar the student model's prediction and the actual result are.

[0162] Step 4F: Calculate the final loss function: Summate the distillation loss function L1 with the hard loss function L2 to obtain the final loss function L of the student network.

[0163] L=αL1+(1-α)L2

[0164] Here, α is the weight of the loss function L1. The student model is trained using the loss function L to obtain the final lightweight model.

[0165] The embodiments of the present invention will now be further described with reference to the accompanying drawings:

[0166] Example 1: The present invention provides a method for human motion recognition in coal mines applicable to edge terminals, the specific steps of which are as follows:

[0167] Step 1: Use underground cameras to collect videos of mine operations, including those of workers, under both well-lit and low-light conditions. The videos contain human movements of the workers during their operations. The collected video data is preprocessed by classifying the data according to the work scenario, removing videos that do not contain any movements, and generating videos based on the principle that each complete production operation action forms a video data set. Text description tags are added to the corresponding production operation actions to obtain a coal mine underground action image sequence dataset. The action image sequence dataset is then divided into a training set and a test set.

[0168] Step 2: Train the Vision Transformer network using the coal mine underground action image sequence dataset, so that after training, the Vision Transformer network can accurately output text descriptions of the underground personnel's production operations based on the input video. The Vision Transformer network is a Vision Transformer network with an improved loss function, including an embedding layer and an encoder layer.

[0169] Step 3: Perform context pruning on the trained Vision Transformer network to make it lightweight, so that the lightweight Vision Transformer network can adapt to video data shot in low-light conditions in coal mines. The context pruning includes neuron pruning and weight pruning.

[0170] Step 4: Distill the teaching model of the lightweight Vision Transformer network to obtain the final lightweight model after concentration;

[0171] Step 5: Set the final lightweight model in the edge terminal in the coal mine, and use the edge terminal to recognize real-time underground coal mine production operation video data; the edge terminal converts the video data to be recognized into image sequence action recognition, and outputs text descriptions of the production operation actions contained in the video.

[0172] The specific steps to achieve step 1 are as follows:

[0173] (1) Standard dustproof and explosion-proof underground coal mine cameras are used and installed in places that can capture personnel and equipment in the production work space, and video data of various production operations are collected. At the same time, video data of some production operations under sufficient lighting conditions are also collected;

[0174] (2) The acquired underground coal mine video data is classified according to the operation scenario. At the same time, the video data is filtered to discard video information that does not contain any action. Each complete action forms a video data and a corresponding action tag.

[0175] (3) The video image information with unclear personnel and equipment edge information in the video image data after the above initial screening is sharpened by spatial domain image enhancement technology.

[0176] (4) The video image information with excessive noise in the video image data after the initial screening is smoothed by spatial domain image enhancement technology.

[0177] (5) Use FFmpeg to trim the video to 2 frames per second to make it a frame sequence;

[0178] (6) Using convolution operation, each frame image is cut into 3×3 blocks, a total of 9 small images, to form an image sequence;

[0179] (7) Divide the obtained coal mine underground motion image sequence dataset into a training set and a test set in an 8:2 ratio; the specific operation for step 2 is as follows:

[0180] (1) Configure the required development environment on a Linux server with a dedicated graphics card;

[0181] (2) Create the Vision Transformer neural network model;

[0182] (3) Input the training set of the image sequence dataset constructed in step 1 into the Vision Transformer neural network model;

[0183] (4) Set the training pause condition. The training pause condition can be set to 50 training cycles.

[0184] (5) Train the network multiple times and save the parameters of the network model with the highest accuracy.

[0185] (6) Test the accuracy of the model with the test set: Input the test set from step 1 into the trained VisionTransformer neural network model, calculate the action recognition accuracy of the VisionTransformer neural network model on the test set, and use it as the evaluation index of the VisionTransformer neural network model.

[0186] (7) Based on the selected evaluation index and the model's output, determine whether the expected results have been achieved. If the expected results have been achieved, proceed to step 3. If the expected results have not been achieved, it is necessary to adjust the parameters of the Vision Transformer neural network model and the size of the input sequence, etc. After adjusting the parameters, repeat step 2. The specific operation to achieve step 3 is as follows:

[0187] (1) As attached Figure 2 As shown, a dataset of motion image sequences collected under sufficient lighting conditions in a coal mine was input into a neural network model for training 50 times to obtain a neural network model under sufficient lighting conditions; variables were set. It is a neuron n i The number of times the output value is non-zero, set Given a threshold of U, calculate the neuron n under sufficient illumination conditions. i The number of times the output is non-zero. If neuron n i The number of times the output is non-zero Therefore, it can be assumed that the neuron has little effect under sufficient lighting conditions, and thus this neuron is grouped into set N. w In, that is:

[0188] n i ∈N w

[0189] Where, n i For the neurons in the human motion recognition model in underground coal mines, the set N w A collection of neurons with low importance levels under sufficient light conditions;

[0190] (2) Input the dataset of coal mine underground motion image sequences collected under low-light conditions with the same sample size into the neural network model for training 50 times to obtain the neural network model under low-light conditions; calculate the number of neurons n under low-light conditions. i The number of times the output is non-zero. If neuron n i The number of times the output is non-zero Therefore, it can be assumed that the neuron has little effect under low illumination conditions, and thus this neuron is grouped into set N. b In, that is:

[0191] n i ∈N b

[0192] Among them, set N b This is a collection of neurons with low importance levels under low illumination conditions.

[0193] (3) The set N of neurons with low importance under sufficient light conditions w The set N of neurons with low importance under low illumination conditions b By taking the intersection, we obtain a set of neurons that are less correlated with the light intensity, and thus obtain a set N of neurons with low importance level.

[0194] (4) Delete neurons in the network that belong to set N;

[0195] (5) Calculate the number of neurons n under the combined condition. i The number of times the output is non-zero.

[0196]

[0197] Neuron n i ∈N Arrange the neurons from largest to smallest and restore the first K% to the model.

[0198] (6) Using the same dataset of human motion images in coal mines, a network with pruned neurons was trained. The threshold for the number of times the weights were non-zero was set to H, and the weights p were calculated. i The number of times it is non-zero If the weight p i The number of times the output is non-zero The weight is summarized into set P, as shown in the following expression:

[0199] p i ∈P

[0200] Where, p iLet P be the weights in the human motion recognition model in underground coal mines, where P is the set of weights with low importance levels.

[0201] (7) Encode the weight value p:

[0202] m0=Γ(p)

[0203] Where m0 is the encoded weight, representing a series of similar weight values. When reading the weight values, they are then restored from the file that stores the encoded weight values.

[0204] p = Γ -1 (m0)

[0205] Among them, Γ -1 (·) is the inverse function of Γ(·), used to recover similar weight values.

[0206] (8) By fine-tuning the accuracy of the model, calculate the storage space occupied by the model, the computational cost of the model, and the accuracy of the model after fine-tuning. Use these three as evaluation indicators of the model compression effect. If these three indicators do not achieve the expected results, continue to repeat the network pruning operation until all three indicators achieve the expected results.

[0207] The specific steps to achieve step 4 are as follows:

[0208] (1) As attached Figure 3 As shown, the coal mine underground action image sequence dataset from step 1 is input into the pruned teacher network and student network respectively to obtain the logit output by the teacher network and student network.

[0209] (2) Introducing an encoder:

[0210] θ i =Θ(logit) i ,I i )

[0211] Each logit and its corresponding index I are encoded and stored, where logit i It is the i-th logit output by the network, I i It is logit i The corresponding index. Encoded logit saves storage space; it can be decoded when needed.

[0212] logit i ,I i =Θ -1 (θ i )

[0213] (3) Modify the classification function of the SoftMax layer in the teacher network and student network as follows:

[0214]

[0215] (4) Calculate the distillation loss function: Decode the encoded and saved teacher logit file, input it into the teacher network at temperature T, and obtain the soft label; set the T of the student network to be consistent with the T of the teacher network, decode the student logit file and input it into the student network to obtain the output after passing through the student network, which is called the soft prediction. Calculate the loss function using the soft label and the soft prediction, which is called the distillation loss function L1:

[0216] L1(p,q)=-∑ x (p(x)logq(x)+(1-p(x))log(1-q(x)))

[0217] Where p(x) is the output probability of the teacher network for each possibility, and q(x) is the output probability of the student network for each possibility.

[0218] (5) Set the temperature T=1 for the softmax function of the student network, and decode the student logit file and input it into the student network again. At this time, the softmax output of the student network is called the hard prediction. Calculate the loss function by combining the hard prediction with the hard label, which is called the hard loss function L2:

[0219] L2(m,n)=-∑ x (m(x)logn(x)+(1-m(x))log(1-n(x)))

[0220] Where m(x) is the probability of the hard label, and n(x) is the output probability of the student network for each possibility when T=1.

[0221] (6) Calculate the final loss function: Summate the distillation loss function L1 and the hard loss function L2 to obtain the final loss function L of the student network:

[0222] L=αL1+(1-α)L2

[0223] Here, α is the weight of the loss function L1. The student model is trained using the loss function L to obtain the final lightweight model.

[0224] The specific steps to achieve step 5 are as follows:

[0225] (1) Convert the video data to be identified into an image sequence, input it into the terminal where the final lightweight model is deployed in step 4 for action recognition, and output the action information contained in the video.

[0226] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for human motion recognition in coal mines suitable for edge terminals, characterized in that, To address the low-lighting environment in coal mines, a lightweight motion recognition model is constructed that can identify human movements in low-light conditions, enabling the recognition of human movements using edge terminals. The specific steps are as follows: Step 1: Use underground cameras to collect videos of mine operations, including those of workers, under both well-lit and low-light conditions. The videos contain human movements of the workers during their operations. The collected video data is preprocessed by classifying the data according to the work scenario, removing videos that do not contain any movements, and generating videos based on the principle that each complete production operation action forms a video data set. Text description tags are added to the corresponding production operation actions to obtain a coal mine underground action image sequence dataset. The action image sequence dataset is then divided into a training set and a test set. Step 2: Train the Vision Transformer network using the coal mine underground action image sequence dataset, so that after the Vision Transformer network is trained, it can accurately output text descriptions of the underground personnel's production operation actions based on the input video. The Vision Transformer network is a Vision Transformer network with an improved loss function, including an embedding layer and an encoder layer. Step 3: Perform context pruning on the trained Vision Transformer network to make it lightweight, so that the lightweight Vision Transformer network can adapt to video data shot in low-light conditions in coal mines. The context pruning includes neuron pruning and weight pruning. Step 4: Distill the teaching model of the lightweight Vision Transformer network to obtain the final lightweight model after concentration; Step 5: Set the final lightweight model in the edge terminal in the coal mine, and use the edge terminal to recognize real-time underground coal mine production operation video data; the edge terminal converts the video data to be recognized into image sequence action recognition, and outputs text descriptions of the production operation actions contained in the video.

2. The method for human motion recognition in coal mines applicable to edge terminals according to claim 1, characterized in that: For video images with unclear edge information of personnel and equipment obtained in step 1, spatial domain image enhancement technology is used to sharpen them. For video images with excessive noise, spatial domain image enhancement technology is used to smooth them. Then, convolution operation is used on the images cropped into frame sequences to cut each frame into 3×3 blocks, a total of 9 small images, forming an image sequence. The obtained coal mine underground action image sequence dataset is divided into training set and test set in an 8:2 ratio.

3. The method for human motion recognition in coal mines applicable to edge terminals according to claim 2, characterized in that: In step 1, collecting and constructing a coal mine underground action image sequence dataset requires preprocessing the video data and converting it into image sequence data. The process for creating the action image sequence dataset is as follows: Step 1A: Divide the preprocessed video into segments of one second each. The frame standard is used to trim the frame into a frame sequence, represented as follows: ;in The total number of frames in the video. The video represents the Frame; Each frame in a frame sequence can be represented as: , in The height and width of the original image. It is the number of channels; Step 1B: Transfer the frame image Cut into Small images are combined to form an image block sequence. Each small image is represented as , Indicates the first The first frame Small image patches The side length of each small image patch is used to obtain the image sequence. The image sequence dataset consists of: ; The method for dividing the training and test sets is as follows: the obtained coal mine underground action image sequence dataset is divided into... , The ratio is divided into training set and test set.

4. The method for human motion recognition in coal mines applicable to edge terminals according to claim 1, characterized in that: Input the test set into the pre-trained Vision Transformer neural network model, calculate the action recognition accuracy of the Vision Transformer neural network model on the test set, and use it as the evaluation metric of the Vision Transformer neural network model. Based on the selected evaluation metric and the model's output, determine whether the expected results have been achieved. If the expected results have been achieved, proceed to step 3. If the expected results have not been achieved, the parameters of the Vision Transformer neural network model and the size of the input sequence need to be adjusted. After adjusting the parameters, repeat step 2.

5. A method for recognizing human actions in coal mines suitable for edge terminals according to claim 4, characterized in that, Step 2 involves training the Vision Transformer network using a dataset of action image sequences from underground coal mines. The specific process is as follows: The Vision Transformer neural network model structure includes: an embedding layer and an encoder layer; The input of the embedding layer is ,in It is the vector value encoded from the input image, that is, the embedding block. This represents the total number of embedded blocks; a linear mapping is performed on the input vector, and the result of the mapping is... , Indicates the first An embedded block, It is the total number of embedded blocks. For fully connected layers; a learnable classification vector is added before the matrix after the embedding block mapping. As a category identifier, the matrix after adding it is: Then add location information. The output is a vector that combines the category identifier matrix and location information: ， The encoder layer consists of a standardized operation layer. Multi-head self-attention layer Multilayer perceptron The composition, in which the input and output of the multi-head self-attention layer are: ， in For the multi-head self-attention layer, It is a standardized operation layer. For the output of the multi-head self-attention layer, The input to the multi-head self-attention layer is used as the input to the multilayer perceptron, and the output of the multi-head self-attention layer is used as the input to the multilayer perceptron. The input and output are represented as follows: ， Finally, the prediction results are output: ， The specific training process is as follows: Step 2A: The obtained coal mine underground motion image sequence data is processed according to... , The dataset is divided into training and test sets, with the following proportions: The image sequence dataset is represented as follows: , From image sequence composition; Step 2B: Define the loss function : ， Step 2C: Use the cross-entropy loss function to represent the difference between the probability of the action category and the true action category, where... It is the number of action types. It is a sample Belongs to the category The probability, It is a sign function, if the sample The true types and categories If the values ​​match, set them to 1; otherwise, set them to 0. During training, the weights and biases of each neuron are continuously updated until the number of training iterations reaches a preset value or the model's prediction accuracy reaches a preset value. After multiple training iterations, the network model parameters with the highest accuracy can be selected and saved. At this point, the coal mine underground action features extracted by the model are the output of the last layer of the multilayer perceptron.

6. A method for human motion recognition in coal mines applicable to edge terminals according to claim 5, characterized in that, The neuron pruning process involves evaluating the importance of neurons and weights in the Vision Transformer neural network model obtained in step 2, determining the set of Vision Transformer neurons that are less correlated with illumination levels, and then lightweighting the model as follows: Step 3A: Obtain the set of non-essential neurons under sufficient lighting conditions: Input the dataset of action image sequences collected under sufficient lighting conditions in the coal mine underground into the neural network model for training. Next, a neural network model under sufficient lighting conditions is obtained; variables are defined. , Neuron The number of times the output value is non-zero, set The threshold is Calculate neurons under adequate lighting conditions The number of times the output is non-zero. If neurons The number of times the output is non-zero If this is the case, then the neuron's effect is minimal under sufficient lighting conditions, and thus the neuron is grouped into a set. In, that is: ， in, For neurons in the human motion recognition model in underground coal mines, a set A collection of neurons with low importance levels under sufficient light conditions; Step 3B: Obtain a set of neurons with low importance levels under low illumination conditions. A dataset of motion image sequences from underground coal mines, collected under low-light conditions and with the same sample size, is input into a neural network model for training. This process yields a neural network model under low-light conditions; neurons under low-light conditions are then calculated. The number of times the output is non-zero. If neurons The number of times the output is non-zero Therefore, it can be assumed that the neuron has little effect under low illumination conditions, and thus the neuron is grouped into a set. In, that is: ， Among them, set This is a collection of neurons with low importance levels under low illumination conditions. Step 3C: Identify the set of neurons with weak correlation to light intensity: This involves grouping neurons with low importance levels under sufficient light conditions. A collection of neurons with low importance levels under low illumination conditions. Taking the intersection yields a set of neurons that are less correlated with light intensity; this operation eliminates the influence of some neurons becoming "pseudo-low importance neurons" due to low light, thus obtaining a set of neurons with low importance levels. ; Step 3D: Identify the network elements belonging to the set. Neuron deletion; Step 3E: Restore some neurons: To prevent a significant drop in accuracy caused by deleting too many neurons at once, the top neurons are preserved. The strategy; the specific steps are as follows: Step 3E-1: Calculate neurons under the combined condition. The number of times the output is non-zero. : , Step 3E-2: Place the neuron according to Arrange from largest to smallest, restore the previous order. The neurons are added to the model.

7. A method for human motion recognition in coal mines applicable to edge terminals according to claim 6, characterized in that, The Vision Transformer neural network model is further lightweighted by evaluating the importance of weights in the pruned network using weight pruning, identifying a set of weights with weak correlation to illumination levels, and then implementing the following method: Step 3F-1: Under the same coal mine human body underground action image sequence dataset, the Vision Transformer neural network with pruned neurons is used for training. During the process, if a certain weight is always zero or close to zero, it is considered that the weight is of little importance and can be removed. That is: set a threshold for the number of times the weight is non-zero. Calculate weights The number of times it is non-zero If weight The number of times the output is non-zero If so, then the effect of this weight is considered very small, and this weight is then grouped into the set. In Chinese, the expression is as follows: ， in, For the weights in the human motion recognition model in underground coal mines, set This is a set of weights with low importance levels. Step 3F-2: Encode and store the weight values: Since the Vision Transformer neural network contains a large proportion of similar weights, storing the same weight values ​​multiple times would waste storage space. Therefore, the weight values ​​can be encoded and stored. Encode: ， in, The encoded weights represent a series of similar weight values. These similar weights are fixed to a single value after encoding, significantly reducing model storage space. When reading the weight values, they are then restored from the file containing the encoded weight values. ， in, for The inverse function of is used to recover similar weight values; Step 3F-3: Fine-tuning to restore model accuracy and obtain the pruned Vision Transformer neural network model: After removing some neurons and weights, the accuracy of the Vision Transformer neural network model decreases. At this point, the pruned Vision Transformer neural network model is retrained using the training set of the coal mine underground action image sequence dataset. The model's accuracy is restored through fine-tuning. The storage space occupied by the model, the computational cost, and the accuracy of the model after fine-tuning are calculated. These three factors are used as evaluation indicators of the model compression effect. If these three indicators do not meet the expected results, the network pruning operation is repeated until all three indicators meet the expected results. The pruned Vision Transformer neural network model that meets the three indicators is output.

8. A method for human motion recognition in coal mines applicable to edge terminals according to claim 7, characterized in that, The aforementioned model distillation process is as follows: Based on a conventional teaching network model, a Vision Transformer neural network model that meets three criteria after pruning is set as the teacher model. A student model is then designed based on this teacher model, directly learning from it without using the original coal mine underground action image sequence dataset. The student model adopts a "teacher model output – encoder – student model output" structure. To make the output distribution of the teacher model easier for the student model to learn, temperature values ​​are set in the SoftMax layers of both the teacher and student models. This is to amplify the loss value corresponding to the probability value of action categories other than top1 when calculating the loss function; The specific steps of the distillation process are as follows: Step 4A: Input the coal mine underground action image sequence dataset from Step 1 into the pruned teacher network and student network respectively, and obtain the outputs of the teacher network and student network. ; Step 4B: Encode and store similar items For both the teacher and student networks, when training data is input, both networks will output a series of... To further reduce storage space consumption, an encoder is introduced: ， For each and the corresponding index Encode and store, where It is the first output of the network indivual , yes The corresponding index; stored in encoding This can save storage space; the codec can decode the data when needed. ， Step 4C: Modify the classification function definitions for the SoftMax layers of the teacher and student networks, and set the temperature values. Each input to the SoftMax classifier is divided by the temperature value. The classification function is: ， Step 4D: Calculate the distillation loss function: The teacher's code saved in Step 4B... File decoding, input temperature is The teacher network at that time obtained soft tags; the student network Configured to work with the teacher network Consistency, and students The file is decoded and input into the student network. The output obtained after passing through the student network is called the soft prediction. The loss function is calculated using the soft label and the soft prediction, and is called the distillation loss function. : ， in The output probability of the teacher network for each possibility. The loss function is the output probability of the student network for each possibility. This represents the error in the similarity between the prediction results of the student model and the teacher model. The smaller the value, the more similar the prediction results of the student model and the teacher model are. Step 4E: Calculate the hard loss function: Set the temperature of the student network's Softmax function. Once again, the students The decoded file is input into the student network. The softmax output of the student network at this point is called the hard prediction. The loss function calculated by combining the hard prediction with the hard label is called the hard loss function. : ， in The probability of being a hard label. For when The student network outputs the probability for each possibility. Since the hard label has only one value of 1 and all other values ​​are 0, therefore... It can be simplified to: ， loss function This represents the error in the similarity between the student model's predictions and the actual results. The smaller the value, the more similar the student model's predictions are to the actual results. Step 4F: Calculate the final loss function: Distill the loss function With hard loss function Summation, used as the loss function of the final student network. : ， in For loss function The weights, using the loss function Train the student model to obtain the final lightweight model.

9. A recognition system using the human motion recognition method for edge terminals in coal mines according to any one of claims 1-8, characterized in that: This includes a sequentially arranged action classification module, a context pruning module, and a teaching model distillation module; The action classification module is used to annotate text descriptions for each action of the workers in the production operation videos captured by the underground camera; The context pruning module is a lightweight action classification module designed for the specific application scenario of underground coal mines. This lightweight action classification module can adapt to video data shot in low-light conditions in underground coal mines. The teaching model distillation module processes the lightweight action classification module to obtain the final lightweight model.

10. A lightweight edge terminal installed underground, on which a computer program is stored, characterized in that: When executed by an edge terminal, the program implements the coal mine underground human motion recognition method as described in any one of claims 1-8.

Images