Lightweight downhole personnel action recognition method and device optimized for dynamic memory

Abstract

A lightweight method and device for underground personnel action recognition with dynamic memory optimization is presented. This method utilizes video data of underground coal mine personnel actions to design a lightweight action recognition model suitable for embedded terminals. First, a data acquisition and preprocessing module constructs a dataset of underground coal mine personnel actions. Second, the dataset is pre-trained using a pre-training module to perform hybrid pre-training of the action recognition model for video action feature recognition tasks. Third, the action recognition model is searched using a lightweight neural network architecture search module. Finally, the lightweight network is compressed using a parameter compression module to obtain a minimal model, which is then applied to an embedded terminal. Advantages: The hybrid pre-training technique improves the model's robustness; the use of parameter compression and dynamic memory loading optimizes the model structure, reduces model size, and shortens search time; and it enables the deployment of complex models on underground embedded terminals while maintaining recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer image recognition, specifically to a lightweight method and apparatus for recognizing downhole personnel movements with dynamic memory optimization. Background Technology

[0002] In recent years, with the rapid development of computer vision and camera technologies, collecting massive amounts of video data, analyzing the data, and extracting useful information has become an important research direction in computer technology. Video data records the actions of people, machinery, and other targets, containing rich spatiotemporal information. It can highly reconstruct events that occur in a video-monitored area over a period of time. Therefore, video data can be widely used in fields such as intelligent transportation, enterprise security, and safe production.

[0003] Coal mine underground production takes place in a complex environment. Tunneling machines are massive, there are numerous underground workers, and the tunnels are narrow; the consequences of a safety accident would be unimaginable. Standardizing the operation of production personnel can significantly reduce the probability of accidents. Surveillance video is a crucial tool for identifying and preventing dangerous operations. Personnel motion recognition technology is an important application of surveillance video, saving manpower and reducing human error. Therefore, applying personnel motion recognition technology to embedded terminals in coal mines has gradually become a popular research direction.

[0004] Currently, most existing model compression methods are based on pruning, distillation, etc. However, due to the special nature of the underground coal mine environment, this rule-based method requires a large amount of expert domain knowledge. If the rules are not well designed, it is easy to cause accuracy collapse. If a micro neural network suitable for embedded terminals is designed directly, the network will be too small and have weak expressive power, and overfitting is also likely to occur during training.

[0005] Therefore, existing technologies of the same type still have the following key limitations:

[0006] 1. Rule-based methods rely on expert domain knowledge, resulting in low universality and limited application scope;

[0007] 2. Due to the unique environment of underground coal mines, traditional model compression techniques have a significant impact on accuracy;

[0008] 3. Ordinary neural network architecture search results in a large amount of computational overhead, and the proxy task is not well adapted to the target task. Summary of the Invention

[0009] Technical Problem: The purpose of this invention is to provide a lightweight downhole personnel action recognition method and device for dynamic memory optimization, in order to solve the problems of large number of parameters, reliance on a large amount of expert experience for network architecture search, and high requirements for computing resources in existing action recognition models.

[0010] Technical Solution: This invention provides a lightweight underground personnel action recognition method for dynamic memory optimization, characterized by: introducing underground coal mine personnel action video data and AVA video data; obtaining a lightweight underground coal mine action recognition model using the underground coal mine personnel action video data and AVA data; performing underground personnel action recognition through a pre-training module, a lightweight neural network architecture search module, a parameter compression module, and a real-time detection module; preprocessing the AVA video data and underground coal mine personnel action video data, then passing them through the pre-training module to obtain an action recognition pre-trained model; passing the action recognition pre-trained model through the lightweight neural network architecture search module to obtain a lightweight action recognition fine-tuning model; and passing the lightweight action recognition fine-tuning model through the parameter compression module to obtain the final lightweight underground coal mine action recognition model.

[0011] Specifically, the steps include the following:

[0012] Step 1: Collect and construct the coal mine underground action image sequence dataset and AVA dataset: Preprocess the collected coal mine underground personnel action video data and AVA video data, convert the video data into image sequences, and divide them into training set and test set to obtain the coal mine underground personnel action image sequence dataset and AVA dataset. Merge the datasets to form a pre-trained augmented dataset.

[0013] Step 2: The pre-trained augmented dataset obtained in Step 1 is passed through the pre-training module to achieve joint learning of the ultra-large-scale network model under the classification task of video action recognition, so as to obtain the action recognition pre-trained model.

[0014] Step 3: The action recognition pre-trained model obtained in Step 2 is passed through the lightweight neural network architecture search module to achieve fast optimal network architecture search and obtain a lightweight action recognition fine-tuning model.

[0015] Step 4: The lightweight motion recognition fine-tuning model obtained in Step 3 is further processed through the parameter compression module to obtain the final lightweight coal mine underground motion recognition model, which is then applied to the embedded terminal.

[0016] Step 5: Recognize the real-time underground coal mine personnel action video data: Convert the video data to be recognized into an image sequence, input it into the terminal of the final lightweight underground coal mine action recognition model obtained in Step 4 for action recognition, and output the action information of all personnel contained in the video in text.

[0017] In step 1, the method for collecting video data of underground coal mine production operations is as follows: a standard dustproof and explosion-proof underground coal mine camera is used and installed in a place that can capture personnel and equipment in the production operation space to collect video data of various production operation actions.

[0018] In step 1, the AVA video data preprocessing method is as follows: the AVA video data is classified according to the work scenario, and the video data is filtered to discard video information that does not contain any action. Each complete action forms a video data and a corresponding action label is defined. After the video data is initially screened, spatial domain image enhancement technology is used to sharpen images with unclear edge information of personnel and equipment, and to smooth video information with too much noise, so as to obtain enhanced video data.

[0019] In step 1, the method for dividing the training set and the test set is as follows: the obtained coal mine underground action image sequence dataset is divided into a fine-tuning training set and a fine-tuning test set according to the ratio of λ and (1-λ); similarly, the pre-training augmentation dataset is divided into a pre-training augmentation training set and a pre-training augmentation test set according to the ratio of λ and (1-λ).

[0020] In step 1, the process of creating the coal mine underground personnel motion image sequence dataset is as follows:

[0021] Step 1A: Trim the preprocessed video of underground coal mine workers' actions according to the standard of t frames per second, so that the video becomes a frame sequence, represented as V={v1,v2,…,v i ,…,v T}; where T is the total number of frames in the video, v i This represents the i-th frame of the video; each frame in the frame sequence is represented as: Where (H,W) are the height and width of the original image, and C is the number of channels;

[0022] Step 1B: Transfer frame image v i Cut into S = (H × W) / P 2 Small images are formed into an image block sequence F. i ={f i1 ,f i2 ,…,f ij ,…,f iS}, where each small image is represented as f iS Let S represent the S-th small image block in the i-th frame, and P be the side length of each small image block. This is obtained from the image sequence F. i The image sequence dataset consists of: X = {F1, F2, ..., F...} i ,…,F n}

[0023] In step 2, obtaining the action recognition pre-trained model requires inputting the pre-trained augmented dataset into the ultra-large-scale network model. The specific process of joint learning of the ultra-large-scale network model is as follows:

[0024] Step 2A: Divide the pre-trained augmentation dataset obtained in Step 1 into a pre-trained augmentation training set and a pre-trained augmentation test set according to the ratio of λ and (1-λ), where λ<1. The image sequence dataset is represented as: X={F1,F2,…,F…} i ,…,F n}, X is composed of image sequence F i composition;

[0025] Step 2B: Establish the pre-training comprehensive loss function. Loss:

[0026]

[0027] Where: N is the number of action types, p ic y is the probability that sample i belongs to class c, and total is the total number of samples in the video dataset; ic The sign function is 1 if the true class of sample i matches class c, and 0 otherwise. The pre-training comprehensive loss function represents the difference between the probability of the action recognition class and the true action class. Step 2C: Input the pre-training augmented training set into the ultra-large-scale network model for pre-training. During pre-training, the model parameters will be continuously updated, and the model accuracy will gradually converge. After multiple training iterations, the converged network model is selected and saved to obtain the action recognition pre-trained model.

[0028] In step 3, obtaining the lightweight action recognition fine-tuning model requires setting a search space and using a lightweight neural network architecture search module to search the pre-trained action recognition model; the lightweight neural network architecture module includes a component compression module, a dynamic memory optimization module, and a fine-tuning module.

[0029] The specific process for setting the search space is as follows:

[0030] The process of transforming the structural parameters in the action recognition pre-trained model obtained in step 2 into candidate elements is as follows:

[0031] Step 3A-1: Let l be the path of the action recognition pre-trained model obtained in Step 2, then l i This represents the i-th path in the action recognition pre-trained model, i.e.:

[0032] M(l,…,l i ,…,l n )

[0033] Where M is the graph composed of paths in the action recognition pre-trained model;

[0034] Step 3A-2: Let Γ be the operator of the action recognition pre-trained model obtained in Step 2, then γ i Let represent the i-th operator in the action recognition pre-trained model, i.e.:

[0035] Γ={γ i}

[0036] Where O is the set of operators in the action recognition pre-trained model;

[0037] Step 3A-3: Denote the operators on the n paths of the action recognition pre-trained model as m. Γ The structure of the entire action recognition pre-trained model can then be represented as:

[0038]

[0039] The component compression module is the process of binarizing and pruning the hidden layers of the feedforward neural network (FFN) in the action recognition pre-training model obtained in step 2. The specific process is as follows:

[0040] Step 3B-1: Set the inflation rate of the multilayer perceptron (MLP) in the action recognition pre-trained model to α, where the inflation rate α controls the number of hidden layers in the FFN.

[0041] Step 3B-2: For the input vector X, the output of the feedforward neural network FFN can be expressed as:

[0042]

[0043] Among them, O FFN The output of the feedforward neural network is represented by t, which represents the number of input channels to the feedforward neural network. α is the dilation rate, and c is the output of the feedforward neural network. in O is the dimension of the channel. t (X) can be represented as:

[0044]

[0045] Where GeLU is the activation function. and Let be the weight matrix of the fully connected layer in the feedforward neural network, and Where αc in W represents the dimension of the hidden layer. fc1 Map the dimension of the input feature X to αc in W fc2 Inversely map the dimension of the input feature X to c. in ;

[0046] Step 3B-3: Binarize the path weights of each hidden layer of the FFN to obtain the output of the BFFN:

[0047]

[0048] Among them, O BFFN (X) represents the output of BFFN, where BFFN is a binarized feedforward neural network. Code The parameters are binarized and automatically optimized during the neural network architecture search process. The parameter size is compressed after passing through the binarized feedforward neural network.

[0049] Step 3B-4: Introduce a variable parameter λ to evaluate the importance of the hidden dimensions in BFFN:

[0050] λ t =Ber(Sigmoid(P) t ))

[0051] Among them, sigmoid(P t ) represents the probability of using a t-dimensional hidden layer, λ t To indicate the importance of the hidden dimension of BFFN, based on λ t Pruning hidden layer dimensions with low importance achieves component compression;

[0052] The dynamic memory optimization module described above is a process that dynamically reduces memory consumption during the neural network architecture search. The specific method is as follows:

[0053] Step 3C-1: Dynamically reduce the number of queries active in memory at the same time: When memory is insufficient, only focus on information from the previous j steps.

[0054] Query k :=[func(Query k-j ),…,func(Query k-1 ),Query k ]

[0055] Among them, Query k Query represents the query operator at time k. k-j This represents the query operator for the first j steps. The function func indicates that the gradient calculation should be stopped. Stopping the gradient calculation can effectively reduce the dependence on previous information during backpropagation and effectively reduce memory consumption.

[0056] Step 3C-2: Similarly, dynamically reduce the number of active keys in memory at the same time: when memory is insufficient, only focus on information from the previous j steps.

[0057] Key k:=[func(Key) k-j ),…,func(Key k-1 ), Key k ]

[0058] Among them, Key k Represents the key operator at time k, Query k-j This represents the key operator for the first j steps, and the function func indicates when to stop calculating the gradient;

[0059] Step 3D: Evaluate the searched network structure: For the model searched for neural network structure, the difference between the expected value and the true value can be calculated using the loss function to evaluate the quality of the searched model. Therefore, the model loss function L is defined as follows:

[0060] L=ρ|F output (X)-F expect (X)|+(1-ρ)L CE

[0061] Wherein, the hyperparameter ρ is the balance operator used to balance L. CE With |F output (X)-F expect (X)|, and 0 < ρ < 1, L CE Let F be the cross-entropy loss function. output (X) represents the actual computational complexity of the model, F expect (X) represents the expected computational complexity of the model;

[0062] Step 3E: Repeat steps 3A-3D until a lightweight neural network model that meets the expected values ​​is obtained;

[0063] The fine-tuning module is a process of fine-tuning the lightweight neural network model obtained in step 3E from the coal mine underground personnel action image sequence dataset obtained in step 1, so that the model is more suitable for the task of coal mine underground action recognition. The specific method is as follows:

[0064] Step 3F-1: Modify the number of fully connected layers in the classifier of the lightweight neural network model obtained in Step 3E to match the number of action categories in the coal mine underground personnel action image sequence dataset;

[0065] Step 3F-2: Modify the fine-tuned learning rate lr ft Learning rate lr for pre-training action recognition pt The value is μ times the value of 1, where 1 < μ < 10, to improve the speed of fine-tuning training, i.e.: lr ft =u·lr pt ;

[0066] Step 3F-3: Cut off backpropagation of encoder and decoder layers in lightweight neural network model: Freeze the parameters of encoder and decoder layers to ensure that only the parameters of classifier are updated during fine-tuning training;

[0067] Step 3F-4: Input the dataset of coal mine personnel action image sequences obtained in Step 1 into the lightweight neural network model for training, and repeat Steps 3F-1 to 3F-3 to obtain the initial coarse and fine-tuned model;

[0068] Step 3F-5: Reduce the learning rate of fine-tuning and restore the backpropagation of the encoder and decoder layers in the coarse-fine-tuned model: Unfreeze the parameters of the encoder and decoder layers, and slowly update all parameters of the entire coarse-fine-tuned model until the accuracy of the coarse-fine-tuned model on the coal mine underground personnel action image sequence dataset converges. At this point, a lightweight action recognition fine-tuned model is obtained.

[0069] In step 4, the process of further processing through the parameter compression module to obtain the final lightweight coal mine underground action recognition model is as follows;

[0070] The parameter compression module specifically involves: redesigning the weight reuse method, optimizing the traditional inter-layer reuse into a fusion of intra-layer and inter-layer reuse, with intra-layer reuse as the primary method and inter-layer reuse as a secondary method. Furthermore, a linear mapping technique is incorporated between inter-layer parameter reuse, and the linear mappings of each layer are not entirely consistent. This redesigned weight reuse method allows each layer to have different parameters, effectively alleviating the problem of decreased model expressiveness due to parameter similarity, and diversifying model parameters while compressing parameters.

[0071] The specific steps are as follows:

[0072] Step 4A: Add linear mapping layers before and after the self-attention module in the lightweight action recognition fine-tuning model obtained in Step 3. This compresses the parameters while diversifying them. The original self-attention module is represented as follows:

[0073] AttScore k =G k V k

[0074]

[0075] Among them, AttScore k G represents the attention value. k V represents the weight. k Represents the Value matrix, Q k Represents the Query matrix. Let represent the key matrix, and d be the scale, representing the dimensions of the query matrix and the key matrix; the self-attention module after adding a linear mapping layer is represented as:

[0076]

[0077] in, This is a linear transformation layer before the softmax layer. This is a linear transformation layer following the softmax layer, where H is the number of attention heads;

[0078] Thus, the final lightweight coal mine underground motion recognition model was obtained.

[0079] A lightweight downhole personnel motion recognition device with dynamic memory optimization that implements the above method includes a data acquisition and preprocessing module, a pre-training module, a lightweight neural network architecture search module, a parameter compression module, and a real-time detection module connected in sequence.

[0080] The data acquisition and preprocessing module includes a dedicated underground camera, a data acquisition and preprocessing processor, built-in memory, and external memory hardware. It is used to acquire and construct underground coal mine action image sequence datasets and AVA datasets. The data acquisition and preprocessing module preprocesses the acquired underground coal mine production operation video data and AVA video data, converting the video data into image sequences and dividing them into training and testing sets to obtain underground coal mine personnel action image sequence datasets and AVA datasets. These datasets are then merged to form a pre-trained augmented dataset.

[0081] The pre-training module includes an image processing unit computing cluster, a pre-training processor, and a storage cluster hardware. It is used to transfer the pre-trained augmented dataset obtained by the data acquisition and pre-processing modules through the pre-training module to achieve joint learning of a large-scale network model under the classification task of video action recognition, thereby obtaining an action recognition pre-trained model.

[0082] The lightweight neural network architecture search module includes an image processing unit computing cluster, a lightweight neural network architecture search processor, and a storage cluster hardware. It is used to quickly search for the optimal network architecture of the action recognition pre-trained model obtained by the pre-training module and obtain a lightweight action recognition fine-tuning model.

[0083] The parameter compression module includes an image processing unit computing cluster, a parameter compression processor, and memory hardware. It is used to further process the lightweight action recognition fine-tuning model obtained by the lightweight neural network architecture search module to obtain the final lightweight coal mine underground action recognition model, which is then applied to an embedded terminal.

[0084] The real-time detection module includes a dedicated underground camera, a microprocessor, and memory hardware. It is used to identify real-time underground coal mine production operation video data; convert the video data to be identified into an image sequence, input it into a terminal with the final lightweight model deployed for action recognition, and output the action information of all personnel contained in the video in text.

[0085] Beneficial Effects: This invention introduces video data of coal mine personnel movements and AVA video data; using this method, a lightweight coal mine underground action recognition model is obtained; the method includes: a pre-training module, a lightweight neural network architecture search module, a parameter compression module, and a real-time detection module; AVA stands for Atomic Visual Actions; after preprocessing the AVA video data and the coal mine personnel movement video data, the pre-training module obtains an action recognition pre-trained model; the lightweight neural network architecture search module then passes the action recognition pre-trained model to obtain a lightweight action recognition fine-tuning model; and the parameter compression module then passes the lightweight action recognition fine-tuning model to obtain the final lightweight coal mine underground action recognition model, providing technical support for safe production operations in coal mines and contributing to the digital and intelligent development of mines. This invention utilizes the concept of training and compression on a high-performance computing platform, followed by deployment on an embedded terminal. The trained neural network results, including neural network architecture search, are compressed and then deployed to an embedded computing terminal in a coal mine. This breaks away from the traditional embedded terminal acquisition-server recognition architecture, allowing direct recognition on the embedded computing terminal, greatly ensuring the robustness and timeliness of action recognition. Because the model undergoes component and parameter compression, the model in this invention reduces computational overhead and storage requirements compared to the original Vision Transformer neural network model, thus alleviating the pressure on the embedded computing terminal.

[0086] The Vision Transformer neural network model is a context-aware neural network model that supports parallel computation. Action recognition requires a neural network model with good context awareness. Furthermore, the Vision Transformer neural network model supports parallel computation, which helps accelerate network training and computation, maximizing the utilization of computing resources on edge computing terminals. Therefore, this invention uses the Vision Transformer neural network model for action recognition.

[0087] Neural architecture search is a method for discovering optimal neural network architectures. It transforms the traditional process of manually designing neural network models into an automated exploration process, enabling the discovery of the optimal architecture that meets specific objectives. After performing module compression and dynamic memory optimization on the neural network architecture search, the GPU memory usage during the search process is reduced, as well as the computational resources required for the search, allowing for a faster acquisition of the optimal model architecture for action recognition in coal mines.

[0088] This invention addresses the problems of existing coal mine underground motion recognition, such as the inability to detect motion in real time, high computational load, reliance on expert experience for traditional model compression techniques, and high computational and storage costs for model architecture search.

[0089] Advantages: This invention has the advantages of reducing the computational load of neural network architecture search, GPU memory usage, improving the real-time performance of action recognition, and saving embedded computing terminal resources.

[0090] Compared with existing technologies, this invention fully considers the business pain point of deploying large-scale neural network models in embedded terminals in underground coal mines. It employs an enhanced pre-training method to improve action recognition accuracy while slowing down the gradient of the final action recognition model's learning curve. It uses specific parameter binarization and pruning techniques to effectively reduce candidate path redundancy during the neural network architecture search process, alleviating the problem of long search times caused by path redundancy. It employs dynamic memory optimization techniques to reduce the number of matrices in an active state at the same time, effectively alleviating the problem of excessive GPU memory consumption during the neural network architecture search process. It redesigns the weighted reuse method, reducing model parameters while maintaining model expressiveness, providing technical support for the application of large-scale neural network models in embedded terminals in underground coal mines, and greatly ensuring the robustness and timeliness of personnel action recognition in underground coal mines. Furthermore, because the neural network architecture search technology of this invention undergoes multiple compression processes, it minimizes the computational load and GPU memory usage during the search process while maintaining accuracy, effectively reducing the search model overhead. This allows for good model compression results even when applied to ultra-large-scale neural network models. Attached Figure Description

[0091] Figure 1 This is a flowchart of the underground personnel action recognition method of the present invention.

[0092] Figure 2 This is a structural diagram of the model of the present invention.

[0093] Figure 3 This is a schematic diagram of the device module of the present invention. Detailed Implementation

[0094] The embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0095] This invention discloses a lightweight underground personnel action recognition method for dynamic memory optimization: It introduces underground coal mine personnel action video data and AVA video data; it uses the underground coal mine personnel action video data and AVA data to obtain a lightweight underground coal mine action recognition model; it performs underground personnel action recognition through a pre-training module, a lightweight neural network architecture search module, a parameter compression module, and a real-time detection module; after preprocessing the AVA video data and underground coal mine personnel action video data, it passes through the pre-training module to obtain an action recognition pre-trained model; the action recognition pre-trained model passes through the lightweight neural network architecture search module to obtain a lightweight action recognition fine-tuning model; the lightweight action recognition fine-tuning model passes through the parameter compression module to obtain the final lightweight underground coal mine action recognition model.

[0096] Specifically, the steps include the following:

[0097] Step 1: Collect and construct the coal mine underground action image sequence dataset and AVA dataset: Preprocess the collected coal mine underground personnel action video data and AVA video data, convert the video data into image sequences, and divide them into training set and test set to obtain the coal mine underground personnel action image sequence dataset and AVA dataset. Merge the datasets to form a pre-trained augmented dataset.

[0098] The method for collecting video data of underground coal mine production operations is as follows: a standard dustproof and explosion-proof underground coal mine camera is used and installed in a place that can capture personnel and equipment in the production operation space to collect video data of various production operation actions.

[0099] The AVA video data preprocessing method described above is as follows: AVA video data is classified according to the work scenario, and the video data is filtered to discard video information that does not contain any action. Each complete action forms a video data record and a corresponding action label is defined. After the video data has passed the above initial screening, spatial domain image enhancement technology is used to sharpen images with unclear edge information of personnel and equipment, and to smooth video information with excessive noise, so as to obtain enhanced video data.

[0100] The method for dividing the training set and the test set is as follows: the obtained coal mine underground action image sequence dataset is divided into a fine-tuning training set and a fine-tuning test set according to the ratio of λ and (1-λ); similarly, the pre-training augmentation dataset is divided into a pre-training augmentation training set and a pre-training augmentation test set according to the ratio of λ and (1-λ).

[0101] The process of creating the aforementioned dataset of motion image sequences of underground coal mine workers is as follows:

[0102] Step 1A: Trim the preprocessed video of underground coal mine workers' actions according to the standard of t frames per second, so that the video becomes a frame sequence, represented as V={v1,v2,…,v i ,…,v T}; where T is the total number of frames in the video, v i This represents the i-th frame of the video; each frame in the frame sequence is represented as: Where (H,W) are the height and width of the original image, and C is the number of channels;

[0103] Step 1B: Transfer frame image v i Cut into S = (H × W) / P 2 Small images are divided into blocks, forming an image block sequence F. i ={f i1 ,f i2 ,…,f ij ,…,f iS}, where each small image is represented as f iS Let S represent the S-th small image block in the i-th frame, and P be the side length of each small image block. This is obtained from the image sequence F. i The image sequence dataset consists of: X = {F1, F2, ..., F...} i ,…,F n}

[0104] Step 2: The pre-trained augmented dataset obtained in Step 1 is passed through the pre-training module to achieve joint learning of the ultra-large-scale network model under the classification task of video action recognition, so as to obtain the action recognition pre-trained model.

[0105] The process of obtaining the action recognition pre-trained model requires inputting the pre-trained augmented dataset into a large-scale network model. The specific process of joint learning of the large-scale network model is as follows:

[0106] Step 2A: Divide the pre-trained augmentation dataset obtained in Step 1 into a pre-trained augmentation training set and a pre-trained augmentation test set according to the ratio of λ and (1-λ), where λ<1. The image sequence dataset is represented as: X={F1,F2,…,F…} i ,…,F n}, X is composed of image sequence F i composition;

[0107] Step 2B: Establish the pre-training comprehensive loss function. Loss:

[0108]

[0109] Where: N is the number of action types, p ic y is the probability that sample i belongs to class c, and total is the total number of samples in the video dataset;ic It is a sign function, which takes the value 1 if the true type of sample i matches the type c, and 0 otherwise; the pre-training comprehensive loss function represents the difference between the probability of action recognition type and the true action type.

[0110] Step 2C: Input the pre-trained augmented training set into the ultra-large-scale network model for pre-training. During pre-training, the model parameters will be continuously updated, and the model accuracy will gradually converge. After multiple training iterations, select the converged network model and save it to obtain the action recognition pre-trained model.

[0111] Step 3: The action recognition pre-trained model obtained in Step 2 is passed through the lightweight neural network architecture search module to achieve fast optimal network architecture search and obtain a lightweight action recognition fine-tuning model.

[0112] To obtain the lightweight action recognition fine-tuning model, it is necessary to set a search space and search the action recognition pre-trained model through a lightweight neural network architecture search module; the lightweight neural network architecture module includes a component compression module, a dynamic memory optimization module, and a fine-tuning module.

[0113] The specific process for setting the search space is as follows:

[0114] The process of transforming the structural parameters in the action recognition pre-trained model obtained in step 2 into candidate elements is as follows:

[0115] Step 3A-1: Let l be the path of the action recognition pre-trained model obtained in Step 2, then l i This represents the i-th path in the action recognition pre-trained model, i.e.:

[0116] M(l,…,l i ,…,l n )

[0117] Where M is the graph composed of paths in the action recognition pre-trained model;

[0118] Step 3A-2: Let Γ be the operator of the action recognition pre-trained model obtained in Step 2, then γ i Let represent the i-th operator in the action recognition pre-trained model, i.e.:

[0119] Γ={γ i}

[0120] Where Γ is the set of operators in the action recognition pre-training model;

[0121] Step 3A-3: Denote the operators on the n paths of the action recognition pre-trained model as m. Γ The structure of the entire action recognition pre-trained model can then be represented as:

[0122]

[0123] The component compression module is the process of binarizing and pruning the hidden layers of the feedforward neural network (FFN) in the action recognition pre-trained model obtained in step 2. FFN stands for Feed-forward Neural Network. The specific process is as follows:

[0124] Step 3B-1: Set the inflation rate of the multilayer perceptron (MLP) in the action recognition pre-trained model to α, where the inflation rate α controls the number of hidden layers in the FFN.

[0125] Step 3B-2: For the input vector X, the output of the feedforward neural network FFN can be expressed as:

[0126]

[0127] Among them, O FFN The output of the feedforward neural network is represented by t, which represents the number of input channels to the feedforward neural network. α is the dilation rate, and c is the output of the feedforward neural network. in O is the dimension of the channel. t (X) can be represented as:

[0128]

[0129] Where GeLU is the activation function. and Let be the weight matrix of the fully connected layer in the feedforward neural network, and Where αc in W represents the dimension of the hidden layer. fc1 Map the dimension of the input feature X to αc in W fc2 Inversely map the dimension of the input feature X to c. in ;

[0130] Step 3B-3: Binarize the path weights of each hidden layer of the FFN to obtain the output of the BFFN:

[0131]

[0132] Among them, O BFFN (X) represents the output of BFFN, where BFFN is a binarized feedforward neural network. Code The parameters are binarized and automatically optimized during the neural network architecture search process. The parameter size is compressed after passing through the binarized feedforward neural network.

[0133] Step 3B-4: Introduce a variable parameter λ to evaluate the importance of the hidden dimensions in BFFN:

[0134] λ t =Ber(Sigmoid(P) t ))

[0135] Among them, Sigmoid(P t ) represents the probability of using a t-dimensional hidden layer, λ t To indicate the importance of the hidden dimension of BFFN, based on λ t Pruning hidden layer dimensions with low importance can effectively compress components.

[0136] The dynamic memory optimization module described above is a process that dynamically reduces memory consumption during the neural network architecture search. The specific method is as follows:

[0137] Step 3C-1: Dynamically reduce the number of queries active in memory at the same time: When memory is insufficient, only focus on information from the previous j steps.

[0138] Query k :=[func(Query k-j ),…,func(Query k-1 ),Query k ]

[0139] Among them, Query k Query represents the query operator at time k. k-j This represents the query operator for the first j steps. The function func indicates that the gradient calculation should be stopped. Stopping the gradient calculation can effectively reduce the dependence on previous information during backpropagation and effectively reduce memory consumption.

[0140] Step 3C-2: Similarly, dynamically reduce the number of active keys in memory at the same time: when memory is insufficient, only focus on information from the previous j steps.

[0141] Key k :=[func(Key) k-j ),…,func(Key k-1 ), Key k ]

[0142] Among them, Key k Represents the key operator at time k, Query k-j This represents the key operator for the first j steps, and the function func indicates when to stop calculating the gradient;

[0143] Step 3D: Evaluate the searched network structure: For the model searched for neural network structure, the difference between the expected value and the true value can be calculated using the loss function to evaluate the quality of the searched model. Therefore, the model loss function L is defined as follows:

[0144] L=ρ|F output (X)-F expect (X)|+(1-ρ)L CE

[0145] Wherein, the hyperparameter ρ is the balance operator used to balance L. CE With |F output (X)-F expect (X)|, and 0 < ρ < 1, L CE Let F be the cross-entropy loss function. output (X) represents the actual computational complexity of the model, F expect (X) represents the expected computational complexity of the model;

[0146] Step 3E: Repeat steps 3A-3D until a lightweight neural network model that meets the expected values ​​is obtained;

[0147] The fine-tuning module is a process of fine-tuning the lightweight neural network model obtained in step 3E from the coal mine underground personnel action image sequence dataset obtained in step 1, so that the model is more suitable for the task of coal mine underground action recognition. The specific method is as follows:

[0148] Step 3F-1: Modify the number of fully connected layers in the classifier of the lightweight neural network model obtained in Step 3E to match the number of action categories in the coal mine underground personnel action image sequence dataset;

[0149] Step 3F-2: Modify the fine-tuned learning rate lr ft Learning rate lr for pre-training action recognition pt The value is μ times the value of 1, where 1 < μ < 10, to improve the speed of fine-tuning training, i.e.: lr ft =u·lr pt ;

[0150] Step 3F-3: Cut off backpropagation of encoder and decoder layers in lightweight neural network model: Freeze the parameters of encoder and decoder layers to ensure that only the parameters of classifier are updated during fine-tuning training;

[0151] Step 3F-4: Input the dataset of coal mine personnel action image sequences obtained in Step 1 into the lightweight neural network model for training, and repeat Steps 3F-1 to 3F-3 to obtain the initial coarse and fine-tuned model;

[0152] Step 3F-5: Reduce the learning rate of fine-tuning and restore the backpropagation of the encoder and decoder layers in the coarse-fine-tuned model: Unfreeze the parameters of the encoder and decoder layers, and slowly update all parameters of the entire coarse-fine-tuned model until the accuracy of the coarse-fine-tuned model on the coal mine underground personnel action image sequence dataset converges. At this point, a lightweight action recognition fine-tuned model is obtained.

[0153] Step 4: The lightweight motion recognition fine-tuning model obtained in Step 3 is further processed through the parameter compression module to obtain the final lightweight coal mine underground motion recognition model, which is then applied to the embedded terminal.

[0154] The process of further processing through the parameter compression module to obtain the final lightweight coal mine underground action recognition model is as follows:

[0155] The parameter compression module specifically involves: redesigning the weight reuse method, optimizing the traditional inter-layer reuse into a fusion of intra-layer and inter-layer reuse, with intra-layer reuse as the primary method and inter-layer reuse as a secondary method. Furthermore, a linear mapping technique is incorporated between inter-layer parameter reuse, and the linear mappings of each layer are not entirely consistent. This redesigned weight reuse method allows each layer to have different parameters, effectively alleviating the problem of decreased model expressiveness due to parameter similarity, and diversifying model parameters while compressing parameters.

[0156] The specific steps are as follows:

[0157] Step 4A: Add linear mapping layers before and after the self-attention module in the lightweight action recognition fine-tuning model obtained in Step 3. This compresses the parameters while diversifying them. The original self-attention module is represented as follows:

[0158] AttScore k =G k V k

[0159]

[0160] Among them, AttScore k G represents the attention value. k V represents the weight. k Represents the Value matrix, Q k Represents the Query matrix. Let represent the key matrix, and d be the scale, representing the dimensions of the query matrix and the key matrix; the self-attention module after adding a linear mapping layer is represented as:

[0161]

[0162] in, This is a linear transformation layer before the softmax layer. This is a linear transformation layer following the softmax layer, where H is the number of attention heads;

[0163] Thus, the final lightweight coal mine underground motion recognition model was obtained.

[0164] Step 5: Recognize the real-time underground coal mine personnel action video data: Convert the video data to be recognized into an image sequence, input it into the terminal of the final lightweight underground coal mine action recognition model obtained in Step 4 for action recognition, and output the action information of all personnel contained in the video in text.

[0165] A lightweight downhole personnel motion recognition device with dynamic memory optimization that implements the above method includes a data acquisition and preprocessing module, a pre-training module, a lightweight neural network architecture search module, a parameter compression module, and a real-time detection module connected in sequence.

[0166] The data acquisition and preprocessing module includes a dedicated underground camera, a data acquisition and preprocessing processor, built-in memory, and external memory hardware. It is used to acquire and construct underground coal mine action image sequence datasets and AVA datasets. The data acquisition and preprocessing module preprocesses the acquired underground coal mine production operation video data and AVA video data, converting the video data into image sequences and dividing them into training and testing sets to obtain underground coal mine personnel action image sequence datasets and AVA datasets. These datasets are then merged to form a pre-trained augmented dataset.

[0167] The pre-training module includes an image processing unit computing cluster, a pre-training processor, and a storage cluster hardware. It is used to transfer the pre-trained augmented dataset obtained by the data acquisition and pre-processing modules through the pre-training module to achieve joint learning of a large-scale network model under the classification task of video action recognition, thereby obtaining an action recognition pre-trained model.

[0168] The lightweight neural network architecture search module includes an image processing unit computing cluster, a lightweight neural network architecture search processor, and a storage cluster hardware. It is used to quickly search for the optimal network architecture of the action recognition pre-trained model obtained by the pre-training module and obtain a lightweight action recognition fine-tuning model.

[0169] The parameter compression module includes an image processing unit computing cluster, a parameter compression processor, and memory hardware. It is used to further process the lightweight action recognition fine-tuning model obtained by the lightweight neural network architecture search module to obtain the final lightweight coal mine underground action recognition model, which is then applied to an embedded terminal.

[0170] The real-time detection module includes a dedicated underground camera, a microprocessor, and memory hardware. It is used to identify real-time underground coal mine production operation video data; convert the video data to be identified into an image sequence, input it into a terminal with the final lightweight model deployed for action recognition, and output the action information of all personnel contained in the video in text.

Claims

1. A lightweight downhole personnel action recognition method optimized for dynamic memory, characterized in that: This study introduces video data of coal mine personnel movements and AVA video data. A lightweight coal mine underground action recognition model is obtained using this data. Underground personnel action recognition is performed through a pre-training module, a lightweight neural network architecture search module, a parameter compression module, and a real-time detection module. After preprocessing the AVA video data and the coal mine underground personnel movement video data, a pre-training module is used to obtain a pre-trained action recognition model. This model is then passed through a lightweight neural network architecture search module to obtain a lightweight fine-tuned action recognition model. Finally, the fine-tuned model is passed through a parameter compression module to obtain the final lightweight coal mine underground action recognition model. Specifically, the steps include the following: Step 1: Collect and construct the coal mine underground action image sequence dataset and AVA dataset: Preprocess the collected coal mine underground personnel action video data and the acquired AVA video data, convert the video data into image sequences, and divide them into training set and test set to obtain the coal mine underground personnel action image sequence dataset and AVA dataset. Merge the datasets to form a pre-trained augmented dataset. Step 2: The pre-trained augmented dataset obtained in Step 1 is passed through the pre-training module to achieve joint learning of the ultra-large-scale network model under the classification task of video action recognition, so as to obtain the action recognition pre-trained model. Step 3: The action recognition pre-trained model obtained in Step 2 is passed through the lightweight neural network architecture search module to achieve fast optimal network architecture search and obtain a lightweight action recognition fine-tuning model. Step 4: The lightweight motion recognition fine-tuning model obtained in Step 3 is further processed through the parameter compression module to obtain the final lightweight coal mine underground motion recognition model, which is then applied to the embedded terminal. The process of further processing through the parameter compression module to obtain the final lightweight coal mine underground action recognition model is as follows: The parameter compression module specifically involves: redesigning the weight reuse method, optimizing the traditional inter-layer reuse into a fusion of intra-layer and inter-layer reuse, with intra-layer reuse as the primary method and inter-layer reuse as a secondary method. Furthermore, a linear mapping technique is incorporated between inter-layer parameter reuse, and the linear mappings of each layer are not entirely consistent. This redesigned weight reuse method allows each layer to have different parameters, effectively alleviating the problem of decreased model expressiveness due to parameter similarity, and diversifying model parameters while compressing parameters. The specific steps are as follows: Step 4A: Add linear mapping layers before and after the self-attention module in the lightweight action recognition fine-tuning model obtained in Step 3. This compresses the parameters while diversifying them. The original self-attention module is represented as follows: AttScore k =G k V k Among them, AttScore k G represents the attention value. k V represents the weight. k Represents the Value matrix, Q k Represents the Query matrix. Let represent the key matrix, and d be the scale, representing the dimensions of the query matrix and the key matrix; the self-attention module after adding a linear mapping layer is represented as: in, This is a linear transformation layer before the softmax layer. This is a linear transformation layer following the softmax layer, where H is the number of attention heads; Thus, the final lightweight coal mine underground motion recognition model was obtained; Step 5: Recognize the real-time underground coal mine personnel action video data: Convert the video data to be recognized into an image sequence, input it into the terminal of the final lightweight underground coal mine action recognition model obtained in Step 4 for action recognition, and output the action information of all personnel contained in the video in text.

2. The lightweight downhole personnel action recognition method optimized for dynamic memory as described in claim 1, characterized in that: In step 1, the method for collecting video data of underground coal mine production operations is as follows: a standard dustproof and explosion-proof underground coal mine camera is used and installed in a place that can capture personnel and equipment in the production operation space to collect video data of various production operation actions.

3. The lightweight downhole personnel action recognition method optimized for dynamic memory as described in claim 1, characterized in that: In step 1, the AVA video data preprocessing method is as follows: the AVA video data is classified according to the work scenario, and the video data is filtered to discard video information that does not contain any action. Each complete action forms a video data and a corresponding action label is defined. After the video data is initially screened, spatial domain image enhancement technology is used to sharpen images with unclear edge information of personnel and equipment, and to smooth video information with too much noise, so as to obtain enhanced video data.

4. A lightweight downhole personnel action recognition method optimized for dynamic memory as described in claim 1, characterized in that: In step 1, the method for dividing the training set and the test set is as follows: the obtained coal mine underground action image sequence dataset is divided into a fine-tuning training set and a fine-tuning test set according to the ratio of λ and (1-λ); Similarly, the pre-trained augmented dataset is divided into a pre-trained augmented training set and a pre-trained augmented test set according to the ratio of λ and (1-λ).

5. A lightweight downhole personnel action recognition method optimized for dynamic memory as described in claim 1, characterized in that: In step 1, the process of creating the coal mine underground personnel motion image sequence dataset is as follows: Step 1A: Trim the preprocessed video of underground coal mine workers' actions according to the standard of t frames per second, so that the video becomes a frame sequence, represented as V={v1,v2,…,v i ,…,v T }; where T is the total number of frames in the video, v i This represents the i-th frame of the video; each frame in the frame sequence is represented as: Where (H,W) are the height and width of the original image, and C is the number of channels; Step 1B: Transfer frame image v i Cut into S = (H × W) / P 2 Small images are divided into blocks, forming an image block sequence F. i ={f i1 ,f i2 ,…,f ij ,…,f iS }, where each small image is represented as f iS Let S represent the S-th small image block in the i-th frame, and P be the side length of each small image block. This is obtained from the image sequence F. i The image sequence dataset consists of: X = {F1, F2, ..., F...} i ,…,F n } 6. A lightweight downhole personnel action recognition method optimized for dynamic memory as described in claim 1, characterized in that: In step 2, obtaining the action recognition pre-trained model requires inputting the pre-trained augmented dataset into the ultra-large-scale network model. The specific process of joint learning of the ultra-large-scale network model is as follows: Step 2A: Divide the pre-trained augmentation dataset obtained in Step 1 into a pre-trained augmentation training set and a pre-trained augmentation test set according to the ratio of λ and (1-λ), where λ<1. The image sequence dataset is represented as: X={F1,F2,…,F…} i ,…,F n }, X is composed of image sequence F i composition; Step 2B: Establish the pre-training comprehensive loss function. Loss: Where: N is the number of action types, p ic y is the probability that sample i belongs to class c. ic is the sign function, which takes the value 1 if the true class of sample i matches the class c, and 0 otherwise; the pre-training comprehensive loss function represents the difference between the probability of action recognition class and the true action class. Step 2C: Input the pre-trained augmented training set into the ultra-large-scale network model for pre-training. During pre-training, the model parameters will be continuously updated, and the model accuracy will gradually converge. After multiple training iterations, select the converged network model and save it to obtain the action recognition pre-trained model.

7. A lightweight downhole personnel action recognition method optimized for dynamic memory as described in claim 1, characterized in that: In step 3, obtaining the lightweight action recognition fine-tuning model requires setting a search space and using a lightweight neural network architecture search module to search the pre-trained action recognition model; the lightweight neural network architecture module includes a component compression module, a dynamic memory optimization module, and a fine-tuning module.

8. A lightweight downhole personnel action recognition method optimized for dynamic memory as described in claim 7, characterized in that: The specific process for setting the search space is as follows: The process of transforming the structural parameters in the action recognition pre-trained model obtained in step 2 into candidate elements is as follows: Step 3A-1: Let l be the path of the action recognition pre-trained model obtained in Step 2, then l i This represents the i-th path in the action recognition pre-trained model, i.e.: M(l,…,l i ,…,l n ) Where M is the graph composed of paths in the action recognition pre-trained model; Step 3A-2: Let Γ be the operator of the action recognition pre-trained model obtained in Step 2, then γ i Let represent the i-th operator in the action recognition pre-trained model, i.e.: C={c i } Where Γ is the set of operators in the action recognition pre-training model; Step 3A-3: Denote the operators on the n paths of the action recognition pre-trained model as m. Γ The structure of the entire action recognition pre-trained model can then be represented as: The component compression module is the process of binarizing and pruning the hidden layers of the feedforward neural network (FFN) in the action recognition pre-training model obtained in step 2. The specific process is as follows: Step 3B-1: Set the inflation rate of the multilayer perceptron (MLP) in the action recognition pre-trained model to α, where the inflation rate α controls the number of hidden layers in the FFN. Step 3B-2: For the input vector X, the output of the feedforward neural network FFN can be expressed as: Among them, O FFN The output of the feedforward neural network is represented by t, which represents the number of input channels to the feedforward neural network. α is the dilation rate, and c is the output of the feedforward neural network. in O is the dimension of the channel. t (X) can be represented as: Where GeLU is the activation function. and Let be the weight matrix of the fully connected layer in the feedforward neural network, and Where αc in W represents the dimension of the hidden layer. fc1 Map the dimension of the input feature X to αc in W fc2 Inversely map the dimension of the input feature X to c. in ; Step 3B-3: Binarize the path weights of each hidden layer of the FFN to obtain the output of the BFFN: Among them, O BFFN (X) represents the output of BFFN, where BFFN is a binarized feedforward neural network. Code The parameters are binarized and automatically optimized during the neural network architecture search process. The parameter size is compressed after passing through the binarized feedforward neural network. Step 3B-4: Introduce a variable parameter λ to evaluate the importance of the hidden dimensions in BFFN: l t =Ber(Sigmoid(P t )) Among them, Sigmoid(P t ) represents the probability of using a t-dimensional hidden layer, λ t To indicate the importance of the hidden dimension of BFFN, based on λ t Pruning hidden layer dimensions with low importance achieves component compression; The dynamic memory optimization module described above is a process that dynamically reduces memory consumption during the neural network architecture search. The specific method is as follows: Step 3C-1: Dynamically reduce the number of queries active in memory at the same time: When memory is insufficient, only focus on information from the previous j steps. Query k :=[func(Query k-j ),…,func(Query k-1 ),Query k ] Among them, Query k Query represents the query operator at time k. k-j This represents the query operator for the first j steps. The function func indicates that the gradient calculation should be stopped. Stopping the gradient calculation can effectively reduce the dependence on previous information during backpropagation and effectively reduce memory consumption. Step 3C-2: Similarly, dynamically reduce the number of active keys in memory at the same time: when memory is insufficient, only focus on information from the previous j steps. Key k :=[func(Key k-j ),…,func(Key k-1 ),Key k ] Among them, Key k Represents the key operator at time k, Query k-j This represents the key operator for the first j steps, and the function func indicates when to stop calculating the gradient; Step 3D: Evaluate the searched network structure: For the model searched for neural network structure, the difference between the expected value and the true value can be calculated using the loss function to evaluate the quality of the searched model. Therefore, the model loss function L is defined as follows: L=ρ|F output (X)-F expect (X)|+(1-ρ)L CE Wherein, the hyperparameter ρ is the balance operator used to balance L. CE With |F output (X)-F expect (X)|, and 0 < ρ < 1, L CE Let F be the cross-entropy loss function. output (X) represents the actual computational complexity of the model, F expect (X) represents the expected computational complexity of the model; Step 3E: Repeat steps 3A-3D until a lightweight neural network model that meets the expected values ​​is obtained; The fine-tuning module is a process of fine-tuning the lightweight neural network model obtained in step 3E from the coal mine underground personnel action image sequence dataset obtained in step 1, so that the model is more suitable for the task of coal mine underground action recognition. The specific method is as follows: Step 3F-1: Modify the number of fully connected layers in the classifier of the lightweight neural network model obtained in Step 3E to match the number of action categories in the coal mine underground personnel action image sequence dataset; Step 3F-2: Modify the fine-tuned learning rate lr ft Learning rate lr for pre-training action recognition pt The value is μ times the value of 1, where 1 < μ < 10, to improve the speed of fine-tuning training, i.e.: lr ft =u·lr pt ; Step 3F-3: Cut off backpropagation of encoder and decoder layers in lightweight neural network model: Freeze the parameters of encoder and decoder layers to ensure that only the parameters of classifier are updated during fine-tuning training; Step 3F-4: Input the dataset of coal mine personnel action image sequences obtained in Step 1 into the lightweight neural network model for training, and repeat Steps 3F-1 to 3F-3 to obtain the initial coarse and fine-tuned model; Step 3F-5: Reduce the learning rate of fine-tuning and restore the backpropagation of the encoder and decoder layers in the coarse-fine-tuned model: Unfreeze the parameters of the encoder and decoder layers, and slowly update all parameters of the entire coarse-fine-tuned model until the accuracy of the coarse-fine-tuned model on the coal mine underground personnel action image sequence dataset converges. At this point, a lightweight action recognition fine-tuned model is obtained.

9. A lightweight downhole personnel motion recognition device for implementing the method of claim 1, characterized in that: It includes a data acquisition and preprocessing module, a pre-training module, a lightweight neural network architecture search module, a parameter compression module, and a real-time detection module connected in sequence; The data acquisition and preprocessing module includes a dedicated underground camera, a data acquisition and preprocessing processor, built-in memory, and external memory hardware. It is used to acquire and construct underground coal mine action image sequence datasets and AVA datasets. The data acquisition and preprocessing module preprocesses the acquired underground coal mine production operation video data and AVA video data, converting the video data into image sequences and dividing them into training and testing sets to obtain underground coal mine personnel action image sequence datasets and AVA datasets. These datasets are then merged to form a pre-trained augmented dataset. The pre-training module includes an image processing unit computing cluster, a pre-training processor, and a storage cluster hardware. It is used to transfer the pre-trained augmented dataset obtained by the data acquisition and pre-processing modules through the pre-training module to achieve joint learning of a large-scale network model under the classification task of video action recognition, thereby obtaining an action recognition pre-trained model. The lightweight neural network architecture search module includes an image processing unit computing cluster, a lightweight neural network architecture search processor, and a storage cluster hardware. It is used to quickly search for the optimal network architecture of the action recognition pre-trained model obtained by the pre-training module and obtain a lightweight action recognition fine-tuning model. The parameter compression module includes an image processing unit computing cluster, a parameter compression processor, and memory hardware. It is used to further process the lightweight action recognition fine-tuning model obtained by the lightweight neural network architecture search module to obtain the final lightweight coal mine underground action recognition model, which is then applied to an embedded terminal. The real-time detection module includes a dedicated underground camera, a microprocessor, and memory hardware. It is used to identify real-time underground coal mine production operation video data; convert the video data to be identified into an image sequence, input it into a terminal with the final lightweight model deployed for action recognition, and output the action information of all personnel contained in the video in text.

Images