A mine personnel illegal action recognition system based on constructive federated learning
Through constructive federated learning and Openpose behavior feature extraction, the problem of difficult identification of miners' violations is solved, efficient monitoring of violations and accident prevention is achieved, and the accuracy and supervision efficiency of the identification model are improved.
Patent Information
- Application Number
- CN202210393664.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-14
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-04-14
AI Technical Summary
The existing technology is difficult to effectively identify and monitor miners’ violations in mines, resulting in frequent safety accidents. In addition, artificial intelligence monitoring methods require a large amount of manual labeling data, making it difficult to train high-performance identification models in a single mining area.
Adopt a mine personnel violation action recognition system based on structural federal learning, through the cooperation of the aggregation cloud and the mining area violation identification terminal, use Openpose to extract the miner's behavior characteristics, combine the parameter transmission module and the network collaborative learning module, jointly train the network model, and replace human monitoring. , improve supervision efficiency.
While ensuring data privacy, network models with better identification performance were jointly trained to reduce accident incidence and improve supervision efficiency, solve the problem of insufficient data in a single mining area, and improve the accuracy and practicality of the identification model.
Smart Images

Figure CN115147749B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of illegal behavior recognition of workers in mining areas, and in particular to a system for identifying illegal actions of workers in mining areas based on constructive federated learning. Background Art
[0002] my country is a vast country with abundant resources. Its mineral resources rank among the top in the world and its potential is enormous. There are countless mined and unmined mining areas, and countless amounts of various ores are mined each year, which continuously inject energy and materials into my country's development.
[0003] Safety has always been a big problem in ore mining and development. Various mining accidents are shocking and pose great danger to the lives of miners and the safety of equipment. Among them, the illegal behavior of workers during operation is a huge hidden danger. The traditional method to eliminate workers' illegal behavior is usually to stare at the monitoring screen manually. This kind of supervision purely relying on monitoring personnel has many disadvantages, such as: long-term high-intensity monitoring by monitoring personnel is prone to visual fatigue, too many monitoring areas to take care of all, etc. If artificial intelligence monitoring methods are used, the supervision effect can be greatly improved, accidents can be prevented, and the safety of workers and equipment can be guaranteed. However, the current artificial intelligence monitoring methods usually require a large amount of manually labeled data to train the recognition network model. It is usually difficult to train a network model with good recognition performance based solely on data in a certain mining area. Summary of the invention
[0004] Purpose of the invention: In view of the defects of the prior art, the present invention provides a system for identifying illegal actions of mine personnel based on constructive federated learning. Since the mines participating in constructive federated learning have similar equipment, similar industrial processes, similar actions during mining operations, and similar characteristics, a network model with better recognition performance can be jointly trained while ensuring the data privacy of each mine, thereby replacing manual monitoring, improving supervision efficiency, and reducing the accident rate.
[0005] The system of the present invention includes an aggregation cloud and a mining area violation identification terminal;
[0006] The aggregation cloud includes a parameter transmission module and a network collaborative learning module;
[0007] The mining area violation identification terminal includes a miner behavior feature acquisition module, a local network model construction module and a behavior recognition module;
[0008] The miner behavior feature acquisition module obtains historical videos of miners' behavior in the belt area of each mining area through RGB cameras installed above the belt conveyors of more than two mining areas, uses human skeleton extraction tools to analyze and extract feature data of miners' behavior in the belt area from video stream data, manually annotates behavior categories and organizes them into local miners' behavior databases in each mining area;
[0009] As a human skeleton extraction tool, Openpose can extract the skeleton nodes of miners in video stream data. According to the extracted skeleton nodes, key frames can be filtered and multi-dimensional behavior features of miners can be extracted. The multi-dimensional features include the relative distance feature value of miners' limbs, the angle feature value of miners' limbs, the inter-frame velocity feature value of miners' limbs, and the inter-frame angular velocity feature value of miners' limbs. The details are as follows:
[0010] Openpose is used to extract the miner's skeleton nodes. 15 skeleton nodes are artificially selected from the extracted miner's skeleton nodes, and they form 14 rigid bodies that can represent the miner's body. The relative distance eigenvalue of the miner's limbs, that is, the length of the 14 rigid bodies of the miner's body, is to calculate the relative distance of the skeleton nodes at both ends of the 14 rigid bodies. For example, the length of a rigid body from the right shoulder to the center of the shoulder is taken as an example. Let the coordinates of the skeleton node at the center of the shoulder be p2(x2,y2), and the coordinates of the skeleton node of the right shoulder be p4(x4,y4). Then the length d2 of the second rigid body is:
[0011] d2=|p2(x2,y2)-p2(x4,y4)|
[0012] where p i (x i ,y i ) represents the two-dimensional coordinates of skeleton node i.
[0013] In order to eliminate the difference in individual skeleton size, the above relative distances are normalized, that is, the relative distances between all skeleton nodes are divided by the relative distance from the center point of the shoulder to the center point of the hip, that is:
[0014]
[0015] Where h represents the relative distance from the center of the shoulder to the center of the hip.
[0016] For the nth frame of the miner's behavior image, the miner's limb relative distance feature value L n It is expressed as:
[0017] L n ={d n,1 ,d n,2 ,...,d n,14 |n=1,2,...,N}
[0018] Where n represents the nth frame image in the total length N of the miner behavior video, d n,i Represents the relative distance feature value of the i-th miner's limbs in the n-th frame image.
[0019] The eigenvalue of the miner's limb angle is the eigenvalue of the angle formed between the miner's limbs. We artificially select 14 key angles of the miner's human skeleton and take the right arm as an example to calculate the eigenvalue of the miner's limb angle θ3. Let the right shoulder skeleton node coordinates be p4 (x4, y4), the right elbow skeleton node coordinates be p5 (x5, y5), and the right hand skeleton node coordinates be p6 (x6, y6). Then the eigenvalue of the miner's limb angle θ3 is:
[0020]
[0021] where p i (x i ,y i ) represents the two-dimensional coordinates of skeleton node i.
[0022] For the nth frame of the miner's behavior image, the miner's limb angle feature value θ n It is expressed as:
[0023] θ n ={θ n,1 ,θ n,2 ,...,θ n,14 |n=1,2,...,N}
[0024] Where n represents the nth frame image in the total length N of the miner behavior video, θ n,i Represents the eigenvalue of the limb angle of the i-th miner in the n-th frame image.
[0025] Use cosine similarity to extract key frames from the video data stream to reduce resource consumption and speed up the recognition of the model. For the two frames of miners’ limb angle feature values:
[0026] θ n-1 (θ n-1,1 ,θ n-1,2 ,...,θ n-1,14 ),θ n (θ n,1 ,θ n,2 ,...,θ n,14 )
[0027] Where n-1 and n represent two adjacent frames in the video, θ n-1,i ,θ n,i They represent the i-th angle eigenvalue in the n-1-th frame and the i-th angle eigenvalue in the n-th frame respectively.
[0028] The similarity between two frames is calculated as follows:
[0029]
[0030] Where cos(θ) represents the similarity between two frames. The closer the value is to 1, the more similar the two frames are. When it exceeds the set threshold, the similar frames are discarded. The threshold is generally obtained through experimental testing.
[0031] The inter-frame miner limb velocity feature value is the average speed of the miner's same skeleton node moving between two frames. Let n-1, n represent the miner's behavior images in two adjacent frames, then the inter-frame displacement S of skeleton node i is n,i for:
[0032] S n,i =|p n,i (x n,i ,y n,i )-p n-1,i (x n-1,i ,y n-1,i )|
[0033] where p n,i (x n,i ,y n,i ) represents the two-dimensional coordinates of the i-th skeleton node in the n-th frame.
[0034] Let the time interval between two frames be t, then the inter-frame speed V of the miner skeleton node i in the nth frame n,i for:
[0035]
[0036] The inter-frame velocity characteristic value of the miner's body in the nth frame is expressed as:
[0037] V n = {V n,1 ,V n,2 ,...,V n,15 |n=1,2,...,N}
[0038] Where n represents the nth frame image in the total length N of the miner behavior video, V n,i Represents the eigenvalue of the miner's limb velocity in the i-th frame of the n-th frame image.
[0039] The inter-frame angular velocity characteristic value of the miner's limbs is the average speed of the change of the same key angle between two frames. Let n-1, n represent the miner's behavior images in two adjacent frames, then the inter-frame change of the key angle i is Ω n,i for:
[0040] Ω n,i =θ n,i -θ n-1,i
[0041] where θ n,i Represents the eigenvalue of the limb angle of the i-th miner in the n-th frame.
[0042] Let the time interval between two frames be t, then the inter-frame angular velocity ω of the miner's limb angle i in the nth frame is n,i for:
[0043]
[0044] The angular velocity characteristic value of the miner's body between frames of the nth frame can be expressed as:
[0045] ω n ={ω n,1 ,ω n,2 ,...,ω n,14 |n=1,2,...,N}
[0046] Where n represents the nth frame image in the total length N of the miner behavior video, ω n,i Represents the angular velocity feature value of the miner's limbs in the i-th frame of the n-th frame image.
[0047] The multi-dimensional feature combination derives the miner behavior feature x in the nth frame n ,Right now:
[0048] x n ={L n ,θ n ,V n ,ω n |n=1,2,...,N}
[0049] ={L n,1 ,L n,2 ,...,L n,14 ,θ n,1 ,θ n,2 ,...,θ n,14 ,V n,1 ,V n,2 ,...,V n,15 ,ω n,1 ,ω n,2 ,...,ω n,14 |n=1,2,...,N}
[0050] Where N represents the total number of miner behavior image frames, L n,i represents the relative distance feature value of the limbs of the i-th miner in the n-th frame, θ n,i represents the eigenvalue of the limb angle of the i-th miner in the n-th frame, V n,i represents the eigenvalue of the miner’s limb velocity between the i-th frames in the n-th frame, ω n,i It represents the angular velocity characteristic value of the miner's limbs in the i-th frame in the n-th frame.
[0051] Assuming that there are N frames of miner behavior videos with behavior category labels in the violation identification terminal k, the local miner behavior database of the violation identification terminal k can be expressed as:
[0052]
[0053] in represents the multi-dimensional features of the i-th frame of the miner behavior image at the violation identification terminal k, Represents the miner behavior category label of the i-th frame.
[0054] The local network model construction module is used to perform constructive federated learning. A constructive federated learning aggregation cloud is set at a trusted third party. Each mining area acts as a mining area violation identification terminal and participates in constructive federated learning. Specifically, the local miner behavior database of each mining area is used as a feature database. Starting from a zero network, hidden nodes are gradually added. Each time a new node is constructed in the local network model, a set of hidden node parameters are randomly generated within a set interval. The output weights and variances of the local network model are calculated according to the generated hidden node parameters. The randomly generated hidden node parameters and the current variance of the local network model are transmitted to the aggregation cloud through the parameter transmission module;
[0055] The parameter transmission module is used for parameter transmission between the local network model construction module and the network collaborative learning module;
[0056] The network collaborative learning module receives the latest hidden layer node parameters and variances randomly generated by each local network model and performs adaptive parameter aggregation to construct aggregated hidden layer nodes, and transmits the aggregated hidden layer nodes to each mining area violation identification terminal through a parameter transmission module;
[0057] After receiving the aggregated hidden layer nodes, the mine area violation identification terminal updates the newly added hidden layer node parameters of each mine area violation identification terminal, and calculates the output weight of the local network model to obtain the belt area miner behavior classification error of the current mine area violation identification terminal. According to the settings of each mine area violation identification terminal, it is judged whether the belt area miner behavior classification error of the current mine area violation identification terminal meets the belt area miner behavior violation action recognition accuracy requirement in the mine area violation identification terminal. If so, the mine area violation identification terminal exits the constructive federated learning; otherwise, the local network model construction module continues to construct new hidden layer nodes for the next round of federated learning.
[0058] After the mine violation identification terminal completes the federated learning, the behavior recognition module determines whether the miner's behavior is in violation of the regulations based on the real-time behavior characteristics of the mining operations in the belt area.
[0059] The local network model construction module uses a random vector function link network as the local network model of each mining area violation identification terminal, and k represents the kth violation identification terminal among all the violation identification terminals participating in the constructed federated learning. When the mining area violation identification terminal k starts to participate in the constructed federated learning, the local network model learning parameters are initialized, including: the expected tolerance ε k , hidden layer parameter random generation range Υ={-λ:+λ}, activation function g(.), initial residual e0=T k , where -λ is the lower limit of the distribution interval of the random parameter, +λ is the upper limit of the distribution interval of the random parameter, T k is the actual category label of the local belt area miner behavior feature data of the violation identification terminal k, ε k represents the expected tolerance set for the violation identification terminal k, the hidden layer parameter random generation range Υ and the activation function g(.) are set the same for all violation identification terminals.
[0060] Taking the construction of a new hidden node L for the mining area violation identification terminal k as an example, a set of hidden node parameters are randomly generated in the symmetric interval Υ={-λ:+λ} that obeys the Gaussian distribution. Use the following formula to calculate the current hidden layer node output matrix of the local network model Current hidden layer output matrix Expanding the Input Matrix and output weights
[0061] The superscript T represents the transpose of a matrix or a vector. represents the Mth dimension input feature of the local network model of the mining area violation identification terminal k, represents the parameters of the Lth hidden node constructed by the local network model of the illegal identification terminal k, represents the hidden layer output matrix of the Lth node calculated according to the parameters of the Lth hidden layer node constructed according to the local network model, represents the parameters of the L-1 hidden nodes before the Lth hidden node determined by constructive federated learning, represents the output matrix of the L-1 hidden layer nodes before the Lth hidden layer node, Represents the current hidden layer output matrix of the local network model, Represents the current expanded input matrix of the local network model, X k It represents the belt area miner behavior database within the violation identification terminal k, and the superscript + indicates finding the generalized inverse of the matrix.
[0062] The current output variance of the local network model for the violation identification terminal k is calculated using the following formula
[0063]
[0064] Where T k Represents the true label of the behavior classification of the miner behavior image in the violation identification terminal k.
[0065] The hidden layer node parameters randomly generated by the local network model and the current variance of the local network model are transmitted to the aggregation cloud through the parameter transmission module.
[0066] The aggregation cloud receives the hidden layer node parameters randomly generated by the local network model of all violation identification terminals, performs adaptive parameter aggregation in the network collaborative learning module, and constructs aggregated hidden layer nodes.
[0067] The network collaborative learning module uses the following formula to construct the adaptive aggregation hidden layer nodes:
[0068]
[0069] in, represents the aggregate hidden layer node parameter after aggregation, K represents the total number of mining area violation identification terminals participating in constructive federated learning, represents the Lth hidden node parameter of the mining area violation identification terminal k, represents the adaptive aggregation parameter of the Lth hidden layer node of the violation identification terminal k, It represents the variance of the Lth hidden layer node of the local network model of the illegal identification terminal k.
[0070] The aggregation cloud transmits the constructed aggregation hidden layer nodes to each violation identification terminal through the parameter transmission module.
[0071] Each violation identification terminal receives the aggregated hidden layer node and updates the Lth hidden layer node parameter of each violation identification terminal.
[0072] The hidden layer output of the Lth hidden layer node of the local network model is calculated using the following formula:
[0073]
[0074] in represents the Lth aggregation hidden layer node parameter determined according to constructive federated learning;
[0075] The local network model output weight matrix of the violation identification terminal is calculated using the following formula
[0076]
[0077] The superscript + is the generalized inverse matrix of the matrix. Represents the current expanded input matrix of the local network model, Represents the current hidden layer output matrix of the local network model, represents the output matrix of the lth hidden layer node of the local network model, represents the parameters of the lth hidden layer node determined by constructive federated learning, X k represents the belt area miner behavior dataset within the violation identification terminal k;
[0078] The output of the local network model for the violation identification terminal k is calculated using the following formula
[0079]
[0080] Then the current local network model recognition error is obtained, and it is judged whether the local network model recognition error of the current violation identification terminal meets the expected recognition error ε of the illegal action of the miner behavior in the belt area of the terminal k If the requirements are met, the terminal exits the constructive federated learning; if not, it continues to construct new hidden layer node parameters and conducts the next round of federated learning.
[0081] After the violation identification terminal completes the constructive federated learning, it identifies the miners' behavior categories based on the real-time behavioral characteristics of the mining operations in the belt area.
[0082] The present invention jointly trains the illegal behavior identification model with the data of multiple mining areas, and the data collected by each mining area will not be leaked to each other, ensuring the privacy of the data of all parties. This method solves the island problem of data from each mining area, that is, each mining area only has the behavior data of miners in this mining area, and the recognition effect of the trained model is limited. The present invention jointly trains a model with better recognition effect while ensuring the data security of each plant area.
[0083] Beneficial effects: The mine personnel illegal action recognition system based on constructive federated learning of the present invention can not only effectively solve the problems of difficulty in real-time monitoring of illegal and dangerous behaviors of mine workers, long-term high-intensity monitoring of monitoring personnel, visual fatigue and excessive burden, etc., but also the federated training improves the accuracy of the recognition model, reduces the accident rate in the mine area, is easy to use, and has strong practicality. BRIEF DESCRIPTION OF THE DRAWINGS
[0084] Figure 1 A schematic diagram of a miner skeleton according to an embodiment of the present invention;
[0085] Figure 2 This is a flow chart of a system for identifying illegal actions of mine personnel based on constructive federated learning according to an embodiment of the present invention. DETAILED DESCRIPTION
[0086] The following is a further description of the embodiments of the present invention in conjunction with the accompanying drawings:
[0087] Matters not covered by the present invention are known technologies.
[0088] Technical basis of this invention: OpenPose is a human posture estimation technology proposed by the CMU Perceptual Computing Laboratory in 2017. The extracted miner skeleton diagram is shown in Figure 1 As shown in the figure. OpenPose is an open source library based on convolutional neural networks and supervised learning and written in the Caffe framework. It can track people's facial expressions, torsos, limbs and even fingers. It is not only suitable for single people but also for multiple people, and has good robustness. It can be called the world's first real-time multi-person two-dimensional pose estimation based on deep learning. It is a milestone in human-computer interaction and provides a high-quality information dimension for machines to understand people.
[0089] A specific embodiment of the present invention, as Figure 2 As shown, a mine personnel illegal action recognition system based on constructive federated learning is disclosed, which is divided into two parts: an aggregation cloud and a mine area illegal action identification terminal. The aggregation cloud includes a parameter transmission module and a network collaborative learning module; the mine area illegal action identification terminal includes a miner behavior feature acquisition module, a local network model construction module and a behavior recognition module. The miner behavior feature acquisition module uses a human skeleton extraction tool to analyze and extract the multi-dimensional feature data of the belt miner behavior from the historical video stream data of the belt area mining operation; the local network model construction module randomly generates new hidden layer nodes of the local network according to the local multi-dimensional feature data of the belt miner behavior; the parameter transmission module receives the parameters of the new hidden layer nodes of the local network model, the current variance of the local network model and the distribution network collaborative learning parameters; the network collaborative learning module performs adaptive parameter aggregation on the new hidden layer nodes of all local networks to construct aggregated hidden layer nodes; the behavior recognition module uses the trained constructive federated learning mine personnel illegal action recognition model to analyze the multi-dimensional features of the miner behavior of the miner behavior feature acquisition module, and gives the final behavior category. The system of the present invention performs the following steps:
[0090] Step 1: Assume that there are 10 mining areas participating in constructive federated learning. Each mining area is equipped with a belt conveyor. The historical videos of miners' working behaviors in the belt areas of each mining area are obtained by installing RGB cameras above the belt conveyors of the 10 mining areas. The human skeleton extraction tool Openpose is used to filter key frames and extract multi-dimensional behavioral features of miners from the video stream data. The multi-dimensional features include the relative distance feature values of miners' limbs, the angle feature values of miners' limbs, the inter-frame limb velocity feature values of miners, and the inter-frame angular velocity feature values of miners' limbs. The behavior categories are manually labeled and organized into a local belt area miner behavior database for each mining area.
[0091] like Figure 1As shown, 15 miner skeleton nodes that can represent the miner's behavior are artificially selected, and the 15 skeleton nodes form 14 segments of rigid bodies that can represent the miner's body. 14 key angles of the miner's body are artificially selected. The 14 segments of the Euclidean distances of the skeleton nodes at both ends of the rigid body are calculated as the relative distance feature values of the miner's limbs. In order to eliminate the differences in the individual skeleton sizes of the miners, the above relative distances are normalized, that is, the relative position difference between all bone points is divided by the relative position difference from the center point of the shoulder to the center point of the hip.
[0092] The 14 angle eigenvalues formed between the 14 rigid bodies are calculated as the angle eigenvalues of the miner's limbs.
[0093] In order to reduce resource consumption and speed up model recognition, cosine similarity is used to extract key frames from video stream data, and only the key frames of miners' actions are retained. The similarity threshold is set to 0.9823, and the next frame of two adjacent frames exceeding the threshold is discarded.
[0094] The average speed of the same skeleton node of the miner between two frames is calculated as the inter-frame miner limb speed feature value. The spatial position of the miner skeleton node is known, and the time interval between two frames of the video stream data is fixed, so the time interval between the two frames can be known after extracting the key frame, so the speed of the miner skeleton node can be obtained by calculation.
[0095] The average speed of the miner's limb angle change between two frames is calculated as the inter-frame miner limb angular velocity characteristic value. The miner's limb angles in two adjacent frames are known, and the time interval between two frames of the video stream data is fixed. Therefore, the time interval between the two frames can be known after extracting the key frame, so the miner's limb angular velocity can be calculated.
[0096] Then the multi-dimensional feature combination of the miner behavior of the n-th frame miner behavior image can be expressed as x n ,Right now:
[0097] x n ={L n ,θ n ,V n ,ω n |n=1,2,...,N}
[0098] ={L n,1 ,L n,2 ,...,L n,14 ,θ n,1 ,θ n,2 ,...,θ n,14 ,V n,1 ,V n,2 ,...,V n,15 ,ω n,1 ,ω n,2 ,...,ω n,14|n=1,2,...,N}
[0099] Where N represents the total number of miner behavior image frames, L n,i represents the relative distance feature value of the limbs of the i-th miner in the n-th frame, θ n,i represents the eigenvalue of the limb angle of the i-th miner in the n-th frame, V n,i represents the eigenvalue of the miner’s limb velocity between the i-th frames in the n-th frame, ω n,i It represents the angular velocity characteristic value of the miner's limbs in the i-th frame in the n-th frame.
[0100] The mining belt area operation behavior characteristics database of the first mining area can be expressed as:
[0101]
[0102] Where N means that there are N frames of miners' belt area behavior video frames in the first mining area, T i 1 Represents the miner behavior category label of the i-th frame in the violation identification terminal 1.
[0103] Step 2: 10 mining areas are used as 10 mining area violation identification terminals to participate in constructive federated learning. Taking the first violation identification terminal as an example, the initialization of the local network model learning parameters includes: expected recognition error ε 1 =0.05, hidden layer parameter random configuration range Υ = {-1:1}, activation function g(.) selects Sigmoid (S curve) function g(x) = 1 / (1+exp(-x)), initial residual e0 = T 1 .
[0104] Taking the first terminal as an example, a local network model is constructed. Starting from the zero network, hidden layer nodes are gradually added. When the hidden layer nodes of the 100th local network model are generated, a set of hidden layer node parameters that obey Gaussian distribution are generated in the adjustable symmetric interval [-1,1] according to the belt area operation behavior characteristic database of miners in this mining area, that is, the input weights and bias Calculate the current hidden layer node output matrix of the local network model Current hidden layer output matrix Expanding the Input Matrix and output weights
[0105]
[0106] The superscript T represents the transpose of a matrix or a vector. represents the i-th dimension input feature of the local network model of the mining area violation identification terminal k, where represents the hidden layer output matrix of the 99 hidden layer nodes before the 100th hidden layer node in the violation identification terminal 1, represents the parameters of the 99 hidden nodes before the 100th hidden node determined by constructive federated learning, represents the hidden layer output matrix of the 100th hidden layer node in the violation identification terminal 1, Represents the 100th optimal hidden layer node parameter in the randomly generated violation identification terminal 1.
[0107] The variance of the current local network model in the violation identification terminal 1 The calculation formula is as follows:
[0108]
[0109] Where T 1 Represents the true behavior classification label of the miner behavior image in the violation identification terminal 1.
[0110] The violation identification terminal 1 transmits its randomly generated 100th hidden layer node parameters and the current local network model variance to the aggregation cloud.
[0111] Step 3: The aggregation cloud receives the hidden layer nodes randomly generated by 10 mining area violation identification terminals and the current variance of each local network model for adaptive parameter aggregation. The aggregation cloud constructs the aggregated hidden layer node parameters by weighted aggregation method. The calculation formula is as follows:
[0112]
[0113] in, Represents the 100th aggregate hidden layer node parameter after aggregation. represents the hidden node parameters randomly generated by the mining area violation identification terminal k, represents the adaptive aggregation parameter of the 100th hidden node of the violation identification terminal k, It represents the variance of the 100th hidden node of the local network model of the illegal identification terminal k.
[0114] The aggregation cloud transmits the constructed aggregation hidden layer node parameters to the local network model construction module of each mining area violation identification terminal.
[0115] Step 4: The first mining area violation identification terminal receives the aggregated hidden layer nodes, updates the parameters of the 100th newly added hidden layer node of its network model, and calculates the hidden layer output of the 100th hidden layer node of the local network model of the mining area violation identification terminal 1 using the following formula:
[0116]
[0117] in Represents the aggregate hidden layer node parameters determined by constructive federated learning;
[0118] The output weight matrix of the 100th hidden node of the local network model of the mining area violation identification terminal 1 is calculated using the following formula:
[0119]
[0120] The superscript + is the generalized inverse matrix of the matrix. Represents the expanded input matrix of the 100th hidden layer node of the local network model, Represents the hidden layer output matrix of the 100th hidden layer node of the local network model.
[0121] The output of the 100th hidden node of the local network model of the violation identification terminal 1 is calculated using the following formula:
[0122] Then the recognition error of the current local network model is obtained. If the recognition error of the current network model is less than the expected recognition error ε 1 , the terminal exits the constructive federated learning; if it is greater than the model's expected recognition error ε 1 , then return to step 2 and continue to randomly generate new hidden layer node parameters for the next round of federated learning.
[0123] Step 5: The mine violation identification terminal network completed by constructive federated learning can distinguish the miners' behavior categories according to the real-time behavioral characteristics of mining operations in the belt area.
[0124] Taking the mine violation identification terminal in the first mining area as an example, after exiting the constructed federated learning, the local network model is fixed. When in use, the human skeleton extraction tool openpose in the miner behavior feature acquisition module extracts the skeleton nodes in the real-time behavior video of the miners in the belt area, and sorts out the real-time multi-dimensional behavior characteristics of the miners in the belt area based on the skeleton nodes. The local network model in the mine violation identification terminal takes the real-time multi-dimensional behavior characteristics of the miners in the belt area as input, and calculates and outputs whether the current behavior of the miners in the belt area is in violation of the regulations.
[0125] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.
Claims
1. A mine personnel illegal action recognition system based on constructive federated learning, characterized in that: Including aggregation cloud and mining area violation identification terminal; The aggregation cloud includes a parameter transmission module and a network collaborative learning module; The mining area violation identification terminal includes a miner behavior feature acquisition module, a local network model construction module and a behavior recognition module; The miner behavior feature acquisition module obtains historical videos of miners' behavior in the belt area of each mining area through RGB cameras installed above the belt conveyors of more than two mining areas, uses human skeleton extraction tools to analyze and extract feature data of miners' behavior in the belt area from video stream data, annotates behavior categories and organizes them into local miners' behavior databases in each mining area; The local network model construction module is used to perform constructive federated learning. Each mining area acts as a mining area violation identification terminal and participates in constructive federated learning, which specifically includes: using the local miner behavior database of each mining area as the feature database, starting from a zero network, and gradually adding hidden layer nodes. Each time a new node is constructed in the local network model, a set of hidden layer node parameters are first randomly generated within a set interval, and the output weights and variances of the local network model are calculated according to the generated hidden layer node parameters. The randomly generated hidden layer node parameters and the current variance of the local network model are transmitted to the aggregation cloud through the parameter transmission module; The parameter transmission module is used for parameter transmission between the local network model construction module and the network collaborative learning module; The network collaborative learning module receives the latest hidden layer node parameters and variances randomly generated by each local network model and performs adaptive parameter aggregation to construct aggregated hidden layer nodes, and transmits the aggregated hidden layer nodes to each mining area violation identification terminal through a parameter transmission module; After receiving the aggregated hidden layer nodes, the mine area violation identification terminal updates the newly added hidden layer node parameters of each mine area violation identification terminal, and calculates the output weight of the local network model to obtain the belt area miner behavior classification error of the current mine area violation identification terminal. According to the settings of each mine area violation identification terminal, it is judged whether the belt area miner behavior classification error of the current mine area violation identification terminal meets the belt area miner behavior violation action recognition accuracy requirement in the mine area violation identification terminal. If so, the mine area violation identification terminal exits the constructive federated learning; otherwise, the local network model construction module continues to construct new hidden layer nodes for the next round of federated learning. After the mine violation identification terminal completes the federated learning, the behavior recognition module determines whether the miner's behavior is in violation of the regulations based on the real-time behavior characteristics of the mining operations in the belt area.
2. The system according to claim 1, characterized in that The local network model construction module uses a random vector function link network as the local network model of each mining area violation identification terminal, and k represents the kth violation identification terminal among all the violation identification terminals participating in the constructed federated learning. When the mining area violation identification terminal k starts to participate in the constructed federated learning, the local network model learning parameters are initialized, including: the expected tolerance ε k , hidden layer parameter random generation range Υ={-λ:+λ}, activation function g(.), initial residual e0=T k , where -λ is the lower limit of the random parameter generation interval, +λ is the upper limit of the random parameter generation interval, T k is the actual category label of the miner behavior feature data in the local belt area of the violation identification terminal k, ε k represents the expected tolerance set for the violation identification terminal k, the hidden layer parameter random generation range Υ and the activation function g(.) are set the same for all violation identification terminals.
3. The system according to claim 2, characterized in that When constructing a new hidden node L at the mining area violation identification terminal k, a set of hidden node parameters is randomly generated in the symmetric interval Υ={-λ:λ} that obeys the Gaussian distribution. Use the following formula to calculate the current hidden layer node output matrix of the local network model Current hidden layer output matrix Expanding the Input Matrix and output weights The superscript T represents the transpose of a matrix or a vector. represents the Mth dimension input feature of the local network model of the mining area violation identification terminal k, represents the parameters of the Lth hidden node constructed by the local network model of the illegal identification terminal k, represents the hidden layer output matrix of the Lth node calculated according to the parameters of the Lth hidden layer node constructed according to the local network model, represents the parameters of the L-1 hidden nodes before the Lth hidden node determined by constructive federated learning, represents the output matrix of the L-1 hidden layer nodes before the Lth hidden layer node, Represents the current hidden layer output matrix of the local network model, Represents the current expanded input matrix of the local network model, X k represents the belt area miner behavior database within the violation identification terminal k, and the superscript + indicates finding the generalized inverse of the matrix; The current output variance of the local network model for the violation identification terminal k is calculated using the following formula Where T k Represents the true label of the behavior classification of the miner behavior image in the violation identification terminal k.
4. The system according to claim 3, characterized in that The network collaborative learning module uses the following formula to construct the adaptive aggregation hidden layer nodes: in, represents the aggregate hidden layer node parameter after aggregation, K represents the total number of mining area violation identification terminals participating in constructive federated learning, represents the Lth hidden node parameter of the mining area violation identification terminal k, represents the adaptive aggregation parameter of the Lth hidden layer node of the violation identification terminal k, It represents the variance of the Lth hidden layer node of the local network model of the illegal identification terminal k.
5. The system according to claim 4, characterized in that The mining area violation identification terminal uses the following formula to calculate the hidden layer output of the Lth hidden layer node of the local network model: in represents the Lth aggregation hidden layer node parameter determined according to constructive federated learning; The local network model output weight matrix of the violation identification terminal is calculated using the following formula The superscript + is the generalized inverse matrix of the matrix. Represents the current expanded input matrix of the local network model, Represents the current hidden layer output matrix of the local network model, represents the output matrix of the lth hidden layer node of the local network model, represents the parameters of the lth hidden layer node determined by constructive federated learning, X k represents the belt area miner behavior dataset within the violation identification terminal k; The output of the local network model for the violation identification terminal k is calculated using the following formula
Citation Information
Patent Citations
Mine personnel monitoring system and method based on computer vision
CN109905675A
Mine personnel violation action recognition system
CN110738163A
A coal mine fully mechanized coal mining face operator violation and dangerous behavior early warning method
CN113111840A
Block chain-based participant-trusted federal learning method and device
CN113467927A
Miner illegal behavior knowledge extraction method based on entity and relation joint learning
CN114239574A
