Machine vision-based method for rapidly analyzing and identifying unsafe behaviors of underground personnel

Abstract

The invention discloses an underground personnel unsafe behavior rapid analysis and recognition method based on machine vision, and relates to the technical field of intelligent safety monitoring, and the method comprises the steps: configuring an environment sensing unit to monitor underground environment parameters in real time, and dynamically adjusting the threshold parameter of an image recognition and posture estimation model according to the monitoring data; a behavior prediction model is constructed, the behavior prediction model is input based on the time sequence data to predict the future unsafe behavior trend of the underground personnel, and early warning is achieved; visual, sound and other non-visual sensor data are integrated, a multi-modal fusion recognition system is constructed, and the recognition capability in a complex environment is improved through synchronous processing and fusion of modal data; the real-time behaviors of the underground personnel are comprehensively analyzed, and once unsafe behaviors are recognized, an early warning mechanism is started immediately; model parameters are dynamically adjusted through the environment sensing unit, and the adaptability of the system to environment changes is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent safety monitoring, and in particular to a method for quickly analyzing and identifying unsafe behaviors of underground personnel based on machine vision. Background Art

[0002] In the mine operation environment, ensuring personnel safety is always the top priority. In recent years, with the rapid development of the Internet of Things, big data and artificial intelligence technologies, intelligent safety monitoring systems have gradually become a key means to improve the safety of underground operations. Among them, machine vision technology has shown great potential in underground personnel behavior monitoring due to its non-contact, high-precision and real-time advantages. In the early days, machine vision was mainly used in product inspection and quality control in industrial automation. In recent years, its application scope has expanded to public safety and medical health. In particular, it has made significant progress in target detection and behavior recognition in complex environments. However, the application of machine vision to the safety behavior analysis of underground personnel still faces many challenges. Summary of the invention

[0003] In view of the above existing problems, the present invention is proposed.

[0004] Therefore, the present invention provides a method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision to solve the problems of complex and changeable underground environment, poor lighting conditions, and high dust concentration. These factors seriously affect the quality of images and the accuracy of recognition. Secondly, traditional single visual recognition methods are difficult to effectively deal with changeable behavior patterns and emergencies, especially in predicting potential unsafe behaviors. Existing systems ignore the impact of environmental factors on behavior and lack the ability to dynamically adjust recognition strategies, resulting in a decrease in recognition rate under specific environments. In addition, most existing technologies focus on single modal information and fail to fully utilize the advantages of multimodal data, which limits the robustness and adaptability of the system. Finally, the lack and imperfection of an immediate warning mechanism means that even if unsafe behaviors are identified, effective measures cannot be taken in a timely manner, reducing the practicality of the system.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision, which comprises:

[0007] Configure the environmental perception unit to monitor the underground environmental parameters in real time, and dynamically adjust the threshold parameters of the image recognition and posture estimation models according to the monitoring data;

[0008] Build a behavior prediction model, and predict the future unsafe behavior trends of underground personnel based on time series data input into the behavior prediction model to achieve early warning;

[0009] Integrate visual, sound and other non-visual sensor data to build a multi-modal fusion recognition system, and improve recognition capabilities in complex environments by synchronously processing and fusing data from various modalities;

[0010] Conduct a comprehensive analysis of the real-time behavior of underground personnel, and immediately activate the early warning mechanism once unsafe behavior is identified.

[0011] As a preferred solution of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision of the present invention, wherein: the configuration environment perception unit monitors the underground environmental parameters in real time, and the specific steps are:

[0012] Environmental sensing units are deployed at key locations underground to collect light intensity and dust concentration data in real time to form original environmental parameter data;

[0013] Convert raw environmental parameter data into a format that can be processed by the behavior prediction model;

[0014] Exploratory data analysis (EDA) was performed on the original environmental parameter data to check the statistical indicators of the minimum, maximum, mean and median of light intensity L and dust concentration D;

[0015] Based on the minimum-maximum normalization method, a quantization function Q is constructed to map the original environmental parameter data to the [0,1] interval;

[0016] For light intensity, the quantization function expression is:

[0017]

[0018] Among them, Q(L) represents the light intensity after quantization processing, L represents the original light intensity value, and 1000 represents the maximum value of the original light intensity data range;

[0019] For dust concentration, the quantitative function expression is:

[0020]

[0021] Where Q(D) represents the dust concentration value after quantization, D represents the original dust concentration value, and D max and D min Respectively represent the maximum and minimum measured values ​​of dust concentration.

[0022] As a preferred solution of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision described in the present invention, in addition to real-time monitoring of light intensity and dust concentration, the environmental perception unit also integrates a temperature and humidity sensor to monitor the temperature and humidity of the underground environment, and incorporates the temperature and humidity data into the environmental parameter Et The specific steps are:

[0023] Add temperature and humidity sensors to the environmental sensing unit to collect temperature and humidity data of the underground environment in real time;

[0024] The original temperature and humidity data are converted into normalized values, and the quantization function expression is:

[0025] Q(T)=(TT min ) / (T max -T min );

[0026] Q(H)=(HH min ) / (H max -H min );

[0027] Among them, Q(T) and Q(H) represent the normalized temperature and humidity values, respectively. min , T max , H min , H max are the minimum and maximum measured values ​​of temperature and humidity, respectively;

[0028] The normalized temperature and humidity data Q(T) and Q(H) are added to the environmental parameter vector Et, and the expanded environmental parameter vector becomes Et=[Q(L), Q(D), Q(T), Q(H)].

[0029] As a preferred solution of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision of the present invention, the threshold parameters of the image recognition and posture estimation model are dynamically adjusted according to the monitoring data, and the specific steps are as follows:

[0030] The threshold function is constructed based on the quantified environmental parameter data, and the adjusted threshold under the current environment is calculated. The expression is:

[0031] A=θ0+f1(Q(L))+f2(Q(D))+fi(Q(L),Q(D))+fT(Q(T))+f h (Q(H));

[0032] Among them, A represents the adjusted threshold function, θ0 represents the specific value of the basic threshold, f1 and f2 represent the illumination influence function and dust influence function respectively, and f i Represents the interaction adjustment term between light and dust, f T represents the temperature influence function, f H Represents the humidity influence function.

[0033] As a preferred solution of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision of the present invention, the specific steps of constructing a behavior prediction model are as follows:

[0034] Define the state space, which includes all the behavior states of underground personnel. Each state in the state space is composed of the personnel's position, posture angle, action time, action trajectory, current light intensity and dust concentration;

[0035] Define the action space, which contains all unsafe behaviors and safe behaviors. Each element in the action space corresponds to a specific action.

[0036] Set a reward function. The reward function gives positive and negative rewards based on whether the predicted behavior is correct or not. The reward function encourages the model to learn the correct unsafe behavior prediction mode and avoid misidentification and omission.

[0037] When the model successfully predicts an impending unsafe behavior, it gives a positive reward;

[0038] When the prediction is wrong and the warning is not issued in time, negative penalties will be imposed;

[0039] A behavior prediction model is constructed using a deep reinforcement learning algorithm. The model learns the optimal strategy by interacting with the environment. The optimal strategy includes taking specific actions in a given state to maximize the expected value of future rewards.

[0040] As a preferred solution of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision of the present invention, wherein: the behavior prediction model based on time series data input predicts the future unsafe behavior trend of underground personnel and realizes early warning, and the specific steps are:

[0041] Collect samples of unsafe behaviors from historical surveillance videos, accident reports, and field observations;

[0042] Sort the historical data into time series, and record the behavior status of specific underground personnel at each time point, including but not limited to location, action characteristics and environmental parameter data;

[0043] Extract behavioral features F from time series data t ;

[0044] Convert behavioral characteristics into numerical form;

[0045] Arrange the quantified features in time series, with each row being the behavioral feature vector of a time point, to form the original feature matrix X, where the number of rows is the number of time points and the number of columns is the number of features;

[0046] Calculate the covariance matrix C of the behavior feature matrix X;

[0047] Perform eigendecomposition on the covariance matrix C to obtain eigenvalues ​​λ1, λ2,...,λ d and the corresponding eigenvectors v1, v2, ..., v d ;

[0048] Arrange the eigenvalues ​​from large to small;

[0049] Select the eigenvectors corresponding to the first k largest eigenvalues, where k needs to be determined based on the cumulative contribution rate, expressed as:

[0050]

[0051] Where d represents the total number of eigenvalues, k is the number of principal components we want to select, and λ i represents the jth eigenvalue, represents the sum of all eigenvalues;

[0052] Use the selected k eigenvectors to construct the transformation matrix V = [v1, v2, ..., v k ], and then transform the original feature matrix X to obtain the reduced dimension matrix X PCA =XV;

[0053] The real-time data is processed and input into the behavior prediction model to predict the future behavior trends of underground personnel, and the early warning mechanism is triggered based on the prediction results.

[0054] As a preferred solution of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision of the present invention, wherein: the integrated visual, sound and other non-visual sensor data are used to construct a multimodal fusion recognition system, and the recognition ability in a complex environment is improved by synchronously processing and fusing each modal data. The specific steps are as follows:

[0055] Independent preprocessing is performed for different modal data;

[0056] For visual data, convolutional neural network (CNN) is used to extract spatiotemporal features in images.

[0057] The sound data is converted into a spectrum through short-time Fourier transform (STFT), and then the sound features are extracted using the deep learning model CRNN.

[0058] For non-visual sensor data, it is directly quantified into numerical features;

[0059] The multimodal fusion function is constructed based on the weighted summation of different modal data. The expression is:

[0060]

[0061] Among them, Fi represents the eigenvector of the i-th mode, w i Represents the weight value of the i-th mode.

[0062] As a preferred solution of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision of the present invention, the specific steps are: the real-time behavior of underground personnel is comprehensively analyzed, and once unsafe behaviors are identified, an early warning mechanism is immediately activated. The specific steps are:

[0063] Based on real-time data input, the formula is dynamically adjusted to calculate the threshold function A of the image recognition and posture estimation model that adapts to the current environment;

[0064] According to the time series data F t and environmental parameter E t Input the behavior prediction model to predict the unsafe behavior trend of underground personnel and adjust the strategy through the reward function;

[0065] Input the visual, sound and other non-visual sensor data into the multimodal fusion function to obtain the fused feature vector FF;

[0066] The fused feature vector FF is input into the recurrent neural network RNN ​​for behavior recognition and classification to obtain the predicted behavior category probability distribution y θ ;

[0067] Combined with the current environmental parameter data, a dynamic threshold function is constructed, the expression is:

[0068] V(k L , k D ,γ)=θ0+k L (1-Q(L)) γ +k D ·Q(D) γ ;

[0069] Among them, V represents the threshold function after dynamic adjustment, kL represents the adjustment coefficient related to light intensity, kD represents the adjustment coefficient related to dust concentration, γ represents the parameter affecting the steepness of the function curve, and θ0

[0070] represents the basic threshold;

[0071] Compare the classification result yθ after multimodal fusion with the dynamic threshold V;

[0072] When the highest classification probability value is greater than or equal to the threshold value V under the current environmental conditions, it is determined that unsafe behavior exists and the early warning mechanism is immediately activated;

[0073] On the contrary, when the highest classification probability value is less than the threshold V under the current environmental conditions, it is determined that the current behavior is within a safe range, no alarm is required, and monitoring is maintained.

[0074] In a second aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as described in the first aspect of the present invention.

[0075] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, wherein: when the computer program is executed by a processor, it implements any step of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as described in the first aspect of the present invention.

[0076] The beneficial effects of the present invention are as follows: by dynamically adjusting the model parameters through the environmental perception unit, the adaptability of the system to environmental changes is enhanced; by constructing a behavior prediction model, the prediction of unsafe behavior trends is realized, and the timeliness of early warning is improved; through multimodal data fusion, the accuracy and robustness of recognition are improved. The present invention belongs to the field of intelligent safety monitoring technology, and is particularly suitable for underground working environments of coal mines and metal mines, and is of great significance for improving the safety of underground operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0078] Figure 1 This is a flow chart of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision in Example 1.

[0079] Figure 2 This is an interval diagram based on current environmental conditions in Example 1. DETAILED DESCRIPTION

[0080] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0081] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0082] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0083] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision, comprising the following steps:

[0084] S1 is equipped with an environmental perception unit to monitor underground environmental parameters in real time and dynamically adjust the threshold parameters of the image recognition and posture estimation models based on the monitoring data;

[0085] Environmental sensing units are deployed at key locations underground to collect light intensity and dust concentration data in real time, ensuring the real-time and accuracy of the data and forming original environmental parameter data;

[0086] Convert raw environmental parameter data into a format that can be processed by the behavior prediction model;

[0087] Exploratory data analysis (EDA) was performed on the original environmental parameter data to check the statistical indicators of the minimum, maximum, mean and median of light intensity L and dust concentration D;

[0088] A quantization function Q is constructed based on the minimum-maximum normalization method to map the original environmental parameter data to the interval [0, 1];

[0089] For light intensity, the quantization function expression is:

[0090]

[0091] Among them, Q(L) represents the light intensity after quantization processing, L represents the original light intensity value, and 1000 represents the maximum value of the original light intensity data range;

[0092] For dust concentration, the quantitative function expression is:

[0093]

[0094] Where Q(D) represents the dust concentration value after quantization, D represents the original dust concentration value, and D max and D min Respectively represent the maximum and minimum measured values ​​of dust concentration;

[0095] Environmental perception unit, including light intensity sensor and dust concentration detector;

[0096] The measurement range of light intensity L is 0lux to 1000lux;

[0097] In addition to real-time monitoring of light intensity and dust concentration, the environmental sensing unit also integrates temperature and humidity sensors to monitor the temperature and humidity of the underground environment and incorporates the temperature and humidity data into the environmental parameter E t The specific steps are:

[0098] Add temperature and humidity sensors to the environmental sensing unit to collect temperature and humidity data of the underground environment in real time;

[0099] The original temperature and humidity data are converted into normalized values, and the quantization function expression is:

[0100] Q(T)=(TT min ) / (T max -T min );

[0101] Q(H)=(HH min ) / (H max -H min );

[0102] Among them, Q(T) and Q(H) represent the normalized temperature and humidity values, respectively. min , T max , H min , H max are the minimum and maximum measured values ​​of temperature and humidity, respectively;

[0103] The normalized temperature and humidity data Q(T) and Q(H) are added to the environmental parameter vector Et, and the expanded environmental parameter vector becomes Et=[Q(L), Q(D), Q(T), Q(H)], so as to more comprehensively reflect the underground environmental conditions;

[0104] Based on the quantified environmental parameter data, a threshold function is constructed to calculate the adjusted threshold in the current environment to dynamically adjust the threshold parameter θ of the image recognition and posture estimation model. This function needs to consider the combined effect of illumination and dust on image recognition accuracy. The expression is:

[0105] A=θ0+f1(Q(L))+f2(Q(D))+fi(Q(L),Q(D))+fT(Q(T))+fh (Q(H));

[0106] Where A represents the adjusted threshold function, which is used to adapt to the changing underground environmental conditions and optimize the performance of the image recognition and posture estimation model. θ0 represents the specific value of the basic threshold. f1 and f2 represent the illumination influence function and dust influence function respectively. i Represents the interaction adjustment term between light and dust, f T represents the temperature influence function, f H represents the humidity influence function;

[0107] The illumination effect function f1 directly affects the clarity and contrast of the image. When the illumination is weak, the image may be blurred, resulting in increased difficulty in recognition. f1(Q(L)) is a function that increases as the illumination intensity decreases to compensate for the difficulty in recognition in low-light environments. The expression is:

[0108] f1(Q(L))=k1*(1-Q(L));

[0109] Among them, k1 is a coefficient that is negatively correlated with light, indicating that the weaker the light, the greater the adjustment amount;

[0110] The dust influence function f2 directly affects the clarity and contrast of the image. When the dust concentration is high, the particles suspended in the air will scatter the light, making it difficult to identify the key feature points. The threshold of f2(Q(D)) needs to be adjusted as the dust concentration increases. The expression is:

[0111] f2(Q(D))=k2*Q(D) p ;

[0112] Among them, k2 represents the coefficient positively correlated with dust concentration, and p is an exponent greater than 1, which is used to represent the nonlinear growth of the impact of increasing dust concentration on identification;

[0113] Light and dust interaction adjustment factor f i , directly affects the recognition effect of light and dust, f i (Q(L), Q(D)) takes into account the interaction between the two to adjust the threshold more accurately, and the expression is:

[0114] f i (Q(L),Q(D))=k i *(1-Q(L))*Q(D);

[0115] Among them, k i represents the combined effect of reduced sunlight and increased dust concentration;

[0116] Temperature influence function f T,Temperature changes affect the performance of the camera, especially in extremely high or low temperature environments, ,the imaging quality and response speed of the camera will be affected, ,which will indirectly affect the accuracy of image recognition.,Considering that high temperature is usually more likely to cause equipment failure than ,low temperature,f T As the temperature increases, the function increases, and the expression is:

[0117] f T (Q(T)) = k T *(Q(T)-Q o );

[0118] Among them, kT represents the coefficient related to temperature, reflecting the adjustment amount when the temperature deviates from the optimal working temperature Q. Q. represents the normalized value of the optimal working temperature of the camera, which is close to 1. When the actual temperature is higher or lower than the optimal temperature, f T As the value increases, the threshold is adjusted to accommodate the recognition difficulties caused by temperature changes;

[0119] Humidity influence function f H The influence of humidity on image recognition is mainly reflected in two aspects: first, in a high humidity environment, water droplets are easily condensed on the lens surface, affecting the image clarity; second, humidity may affect the stability and life of electronic equipment. Therefore, fH is a function that adjusts the threshold as humidity increases to cope with the recognition challenges of a high humidity environment. The expression is:

[0120] f H (Q(H)) = k H *(Q(H)-Q oH ) q ;

[0121] Among them, kH represents the coefficient related to humidity, which represents the basic adjustment strength of humidity on recognition, Q. H It represents the normalized value of the optimal working range of humidity. q represents an exponent greater than 1, which is used to represent the nonlinear growth of the influence of humidity on recognition, especially in high humidity conditions. When the humidity deviates from the optimal range, the value of fH increases, prompting the model to adjust the recognition threshold when the humidity has a greater impact.

[0122] S2 builds a behavior prediction model, which predicts the future unsafe behavior trends of underground personnel based on time series data input into the behavior prediction model to achieve early warning;

[0123] Define the state space, which includes all the behavior states of underground personnel. Each state in the state space is composed of the personnel's position, posture angle, action time, action trajectory, current light intensity and dust concentration;

[0124] Define the action space, which contains all unsafe behaviors and safe behaviors. Each element in the action space corresponds to a specific action.

[0125] Set a reward function. The reward function gives positive and negative rewards based on whether the predicted behavior is correct or not. The reward function encourages the model to learn the correct unsafe behavior prediction mode and avoid misidentification and omission.

[0126] When the model successfully predicts an impending unsafe behavior, it gives a positive reward;

[0127] When the prediction is wrong and the warning is not given in time, negative penalties are given. This step is crucial for using reinforcement learning algorithms because it guides the model to obtain the best long-term benefits when taking different actions in different states;

[0128] A behavior prediction model is built using a deep reinforcement learning algorithm. The model learns the optimal strategy by interacting with the environment. The optimal strategy involves taking specific actions in a given state to maximize the expected value of future rewards.

[0129] The historical data is input into the behavior prediction model to complete the training. The state is updated in real time according to the time series data Ft, including the human posture characteristics, action trajectory and environmental parameter weighting factors obtained through PCA processing. After each interaction, the model adjusts its strategy according to the reward function;

[0130] Specific actions include, but are not limited to, not wearing a helmet, sleeping on duty, climbing guardrails, illegally riding on a vehicle, illegally riding on a belt, and working in a dangerous area;

[0131] The behavior prediction model is expressed as:

[0132]

[0133] in, represents the predicted behavior state of underground personnel at time t+1. This is a vector representing the probability distribution of the predicted behavior. θ represents the behavior prediction model based on deep reinforcement learning, which depends on the parameters θ, F t represents the behavior feature vector extracted at time point t, including human posture features, motion trajectory features and behavior duration processed by PCA, E t Represents the environmental parameter vector at time point t, which is mainly composed of quantized light intensity and dust concentration, and is weighted by the environmental parameter weighting factor W env Consider the impact of environmental factors on behavior. is the reward function, y t +1 indicates the behavior state at the next time point;

[0134] Collect samples of unsafe behaviors from historical surveillance videos, accident reports, and field observations;

[0135] Sort the historical data into time series, and record the behavior status of specific underground personnel at each time point, including but not limited to location, action characteristics and environmental parameter data;

[0136] Extract behavioral features F from time series data t , the posture feature is based on the coordinates of key points, the action trajectory records the changes of key points over time, and the action time marks the duration of the behavior. The expression is:

[0137] F t =f PCA (T p，t , T t，t , D t ; W env (Q(L), Q(D)));

[0138] Among them, F t represents the behavior feature vector extracted at time point t, f PCA It means that the most representative feature vector under the current environmental conditions is generated through PCA dimensionality reduction processing, T p , t represents the human posture feature at time t, T t , t represents the motion trajectory characteristics at the same time point. By calculating the difference in displacement over time, the dynamic changes of individual actions can be quantified. t Indicates the duration of the behavior, W env represents the weighting factor of environmental parameters;

[0139] The behavior features are converted into numerical form, the posture features can be converted into posture angles, the action trajectory is calculated by the difference between the trajectory points, and the action time is directly quantified into seconds;

[0140] Arrange the quantified features in time series, with each row being the behavioral feature vector of a time point, to form the original feature matrix X, where the number of rows is the number of time points and the number of columns is the number of features;

[0141] Calculate the covariance matrix C of the behavior feature matrix X. The covariance matrix reflects the relationship between features and is expressed as:

[0142]

[0143] Where n represents the number of time points, and XT represents the transpose of X;

[0144] Perform eigendecomposition on the covariance matrix C to obtain eigenvalues ​​λ1, λ2,...,λ d and the corresponding eigenvectors v1, v2, ..., v d;

[0145] Arrange the eigenvalues ​​from large to small;

[0146] Select the eigenvectors corresponding to the first k largest eigenvalues, where k needs to be determined based on the cumulative contribution rate, expressed as:

[0147]

[0148] Among them, d represents the total number of eigenvalues, k is the number of principal components we want to select, which needs to meet the requirement that the cumulative contribution rate reaches the preset threshold, and λ i represents the jth eigenvalue, represents the sum of all eigenvalues;

[0149] Use the selected k eigenvectors to construct the transformation matrix V = [v1, v2, ..., v k ], and then transform the original feature matrix X to obtain the reduced dimension matrix X PCA =XV;

[0150] The real-time data is processed and input into the behavior prediction model to predict the future behavior trend of underground personnel and trigger the early warning mechanism based on the prediction results;

[0151] Samples of unsafe behaviors include, but are not limited to, not wearing a helmet, sleeping on duty, climbing guardrails, illegally riding on a vehicle, illegally riding on a belt, and unsafe behaviors such as working in dangerous areas;

[0152] Behavioral characteristics, including human posture characteristics, movement trajectory and movement time;

[0153] The covariance matrix C is subjected to eigendecomposition. The goal of eigenvalue decomposition is to find a set of eigenvalues ​​and corresponding eigenvectors, expressed as:

[0154] Cv i =λ i v i ;

[0155] Among them, v i represents the i-th eigenvector of the covariance matrix C, λ i represents the i-th eigenvalue of the covariance matrix C;

[0156] Based on the practical experience of using statistics and data analysis of historical data, the value of the cumulative contribution rate is 95%;

[0157] Early warning mechanisms, including automatic emergency measures such as sound and light alarms and instant messaging notifications, prevent potential unsafe incidents from occurring.

[0158] S3 integrates visual, sound and other non-visual sensor data to build a multi-modal fusion recognition system. By synchronously processing and fusing data from each modality, it improves recognition capabilities in complex environments.

[0159] Independent pre-processing of different modal data, including visual, acoustic and other non-visual sensor data;

[0160] For visual data, convolutional neural network (CNN) is used to extract spatiotemporal features in images.

[0161] The sound data is converted into a spectrum through short-time Fourier transform (STFT), and then the sound features are extracted using the deep learning model CRNN.

[0162] For non-visual sensor data, including temperature and humidity, they are directly quantified and processed into numerical features;

[0163] The multimodal fusion function is constructed based on the weighted summation of different modal data. The expression is:

[0164]

[0165] Among them, F i represents the eigenvector of the i-th mode, w i Represents the weight value of the i-th mode. This function takes into account the impact of light and dust concentration on the recognition efficiency of different modes.

[0166] S4 conducts comprehensive analysis of the real-time behavior of underground personnel and immediately activates the early warning mechanism once unsafe behavior is identified;

[0167] Based on real-time data input, the formula is dynamically adjusted to calculate the threshold function A of the image recognition and posture estimation model that adapts to the current environment;

[0168] According to the time series data F t and environmental parameter E t Input the behavior prediction model to predict the unsafe behavior trend of underground personnel, and adjust the strategy through the reward function to improve the prediction accuracy. Construct a recursive neural network RNN ​​based on sequence modeling and time series analysis;

[0169] Input the visual, sound and other non-visual sensor data into the multimodal fusion function to obtain the fused feature vector FF;

[0170] The fused feature vector FF is input into the recurrent neural network RNN ​​for behavior recognition and classification to obtain the predicted behavior category probability distribution y θ ;

[0171] Combined with the current environmental parameter data, a dynamic threshold function is constructed. The dynamic threshold function can dynamically adjust the recognition threshold according to the light intensity and dust concentration. The expression is:

[0172] V(k L , k D ,γ)=θ0+k L (1-Q(L)) γ +k D ·Q(D) γ ;

[0173] Among them, V represents the threshold function after dynamic adjustment, kL represents the adjustment coefficient related to light intensity, which determines the influence of light intensity change on threshold adjustment, kD represents the adjustment coefficient related to dust concentration, which controls the amplitude of threshold adjustment caused by dust concentration change, γ represents the parameter affecting the steepness of the function curve, and θ0 represents the basic threshold;

[0174] Compare the classification result yθ after multimodal fusion with the dynamic threshold V;

[0175] When the highest classification probability value is greater than or equal to the threshold value V under the current environmental conditions, it is determined that unsafe behavior exists and the early warning mechanism is immediately activated;

[0176] On the contrary, when the highest classification probability value is less than the threshold V under the current environmental conditions, it is determined that the current behavior is within the safe range, no alarm is required, and monitoring is maintained;

[0177] Recurrent neural network RNN, the expression of a single-layer RNN is:

[0178] h t =σ(W h h t-1 +W x x t +b h );

[0179] Among them, h t represents the hidden state vector at time step t, σ represents the nonlinear activation function tanh, W h Represents the weight matrix from hidden state to hidden state, W x represents the weight matrix input to the hidden state, b h represents the hidden layer bias vector;

[0180] Recurrent neural network RNN, the output layer expression is:

[0181]

[0182] Among them, y t θrepresents the output layer of time step t, i.e., the classification result, θ represents the set of model parameters, W y represents the weight matrix from the hidden state to the output layer, b y Represents the output layer bias vector.

[0183] This embodiment also provides a computer device, which is suitable for the method of rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the method of rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as proposed in the above embodiment.

[0184] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0185] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for quickly analyzing and identifying unsafe behaviors of underground personnel based on machine vision as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, disk or optical disk.

[0186] In summary, the present invention achieves the following advantages: the environmental perception unit dynamically adjusts the model parameters to enhance the system's adaptability to environmental changes; the behavior prediction model is constructed to achieve the prediction of unsafe behavior trends and improve the timeliness of early warning; and the multimodal data fusion improves the accuracy and robustness of recognition. The present invention belongs to the field of intelligent safety monitoring technology, and is particularly suitable for underground working environments in coal mines and metal mines, and is of great significance for improving the safety of underground operations.

[0187] Example 2

[0188] Referring to Table 1, which is the second embodiment of the present invention, in order to further verify the advancement of the present invention, experimental simulation data of a method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision are provided.

[0189] In order to verify the effectiveness of the present invention, a series of experiments were designed to compare the performance of identifying unsafe behaviors of personnel in complex underground environments between the traditional method and the present invention. The experiments were conducted in a simulated mine environment, and four key locations were set up. Each location was equipped with an environmental perception unit, including light intensity, dust concentration, temperature and humidity sensors. The experiment lasted for two weeks, and continuous data was recorded for 24 hours every day. During this period, multiple staff members were arranged to perform a series of pre-set behaviors, including safe and unsafe behaviors, to ensure the diversity and richness of the data.

[0190] During the experiment, the data collected by the environmental perception unit was first preprocessed, including the conversion, normalization and statistical analysis of the original data. Subsequently, a threshold function was constructed based on the quantified environmental parameters, and the threshold parameters of the image recognition and posture estimation models were dynamically adjusted. At the same time, a behavior prediction model was constructed, and the optimal strategy was learned through a deep reinforcement learning algorithm to predict the future unsafe behavior trends of underground personnel. In addition, visual, sound and other non-visual sensor data were integrated to construct a multimodal fusion recognition system, which improved the recognition ability in complex environments. Finally, a comprehensive analysis of the real-time behavior of underground personnel was carried out, and once unsafe behavior was identified, the early warning mechanism was immediately activated.

[0191] The details are shown in Table 1:

[0192] Table 1 Experimental record table

[0193]

[0194]

[0195] Through comparative analysis, it can be clearly seen that the present invention shows significant advantages in identifying unsafe behaviors of underground personnel. On all test dates, the prediction accuracy of the present invention is always maintained between 92.1% and 96.3%, which is much higher than the 83.7% to 87.5% of the traditional method. Especially under the conditions of low light intensity (such as 2023-03-04) and high dust concentration (such as 2023-03-02), the present invention can still maintain a high recognition accuracy, which is due to the real-time monitoring and dynamic adjustment of environmental parameters by the environmental perception unit, and the fusion recognition of multimodal data.

[0196] Warning response time is also an important indicator for measuring the performance of the recognition system. Experimental data show that the warning response time of the present invention is between 1.2 and 1.4 seconds on average. However, due to the lack of dynamic adjustment and multimodal fusion, the response time of traditional methods is usually longer. Rapid warning response is crucial to preventing unsafe behaviors from happening in a timely manner and helps reduce potential accident risks.

[0197] In summary, the present invention not only surpasses the traditional methods in recognition accuracy, but also performs well in early warning response speed, which fully demonstrates the innovation and advantages of the present invention in the rapid analysis and identification of unsafe behaviors of personnel in complex underground environments.

[0198] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision, characterized by: include, Configure the environmental perception unit to monitor the underground environmental parameters in real time, and dynamically adjust the threshold parameters of the image recognition and posture estimation models according to the monitoring data; Build a behavior prediction model, and predict the future unsafe behavior trends of underground personnel based on time series data input into the behavior prediction model to achieve early warning; Integrate visual, sound and other non-visual sensor data to build a multi-modal fusion recognition system, and improve recognition capabilities in complex environments by synchronously processing and fusing data from various modalities; Conduct a comprehensive analysis of the real-time behavior of underground personnel, and immediately activate the early warning mechanism once unsafe behavior is identified.

2. The method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as claimed in claim 1, characterized in that: The configuration of the environment sensing unit to monitor underground environmental parameters in real time includes the following specific steps: Environmental sensing units are deployed at key locations underground to collect light intensity and dust concentration data in real time to form original environmental parameter data; Convert raw environmental parameter data into a format that can be processed by the behavior prediction model; Exploratory data analysis (EDA) was performed on the original environmental parameter data to check the statistical indicators of the minimum, maximum, mean and median of light intensity L and dust concentration D; Based on the minimum-maximum normalization method, a quantization function Q is constructed to map the original environmental parameter data to the [0,1] interval; For light intensity, the quantization function expression is: Among them, Q(L) represents the light intensity after quantization processing, L represents the original light intensity value, and 1000 represents the maximum value of the original light intensity data range; For dust concentration, the quantitative function expression is: Where Q(D) represents the dust concentration value after quantization, D represents the original dust concentration value, and D max and D min Respectively represent the maximum and minimum measured values ​​of dust concentration.

3. The method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as claimed in claim 2, characterized in that: In addition to real-time monitoring of light intensity and dust concentration, the environmental sensing unit also integrates temperature and humidity sensors to monitor the temperature and humidity of the underground environment and incorporates the temperature and humidity data into the environmental parameter E t The specific steps are: Add temperature and humidity sensors to the environmental sensing unit to collect temperature and humidity data of the underground environment in real time; The original temperature and humidity data are converted into normalized values, and the quantization function expression is: Q(T)=(T-T min ) / (T max -T min ); Q(H)=(H-H min ) / (H max -H min ); Among them, Q(T) and Q(H) represent the normalized temperature and humidity values, respectively. min , T max , H min , H max are the minimum and maximum measured values ​​of temperature and humidity, respectively; The normalized temperature and humidity data Q(T) and Q(H) are added to the environmental parameter vector Et, and the expanded environmental parameter vector becomes Et=[Q(L), Q(D), Q(T), Q(H)].

4. The method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as claimed in claim 3, characterized in that: The specific steps of dynamically adjusting the threshold parameters of the image recognition and posture estimation model according to the monitoring data are as follows: The threshold function is constructed based on the quantified environmental parameter data, and the adjusted threshold under the current environment is calculated. The expression is: A=θ0+f1(Q(L))+f2(Q(D))+f i (Q(L),Q(D))+f T (Q(T))+f H (Q(H)); Among them, A represents the adjusted threshold function, θ0 represents the specific value of the basic threshold, f1 and f2 represent the illumination influence function and dust influence function respectively, and f i Represents the interaction adjustment term between light and dust, f T represents the temperature influence function, f H Represents the humidity influence function.

5. The method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as claimed in claim 4, characterized in that: The specific steps of constructing the behavior prediction model are as follows: Define the state space, which includes all the behavior states of underground personnel. Each state in the state space is composed of the personnel's position, posture angle, action time, action trajectory, current light intensity and dust concentration; Define the action space, which contains all unsafe behaviors and safe behaviors. Each element in the action space corresponds to a specific action. Set a reward function. The reward function gives positive and negative rewards based on whether the predicted behavior is correct or not. The reward function encourages the model to learn the correct unsafe behavior prediction mode and avoid misidentification and omission. When the model successfully predicts an impending unsafe behavior, it gives a positive reward; When the prediction is wrong and the warning is not issued in time, negative penalties will be imposed; A behavior prediction model is constructed using a deep reinforcement learning algorithm. The model learns the optimal strategy by interacting with the environment. The optimal strategy includes taking specific actions in a given state to maximize the expected value of future rewards.

6. The method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as claimed in claim 5, characterized in that: The behavior prediction model based on time series data input predicts the future unsafe behavior trend of underground personnel and realizes early warning. The specific steps are: Collect samples of unsafe behaviors from historical surveillance videos, accident reports, and field observations; Sort the historical data into time series, and record the behavior status of specific underground personnel at each time point, including but not limited to location, action characteristics and environmental parameter data; Extract behavioral features F from time series data t ; Convert behavioral characteristics into numerical form; Arrange the quantified features in time series, with each row being the behavioral feature vector of a time point, to form the original feature matrix X, where the number of rows is the number of time points and the number of columns is the number of features; Calculate the covariance matrix C of the behavior feature matrix X; Perform eigendecomposition on the covariance matrix C and obtain the eigenvalues ​​λ1,λ2,...,λ d and the corresponding eigenvectors v1,v2,...,v d ; Arrange the eigenvalues ​​from large to small; Select the eigenvectors corresponding to the first k largest eigenvalues, where k needs to be determined based on the cumulative contribution rate, expressed as: Where d represents the total number of eigenvalues, k is the number of principal components we want to select, and λ j represents the jth eigenvalue, represents the sum of all eigenvalues; Use the selected k eigenvectors to construct the transformation matrix V = [v1, v2, ..., v k ], and then transform the original feature matrix X to obtain the reduced dimension matrix X PCA =XV; The real-time data is processed and input into the behavior prediction model to predict the future behavior trends of underground personnel, and the early warning mechanism is triggered based on the prediction results.

7. The method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as claimed in claim 6, characterized in that: The above-mentioned integration of visual, sound and other non-visual sensor data to build a multi-modal fusion recognition system improves the recognition capability in complex environments by synchronously processing and fusing data from each modality. The specific steps are as follows: Independent preprocessing is performed for different modal data; For visual data, convolutional neural network (CNN) is used to extract spatiotemporal features in images. The sound data is converted into a spectrum through short-time Fourier transform (STFT), and then the sound features are extracted using the deep learning model CRNN. For non-visual sensor data, it is directly quantified into numerical features; The multimodal fusion function is constructed based on the weighted summation of different modal data. The expression is: Among them, F i represents the eigenvector of the i-th mode, w i Represents the weight value of the i-th mode.

8. The method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as claimed in claim 7, characterized in that: The specific steps are: the real-time behavior of underground personnel is comprehensively analyzed, and once unsafe behavior is identified, the early warning mechanism is immediately activated. The specific steps are: Based on real-time data input, the formula is dynamically adjusted to calculate the threshold function A of the image recognition and posture estimation model that adapts to the current environment; According to the time series data F t and environmental parameter E t Input the behavior prediction model to predict the unsafe behavior trend of underground personnel and adjust the strategy through the reward function; Input the visual, sound and other non-visual sensor data into the multimodal fusion function to obtain the fused feature vector FF; The fused feature vector FF is input into the recurrent neural network RNN ​​for behavior recognition and classification to obtain the predicted behavior category probability distribution y θ ; Combined with the current environmental parameter data, a dynamic threshold function is constructed, the expression is: V(k L ,k D ,γ)=θ0+k L ·(1-Q(L)) γ +k D ·Q(D) γ ; Among them, V represents the dynamically adjusted threshold function, k L Represents the adjustment coefficient related to light intensity, k D represents the adjustment coefficient related to dust concentration, γ represents the parameter affecting the steepness of the function curve, and θ0 represents the basic threshold; The classification result y after multimodal fusion θ , compared with the dynamic threshold V; When the highest classification probability value is greater than or equal to the threshold value V under the current environmental conditions, it is determined that unsafe behavior exists and the early warning mechanism is immediately activated; On the contrary, when the highest classification probability value is less than the threshold V under the current environmental conditions, it is determined that the current behavior is within a safe range, no alarm is required, and monitoring is maintained.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for rapid analysis and identification of unsafe behaviors of underground personnel based on machine vision as described in any one of claims 1 to 8 are implemented.

Images