A method for detecting personnel falling in coal mining face based on multi-sensor fusion

Abstract

The present invention belongs to the technical field of coal mine production safety, and discloses a method for detecting personnel falls on coal mining working faces based on multi-sensor fusion, aiming to solve the problems of poor accuracy and reliability of personnel fall detection using a single sensor in complex coal mining environments, as well as the lack of near- and long-range multi-sensor data fusion detection methods. The method for detecting personnel falls on coal mining working faces based on multi-sensor fusion described in the present invention can quickly and accurately determine whether personnel on coal mining working faces have fallen through processes such as millimeter wave radar feature extraction, laser radar feature extraction, radio frequency device feature extraction, feature vector splicing and prediction, and realizes efficient monitoring and early warning of personnel safety status, effectively protects the life safety of operating personnel, and greatly improves the safety and intelligence level of coal mining operations, and has significant practical value and broad application prospects.

Description

Technical Field

[0001] The present invention belongs to the technical field of coal mine production safety, and in particular relates to a method for detecting personnel falling in a coal mining face based on multi-sensor fusion. Background Art

[0002] In coal mining operations, the safety of coal mining faces has always been a focus of the industry. Due to the unique complexity of the coal mining working environment, such as low brightness, high humidity, diffuse dust, narrow space and irregular terrain, workers face a high risk of falling when working in it.

[0003] Traditional personnel safety monitoring methods have exposed many defects in harsh environments such as underground coal mines. For example, video monitoring systems based on visible light are difficult to clearly capture the exact position and movement details of personnel under low-light conditions. High humidity and the presence of dust will further reduce the clarity and recognition of the image, resulting in frequent omissions or misjudgments of dangerous situations such as personnel falling. At the same time, relying on a single sensor technology, such as a simple infrared sensing device, although it has a certain effect in a low-light environment, it is unable to fully and accurately obtain rich information such as the posture, speed, and position changes of personnel, making it difficult to accurately determine whether a person is in the process of falling, and unable to meet the complex and changeable safety monitoring needs of coal mining working faces. In addition, with the continuous improvement of the mechanization and automation of coal mining operations, the equipment and personnel activities at the work site are more frequent and complex. The existing simple monitoring methods are far from meeting the requirements for personnel safety protection in coal mining working faces under the new situation.

[0004] Therefore, it is urgent to develop a method for detecting personnel falls in coal mining faces that can overcome the interference of harsh conditions such as low brightness, high humidity and high dust concentration in mines, integrate a variety of advanced sensor technologies and have high precision and high reliability. This is of vital practical significance for effectively preventing personnel fall accidents and improving the overall safety of coal mining. Summary of the invention

[0005] The purpose of the present invention is to propose a method for detecting personnel falls in coal mining working faces based on multi-sensor fusion, so as to solve the problem that the existing means of detecting personnel falls in coal mining working faces are restricted by complex environments, resulting in poor accuracy and reliability.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] A method for detecting personnel falling in a coal mining face based on multi-sensor fusion comprises the following steps:

[0008] Step 1. Build a coal mining face personnel fall detection model based on multi-sensor fusion;

[0009] The model includes a millimeter wave radar feature extraction module, a laser radar feature extraction module, a radio frequency device feature extraction module, a feature vector splicing module and a prediction module;

[0010] Step 2. The millimeter-wave radar feature extraction module uses the millimeter-wave radar to detect the fall of a person on the target, measures the radial velocity, movement direction, and velocity change rate of the target, and then obtains the millimeter-wave radar feature vector after quantization encoding;

[0011] Step 3. The laser radar feature extraction module uses the laser radar to detect the fall of the target, clusters the collected point cloud, extracts the human body contour, calculates the geometric features of the human body contour, and then normalizes it to form a laser radar feature vector;

[0012] Step 4. The RF device feature extraction module uses the RF device to detect the fall of a person on the target, extracts the tilt angle features of the human body posture, and performs normalization encoding to obtain a RF device feature vector;

[0013] Step 5. The feature vector concatenation module concatenates the millimeter wave radar feature vector, the laser radar feature vector, and the radio frequency device feature vector to obtain a concatenated joint feature vector;

[0014] Step 6. The prediction module uses the support vector machine (SVM) as a classification model and inputs the joint feature vector into the support vector machine for personnel fall prediction;

[0015] Step 7. Train the constructed coal mining face personnel fall detection model based on multi-sensor fusion;

[0016] Step 8. Input the data collected in real time by the millimeter wave radar, lidar, and radio frequency device into the trained coal mining face personnel fall detection model based on multi-sensor fusion to determine whether a person in the coal mining face has fallen.

[0017] In addition, based on the above-mentioned method for detecting personnel falling in a coal mining face based on multi-sensor fusion, the present invention also proposes a computer device, which includes a memory and one or more processors;

[0018] The memory stores executable codes, and when the processor executes the executable codes, it is used to implement the steps of the above-mentioned method for detecting personnel falling on a coal mining face based on multi-sensor fusion.

[0019] In addition, based on the above-mentioned method for detecting falls of personnel in coal mining working faces based on multi-sensor fusion, the present invention also proposes a computer-readable storage medium on which a program is stored; when the program is executed by a processor, it is used to implement the steps of the above-mentioned method for detecting falls of personnel in coal mining working faces based on multi-sensor fusion.

[0020] The present invention has the following advantages:

[0021] (1) The present invention proposes a multi-sensor fusion detection system that can simultaneously collect the personnel movement features extracted by the millimeter wave radar at a long distance, the human body contour features extracted by the laser radar at a medium and long distance, and the human body posture tilt angle features extracted by the radio frequency device at a medium and short distance, enrich the personnel status information dimension, and process and fuse the data of each sensor to solve the problem of one-sided information of a single sensor in the complex coal mining environment, so as to achieve the personnel fall detection task more accurately.

[0022] (2) The present invention proposes a highly adaptable multi-sensor fusion detection system, which uses millimeter-wave radar to reduce the impact of the high-dust and harsh environment of the coal mining face on the detection effect, uses laser radar to improve the accuracy of judging whether the falling target is a human body, and uses a radio frequency device to accurately judge whether a person is in a falling state, effectively reducing the false alarm rate of person fall detection.

[0023] (3) The present invention proposes an optimized feature vector fusion and model training method, which can not only effectively integrate multi-sensor feature vectors, but also optimize the support vector machine model by constructing a targeted Lagrangian function to solve the dual problem. This improves the model's classification accuracy of personnel status without relying on a large amount of sample data.

[0024] (4) The present invention proposes an intelligent real-time detection and decision-making mechanism, which can not only collect and process sensor data in real time to quickly determine whether a person is in a falling state and reduce the false alarm rate, but also use the Sigmoid function to convert the decision result into a falling probability to improve the accuracy of the judgment. By predicting the falling position and direction and the self-diagnosis function, the system stability and practicality can be enhanced while ensuring the safety of personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The present invention is a flowchart of a method for detecting personnel falling in a coal mining face based on multi-sensor fusion in an embodiment of the present invention.

[0026] Figure 2 It is a schematic diagram of the operation of the personnel fall detection device in the embodiment of the present invention at the working site of the coal mining face.

[0027] Figure 3It is a module diagram of a coal mining face personnel fall detection model based on multi-sensor fusion in an embodiment of the present invention.

[0028] Figure 4 The flowchart of determining whether a person falling accident occurs in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0030] Example 1

[0031] In this embodiment, a method for detecting personnel falls in coal mining working faces based on multi-sensor fusion is proposed, aiming to solve the problems of poor accuracy and reliability of personnel fall detection using a single sensor in complex coal mining environments, as well as the lack of near-range and long-range multi-sensor data fusion detection methods. This method effectively protects the life safety of workers through five parts: millimeter wave radar feature extraction and processing, lidar feature extraction and processing, radio frequency device feature extraction and processing, feature vector splicing and model training, and real-time detection and decision-making.

[0032] In the method of personnel fall detection in coal mining working face based on multi-sensor fusion, laser radar, millimeter wave radar and radio frequency device are integrated to work together. First, in the process of millimeter wave radar feature extraction and processing, millimeter wave radar is used to detect personnel fall of distant targets. Based on antenna array technology and Doppler effect, the radial velocity, movement direction and velocity change rate of the human body are accurately measured. After quantization encoding, the millimeter wave radar feature vector is obtained to achieve accurate detection of human body falling at a long distance.

[0033] In the process of laser radar feature extraction and processing, laser radar is used to detect human falls on medium and long-distance targets. The DBSCAN algorithm, which is a density-based clustering algorithm, is used to cluster the collected point cloud data, accurately extract the human body contour, and calculate geometric features such as circumference, area, and center of gravity coordinates. After normalization, the laser radar feature vector is formed to achieve accurate judgment of whether the fallen object is a human body.

[0034] In the process of RF device feature extraction and processing, the RF device is used to detect the fall of a person at close range. Based on the antenna phase difference, the RF device extracts the tilt angle feature of the human body posture, that is, the tilt angle of the human body relative to the RF device, and normalizes the encoding to achieve accurate judgment of whether the human body is in a falling state.

[0035] Then, the feature vectors of the above three sensors are spliced, and the model is trained using the support vector machine (SVM). The dual problem is derived and solved using the Lagrange multiplier method to determine the optimal parameters of the model.

[0036] In the real-time detection and decision-making stage, each sensor continuously collects data, extracts and splices feature vectors in real time, and then inputs them into the trained SVM model. The Sigmoid function is used to convert the decision function value into the probability of personnel falling, and then compared with the preset threshold, so as to quickly and accurately determine whether the personnel in the coal mining face have fallen, and realize efficient monitoring and early warning of the safety status of personnel. The personnel fall detection method in the coal mining face based on multi-sensor fusion has greatly improved the safety and intelligence level of coal mining operations, and has significant practical value and broad application prospects.

[0037] Figure 2 The figure is a schematic diagram of the operation of the personnel fall detection device in the embodiment of the present invention at the coal mining face work site, including the installation position of the personnel fall detection device and the working distance range of each sensor. In this embodiment, the personnel fall detection device is installed above the coal mine belt conveyor at the coal mining face, and the coal mine crusher is connected behind the coal mining transfer point. The device uses millimeter wave radar, laser radar and radio frequency device to warn of the occurrence of personnel falls, and timely detects personnel falls when an accident occurs, so as to take corresponding alarms and take rescue measures such as shutting down the belt conveyor in time. Among them, the millimeter wave radar is used for long-distance detection of 40 to 150 meters, the laser radar is used for medium and long-distance detection of 10 to 150 meters, and the radio frequency device is used for medium and short-distance detection of 0 to 100 meters.

[0038] like Figure 1 As shown, a method for detecting personnel falling in a coal mining face based on multi-sensor fusion includes the following steps:

[0039] Step 1. Build a coal mining face personnel fall detection model based on multi-sensor fusion;

[0040] The model includes a millimeter wave radar feature extraction module, a lidar feature extraction module, a radio frequency device feature extraction module, a feature vector splicing module and a prediction module.

[0041] Step 2. The millimeter-wave radar feature extraction module uses the millimeter-wave radar to detect the fall of a person on the target, measures the radial velocity, movement direction, and velocity change rate of the target, and then obtains the millimeter-wave radar feature vector after quantization encoding.

[0042] Millimeter-wave radars are installed at selected locations on the coal mining face to effectively detect the movement of personnel at a distance. Millimeter-wave radars transmit millimeter-wave signals through antenna arrays and obtain information such as the target's radial velocity and direction of movement based on the Doppler frequency shift and phase difference of the echo signal.

[0043] Step 2.1. Estimation of radial velocity and motion direction.

[0044] According to the Doppler effect formula of millimeter wave radar: .

[0045] in, is the Doppler shift, is the radial velocity of the target relative to the millimeter-wave radar, is the millimeter wave wavelength, is the transmitting signal frequency.

[0046] The radial velocity of the target is calculated by detecting the Doppler frequency shift of the echo signal.

[0047] In order to obtain the moving direction of the target, an antenna array consisting of multiple antennas is used for angle estimation. A uniform linear array (ULA) is used with an antenna spacing of According to the phase difference method, for the signals received by two adjacent antennas, the phase difference is It is expressed as: .

[0048] in, is the direction angle of the target. By measuring the phase difference, the direction angle can be calculated .

[0049] Step 2.2. Feature quantization and encoding.

[0050] The radial velocity Quantize according to the preset resolution, with Δv as a quantization interval of speed, assuming that the speed quantized value is . Set the direction angle Quantify it with Δ is a quantized interval of direction, Indicates the direction interval. At the same time, the velocity change rate, i.e. acceleration, is calculated by dividing the velocity difference of adjacent time steps by the time interval. , and then the acceleration Quantization is performed with Δ is a quantized interval of acceleration, and the quantized acceleration value is .

[0051] The quantized speed, direction and acceleration features are encoded using one-hot encoding to obtain the encoded millimeter-wave radar feature vector Assuming the speed is quantization intervals, directions quantization intervals, the acceleration is quantization intervals, then the encoded millimeter-wave radar feature vector The dimension is .

[0052] Step 3. The LiDAR feature extraction module uses LiDAR to detect human falls, clusters the collected point clouds, extracts the human body contour, calculates the geometric features of the human body contour, and then normalizes it to form a LiDAR feature vector.

[0053] Install a laser radar at a suitable location on the coal mining face so that it can collect point cloud data within a medium and long distance and accurately determine whether the falling target is a human body. The laser radar emits a laser beam at a preset frequency and receives the reflected light signal to obtain the three-dimensional point cloud data of the coal mining face.

[0054] Step 3.1. Point cloud clustering and contour extraction.

[0055] The DBSCAN algorithm is used to perform cluster analysis on the collected point cloud data to separate the point cloud data belonging to the human body. The appropriate neighborhood radius (such as 0.5 meters) and the minimum number of points (such as 10 points) are set as parameters of the DBSCAN algorithm to adapt to the approximate size and distribution of personnel in the coal mining face.

[0056] For the clustered human point cloud ,in , is the number of human body points, Indicates The three-dimensional coordinates of the points, , , Respectively represent Points axis, axis, Axis coordinates.

[0057] The human body contour is extracted by calculating the convex hull of the point cloud. In this embodiment, the algorithm for calculating the convex hull of the point cloud adopts the Graham scanning method to obtain the vertex set of the human body contour. ,in , is the number of contour vertices, Indicates The 3D coordinates of the contour vertices, , , Respectively represent of contour vertices axis, axis, Axis coordinates.

[0058] Step 3.2. Contour feature calculation and normalization.

[0059] Calculate geometric features of contours, such as perimeter ,area and the barycentric coordinates .perimeter It is obtained by calculating the sum of the Euclidean distances between the contour vertices.

[0060] .

[0061] For area Use a triangulation-based method to calculate, select a vertex (such as the contour vertex ) as the starting point, and then connect the contour vertex with other non-adjacent contour vertices in sequence to divide the polygon into M-2 triangles. Usually, the human body contour is similar to a convex polygon. This operation can be performed for convex polygons. For concave polygons, some additional processing may be required to ensure that the divided triangles are all inside the polygon.

[0062] A triangle obtained by segmenting one of the human body contour polygons For example, M-1; in three-dimensional space, a triangle Area Calculated by the vector cross product method, assuming:

[0063] , .

[0064] Then the triangle The area calculation formula is: .

[0065] in, Representation vector and vector The result is a cross product vector, and Represents the modulus or length of this cross product vector.

[0066] For the vector cross product operation:

[0067] = .

[0068] After calculating the area of ​​each triangle, the area of ​​the entire polygon, i.e. the outline of the human body, is It is the sum of the areas of all triangles, that is:

[0069] .

[0070] Barycentric coordinates The calculation formula is:

[0071] , , .

[0072] For geometric features including perimeter ,area and the barycentric coordinates Normalize it to be between [0, 1]. For example, for the perimeter , assuming that the maximum perimeter observed in the training data set is , the minimum value is , then the normalized perimeter for:

[0073] .

[0074] Similarly, for area and the barycentric coordinates A similar normalization process is also performed to obtain the normalized feature vector , as the lidar feature vector.

[0075] in, is the normalized area, are the normalized barycentric coordinates.

[0076] Step 4. The RF device feature extraction module uses the RF device to detect the fall of a person on the target, extracts the tilt angle features of the human body posture, and performs normalized encoding to obtain the RF device feature vector.

[0077] A radio frequency device is arranged at the coal mining face. The radio frequency device transmits radio frequency signals of a preset frequency and receives signals reflected back by the human body, so as to accurately judge whether the human body is in a falling posture at medium and close distances.

[0078] Step 4.1. Human body posture tilt angle feature extraction.

[0079] The radio frequency device calculates the posture information of the human body, that is, the tilt angle of the human body relative to the radio frequency device, by analyzing the signal phase difference of multiple antennas.

[0080] The phase difference measured by receiving signals from two adjacent antennas is , then the inclination angle of the human body relative to the RF device for:

[0081] .

[0082] in, is the antenna spacing of the RF device, is the wavelength of the RF signal.

[0083] Step 4.2. Feature normalization and encoding.

[0084] The extracted human body posture tilt angle features are normalized to convert the tilt angle Normalized to [0,1], assuming the maximum tilt angle is , the minimum tilt angle is , then the normalized tilt angle for:

[0085] .

[0086] Encode the normalized posture features, for example, using a fixed-length binary vector. Assume that The binary bit represents the tilt angle, according to The value of determines the value of each bit in the binary vector, and obtains the RF device feature vector .

[0087] Step 5. The feature vector concatenation module concatenates the millimeter wave radar feature vector, the lidar feature vector, and the radio frequency device feature vector to obtain a concatenated joint feature vector.

[0088] Assume that the feature vector extracted by the lidar is ,in is the dimension of the lidar feature vector, to Respectively The first to nth components of these geometric features include the normalized perimeter mentioned above ,area And the center of gravity coordinates .

[0089] The feature vector detected by the millimeter wave radar is , k is the dimension of the millimeter wave radar feature vector, to Respectively The first to kth components of contain the speed, direction, and acceleration features after one-hot encoding.

[0090] The feature vector obtained by the radio frequency device is , is the dimension of the RF device feature vector, to Respectively The 1st to rth components of contain the normalized and encoded tilt angle posture features.

[0091] The concatenated joint feature vector for:

[0092] .

[0093] Joint eigenvector The dimension is n+k+r.

[0094] Step 6. The prediction module uses the support vector machine (SVM) as the classification model and inputs the joint feature vector into the support vector machine for personnel fall prediction.

[0095] The method of the present invention uses a support vector machine as a classification model.

[0096] The goal of SVM is to find a hyperplane that separates samples of different categories and maximizes the margin. In this embodiment, the categories are people falling and people not falling. For linearly separable data, the hyperplane equation is:

[0097] .

[0098] in, is the weight vector, the weight vector The dimensions and joint eigenvector of same, is the input feature vector. Here, the input feature vector is , is the bias term.

[0099] Step 7. Train the constructed coal mining face personnel fall detection model based on multi-sensor fusion.

[0100] For the training sample set ,in It is joint eigenvectors, is the category label corresponding to the sth joint feature vector, , Indicates that the person has not fallen. Represents a person falling, and the goal of SVM is to minimize the following objective function: .

[0101] The constraints are:

[0102] .

[0103] in, K, K represents the number of samples in the training sample set.

[0104] The slack variable is introduced to deal with the nonlinear separable case, namely the soft margin SVM.

[0105] In actual situations, the data may not be completely linearly separable, so a slack variable is introduced , the objective function becomes:

[0106] .

[0107] The constraints become: .

[0108] in, , is the penalty parameter, which controls the degree of penalty for misclassification. The larger it is, the heavier the penalty for misclassification, and the model will try to classify all samples correctly; The smaller it is, the model allows more sample classification errors to obtain a looser interval.

[0109] Then the support vector machine model is optimized according to the following steps.

[0110] Step 7.1. Dual problem and solution.

[0111] By using the Lagrange multiplier method, the above constrained optimization problem is transformed into a dual problem for solution. Introducing Lagrange multipliers, Lagrange function It is expressed as:

[0112] .

[0113] in, , is the Lagrange multiplier, , .

[0114] For the Lagrangian function , respectively, about , and Taking the partial derivative and setting it to 0, we get:

[0115] .

[0116] .

[0117] .

[0118] Substituting the result into the Lagrangian function, we get the dual problem:

[0119] .

[0120] in, K, K; , is the Lagrange multiplier, , ; is the joint feature vector of the pth sample, is the category label corresponding to the pth joint feature vector; is the joint feature vector of the qth sample, is the category label corresponding to the qth joint feature vector.

[0121] The constraints are:

[0122] .

[0123] Step 7.2. Transformation to quadratic programming form.

[0124] The dual problem can be expressed in the standard form of quadratic programming. , then the objective function can be written as form.

[0125] in, It's K The matrix , , .

[0126] The constraints can be written as and form.

[0127] in, K The identity matrix of .

[0128] Step 7.3. Solve using quadratic programming algorithm.

[0129] The quadratic programming algorithm solves the problem through iteration, looking for a series of points within the feasible domain to approach the optimal solution.

[0130] First, we need to determine an initial feasible point , , to Respectively The first to the Kth components of the initial feasible point satisfy all the constraints and .

[0131] Then, at each iteration, we update The barrier function is usually a logarithmic function, such as , transforming a constrained problem into an unconstrained problem.

[0132] Calculate the gradient and Hessian matrix Hessian matrix, gradient and the Hessian matrix Used to determine the search direction. For the above objective function , the pth element of the gradient is:

[0133] .

[0134] Update according to search direction and step size , and repeat this process until the stopping criterion is met, such as the norm of the gradient is less than a preset threshold or the change of the objective function is less than a preset threshold.

[0135] Step 7.4. Compute decision function and classification.

[0136] Find Then, calculate the weight vector and the bias term . For the weight vector Calculated by the following formula:

[0137] .

[0138] For support vectors, A sample of Calculated by the following formula:

[0139] .

[0140] in, is the index set of support vectors, and are the category labels and feature vectors corresponding to the support vectors; h represents the value in the support vector index set, represents the category label of the hth element in the support vector index set, represents the joint eigenvector of the hth element in the support vector index set.

[0141] The final decision function for:

[0142] .

[0143] For a new joint eigenvector ,if , it is judged as not falling, that is, , is the joint eigenvector The corresponding category label; if , it is judged as falling, that is 1.

[0144] In this embodiment, model evaluation and hyperparameter adjustment are also performed, and the specific process is as follows:

[0145] The K-fold cross validation formula was used for model evaluation.

[0146] The labeled training data set D is divided into There are subsets of similar size, and the dataset size is , each subset size is approximately .for =1,2,3…, , will The subsets are used as validation sets, and the rest The model is trained on the validation set and its performance is evaluated.

[0147] Commonly used evaluation indicators include accuracy, precision, recall and F1-Score.

[0148] The accuracy is expressed as: .

[0149] in, That is, True Positive is a true example. At this time, the actual label category is drop, and the model's predicted label category is drop. That is, True Negative is a true negative example. At this time, the actual label category is not dropped, and the model's predicted label category is not dropped. That is, False Positive is a false positive example. At this time, the actual label category is no drop, and the model's predicted label category is drop. That is, False Negative is a false negative example. At this time, the actual label category is drop, and the model's predicted label category is no drop.

[0150] The precision is expressed as: .

[0151] The recall rate is expressed as: .

[0152] The F1-score is expressed as: .

[0153] The process of hyperparameter tuning is:

[0154] Choose from a range of different Values, such as , for each Values, use fold cross validation to evaluate model performance. Record different Evaluation indicators under different values, such as average accuracy, average F1-score, etc.

[0155] Based on the evaluation results, the value that gives the best model performance is selected as the final penalty parameter. When the support vector machine model has the highest average F1-score in cross validation, then choose As the final penalty parameter, it is used for the final model training.

[0156] Step 8. Input the data collected in real time by the millimeter wave radar, lidar, and radio frequency device into the trained coal mining face personnel fall detection model based on multi-sensor fusion to determine whether a person in the coal mining face has fallen.

[0157] Specifically, the millimeter-wave radar, lidar, and radio frequency device input the real-time collected data into the trained coal mining face personnel fall detection model based on multi-sensor fusion, predicts the probability of personnel falling according to the decision function value, and determines whether personnel in the coal mining face have fallen.

[0158] The process of real-time data collection and feature extraction is as follows:

[0159] Step 8.1. For millimeter wave radar, at time , according to the received echo signal, through the Doppler effect formula Calculate the target at time The radial velocity ,in For the moment Doppler frequency shift, is the millimeter wave wavelength, is the transmitting signal frequency.

[0160] The target's moving direction is calculated by using the phase difference method using the antenna array. Assuming that the antenna spacing is ,time The phase difference is , then the time Direction angle for:

[0161] .

[0162] The moment Speed Quantize according to the preset resolution, is a quantization interval, and the quantized moment The speed is ,time Direction angle Also according to each Quantize an interval, the quantized moment The direction is , while calculating the time The rate of change of speed, that is, the time Acceleration , and quantized as , and get the quantized moment Then perform one-hot encoding to obtain the acceleration value of the millimeter-wave radar at time The eigenvector of .

[0163] Step 8.2. For the laser radar, collect point cloud data in real time and cluster it using the DBSCAN algorithm to obtain the human point cloud. Assume that at time , the clustered human point cloud data is ,in , It's time The number of human body points, Indicates Point in time The three-dimensional coordinates of , , Respectively represent Point in time of axis, axis, Axis coordinates.

[0164] Extracting time using convex hull algorithm The vertex set of the human body contour ,in , is the number of contour vertices, Indicates The 3D coordinates of the contour vertices, , , Respectively represent of contour vertices axis, axis, axis coordinates, and then calculate at time The perimeter of the contour ,area and the barycentric coordinates .

[0165] At the moment Circumference The calculation formula is:

[0166] .

[0167] time Area Calculated by a triangulation-based algorithm.

[0168] Choose a contour vertex (such as the moment The contour vertex ) as the starting point, and sequentially connect the contour vertex with other non-adjacent vertices to split the polygon into The same operation can generally be applied to human body contours that are approximately convex polygons.

[0169] For the moment One of the triangles ,set up:

[0170] .

[0171] .

[0172] in, .

[0173] In three-dimensional space, time The triangle Area Computed via vector cross product:

[0174] .

[0175] in, Represents the modulus or length of the cross product vector.

[0176] The vector cross product operation is:

[0177] .

[0178] The entire polygon, i.e. the human body contour, at the moment Area It is the sum of the areas of all these triangles, that is:

[0179] .

[0180] time The centroid coordinates of The calculation formula is:

[0181] , , .

[0182] in, It is The triangle formed by the contour vertices at time area.

[0183] These features are normalized, assuming that the maximum perimeter determined during the training phase is , the minimum perimeter is , then the normalized moment Circumference for:

[0184] .

[0185] Likewise, for the moment Area and the barycentric coordinates Normalize and get the time The laser radar feature vector : .

[0186] in, is the normalized moment The area of is the normalized moment The centroid coordinates of .

[0187] Step 8.3. For the RF device, at time Calculate the time using the signal phase difference of multiple antennas The tilt angle of the human body , assuming the antenna spacing is , the measured moment The phase difference is , then the time The tilt angle for:

[0188] .

[0189] The moment The tilt angle Normalized to [0,1], assuming that the maximum tilt angle determined during the training phase is , the minimum tilt angle is , then the normalized moment The tilt angle for:

[0190] .

[0191] right Encoding is performed to obtain the radio frequency device at time The eigenvector of .

[0192] Step 8.4. Feature vector concatenation and decision making.

[0193] The moment The laser radar feature vector ,time The millimeter wave radar feature vector and time The RF device feature vector , spliced ​​into a moment in sequence The joint eigenvector of :

[0194] .

[0195] The moment The joint eigenvector of Input into the trained support vector machine model, the decision function of SVM is:

[0196] .

[0197] in, is the weight vector obtained through training, is the bias term.

[0198] Make a judgment based on the value of the decision function:

[0199] if , it is determined that the person has not fallen. .

[0200] if , it is judged as a person falling. .

[0201] in, It's time The joint eigenvector of The corresponding category label.

[0202] The probability of falling can also be evaluated according to the value of the decision function, and a falling probability can be preset. , use the logistic function, namely the Sigmoid function, to perform the conversion:

[0203] .

[0204] For example, if The value of is 0, then , which means that there is a 50% chance that the person is in a falling state. The value of is 3, then , indicating that the probability of a person falling is 95.26%; on the contrary, if The value of is -3, then , the probability of people falling is only 4.74%.

[0205] According to the actual application scenario, preset a threshold (For example =0.7) to determine whether action is needed. When the system determines that the possibility of a person falling is high, it triggers an alarm and takes appropriate rescue measures. When the system considers that the person is in normal condition or the possibility of falling is low, it maintains the monitoring status.

[0206] Example 2

[0207] This embodiment 2 describes a computer device, which includes a memory and one or more processors.

[0208] An executable code is stored in the memory. When the processor executes the executable code, it is used to implement the steps of the method for detecting personnel falling on a coal mining face based on multi-sensor fusion in the above-mentioned embodiment 1.

[0209] In this embodiment, the computer device is any device or apparatus with data processing capability, which will not be described in detail here.

[0210] Example 3

[0211] This embodiment 3 describes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used for the steps of a method for detecting a person falling on a coal mining face based on multi-sensor fusion.

[0212] The computer-readable storage medium may be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc., equipped on the device.

[0213] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A method for detecting personnel falling in coal mining working face based on multi-sensor fusion, characterized in that: The steps include: Step 1. Build a coal mining face personnel fall detection model based on multi-sensor fusion; Step 2. Use the millimeter wave radar to detect the fall of a person on the target, measure the radial velocity, movement direction and velocity change rate of the target, and then obtain the millimeter wave radar feature vector after quantization encoding; Step 3. Use the laser radar to detect the fall of a person on the target, cluster the collected point cloud, extract the human body contour, calculate the geometric features of the human body contour, and then normalize it to form a laser radar feature vector; Step 4. Use the radio frequency device to detect the fall of the target, extract the tilt angle features of the human body posture, and perform normalization encoding to obtain the feature vector of the radio frequency device; Step 5. Concatenate the millimeter wave radar feature vector, the laser radar feature vector, and the radio frequency device feature vector to obtain a concatenated joint feature vector; Step 6. Use the support vector machine (SVM) as a classification model and input the joint feature vector into the support vector machine for personnel fall prediction; Step 7. Train the constructed coal mining face personnel fall detection model based on multi-sensor fusion; Step 8. Input the data collected in real time by the millimeter wave radar, lidar, and radio frequency device into the trained coal mining face personnel fall detection model based on multi-sensor fusion to determine whether a person in the coal mining face has fallen.

2. The method for detecting personnel falling in coal mining working face based on multi-sensor fusion according to claim 1 is characterized in that: The step 2 is specifically as follows: Millimeter-wave radar is installed on the coal mining face; the millimeter-wave radar transmits millimeter-wave signals through the antenna array, and obtains the radial velocity and movement direction information of the target based on the Doppler frequency shift and phase difference of the echo signal; Step 2.

1. Estimate the radial velocity and direction of motion of the target; according to the Doppler effect formula of millimeter wave radar: ； in, is the Doppler frequency shift, is the radial velocity of the target relative to the millimeter-wave radar, is the millimeter wave wavelength, is the frequency of the transmitted signal; the radial velocity of the target is calculated by detecting the Doppler frequency shift of the echo signal ; An antenna array consisting of multiple antennas is used for angle estimation. The antenna array is a uniform linear array with an antenna spacing of According to the phase difference method, for the signals received by two adjacent antennas, the phase difference It is expressed as: ； in, is the direction angle of the target, by measuring the phase difference , calculate the direction angle ; Step 2.

2. Feature quantization and encoding; The radial velocity Quantize according to the preset resolution, with Δv as a speed quantization interval. The speed quantization interval is The quantized radial velocity is ; Set the direction angle Quantify it with Δ is a quantization interval of the direction, and the quantization interval of the direction angle is , and obtain the quantized direction angle interval ; Calculate the rate of change of velocity, i.e. acceleration, by dividing the velocity difference between adjacent time steps by the time interval , and then the acceleration Quantization is performed with Δ is a quantization interval of acceleration. The quantization interval of acceleration is The quantized acceleration value is ; The quantized speed, direction and acceleration features are encoded using one-hot encoding to obtain the encoded millimeter-wave radar feature vector , the encoded millimeter-wave radar feature vector The dimension is .

3. The method for detecting personnel falling in coal mining working face based on multi-sensor fusion according to claim 1 is characterized in that: The step 3 is specifically as follows: A laser radar is installed on the coal mining face to determine whether the falling target is a human body; the laser radar emits a laser beam at a preset frequency and receives the reflected light signal to obtain the three-dimensional point cloud data of the coal mining face; Step 3.

1. Point cloud clustering and contour extraction; The preset neighborhood radius and minimum number of points are used as parameters of the DBSCAN algorithm, and the collected point cloud data are clustered using the DBSCAN algorithm to separate the point cloud data belonging to the human body; The clustered human point cloud data is expressed as: ； in, , is the number of human body points, Indicates The three-dimensional coordinates of individual body points, , , Respectively represent Personal point axis, axis, Axis coordinates; The human body contour is extracted by calculating the convex hull of the point cloud. The algorithm for calculating the convex hull of the point cloud adopts the Graham scanning method, and the vertex set of the human body contour is expressed as: ； in, , is the number of contour vertices, Indicates The 3D coordinates of the contour vertices, , , Respectively represent of contour vertices axis, axis, Axis coordinates; Step 3.

2. Calculation and normalization of contour geometric features; Calculate geometric features of human body contours, including circumference ,area and the barycentric coordinates ; The perimeter is obtained by calculating the sum of the Euclidean distances between the contour vertices : ； For area The calculation is performed using a method based on triangulation. A contour vertex is selected as the starting point, and then the contour vertex is sequentially connected with other non-adjacent contour vertices, so as to divide the polygon of the human body contour into M-2 triangles. For the concave polygon of the human body contour, additional processing is performed to ensure that the divided triangles are all inside the polygon. The triangles obtained by polygon segmentation of the human body contour ,in, M-1; In three-dimensional space, a triangle The area of ​​is calculated by the vector cross product method, assuming: ， ； Then the triangle Area The calculation formula is: ； in, Representation vector and vector The result is a cross product vector, represents the modulus of the cross product vector, that is, the length of the cross product vector; For the vector cross product operation: = ； After calculating the area of ​​each triangle, calculate the polygonal area of ​​the entire human body outline , which is the sum of the areas of all triangles, we get: ; Barycentric coordinates The calculation formula is: ， ， ； Geometric features of the human body contour include circumference ,area and the barycentric coordinates Normalize it so that its range is between [0, 1]; For circumference , let the maximum perimeter be , the minimum value is , then the normalized perimeter for: ； For area and the barycentric coordinates Perform normalization to obtain the normalized feature vector: ； in is the laser radar feature vector, is the normalized area, are the normalized barycentric coordinates.

4. The method for detecting personnel falling in coal mining working face based on multi-sensor fusion according to claim 1 is characterized in that: The step 4 is specifically as follows: A radio frequency device is arranged at the coal mining face. The radio frequency device transmits a radio frequency signal of a preset frequency and receives the signal reflected back by the human body to accurately judge whether the target is in a falling posture; Step 4.

1. Extract human body posture tilt angle features; The radio frequency device calculates the posture information of the human body, that is, the tilt angle of the human body relative to the radio frequency device, by analyzing the signal phase difference of multiple antennas; The phase difference measured by receiving signals from two adjacent antennas is , then the inclination angle of the human body relative to the RF device for: ； in, is the antenna spacing of the RF device, is the wavelength of the RF signal; by measuring the phase difference To calculate the inclination angle of the human body relative to the RF device ; Step 4.

2. Feature normalization and encoding; The extracted human body posture tilt angle feature, that is, the tilt angle of the human body relative to the radio frequency device Normalize the tilt angle Normalized to [0, 1], the maximum tilt angle is , the minimum tilt angle is , then the normalized tilt angle for: ； The normalized human body posture tilt angle feature is the normalized tilt angle To encode, use A binary vector represents the tilt angle of the human body relative to the RF device. The value of determines the binary vector The value of each bit in the RF device is obtained .

5. The method for detecting personnel falling in coal mining face based on multi-sensor fusion according to claim 1 is characterized in that: The step 5 is specifically as follows: The feature vector extracted by the laser radar is ,in is the dimension of the lidar feature vector, to Respectively The 1st to nth components of the laser radar extracted feature vector Contains the normalized perimeter ,area And the center of gravity coordinates The geometric characteristics of The feature vector detected by the millimeter wave radar is , k is the dimension of the millimeter wave radar feature vector, to Respectively The 1st to kth components of the millimeter-wave radar detected feature vector Contains one-hot encoded speed, direction, and acceleration features; The feature vector obtained by the radio frequency device is , is the dimension of the RF device feature vector, to Respectively The 1st to rth components, the characteristic vector obtained by the radio frequency device Contains normalized and encoded human body posture tilt angle features; The millimeter wave radar feature vector, the laser radar feature vector and the radio frequency device feature vector are spliced ​​to obtain the spliced ​​joint feature vector for: ； Joint eigenvector The dimension is n+k+r.

6. The method for detecting personnel falling in coal mining working face based on multi-sensor fusion according to claim 5 is characterized in that: The step 6 is specifically as follows: The support vector machine is used for model prediction. The goal of SVM is to find a hyperplane that separates samples of different categories, i.e., samples of people falling and samples of people not falling, and maximizes the interval margin. For linearly separable data, the hyperplane equation is expressed as: ； in, is the weight vector, the weight vector The dimensions and joint eigenvector of same, is the input feature vector, the input feature vector is the joint feature vector , is the bias term.

7. The method for detecting personnel falling in coal mining working face based on multi-sensor fusion according to claim 6 is characterized in that: The step 7 is specifically as follows: For the training sample set ,in K, K represents the number of samples in the training sample set; is the joint feature vector of the sth sample, is the category label corresponding to the sth joint feature vector, , Indicates that the person has not fallen. Indicates that a person has fallen; The goal of SVM is to minimize the following objective function: ; The constraints are: ; Introduce slack variables to handle nonlinear separable situations, soft margin SVM, slack variables are expressed as , the objective function becomes: ； The constraints become: ; in, , is the penalty parameter, which is used to control the degree of penalty for misclassification; Through the Lagrange multiplier method, the constrained optimization problem is transformed into a dual problem for solution to determine the parameters of the model; the Lagrange multiplier and the Lagrange function are introduced It is expressed as: ； in, , is the Lagrange multiplier, , ; For the Lagrangian function , respectively, about , and Taking the partial derivative and setting it to 0, we get: ； ； ； Substituting the result into the Lagrangian function, we get the dual problem: ； in, K, K; , is the Lagrange multiplier, , ; is the joint feature vector of the pth sample, is the category label corresponding to the pth joint feature vector; is the joint feature vector of the qth sample, is the category label corresponding to the qth joint feature vector; The constraints are: ； Write the dual problem in standard form as a quadratic program; set up , the objective function is written as form; in, K The matrix of , ; ; The constraints are written as: , ; in, for The identity matrix of The quadratic programming algorithm solves the problem through iteration, looking for a series of points within the feasible domain to approach the optimal solution; First, determine an initial feasible point , , to Respectively The first to the Kth components of the initial feasible point satisfy all the constraints and ; In each iteration, we update ;Through the barrier function, the constrained problem is transformed into an unconstrained problem; Computing Gradients and the Hessian matrix , the gradient and Hessian matrix are used to determine the search direction; for the objective function , the pth element of the gradient is: ； According to the search direction and step size Update until the stopping criterion is met, where the norm of the gradient is less than a preset threshold or the change of the objective function is less than a preset threshold; Compute decision functions and classifications; Find Then, calculate the weight vector and the bias term ; For the weight vector Calculated by the following formula: ； For support vectors, A sample of Calculated by the following formula: ； in, is the index set of support vectors, and are the category labels and feature vectors corresponding to the support vectors; h represents the value in the support vector index set, represents the category label of the hth element in the support vector index set, represents the joint eigenvector of the hth element in the support vector index set; Get the final decision function for: ； For the joint eigenvector ,if , it is determined that the person has not fallen, that is, , is the joint eigenvector The corresponding category label; if , it is judged as a person falling, that is, 1.

8. The method for detecting personnel falling in coal mining working face based on multi-sensor fusion according to claim 1 is characterized in that: The step 8 is specifically as follows: Step 8.

1. For millimeter wave radar, at time According to the received echo signal, the Doppler effect formula is used to calculate the target at time The radial velocity , the Doppler effect formula is: ； in, For the moment Doppler frequency shift, is the millimeter wave wavelength, is the frequency of the transmitted signal; The target’s moving direction is calculated by using the phase difference method using an antenna array. The antenna spacing is ,time The phase difference is , then the time Direction angle for: ； The moment Speed Quantize according to the preset resolution, is a quantization interval, and the quantized moment The speed is ; will be the moment Direction angle According to each Quantize an interval and get the quantized moment The direction is ; Calculate time The rate of change of speed, that is, the time Acceleration , and quantize it to get the quantized moment The acceleration value ; then , , Perform one-hot encoding to obtain the millimeter-wave radar at time The eigenvector of ; Step 8.

2. For the laser radar, collect point cloud data in real time, and cluster the human body point cloud through the DBSCAN algorithm; at time , the clustered human point cloud data is: ； in, , It's time The number of human body points, Indicates Personal point at the moment The three-dimensional coordinates of , , Respectively represent Personal point at the moment of axis, axis, Axis coordinates; Extracting time using convex hull algorithm The vertex set of the human body contour: ； in, , is the number of contour vertices, Indicates The 3D coordinates of the contour vertices, , , Respectively represent of contour vertices axis, axis, Axis coordinates, calculated at time The perimeter of the contour ,area and the barycentric coordinates ; At the moment Circumference The calculation formula is: ； time Area Calculated based on triangulation algorithm; Choose a contour vertex as the starting point, connect the contour vertex with other non-adjacent vertices in sequence, and split the polygon into triangle; for the moment The triangle ,set up: ； ； in, ; In three-dimensional space, time The triangle Area Computed via vector cross product: ； in, represents the modulus of the cross product vector, that is, the length of the cross product vector; The vector cross product operation is: ； The entire polygon, i.e. the human body contour, at the moment Area is the sum of the areas of all triangles, that is: ； time The centroid coordinates of The calculation formula is: ， ， ； The maximum circumference determined during the training phase is , the minimum perimeter is , then the normalized moment Circumference for: ； About the moment Area and the barycentric coordinates Normalize it and get the time The laser radar feature vector It is expressed as: ； in, is the normalized moment The area of is the normalized moment The centroid coordinates of Step 8.

3. For the RF device, at time Calculate the time using the signal phase difference of multiple antennas The tilt angle of the human body , the antenna spacing is , the measured moment The phase difference is , the wavelength of the RF signal is , then the time The tilt angle of the human body for: ； The moment The tilt angle of the human body Normalized to [0,1], the maximum tilt angle determined during the training phase is , the minimum tilt angle is , then the normalized moment The tilt angle for: ； right Encode and obtain the RF device at time The eigenvector of ; Step 8.

4. Feature vector concatenation and decision making; The moment The laser radar feature vector ,time The millimeter wave radar feature vector and time The RF device feature vector , and are stitched together into a moment The joint eigenvector of : ； The moment The joint eigenvector of Input into the trained support vector machine model, the decision function of SVM is: ； in, is the weight vector obtained through training, is the bias term; Make a judgment based on the value of the decision function: if , it is determined that the person has not fallen. ; if , it is judged as a person falling. ; in, It's time The joint eigenvector of The corresponding category label; According to the value of the decision function, the possibility of falling is evaluated and a falling probability is preset. , use the logistic function, namely the Sigmoid function, to perform the conversion: ； According to the actual application scenario, preset a threshold to determine whether action is needed; when When the possibility of a person falling is high, the alarm is triggered and corresponding rescue measures are taken; When the person is in normal condition or has a low probability of falling, the monitoring status is maintained.

9. A computer device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the steps of the method for detecting personnel falling in a coal mining face based on multi-sensor fusion as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by the processor, the steps of the method for detecting personnel falling in a coal mining face based on multi-sensor fusion as described in any one of claims 1 to 8 are implemented.

Images