Computer vision-based construction site worker fall precursor detection method

Abstract

The application relates to a computer vision-based construction site worker fall precursor detection method, which comprises the following steps: acquiring a construction site video stream; establishing a human body three-dimensional posture estimation model and training the same to extract the coordinate values of worker human body skeleton joint nodes in a three-dimensional coordinate system; using a principal component analysis method to reduce the dimension of the worker human body skeleton joint node coordinate matrix and selecting human body skeleton joint nodes that have positive or negative influences on the human body three-dimensional posture estimation model; fusing the three-dimensional coordinate information of the human body skeleton joint nodes with positive influences based on a weight function, constructing an improved Fusion-KNN-based machine learning model and training the same to obtain an optimal weight function; and detecting worker fall precursors based on the machine learning model and outputting the state of the worker human body posture: stable, out of balance or falling. Compared with the prior art, the application is accurate in prediction and can obtain the falling risk degree of the worker in advance, thereby realizing the pre-prevention of construction site worker falling accidents.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a construction site worker fall precursor detection method based on computer vision. BACKGROUND

[0002] Fall accidents, including fatal, non-fatal and attempted accidents, have a serious negative impact on the construction industry. Fall accidents are one of the main causes of casualties in the construction industry. In China, Europe, the United States and Singapore, fatal fall accidents account for more than 30% of the construction industry. At the same time, non-fatal fall accidents are also common on construction sites, which can cause workers to be injured and increase the number of days off work, which may cause delays in construction progress and thus cause huge economic losses. In addition, for attempted fall accidents, although they are not widely reported due to their minor consequences, their frequent occurrence on construction sites reveals a high level of safety risk on construction sites. Therefore, fall accidents have a significant impact on the construction period and cost of construction projects, but there is a lack of all-weather and all-round management means and methods.

[0003] Recognizing, detecting and analyzing fall precursors (such as loss of balance and sudden shaking) can track precursor events before a fall accident, which can help managers avoid the next fall accident on a construction site. In addition, if a worker's fall precursor event lasts for a long time, timely detection and alarm can prevent serious consequences of a fall accident. In terms of the severity of the consequences of a fall accident, the proportion of fatal accidents is not high, and most are attempted accidents or minor injuries. Therefore, non-fatal and attempted accidents do not necessarily result in serious consequences, but still require identification, detection and analysis.

[0004] In recent years, with the continuous development of computer hardware and software technology and the improvement of computing power, computer vision and deep learning technology have rapidly grown in various fields. Face recognition, autonomous driving and other fields have achieved remarkable results, and target detection in some specific fields has even approached or exceeded the recognition accuracy of humans. In the field of civil engineering construction safety management, there are mature computer vision-based safety helmet and protective clothing detection technologies. Traditional fall detection methods are based on sensors, which are costly and may interfere with workers' normal construction production activities. There is a lack of a computer vision-based fall detection method that achieves high detection accuracy, is non-intrusive and has a relatively low deployment cost. SUMMARY

[0005] The purpose of the present application is to provide a computer vision-based construction site worker fall precursor detection method, which uses a non-contact method of 3D pose estimation algorithm and machine learning model to identify, detect and analyze the construction site worker fall precursor, improve the detection accuracy, reduce the detection cost, and avoid interference with the construction site.

[0006] The purpose of the present application can be achieved by the following technical solutions:

[0007] A computer vision-based construction site worker fall precursor detection method, comprising the following steps:

[0008] Step 1) Obtain the construction site video stream as the training data set;

[0009] Step 2) Take the picture shot by the monocular camera as the object, establish a human body three-dimensional pose estimation model based on the VideoPose3D algorithm and train it;

[0010] Step 3) Based on the human body three-dimensional pose estimation model, extract the coordinate values of the worker's human body skeleton joint in the three-dimensional coordinate system, and generate the worker's human body skeleton joint coordinate matrix;

[0011] Step 4) Use the principal component analysis (PCA) method to reduce the dimension of the worker's human body skeleton joint coordinate matrix, visualize the worker's fall process, and carry out a pre-experiment to select the human body skeleton joints that have positive or negative effects on the human body three-dimensional pose estimation model;

[0012] Step 5) Based on the traditional nearest neighbor (KNN) model, fuse the three-dimensional coordinate information of the human body skeleton joints that have positive effects on the human body three-dimensional pose estimation model in a weighted manner based on a weight function, construct a machine learning model based on the improved Fusion-KNN and train it to obtain the optimal weight function;

[0013] Step 6) Detect the construction site worker fall precursor based on the trained machine learning model, and output the worker's body posture state, which includes stable, unbalanced and falling.

[0014] The VideoPose3D algorithm is a two-stage algorithm,

[0015] In the first stage, the Mask R-CNN two-dimensional human pose estimation algorithm is used as a two-dimensional human body skeleton joint detector to detect continuous two-dimensional human body skeleton joint motion trajectories from the input video;

[0016] In the second stage, a continuous frame convolutional neural network based on the WaveNet framework is used to perform convolution operation on the time series two-dimensional human skeleton joint coordinate sequence in the time dimension to extract features, and accordingly, the three-dimensional human pose in the input video is predicted.

[0017] The three-dimensional pose estimation model uses labeled real 3D human pose data when being trained, and the loss function thereof is the weighted average of each joint position error:

[0018]

[0019] Wherein, y z is the depth of the human skeleton joint in the camera space, f(x) is the predicted value of the pose estimation model, and y is the real value of the pose coordinates.

[0020] The specific process of reducing the dimension of the worker human skeleton joint coordinate matrix by using the principal component analysis method is:

[0021] The given training sample X m×n is composed of m samples x1, x2, …, x i , x m , …, x i , wherein x i represents the i-th sample, x n ∈R j ,

[0022]

[0023] Let:

[0024] Y=(X-μ)A

[0025] Wherein, Y is the new feature matrix after transformation, A is the feature transformation matrix, and μ is the sample mean;

[0026] A is an n×n orthogonal matrix composed of the eigenvectors of the sample covariance matrix:

[0027] ;

[0028] The solution method of the eigenvectors of the sample covariance matrix C is:

[0029] λ j a j =Ca j j=1,…,n

[0030] Wherein, λ j is an eigenvalue of the sample covariance matrix C, a j is the corresponding eigenvector, and the orthogonal transformation matrix A is:

[0031] A=(a1,L,a n )

[0032] definition Main component y k Contribution rate, Principal components y1, y2, ..., y k The cumulative contribution rate is calculated, and the value of k is determined based on the required cumulative contribution rate. That is, the first k new features are retained as principal components.

[0033] The specific process of visualizing and analyzing the worker's fall is as follows:

[0034] In the two-dimensional visualization based on principal component analysis, the first principal component is used as the horizontal axis and the second principal component is used as the vertical axis for visualization analysis to determine the transition characteristics between the three states of the fall process (i.e., stability, loss of balance, and fall).

[0035] The specific process of conducting preliminary experiments and selecting human skeletal joints that have a positive or negative impact on the human three-dimensional pose estimation model is as follows:

[0036] A preliminary 3D human pose estimation model is established, and the accuracy of the model for each combination of human skeletal joints is calculated. Specifically, a given human skeletal joint `joint_i` is selected, and all possible combinations of it with other human skeletal joints are enumerated. Each human skeletal joint has two selection options: accepting and rejecting. All these combinations are then substituted into the 3D human pose estimation model, and the average accuracy of all combinations containing `joint_i` is obtained. And the average accuracy of all combinations that do not contain joint_i. For each human skeleton joint_i, if the following conditions are met... If the joint_i of the human skeleton has a positive impact on the entire process of a worker falling at a construction site, then the joint_i of the human skeleton is determined to be a key node in the human three-dimensional pose estimation model; otherwise, the joint_i of the human skeleton is determined not to be a key node in the human three-dimensional pose estimation model.

[0037] The method of fusing the 3D coordinate information of human skeletal joints that positively impacts the 3D pose estimation model based on a weighted function is implemented using Mahalanobis distance. The specific process is as follows:

[0038] In video data, each frame is a sample, assuming the number of samples is m, s n If it is one of all samples, its mathematical description is as follows:

[0039] s n =(s xn ,s y n ,s z n )

[0040] s j n =(s j1 n ,s j2 n ,…,s jd n j = x, y, z

[0041] Among them, s x n ,s y n ,s z n These are the components of each sample along the x, y, and z axes, and each component is a d-dimensional vector;

[0042] The Mahalanobis distance between the components of the sample in different directions is expressed as:

[0043]

[0044] Where n and k are two distinct sequence numbers in the sample set, σ(s) j n ,k j n ) is sample s n and sample s k Standard deviation in a certain direction;

[0045] Weighted distance between samples By d x n,k ,d y n,k ,d z n,k Weighted average yields:

[0046]

[0047] in, It is s n With s k The weighted distance between them, w x ,w y ,w z It represents the weights of the x, y, and z coordinates.

[0048] The machine learning model based on the improved Fusion-KNN uses the weighted distance between samples and the true label to calculate the probability of the corresponding category and predict the probability. is the real label of k nearest neighbor samples is the proportion of samples of class i

[0049]

[0050] where, δ i is the correct number of prediction as class i, k represents the number of k nearest neighbor samples, is the real label of sample t; class i corresponds to three states of worker human posture: stable normal, lose balance LOB and fall.

[0051] The machine learning model based on the improved Fusion-KNN compares each prediction probability , finds the maximum value among them, and obtains the prediction label through the sample label corresponding to the maximum value:

[0052]

[0053] where, k is the hyperparameter of the KNN machine learning model.

[0054] The machine learning model is trained to maximize the score function, and the optimal weight function that maximizes the number of correct model predictions is obtained:

[0055]

[0056] s.t.w x +w y +w z = 1

[0057] Compared with the prior art, the present application has the following beneficial effects:

[0058] (1) The present application proposes a method for detecting the pre-fall of workers on construction sites based on computer vision and machine learning technology, which is used to determine the posture state of workers in stable, lose balance or fall, and realizes automatic detection of the whole process of worker falling without affecting the construction process.

[0059] ​(2) The application generates a three-dimensional skeleton joint coordinate matrix of workers in the construction site based on the three-dimensional pose estimation model of human body based on computer vision, and then uses principal component analysis (PCA) to reduce the dimension of the high-dimensional coordinate matrix data and visualize it. The process is screened for workers whose skeleton joint nodes have a positive impact on the process, and then the three-dimensional coordinate information of the x, y, and z axes of the workers' skeleton joint nodes with a positive impact is fused into the Fusion-KNN machine learning model in a weighted manner, an improved Fusion-KNN machine learning model is constructed, which is used to output the posture state (stable, lose balance or fall) of the worker. Through experimental verification, the accuracy of the model reaches 85.02%, and the detection accuracy is high.

[0060] (3) The application only needs to arrange a camera in the construction site to realize detection, and the hardware cost is low. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The method flowchart of the application;

[0062] Figure 2 The human joint model schematic diagram;

[0063] Figure 3 The three-dimensional joint coordinate matrix (z-axis) schematic diagram;

[0064] Figure 4 The PCA visualization result graph of the slipping process in an embodiment;

[0065] Figure 5 The PCA visualization result graph of the tripping process in an embodiment;

[0066] Figure 6 The pre-experiment result graph in an embodiment;

[0067] Figure 7 The optimal weight solving result graph in an embodiment;

[0068] Figure 8 The real label and prediction result graph in an embodiment, wherein (a) is the real label, and (b) is the prediction result. DETAILED DESCRIPTION

[0069] The application will be described in detail below in combination with the drawings and specific embodiments. The present embodiment is implemented on the basis of the technical solution of the application, and gives a detailed implementation manner and specific operation process, but the protection scope of the application is not limited to the following embodiments.

[0070] The present embodiment provides a computer vision-based construction site worker fall precursor detection method, as shown in Figure 1As shown, comprising the following steps:

[0071] Step 1) Obtain the construction site video stream as the training data set.

[0072] In this embodiment, the data source for detecting the premonitory signs of a construction site worker falling is the construction site video stream captured by a monocular camera. Therefore, human pose estimation of the worker target in the video stream is first needed.

[0073] Step 2) Based on the VideoPose3D algorithm, a three-dimensional human pose estimation model for the construction site is established and trained using pictures captured by the monocular camera as the object.

[0074] The VideoPose3D algorithm mainly estimates the three-dimensional human pose of a single person in a video, and it is a two-stage algorithm.

[0075] In the first stage, a two-dimensional human pose estimation algorithm such as Mask R-CNN is used as a two-dimensional human skeleton joint detector. This detector can detect continuous two-dimensional human skeleton joint motion trajectories from the input video.

[0076] In the second stage, a continuous frame convolutional neural network based on the WaveNet framework is used to perform convolution operations on the time-series two-dimensional human skeleton joint coordinate sequence in the time dimension to extract features, and based on this, the three-dimensional human pose in the input video is predicted. In addition, in the training process of this three-dimensional human pose prediction neural network, in addition to using the continuous two-dimensional human skeleton joint motion trajectory (i.e. the time-series coordinate sequence) prediction result of the first stage as input, the 2D human skeleton joint data in the GroundTruth algorithm can also be directly used as the input source.

[0077] When training the above three-dimensional human pose estimation model, real 3D human pose data with labels is used, and the loss function is the weighted mean per joint position error (WMPJPE).

[0078]

[0079] where y z is the depth of the human skeleton joint in the camera space, and the meaning represented is that a lower weight is given to the human skeleton joint farther from the camera. This is because when the human body is farther from the camera, the same absolute deviation will result in a smaller observed deviation after being projected onto the screen; moreover, the farther the human body is from the camera, the more difficult it is to estimate an accurate trajectory; therefore, the weight given should gradually decrease as the distance between the human body and the camera increases. f(x) is the predicted value of the pose estimation model, and y is the true value of the pose coordinates.

[0080] Step 3) Based on the human three-dimensional pose estimation model, the coordinates of the worker's human skeleton joint nodes in the three-dimensional coordinate system are extracted, and a worker's human skeleton joint node coordinate matrix is generated.

[0081] After the human three-dimensional pose estimation is completed by using the VideoPose3D model, the coordinate values of 17 human skeleton joint nodes (as shown in Figure 2 ) of each frame image in the three-dimensional space coordinate system are obtained. Assuming that the total number of frames of the input video is Frame, the output raw data is a three-dimensional matrix of 17x3xFrame. Among them, Frame is the time sequence dimension, and 17x3 is the spatial coordinate dimension. Figure 3 The human three-dimensional skeleton joint coordinate matrix in the z-axis direction is shown. Each frame is considered separately below to measure the spatial characteristics of the human three-dimensional pose.

[0082] In each frame of data, the total number of human skeleton feature points based on Human3.6M is 17, as shown in Figure 2 , so the data output by the human three-dimensional pose estimation is a 17-dimensional high-dimensional vector in three directions. Because the data dimension is too high, it is not conducive to model establishment, and data redundancy and noise may occur, so data preprocessing is needed. As Figure 2 known, since the human skeleton joint node "0-hip" is the coordinate origin of the three-dimensional space coordinate system, its three-dimensional coordinate value is directly close to 0, so only the human skeleton joint nodes other than the "0-hip" node are considered in the subsequent processing, and the total number is 16.

[0083] Step 4) Use the principal component analysis (PCA) method to reduce the dimension of the worker's human skeleton joint coordinate matrix, perform visual analysis on the worker's falling process, and carry out a pre-experiment to select the human skeleton joint nodes that have a positive or negative impact on the human three-dimensional pose estimation model.

[0084] Using the principal component analysis (PCA) method, the human three-dimensional pose estimation high-dimensional matrix obtained above is reduced in dimension, and the worker's falling process (i.e. stable-lose balance-fall) is visualized. The specific process of dimension reduction and visualization is as follows:

[0085] The training sample X m×n is composed of m samples x1, x2, …, x i , x m , where x i represents the i-th sample, x i ∈R n .

[0086]

[0087] Let

[0088] Y = (X - μ)A (3)

[0089] where Y is the new feature matrix after transformation, A is the feature transformation matrix, and μ is the sample mean. The meaning of formula (3) is to solve an optimal orthogonal transformation A, so that the variance of the new feature matrix Y is maximized. Here, the orthogonal transformation is used to ensure that the new features are not correlated. According to the viewpoint of probability statistics, the greater the variance of the new feature, the more information it contains, and the more important the feature is.

[0090] A is an n x n orthogonal matrix composed of the eigenvectors of the sample covariance matrix

[0091]

[0092] The solution method of the sample covariance matrix C eigenvector is:

[0093] λ j a j =Ca j j = 1,...,n (5)

[0094] where λ j is an eigenvalue of the sample covariance matrix C, a j is the corresponding eigenvector, and the orthogonal transformation matrix A is:

[0095] A = (a1,L,a n ) (6)

[0096] Define the contribution rate of the principal component y k as , and the cumulative contribution rate of the principal components y1,y2,…,y k as . Accordingly, the value of k can be calculated according to the required cumulative contribution rate, that is, the first k new features are retained as principal components.

[0097] In two-dimensional visualization based on principal component analysis (PCA), PC0 represents the first principal component, and PC1 represents the second principal component. As can be seen, the 16-dimensional high-dimensional matrix of human three-dimensional pose estimation output data can be reduced in dimension by principal component analysis (PCA), and its data features are expressed in two dimensions by the two most prominent principal components, as shown in Figure 4 , Figure 5 , Figure 4 , Figure 5It can be seen that the whole process from "normal" to "loss of balance" to "fall" has obvious linear transition characteristics. This can be more intuitive to verify that it is feasible to extract the characteristics of the whole process of the construction site worker falling by using the mathematical theory-based method.

[0098] Subsequently, pre-experiments were carried out, and human skeleton joints that have positive or negative effects on the above model were selected. The specific process is as follows:

[0099] A three-dimensional human posture estimation model was initially established, and the model accuracy of each combination of 16 human skeleton joints was solved. That is, a certain human skeleton joint joint_i is selected, and all combination modes of joint_i and the other 15 human skeleton joints are exhausted. For the other 15 human skeleton joints, each human skeleton joint has two selection modes, namely, to be adopted or not to be adopted, so there are combination modes containing joint_i. Similarly, there are combination modes not containing joint_i. By substituting all the above combination modes (i.e., 32768 x 2 = 65536 combination modes) into the three-dimensional human posture estimation model, the accuracy average of all combination modes containing joint_i and the accuracy average of all combination modes not containing joint_i can be obtained. Figure 6

[0100] For each human skeleton joint joint_i, if

[0101]

[0102] it is judged that the human skeleton joint joint_i has a positive effect on the whole process of "stable-lose balance-fall" of the construction site worker, and therefore it is determined that the human skeleton joint joint_i is a key node of the three-dimensional human posture estimation algorithm model.

[0103] On the contrary, if

[0104]

[0105] it is judged that the human skeleton joint joint_i has a negative effect on the whole process of "stable-lose balance-fall" of the construction site worker, and therefore it is determined that the human skeleton joint joint_i is not a key node of the three-dimensional human posture estimation algorithm model.

[0106] ​Step 5) On the basis of a traditional nearest neighbor (KNN) model, the three-dimensional coordinate information of the human skeleton joint which has a positive effect on the human three-dimensional pose estimation model is fused in a weighted manner based on a weight function, an improved Fusion-KNN-based machine learning model is constructed and trained, and an optimal weight function is obtained.

[0107] The main process of the common K nearest neighbor (KNN) algorithm is as follows: all training samples (s 1 ,s 2 , K, s m are stored in an n-dimensional model space. For a sample to be classified, the model space is searched, and the K nearest neighbors of the sample are found according to the approximation degree. According to the K nearest neighbors, the type of the sample to be classified is predicted using a certain voting strategy. Among them, there are many methods to measure the approximation degree between samples, mainly Euclidean distance and chord similarity. The calculation formula of Euclidean distance is shown in formula (9).

[0108]

[0109] Among them, is the Euclidean distance between v1 and v2.

[0110] However, the above method can only process samples composed of a single vector, that is, the matrix R d×1 In the present application, each sample is composed of 3 vectors from 3 directions (x, y and z). Therefore, the matrix of each sample is R d×3 , which makes the above common vector-based calculation method of sample distance not applicable. In view of this, the present application fuses the three-dimensional coordinate information of the worker human skeleton joint x, y and z axis which has a positive effect on the worker three-dimensional pose estimation in an improved Fusion-KNN machine learning model in a weighted manner. The specific process is as follows:

[0111] In video data, each frame is a sample. Assuming that the number of samples is m, s n is one of all samples, its mathematical description is as follows:

[0112] s n = (s x n ,s y n ,s z n )s x n ,s y n ,s zn ∈ 16×1 s n ∈ 16×3 (10)

[0113] s j n =(s j1 n ,s j2 n ,L,s jd n )d=16 j=x,y,z (11)

[0114] Among them, s x n ,s y n ,s z n It represents the components of each sample in three directions (x, y, z axes), and each component is a 16-dimensional vector. j n,k It is the Mahalanobis distance, which is generated based on the Euclidean distance, but is more suitable for non-two-dimensional sample spaces than the Euclidean distance. Its mathematical description is as follows:

[0115]

[0116] Where n and k are two distinct sequence numbers in the sample set. σ(s j n ,k j n ) is sample s n and sample s k The standard deviation in a certain direction (x, y, or z axis).

[0117] It can be made by d x n,k ,d y n,k ,d z n,k The weighted average is as follows.

[0118]

[0119] in It is s n With s k The weighted distance between them, w x ,w y ,w z It represents the weights of the x, y, and z coordinates.

[0120] After obtaining samples s nAfter the distance between the other to-be-predicted samples, the probability of the corresponding class can be calculated according to the distance and the true label. That is, the prediction probability The real label in the k nearest neighbor samples is the proportion of the sample of class i , as shown below.

[0121]

[0122] wherein, δ i is the correct number of prediction as class i, and class i corresponds to three states of the worker's body posture: stable normal, loss of balance LOB, and fall; k is the number of k nearest neighbor samples, is the true label of sample t.

[0123] Finally, each prediction probability is compared to find the maximum value. The prediction label is obtained by the sample label corresponding to the maximum value, as shown below.

[0124]

[0125] wherein, k is the hyperparameter of the KNN machine learning model.

[0126] In the machine learning model, a loss function is often used to optimize the objective function. The loss function proposed in the present application is as shown below.

[0127]

[0128] wherein, δ is the total number of correct predictions of all classes, is the true label of the k nearest neighbor samples of s n . Because the goal of the loss function is to minimize the number of prediction errors, the current w x , w y , w z values can be obtained.

[0129] In the present application, in order to optimize the code and enhance the logicality of modeling, when establishing the optimization function model, the loss function is not used, but the model is optimized by taking the score function as the target, that is, maximizing the score function, so as to obtain the weight coefficient that makes the model prediction correct number the most, as shown in formula (17) and Figure 7 .

[0130]

[0131] Step 6) Detecting the worker fall precursor on the construction site of the construction project based on the trained machine learning model, and outputting the state of the worker's body posture.

[0132] After obtaining the optimal weights of x, y, z directions through the optimization function, the optimal weight coefficients are used to realize the information fusion of the three coordinate directions. In the three stages of the whole process of worker falling down, i.e. stable-lose balance-fall down, the weight of the coordinate value of each human skeleton joint in the vertical direction (i.e. z direction) is different from that in the horizontal direction (i.e. x, y direction) in theory. In the whole process of worker falling down, the z coordinate value of each human skeleton joint presents a significant decreasing trend. In theory, the change of the coordinate value of each human skeleton joint in the vertical direction (i.e. z direction) will play a dominant role in the whole process of worker falling down. Therefore, the above-mentioned loss function optimization method used by the present application can quantitatively calculate the dominant degree of the three coordinate axis directions in a relatively accurate manner. Accordingly, through pre-experiment, the coordinate axis direction which plays a control role in the three stages of stable-lose balance-fall down of worker falling down can be obtained, so that the construction site data collection and data processing work can be carried out in a targeted manner.

[0133] After obtaining the optimal weights of x, y, z directions through the iteration of the above-mentioned loss function, the w x , w y , w z obtained by taking the minimum value of the loss function are used to calculate the information fusion of the three coordinate directions x, y and z according to formula (13). Then, the final predicted category (i.e. stable, lose balance, fall down) and the confidence probability thereof are calculated according to formula (15) and formula (14), as shown in formula (16). Figure 8

[0134] In summary, the improved Fusion-KNN model constructed by using the present application can be used for detecting the pre-fall of workers in construction sites and outputting the human posture state (stable, lose balance or fall down).

[0135] The present application takes the action posture of workers in construction sites as the detection and analysis object, and proposes a non-contact method based on computer vision to directly detect the pre-fall of workers. First, a three-dimensional posture estimation algorithm model is used to generate the skeleton joint coordinate matrix of workers. Second, the matrix is preprocessed through principal component analysis and pre-experiment. Finally, an improved K nearest neighbor machine learning probability model which fuses the three-dimensional direction information of x, y and z axes is established to output the action posture state of workers, i.e. stable, lose balance and fall down. Experiments show that the accuracy of the model proposed by the present application reaches 85.02%. The pre-fall detection method of workers in construction sites proposed by the present application can extract the falling risk of workers in the pre-fall and post-fall stages, and provides an important basis for the safety risk assessment and management of workers in construction sites. ​

[0136] The preferred embodiments of the application have been described above in detail. It should be understood that modifications and variations to the preferred embodiments could be made by those skilled in the art in light of the teachings above. It is therefore contemplated that the application can encompass other variations and modifications that fall within the scope of the claims.

Claims

1. A method for detecting pre-fall warning signs in construction workers at a building construction site based on computer vision, characterized in that, Includes the following steps: Step 1) Obtain the video stream from the construction site as the training dataset; Step 2) Using images captured by a monocular camera as the object, establish and train a three-dimensional human pose estimation model for the construction site based on the VideoPose3D algorithm; Step 3) Based on the human body three-dimensional pose estimation model, extract the coordinate values ​​of the joints of the worker's human skeleton in the three-dimensional coordinate system and generate the coordinate matrix of the joints of the worker's human skeleton. Step 4) Use principal component analysis to reduce the dimensionality of the coordinate matrix of the worker's human skeleton joints, visualize the worker's fall process, and conduct preliminary experiments to select human skeleton joints that have a positive or negative impact on the human three-dimensional posture estimation model. The specific process of conducting preliminary experiments and selecting human skeletal joints that have a positive or negative impact on the human three-dimensional pose estimation model is as follows: A preliminary three-dimensional human pose estimation model is established, and the model accuracy for each combination of human skeletal joints is calculated, i.e., a specific human skeletal joint is selected. joint_i This involves exhaustively listing all possible combinations of the joint with other human skeletal joints. For each human skeletal joint, there are two selection methods: accepting and not accepting. Substituting all these combinations into the human 3D pose estimation model yields the result containing... joint_i Average accuracy of all combinations and does not contain joint_i Average accuracy of all combinations For each human skeletal joint joint_i If satisfied Then determine the joints of the human skeleton. joint_i It has a positive impact on the entire process of falls among construction workers, and helps to determine the joint points of the human skeleton. joint_i These are key nodes in the human body 3D pose estimation model; conversely, they determine the joints of the human skeleton. joint_i It is not a critical node in the human 3D pose estimation model; Step 5) Based on the traditional nearest neighbor model, the three-dimensional coordinate information of human skeleton joints that have a positive impact on the human three-dimensional pose estimation model is fused in a weighted manner based on the weight function, and a machine learning model based on the improved Fusion-KNN is constructed and trained to obtain the optimal weight function. The method of fusing the 3D coordinate information of human skeletal joints that positively impacts the 3D pose estimation model based on a weighted function is implemented using Mahalanobis distance. The specific process is as follows: In video data, each frame is a sample, assuming the number of samples is... m , s n If it is one of all samples, its mathematical description is as follows: in, Each sample is in x,y,z Components along the axial direction, each component is a d dimensional vector; The Mahalanobis distance between the components of the sample in different directions is expressed as: in, n , k These are two different sequence numbers in the sample set. It is a sample s n and samples s k Standard deviation in a certain direction; Weighted distance between samples Depend on , , Weighted average yields: in, yes s n and s k The weighted distance between them w x , w y , w z yes x , y , z The weights of the three directional coordinates; Step 6) Detect the signs of a worker falling at a construction site based on the trained machine learning model, and output the state of the worker's body posture, including stable, loss of balance, and falling.

2. The method for detecting pre-fall warning signs of construction workers based on computer vision according to claim 1, characterized in that, The VideoPose3D algorithm is a two-stage algorithm. In the first stage, the Mask R-CNN 2D human pose estimation algorithm is used as a 2D human skeleton joint detector to detect the continuous 2D human skeleton joint motion trajectory from the input video. In the second stage, a continuous-frame convolutional neural network based on the WaveNet framework is used to perform convolution operations on the temporal two-dimensional human skeleton joint coordinate sequence in the time dimension to extract features, and based on this, to predict the three-dimensional human pose in the input video.

3. The method for detecting pre-fall warning signs of construction workers based on computer vision according to claim 1, characterized in that, The 3D pose estimation model is trained using labeled real 3D human pose data, and its loss function is a weighted average of the position error per joint. in, It is the depth of the human skeletal joints in camera space. These are the predicted values ​​from the attitude estimation model. These are the actual values ​​of the attitude coordinates.

4. The method for detecting pre-fall warning signs of construction workers based on computer vision according to claim 1, characterized in that, The specific process of using principal component analysis to reduce the dimensionality of the coordinate matrix of the joints of the worker's human skeleton is as follows: Given training samples Depend on m Sample x 1 ,x 2, …, x i …, x m Composition, in which x i Indicates the first i One sample, , set up: in, Y It is the new characteristic matrix after transformation. A It is the characteristic transformation matrix. It is the sample mean; A It is n×n The orthogonal matrix is ​​derived from the sample covariance matrix: The eigenvectors are composed of; Sample covariance matrix C The method for solving the eigenvectors is as follows: in, It is the sample covariance matrix C One of the eigenvalues, If the corresponding eigenvectors are the orthogonal transformation matrix, then... A for: definition Main component y k Contribution rate, Main component y 1 ,y 2 ,…,y k The cumulative contribution rate is calculated based on the required cumulative contribution rate. k The value that is retained before k One new feature is used as the principal component.

5. The method for detecting pre-fall warning signs of construction workers based on computer vision according to claim 1, characterized in that, The specific process of visualizing and analyzing the worker's fall is as follows: In a two-dimensional visualization based on principal component analysis, the first principal component is used as the horizontal axis and the second principal component as the vertical axis for visualization analysis to determine the transition characteristics between the three states of the fall process.

6. The method for detecting pre-fall warning signs of construction workers based on computer vision according to claim 1, characterized in that, The machine learning model based on the improved Fusion-KNN uses the weighted distance between samples and the true label to calculate the probability of the corresponding category and predict the probability. yes k The true label of the nearest neighbor samples for The proportion of the sample: in, It is predicted as a category i The correct quantity, k express k The number of nearest neighbor samples, It is a sample t Authentic labels; categories i The three states corresponding to the worker's human posture: stable normal Loss of balance LOB and falling fall .

7. The method for detecting pre-fall warning signs of construction workers based on computer vision according to claim 6, characterized in that, The machine learning model based on the improved Fusion-KNN predicts each probability. Compare the samples, find the maximum value, and obtain the predicted label using the sample label corresponding to the maximum value: in, k yes KNN Hyperparameters of machine learning models.

8. A method for detecting pre-fall warning signs of construction workers based on computer vision according to claim 7, characterized in that, The machine learning model is trained with the goal of maximizing the score function, resulting in an optimal weight function that maximizes the number of correct predictions. 。

Images