Construction site personnel management method and system based on neural network

The CNN-LSTM based system addresses the inefficiencies of manual video inspection by providing real-time detection and warning of unsafe behaviors, enhancing construction site safety and management efficiency.

CN119068546AInactive Publication Date: 2024-12-03GUANGZHOU VOCATIONAL COLLEGE OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411061369.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional manual inspection of construction site surveillance videos for unsafe worker behaviors is labor-intensive, prone to fatigue-induced misjudgment, and unable to timely identify and respond to unsafe actions, leading to potential accidents.

Method used

A construction site personnel management system using a convolutional neural network (CNN) with an attention mechanism and long short-term memory (LSTM) network to extract spatial and temporal features from video data, enabling real-time monitoring and early warning of unsafe behaviors.

Benefits of technology

The system accurately identifies and responds to unsafe behaviors in real-time, reducing accident probability, improving safety management efficiency, and promoting a culture of safety awareness among workers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068546A_ABST
    Figure CN119068546A_ABST
Patent Text Reader

Abstract

The invention discloses a construction site personnel management method and system based on a neural network. The invention relates to the technical field of constructional engineering. An attention mechanism module is arranged behind the convolutional layer of the CNN. The module can emphasize key areas related to unsafe behaviors by calculating the weight of the feature map. The attention mechanism adopts a hard attention form. A feature map output by any convolutional layer of the CNN is set as F belonging to RH * W * C, wherein H, W and C respectively represent the height, width and channel number of the feature map. The attention mechanism module aims to calculate a weight for each position of the feature map so as to emphasize key areas related to unsafe behaviors. By means of the real-time monitoring and early warning system, unsafe behaviors of constructors can be found and corrected in time, potential safety hazards are effectively avoided, and the accident occurrence probability is remarkably reduced. By operating the scheme for a long time, the constructors can gradually realize the safety of their behaviors.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of construction engineering, in particular to the application of neural network technology in construction engineering, and in particular to a construction site personnel management method and system based on neural network. Background Art

[0002] Unsafe behaviors of construction site workers are one of the main reasons for frequent construction safety accidents. These unsafe behaviors include but are not limited to illegal operations and failure to wear safety protection equipment.

[0003] The traditional method of manually inspecting surveillance videos to punish and reward has many disadvantages. Viewing massive and complex surveillance videos requires a large number of staff to invest a lot of time to participate. Not only is the labor cost high, but long-term viewing of surveillance videos can easily induce fatigue and lead to misjudgment. In addition, it is difficult for manual methods to quickly identify relevant information from a large amount of data in a timely manner. It takes a certain period of time and is easy to miss the best investigation period, resulting in an inability to respond in time when an accident occurs.

[0004] To this end, the present invention proposes a construction site personnel management method and system based on neural network. Summary of the invention

[0005] In view of this, the embodiment of the present invention hopes to provide a construction site personnel management method and system based on a neural network to solve or alleviate the technical problems existing in the prior art, namely, how to intelligently judge and classify the unsafe behaviors of construction site personnel, provide auxiliary decision support for personnel management, and at least provide a beneficial option for this; the technical solution of the present invention is implemented as follows:

[0006] First, the construction site personnel management method based on neural network:

[0007] 1. Overview:

[0008] The present invention aims to solve the above technical problems. Convolutional neural network (CNN) is used to extract features from video data of construction sites, an attention mechanism is introduced to emphasize key areas related to unsafe behaviors, and a long short-term memory (LSTM) network is used to capture the time dependency in feature sequences, ultimately achieving accurate identification and real-time monitoring of unsafe behaviors of construction workers.

[0009] (II) Technical solution:

[0010] In order to achieve the above technical objectives, the present invention needs to perform the following steps S1 to S4.

[0011] 2.1 Step S1, data preparation and preprocessing:

[0012] Install surveillance cameras at the construction site to collect video data V containing the construction workers' behavior:

[0013] V={v 1 ,v 2 ,…,v n};

[0014] where v i represents the i-th video.

[0015] The collected video data V is labeled to clearly mark the unsafe behaviors of the construction workers:

[0016] L = {l 1 ,l 2 ,…,l n}

[0017] Among them, l i is with video v i The corresponding annotation information includes the type and timestamp of the unsafe behavior.

[0018] Split the video data into keyframes and sub-video segments. Keyframes select frames with large changes in the video, and sub-video segments are divided according to time intervals or event trigger conditions. At the same time, normalize the keyframes and sub-video segments to have the same size and pixel value range:

[0019] Use the inter-frame difference method to select the changed frames in the video as key frames, denoted as: K = {k 1 ,k 2 ,…,k m};

[0020] where k j represents the jth key frame.

[0021] The video is divided into sub-video segments according to the time interval or event triggering condition, denoted as: S = {s 1 ,s 2 ,…,s p};

[0022] where s q Represents the qth sub-video segment.

[0023] Finally, the input feature vector x is formed as the input of the convolutional neural network in step S2.

[0024] x={k 1 ,s 1},{k 2 ,s 2},...,{k m ,s m};

[0025] Where m is the sample {k i ,s i}.

[0026] 2.2 Step S2, read the convolutional neural network (CNN):

[0027] The architecture of a convolutional neural network consists of convolutional layers, pooling layers, and activation function layers. The convolutional layers are used to extract spatial features from images, and the pooling layers are used to reduce feature dimensions and retain important information.

[0028] 2.2.1 Step S200, layer stacking:

[0029] Convolutional layer C i , Pooling layer P i And the activation function layer R i The number of activation functions is 3. The activation function layer uses the ReLU activation function to increase the nonlinear expression ability of the model.

[0030]

[0031] Each layer receives the output of the previous layer as input and produces new feature maps or activation values ​​as output.

[0032] 2.2.2 Step S201, execute the attention mechanism:

[0033] The convolutional layer of CNN is followed by an attention mechanism module. This module can emphasize key areas related to unsafe behaviors by calculating the weights of feature maps. The attention mechanism adopts the form of hard attention.

[0034] Assume that the feature map output by any convolutional layer of CNN is F∈R H×W×C , where H, W, and C represent the height, width, and number of channels of the feature map, respectively. The goal of the attention mechanism module is to calculate a weight for each position of the feature map to emphasize the key areas related to unsafe behaviors.

[0035] The hard attention mechanism selects certain regions in the feature map and sets other regions to zero: M∈{0,1} H×W ;

[0036] Among them, 1 in the mask M represents the selected area, and 0 represents the unselected area.

[0037] Therefore, the output F′ of the attention mechanism module is: F′=F⊙M;

[0038] Where ⊙ represents element-by-element multiplication. In practical applications, the mask M is calculated from the feature map F through threshold processing, region selection algorithm, etc.

[0039] 2.2.3 Step S202, execute LSTM network:

[0040] The LSTM network is used to process the feature sequence extracted by CNN. The LSTM network can capture the temporal dependency in the feature sequence.

[0041] Assume that the feature sequence extracted by CNN is X = {x 1 ,x 2 ,…,x T}, where x t ∈R d represents the feature vector at time point t, T is the length of the sequence, and d is the dimension of the feature vector.

[0042] The goal of the LSTM network is to process this feature sequence and capture the temporal dependencies therein.

[0043] The LSTM network consists of multiple LSTM unit layers, each of which contains a forget gate, an input gate, and an output gate structure. At time point t, the calculation of the LSTM unit is:

[0044]

[0045] Where: f t 、i t , o t Represent the activation vectors of the forget gate, input gate, and output gate respectively; σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function; W f , W i , W o , W C Represents the weight matrices of the forget gate, input gate, output gate and candidate cell state respectively; b f 、b i 、b o 、b C Respectively represent the bias vectors of the forget gate, input gate, output gate and candidate cell state; ⊙ represents element-by-element multiplication; C t represents the cell state at time point t; h t represents the hidden state at time point t.

[0046] The output of the LSTM network is the hidden state h at the last time point T , or it can be a sequence of hidden states at all time points {h 1 ,h 2 ,…,h T}, depending on the needs of the task.

[0047] When processing the feature sequences extracted by CNN, the LSTM network is able to capture the temporal dependencies in the feature sequences, thus providing temporal information about the behavior of personnel at the construction site.

[0048] 2.2.4 Step S203, fusion:

[0049] The CNN is implemented through a fully connected layer and the output features of the LSTM are fused, and the features of different layers are mapped to the same dimension and weighted summed. After feature fusion, a fully connected layer and a softmax classifier are added to classify and predict unsafe behaviors.

[0050] First, the feature map realized by CNN through the fully connected layer is fused with the output features of LSTM. Let the feature vector obtained by CNN through the fully connected layer be z CNN ∈Rd CNN , the output feature vector of LSTM is z LSTM ∈RdL STM , where d CNN and d LSTM are the dimensions of CNN and LSTM output features respectively.

[0051] In order to fuse these two feature vectors, they are mapped to the same dimension d by adding an additional fully connected layer. The mapped feature vector is:

[0052]

[0053] Next, the mapped feature vectors are weighted summed to obtain the fused feature vector z fuse :

[0054]

[0055] Where α is a learnable parameter used to balance the importance of CNN and LSTM features.

[0056] After feature fusion, a fully connected layer and a softmax classifier are added to classify and predict unsafe behaviors. The fully connected layer can convert the fused feature vector z fused Mapped to the category space, we get the category score vector s∈R Nc , where Nc is the number of categories. The softmax classifier can convert the category score vector into a probability distribution p∈R Nc , where each element p i represents the probability of belonging to category i.

[0057] Specifically, the calculation of the fully connected layer and the softmax classifier is:

[0058] s=W fc ·zfused +b fc ;

[0059]

[0060] Where W fc and b fc are the weight matrix and bias vector of the fully connected layer respectively. Finally, the probability distribution p can be used to classify and predict unsafe behaviors.

[0061] 2.4 Model training:

[0062] Although the above steps use a convolutional neural network, the convolutional neural network still needs to be trained before it can be actually deployed. The methods include:

[0063] 2.4.1 First step:

[0064] Divide the labeled video dataset D into training set D train , test set D test and validation set D val .

[0065] 2.4.2 Step 2:

[0066] Using the training set D train The model is trained with data. The back propagation algorithm and gradient descent optimizer are used to update the model parameters during the training process.

[0067] 2.4.3 Step 3:

[0068] When the model converges, stop training.

[0069] 2.5 Step S3, module deployment:

[0070] The convolutional neural network that has passed the test is deployed to the construction site to monitor the unsafe behavior of construction workers in real time and issue early warning information.

[0071] Suppose the trained and tested convolutional neural network model is M, which can receive the real-time video stream V of the construction site as input and output the prediction P of the unsafe behavior of the construction workers. The steps of model deployment are:

[0072] Deploy(M,V)→Monitor(V)→P→Alert(P);

[0073] Where: Deploy(M,V) means deploying model M to the construction site and connecting to the real-time video stream V. Monitor(V) means real-time monitoring of the construction workers' behaviors in the video stream V. P is the prediction result of model M on the unsafe behaviors of construction workers in the video stream V. Alert(P) means issuing corresponding warning information based on the prediction result P.

[0074] (III) Mechanism for solving technical problems:

[0075] The designed CNN architecture is used to extract features from the preprocessed video frames. Through multiple layers of convolutional layers and pooling layers, CNN can automatically learn and extract spatial features in images, such as edges and textures. These features are crucial for identifying unsafe behaviors.

[0076] The attention mechanism module is introduced after one or some convolutional layers of CNN. By calculating the weights of the feature map, this module can automatically identify and emphasize the key areas related to unsafe behaviors, thereby improving the model's sensitivity to unsafe behaviors and the accuracy of recognition. The feature sequence extracted by CNN is input into the LSTM network. Through its unique forget gate, input gate, and output gate structure, LSTM can capture the time dependency in the feature sequence and understand the evolution of unsafe behaviors over time. In this way, the model can not only identify unsafe behaviors in a single frame, but also conduct coherent analysis and judgment of unsafe behaviors in multiple consecutive frames.

[0077] The model can learn and understand data at multiple levels, from low to high dimensions. Low-level features (such as edges and textures) are used to capture basic information about the image; high-level features (such as objects and scenes) are used to understand complex relationships and contextual information in the image. This multi-level learning mechanism helps the model more accurately judge and classify unsafe behaviors.

[0078] The trained model is deployed to the monitoring system of the construction site to monitor the behavior of construction workers in real time. Once unsafe behavior is detected, the system immediately issues an early warning message to remind the on-site management staff to take timely measures to intervene and correct it.

[0079] Second, the construction site personnel management system based on neural network:

[0080] like Figure 2 As shown, the system includes a processor and a register coupled to the processor, wherein the register stores program instructions, and when the program instructions are executed by the processor, the processor executes the construction site personnel management method as described above.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] 1. Improve the safety of construction sites: The present invention can timely discover and correct unsafe behaviors of construction workers through real-time monitoring and early warning systems, effectively avoid potential safety hazards, and significantly reduce the probability of accidents. Long-term operation of this solution can make construction workers gradually aware of the safety of their own behavior, and then consciously abide by safety regulations and form a good safety culture.

[0083] 2. Improve safety management efficiency: The present invention can automatically monitor the construction site around the clock and analyze the behavior of construction workers in real time, greatly reducing the workload of safety management personnel.

[0084] 3. Promote the transformation to intelligent management: This invention integrates advanced technologies such as convolutional neural networks, attention mechanisms, and long short-term memory networks, and is expected to promote the construction industry to develop in a more intelligent and efficient direction, and improve the safety management level of the entire industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0086] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0087] Figure 2 It is a schematic diagram of the system composition of the present invention;

[0088] Figure 3 Schematic diagram of the CNN neural network architecture of the present invention. DETAILED DESCRIPTION

[0089] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below;

[0090] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0091] Embodiment 1: Figure 1 and 3 As shown, this embodiment discloses a construction site personnel management method based on a neural network, which includes the following steps S1 to S3.

[0092] In this embodiment, regarding step S1, data preparation and preprocessing: a surveillance camera is installed at the construction site to collect video data V containing the behavior of construction workers: V = {v 1 ,v 2 ,…,v n};

[0093] where v i represents the i-th video.

[0094] The collected video data V is labeled to clearly mark the unsafe behaviors of the construction workers: L = {l 1 ,l 2 ,…,l n};

[0095] Among them, l i is with video v i The corresponding annotation information includes the type and timestamp of the unsafe behavior.

[0096] Split the video data into keyframes and sub-video segments. Keyframes select frames with large changes in the video, and sub-video segments are divided according to time intervals or event trigger conditions. At the same time, normalize the keyframes and sub-video segments to have the same size and pixel value range:

[0097] Use the inter-frame difference method to select the changed frames in the video as key frames, denoted as: K = {k 1 ,k 2 ,…,k m}; The inter-frame difference method detects motion or changes in the video by calculating the differences between adjacent frames, thereby selecting key frames.

[0098] where k j represents the jth key frame.

[0099] The video is divided into sub-video segments according to the time interval or event triggering condition, denoted as: S = {s 1 ,s 2 ,…,s p}; The time interval can be fixed or determined according to the dynamic changes of the video content. The event trigger condition can be the occurrence of a specific behavior or action. q Represents the qth sub-video segment.

[0100] Finally, the input feature vector x is formed as the input of the convolutional neural network in step S2:

[0101] x={k 1 ,s 1},{k 2 ,s 2},...,{k m ,s m};

[0102] Where m is the sample {k i ,s i}.

[0103] It is understandable that accurate data annotation ensures that the data used in model training is accurate and reliable, thereby improving the performance of the model. By normalizing key frames and sub-video clips, the size and pixel value differences between different videos can be eliminated, allowing the model to better generalize to new, unseen data. Splitting video data into key frames and sub-video clips can reduce the amount of data processed by the model, thereby improving training efficiency. At the same time, the selection of key frames and sub-video clips also retains important information in the video, which helps the model better learn the characteristics of unsafe behaviors.

[0104] In this embodiment, regarding step S2, a convolutional neural network (CNN) is read: the architecture of a convolutional neural network includes a convolution layer, a pooling layer, and an activation function layer. The convolution layer is used to extract spatial features in an image, and the pooling layer is used to reduce feature dimensions and retain important information. Each layer receives the output of the previous layer as input and generates a new feature map or activation value as output.

[0105] Specifically, step S200, layer stacking: convolution layer C i , Pooling layer P i And the activation function layer R i The number of activation functions is 3. The activation function layer uses the ReLU activation function to increase the nonlinear expression ability of the model.

[0106]

[0107] Each layer receives the output of the previous layer as input and produces new feature maps or activation values ​​as output.

[0108] Specifically, step S201, executes the attention mechanism: the convolutional layer of CNN has an attention mechanism module. This module can emphasize the key areas related to unsafe behaviors by calculating the weights of the feature map. The attention mechanism adopts the form of hard attention, that is, certain areas in the feature map are selected and other areas are set to zero. This can help the model pay more attention to the image areas that are closely related to unsafe behaviors. The output of the attention mechanism module is obtained by element-by-element multiplication of the feature map and the mask, and the mask is calculated from the feature map through threshold processing, region selection algorithm, etc.

[0109] Assume that the feature map output by any convolutional layer of CNN is F∈R H×W×C , where H, W, and C represent the height, width, and number of channels of the feature map, respectively. The goal of the attention mechanism module is to calculate a weight for each position of the feature map to emphasize the key areas related to unsafe behaviors.

[0110] The hard attention mechanism selects certain regions in the feature map and sets other regions to zero: M∈{0,1} H×W ;

[0111] Among them, 1 in the mask M represents the selected area, and 0 represents the unselected area.

[0112] Therefore, the output F′ of the attention mechanism module is: F′=F⊙M;

[0113] Where ⊙ represents element-by-element multiplication. In practical applications, the mask M is calculated from the feature map F through threshold processing, region selection algorithm, etc.

[0114] Specifically, step S202 executes the LSTM network: the LSTM network is used to process the feature sequence extracted by CNN. The LSTM network can capture the time dependency in the feature sequence. Suppose the feature sequence extracted by CNN is X = {x 1 ,x 2 ,…,x T}, where x t ∈R d represents the feature vector at time point t, T is the length of the sequence, and d is the dimension of the feature vector.

[0115] The goal of the LSTM network is to process this feature sequence and capture the temporal dependencies in it. The LSTM network consists of multiple LSTM unit layers, each of which contains a forget gate, an input gate, and an output gate structure. At time point t, the calculation of the LSTM unit is:

[0116]

[0117] Where: f t 、it , o t Represent the activation vectors of the forget gate, input gate, and output gate respectively; σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function; W f , W i , W o , W C Represents the weight matrices of the forget gate, input gate, output gate and candidate cell state respectively; b f 、b i 、b o 、b C Respectively represent the bias vectors of the forget gate, input gate, output gate and candidate cell state; ⊙ represents element-by-element multiplication; C t represents the cell state at time point t; h t represents the hidden state at time point t.

[0118] It should be noted that the output of the LSTM network is the hidden state h at the last time point T , or it can be a sequence of hidden states at all time points {h 1 ,h 2 ,…,h T}, depending on the needs of the task. When processing the feature sequence extracted by CNN, the LSTM network is able to capture the temporal dependencies in the feature sequence, thus providing temporal information about the behavior of personnel on the construction site.

[0119] Specifically, step S203, fusion: CNN is implemented through a fully connected layer and the output features of LSTM are fused, and the features of different layers are mapped to the same dimension and weighted summed. After feature fusion, a fully connected layer and a softmax classifier are added to perform classification prediction of unsafe behaviors.

[0120] First, the feature map realized by CNN through the fully connected layer is fused with the output features of LSTM. Let the feature vector obtained by CNN through the fully connected layer be z CNN ∈Rd CNN , the output feature vector of LSTM is z LSTM ∈RdL STM , where d CNN and d LSTM are the dimensions of CNN and LSTM output features respectively.

[0121] In order to fuse these two feature vectors, they are mapped to the same dimension d by adding an additional fully connected layer. The mapped feature vector is:

[0122]

[0123] Next, the mapped feature vectors are weighted summed to obtain the fused feature vector z fuse :

[0124]

[0125] Where α is a learnable parameter used to balance the importance of CNN and LSTM features.

[0126] After feature fusion, a fully connected layer and a softmax classifier are added to classify and predict unsafe behaviors. The fully connected layer can convert the fused feature vector z fused Mapped to the category space, we get the category score vector s∈R Nc , where Nc is the number of categories. The softmax classifier can convert the category score vector into a probability distribution p∈R Nc , where each element p i represents the probability of belonging to category i.

[0127] Specifically, the calculation of the fully connected layer and the softmax classifier is:

[0128] s=W fc ·z fused +b fc ;

[0129]

[0130] Where W fc and b fc are the weight matrix and bias vector of the fully connected layer respectively. Finally, the probability distribution p can be used to classify and predict unsafe behaviors.

[0131] It is understandable that through the use of layer stacking and ReLU activation function, CNN is able to extract deep features in the image and improve the model's ability to identify unsafe behaviors. The attention mechanism module can emphasize key areas related to unsafe behaviors, helping the model to pay more attention to important image information, thereby improving classification accuracy. The LSTM network can capture the temporal dependencies in feature sequences and provide temporal information about the behavior of construction site personnel, which helps the model better understand the development process of unsafe behaviors. Through feature fusion and softmax classifier, the model can effectively combine the features of CNN and LSTM, and perform classification and prediction of unsafe behaviors, providing auxiliary decision support for personnel management.

[0132] Furthermore, the Python program of the CNN model in step S2 is as follows:

[0133]

[0134]

[0135]

[0136]

[0137]

[0138] In the above program, SimpleCNN is a convolutional neural network that extracts features from images. AttentionModule is an attention mechanism module that receives the feature map extracted by CNN and applies a mask to emphasize the key areas related to unsafe behaviors. The mask is randomly generated, but in practical applications, it should be calculated based on the content of the feature map. LSTMNetwork is an LSTM network that processes the feature sequence extracted by CNN. It captures the temporal dependencies in the feature sequence and provides temporal information about the behavior of people on the construction site. UnsafeBehaviorClassifier is a classifier that fuses the outputs of CNN and LSTM and makes classification predictions for unsafe behaviors. It contains two fully connected layers that map the features of CNN and LSTM to the same dimension and perform weighted summation. Then, it adds an additional fully connected layer and softmax layer to make classification predictions.

[0139] In this embodiment, regarding step S3, model deployment: deploy the tested convolutional neural network to the construction site to monitor the unsafe behaviors of construction workers in real time and issue warning information. Let the trained and tested convolutional neural network model be M, which can receive the real-time video stream V of the construction site as input and output the prediction P of the unsafe behaviors of construction workers. The steps of model deployment are:

[0140] Deploy(M,V)→Monitor(V)→P→Alert(P);

[0141] Where: Deploy(M,V) means deploying model M to the construction site and connecting to the real-time video stream V. Monitor(V) means real-time monitoring of the construction workers' behaviors in the video stream V. P is the prediction result of model M on the unsafe behaviors of construction workers in the video stream V. Alert(P) means issuing corresponding warning information based on the prediction result P.

[0142] Furthermore, the Python execution program of step S3 is as follows:

[0143]

[0144]

[0145]

[0146] In the above program: Use OpenCV's VideoCapture class to initialize the video stream. You need to replace it with your video stream source, such as the RTSP stream address or local video file path.

[0147] Real-time monitoring and early warning: In an infinite loop, we continuously read new frames from the video stream. Each frame is preprocessed, such as resizing and normalizing, to adapt to the input requirements of the model. The preprocessed frame is converted into a Tensor and added to a batch (batch size is 1). The model is used to predict the frames in the batch and obtain the prediction results.

[0148] Based on the prediction results, we determine whether the construction workers have unsafe behavior. If so, we issue a warning message. When the 'q' key is pressed or the video stream ends, we release the video stream resources and destroy all windows created by OpenCV.

[0149] Embodiment 2: Although embodiment 1 uses a convolutional neural network, the convolutional neural network still needs to be trained before it can be actually deployed. The method includes:

[0150] Step 1: Divide the labeled video dataset D into training set D train , test set D test and validation set D val The labeling work involves marking each frame in the video to indicate which behaviors are unsafe. The data set is divided to evaluate the performance of the model during the training process and to prevent the model from overfitting. The training set is used to train the model, the test set is used to evaluate the generalization ability of the model, and the validation set is used to adjust the model parameters during the training process.

[0151] Step 2: Use the training set D train The CNN model is trained with the data. During the training process, the back propagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters. The gradient descent optimizer is used to update the model parameters to minimize the loss function.

[0152] Step 3: When the performance of the model on the validation set Dval no longer improves, the model is considered to have converged. Stop the training process and save the current model parameters.

[0153] The python program of the training method provided in this embodiment is as follows:

[0154]

[0155]

[0156]

[0157] In the above program: Use transforms to preprocess the dataset. Use the customized YourDataset class to load the training set, test set, and validation set.

[0158] Use DataLoader to package the dataset into batches for iteration during training.

[0159] Instantiate the custom CNN model class YourCNN. Use the cross entropy loss function as the loss function and the Adam optimizer to update the model parameters. In each epoch, traverse the training set, perform forward propagation and backward propagation, and update the model parameters. After each epoch, use the validation set to evaluate the model performance and print the loss value.

[0160] Finally, save the trained model parameters to a file for subsequent use.

[0161] The above embodiments only express the implementation methods of the relevant practical applications of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0162] For those skilled in the art, it can be further appreciated that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0163] At the same time, those skilled in the art can understand that all or part of the processes in all the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

Claims

1. A construction site personnel management method based on a neural network, characterized in that: The steps include: S1, collect video data V containing construction workers' behaviors for annotation, and divide the video data into key frames and sub-video segments to form an input feature vector x; S2, read the convolutional neural network: convolution layer C i , Pooling layer P i And the activation function layer R i The number of them is 3; the activation function layer uses the ReLU activation function; S3 deploys the tested convolutional neural network to the construction site to monitor the unsafe behaviors of construction workers in real time and issue early warning information.

2. The construction site personnel management method according to claim 1, characterized in that: In S1, the marking and segmentation methods are: L = {l1, l2, ..., l n }; l i is with video v i Corresponding annotation information; K = {k1, k2, ..., k m }, k j represents the jth key frame; the video is divided into sub-video segments according to the time interval or event triggering condition, denoted as S = {s1, s2, ..., s p },s q Represents the qth sub-video segment.

3. The construction site personnel management method according to claim 2, characterized in that: In S1, the feature vector x is: x = {k1, s1}, {k2, s2}, ..., {k m ,s m }; where m is the sample {k i ,s i }.

4. The construction site personnel management method according to claim 1, characterized in that: In S2, the convolution layer C i , Pooling layer P i And the activation function layer R i The number of them is 3: Each layer receives the output of the previous layer as input and produces new feature maps or activation values ​​as output.

5. The construction site personnel management method according to claim 4, characterized in that: In S2, the convolutional layer is followed by an attention mechanism module, which emphasizes the key areas related to unsafe behaviors by calculating the weights of the feature maps: the feature map output by any convolutional layer is F∈R H×W×C , where H, W and C represent the height, width and number of channels of the feature map respectively; Select some regions in the feature map and set other regions to zero: M∈{0,1} H×W ; Among them, 1 in the mask M represents the selected area, and 0 represents the unselected area; The output F′ of the attention mechanism module is: F′=F⊙M; where ⊙ represents element-wise multiplication.

6. The construction site personnel management method according to claim 4, characterized in that: In the S2, an LSTM network for processing the extracted feature sequence is also included: The LSTM network includes multiple LSTM unit layers, each of which contains a forget gate, an input gate, and an output gate structure. At time point t, the calculation of the LSTM unit is: Where: f t 、i t , o t Represent the activation vectors of the forget gate, input gate, and output gate respectively; σ represents the sigmoid activation function; tanh represents the hyperbolic tangent activation function; W f , W i , W o , W C Represents the weight matrices of the forget gate, input gate, output gate and candidate cell state respectively; b f 、b i 、b o 、b C Respectively represent the bias vectors of the forget gate, input gate, output gate and candidate cell state; ⊙ represents element-by-element multiplication; C t represents the cell state at time point t; h t represents the hidden state at time point t.

7. The construction site personnel management method according to claim 6, characterized in that: The output of the LSTM network is the hidden state h at the last time point T , or it can be a sequence of hidden states at all time points {h1,h2,…,h T }.

8. The construction site personnel management method according to claim 4, characterized in that: In S2, the output features of CNN and LSTM are fused through a fully connected layer, and the features of different layers are mapped to the same dimension and weighted summed. After the feature fusion, a fully connected layer and a softmax classifier are added to perform classification prediction of unsafe behaviors.

9. The construction site personnel management method according to claim 1, characterized in that: In S3, the trained and tested convolutional neural network model is assumed to be M, and the prediction P of the unsafe behavior of the construction personnel is outputted: Deploy(M,V)→Monitor(V)→P→Alert(P); Among them: Deploy(M,V) means deploying the model M to the construction site and connecting to the real-time video stream V; Monitor(V) means real-time monitoring of the construction workers' behavior in the video stream V; P is the prediction result of the model M on the unsafe behavior of the construction workers in the video stream V; Alert(P) means issuing corresponding warning information based on the prediction result P.

10. A construction site personnel management system based on a neural network, characterized by: The system includes a processor and a register coupled to the processor, wherein the register stores program instructions, and when the program instructions are executed by the processor, the processor executes the construction site personnel management method as described in any one of claims 1-9.