Construction site abnormal behavior detection method and system based on FCOS target detection

Abstract

The application discloses a construction site abnormal behavior detection method and system based on FCOS target detection, relates to the technical field of construction site abnormal behavior detection, acquires a construction site abnormal behavior sample image and a corresponding sample label file as a first training data set, and trains an FCOS deep learning network model; according to an identification result, a positioning result and a feature output by the FCOS deep learning network model, the relationship between a worker and an abnormal behavior marker in the construction site abnormal behavior sample image is calibrated; the calibration result, the positioning result and the feature are taken as a second training data set, and an MLP multilayer perceptron is trained; two levels of networks are combined to obtain an abnormal behavior detection model, a to-be-detected image is detected, and a detection result is obtained. The application can realize automatic detection of construction site abnormal behavior, has the advantages of high accuracy, good stability, strong anti-interference capability, high universality, high speed and the like, and can play an effective role in actual application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of construction site abnormal behavior detection, and more particularly to a construction site abnormal behavior detection method and system based on FCOS target detection. BACKGROUND

[0002] In recent years, with the continuous promotion of construction site safety production risk management policy, various construction units have adopted various control technologies and management strategies to ensure the safety of construction personnel and minimize the risk of production safety accidents. Under such background, the concept of "smart construction site" has gradually attracted widespread attention from various levels of government and various enterprises. In fact, smart construction site refers to the use of existing advanced technologies to realize information management of construction site, and gradually replace "people defense" with "technology defense". With the help of deep learning method, the personnel on the construction site are controlled, and whether they are performing abnormal and illegal behavior is detected, and timely alarm is given, which helps to effectively improve the management level of smart construction site and reduce the incidence of safety accidents.

[0003] Therefore, how to provide an efficient and accurate smart construction site abnormal behavior detection method is a problem that those skilled in the art need to solve. SUMMARY

[0004] Therefore, the present application provides a construction site abnormal behavior detection method and system based on FCOS target detection, which can accurately detect abnormal behavior of construction personnel and has good stability.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] A construction site abnormal behavior detection method based on FCOS target detection, comprising the following steps:

[0007] Establish a first training data set, which includes construction site abnormal behavior sample images and corresponding sample label files;

[0008] Train the FCOS deep learning network model using the first training data set;

[0009] Input the first training data set into the trained FCOS deep learning network model again to obtain the recognition result, positioning result and feature of the construction personnel and abnormal behavior markers in the construction site abnormal behavior sample image; calibrate the relationship between the construction personnel and abnormal behavior markers in the construction site abnormal behavior sample image according to the recognition result to obtain a calibration result; and use the calibration result, positioning result and feature as a second training data set;

[0010] Train the MLP multilayer perceptron using the second training data set;

[0011] The trained FCOS deep learning network model and the trained MLP multi-layer perceptron are combined as an abnormal behavior detection model; and the to-be-tested image is input into the abnormal behavior detection model to obtain a detection result.

[0012] Optionally, the construction site abnormal behavior sample image includes two large categories of crossing and moving, and four small categories of crossing a safety fence, moving a safety fence, moving a cable support frame, and moving a fire hydrant.

[0013] The abnormal behavior markers include a safety fence, a cable support frame, and a fire hydrant.

[0014] The abnormal behaviors include crossing a safety fence, moving a safety fence, moving a cable support frame, and moving a fire hydrant.

[0015] Optionally, the sample label file satisfies an xml label file standard of a Pascal VOC format, and the content includes an image name, an image path, a height and a width of the image, and coordinates of a real target frame.

[0016] Optionally, the FCOS deep learning network model is composed of three parts: a backbone network, a feature pyramid, and a detection module; the backbone network performs feature extraction on an input image to obtain three layers of feature maps, the three layers of feature maps are sent to the feature pyramid for feature fusion, and five layers of fused feature maps obtained are sequentially sent to the detection module for pixel-by-pixel regression to obtain recognition results and positioning results.

[0017] Optionally, ResNet50 is selected as the backbone network of the FCOS deep learning network model.

[0018] Optionally, in the detection module, each feature map has two branches, classification and regression, which respectively pass through convolution operation, the classification branch outputs a predicted classification result and a center degree, and the regression branch outputs a regression result.

[0019] The classification result is the recognition result; the regression result is the positioning result, which is represented by a vector t*=(l*,t*,r*,b*), and parameters l*, t*, r*, and b* respectively represent distances of four edges of the recognition frame to four edges of the image; the center degree is located in the classification branch and is used to reduce the weight of the recognition frame far away from the target center, and the calculation formula is as follows:

[0020]

[0021] Optionally, the method for relationship calibration of the construction site abnormal behavior sample image and the abnormal behavior markers is as follows: 0 indicates that there is no abnormal behavior relationship between the two, 1 indicates crossing a safety fence, 2 indicates moving a safety fence, 3 indicates moving a cable support frame, and 4 indicates moving a fire hydrant.

[0022] Optionally, the MLP multilayer perceptron is composed of an input layer, a plurality of hidden layers, and an output layer, and each layer of neurons is fully interconnected with the next layer of neurons, but there is no neuron interconnection between the same layer.

[0023] Optionally, the detection result includes a work personnel bounding box and coordinates, an abnormal behavior marker bounding box and coordinates, an abnormal behavior bounding box and coordinates, and a relationship calibration result between the work personnel and the abnormal behavior marker.

[0024] A construction site abnormal behavior detection system based on FCOS target detection, comprising:

[0025] A first training data set acquisition module is configured to establish a first training data set, wherein the first training data set includes construction site abnormal behavior sample images and corresponding sample label files;

[0026] An FCOS deep learning network model training module is configured to train an FCOS deep learning network model using the first training data set;

[0027] A second training data set acquisition module is configured to input the first training data set into the trained FCOS deep learning network model again to obtain identification results, positioning results, and features of work personnel and abnormal behavior markers in the construction site abnormal behavior sample images; calibrate the relationship between the work personnel and the abnormal behavior markers in the construction site abnormal behavior sample images according to the identification results to obtain calibration results; and use the calibration results, the positioning results, and the features as a second training data set;

[0028] An MLP multilayer perceptron training module is configured to train an MLP multilayer perceptron using the second training data set;

[0029] A detection module is configured to combine the trained FCOS deep learning network model and the trained MLP multilayer perceptron as an abnormal behavior detection model; and input a to-be-detected image into the abnormal behavior detection model to obtain a detection result.

[0030] According to the above technical solution, the present application provides a construction site abnormal behavior detection method and system based on FCOS target detection, which has the following advantages compared with the prior art:

[0031] The present application improves the FCOS target detection algorithm, and can realize automatic detection of construction site abnormal behavior by combining a two-level network structure, has high accuracy, good stability, strong anti-interference ability, high universality, and good robustness, and can be applied to practical scenarios to reduce labor costs and reduce safety hazards.

[0032] The method has high accuracy and recognition precision, and due to the characteristics of the network structure, the operation speed is faster, the abnormal behavior detection alarm task can be efficiently realized, and effective role is played in actual application.

[0033] Through training, the method can also be used for detecting more types of abnormal behaviors in different environments, and has a wide application scene. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only a part of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0035] Figure 1 Typical sample image of the abnormal behavior data set of the embodiment of the present application;

[0036] Figure 2 FCOS deep learning network model schematic diagram of the embodiment of the present application;

[0037] Figure 3 MLP network structure of the embodiment of the present application;

[0038] Figure 4 Overall network structure of the abnormal behavior detection algorithm of the embodiment of the present application;

[0039] Figure 5 Abnormal behavior detection result of the embodiment of the present application;

[0040] Figure 6 Method step schematic diagram of the present application;

[0041] Figure 7 System module schematic diagram of the present application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0043] The embodiment of the present application discloses a construction site abnormal behavior detection method based on FCOS target detection, referring to Figure 6 , comprising the following steps:

[0044] Step 1, establishing a first training data set.

[0045] The abnormal behavior sample image of the construction site is collected mainly by video monitoring and unmanned aerial vehicle on-site collection, and the picture is clear and basically unobstructed. Then, a corresponding sample label file is made for each image, as shown in Figure 1 , to obtain the first training data set.

[0046] In the embodiment, the abnormal behavior sample image of the construction site includes two large categories of crossing and moving, and four small categories of crossing safety fence, moving safety fence, moving cable support frame and moving fire hydrant; the sample label file meets the xml label file standard of PascalVOC format, and the content includes image name, image path, height and width of the image, and coordinates of the real target box. The coordinates of the target box can be represented as (x11, x12, x13, x14), which includes four coordinate values representing the top coordinates of the upper left corner, the lower left corner, the upper right corner and the lower right corner.

[0047] The abnormal behavior markers include safety fence (aqwl), cable support frame (dlzcj) and fire hydrant (xfc); the abnormal behaviors include crossing safety fence (ky_aqwl), moving safety fence (yd_aqwl), moving cable support frame (yd_dlzcj) and moving fire hydrant (yd_xfc).

[0048] Step 2, training the FCOS deep learning network model using the first training data set.

[0049] The FCOS target detection algorithm is a pixel-by-pixel target detection algorithm based on FCN, which does not depend on anchor frame and proposal frame, and realizes capacity reduction and algorithm simplification without affecting the recognition accuracy. Referring to Figure 2 , the FCOS deep learning network model in the application is composed of three parts: backbone network (Backbone), feature pyramid (Feature Pyramid) and detection module (head); the backbone network extracts features from the input original image to obtain feature maps, among which three feature maps (C3, C4, C5) are sent to the feature pyramid for feature fusion, and finally five fused feature maps (P3, P4, P5, P6, P7) are obtained. The detection module is sent in turn to obtain the recognition result and positioning result by pixel-by-pixel regression.

[0050] ResNet50 is selected as the backbone network of the FCOS deep learning network model, which has two main characteristics: one is to contain residual modules, and the other is to have a relatively deep network structure. The role of the residual module is to achieve the identity mapping function when the shallow output is mature enough and the deep layer can achieve the identity mapping function during forward propagation; during back propagation, the gradient is conducted to train the deep model.

[0051] The feature pyramid mainly solves the multi-scale prediction problem in target detection, and the most important one is to use the top-down structure with lateral connection to construct high-level semantic feature maps at various scales. The top-down structure is to obtain a feature map with larger scale and stronger semantics by upsampling from a higher layer of the pyramid, thereby generating a feature with higher resolution. Each lateral connection merges the feature maps with the same size from the bottom-up channel and the top-down channel. The bottom-up feature map has lower semantics, but it can activate more accurate positioning due to fewer downsampling times.

[0052] In the detection module, each feature map has two branches, classification and regression, which respectively output the predicted classification results, center degree and regression results through convolution operation. The classification result is the recognition result; the regression result is the positioning result, represented by a vector t*=(l*,t*,r*,b*), and the parameters l*,t*,r*,b* represent the distances from the four edges of the recognition box to the four edges of the image, respectively. The biggest feature of the FCOS deep learning network model is to add the concept of center degree (Center-ness), which is located in the classification branch, used to reduce the weight of the recognition box far from the target center, and improve the accuracy of the final NMS filtering of low-quality prediction bounding boxes. The calculation formula is:

[0053]

[0054] The center degree is between 0 and 1, so it can be trained by binary cross entropy (BCE) and added to the loss equation. Therefore, the loss function of the model algorithm of the present application includes center loss, classification loss, regression loss, classification loss and center loss.

[0055] Step 3, establish a second training data set.

[0056] The first training data set is input into the trained FCOS deep learning network model again to obtain the recognition result, positioning result and feature of the work personnel and abnormal behavior markers in the construction site abnormal behavior sample image; according to the recognition result, the relationship between the work personnel and the abnormal behavior markers in the construction site abnormal behavior sample image is calibrated to obtain a calibration result, 0 indicating no abnormal behavior relationship between the two, 1 indicating crossing the safety fence, 2 indicating moving the safety fence, 3 indicating moving the cable support frame, and 4 indicating moving the fire hydrant. See Table 1 for calibration examples.

[0057] Table 1

[0058]

[0059] The calibration result, positioning result and feature are used as the second training data set.

[0060] Step 4, training the MLP multi-layer perceptron using the second training data set.

[0061] The MLP multi-layer perceptron is a supervised learning neural network model composed of an input layer, several hidden layers, and an output layer, see Figure 3 . Each layer of neurons is fully interconnected with the next layer of neurons, but there is no interconnection between neurons in the same layer. The forward propagation process is that all inputs are received through the input layer, and the hidden layer contains functional neurons with activation functions, which are calculated and processed according to the corresponding weight of each connection, i.e., the bias of the neuron, and finally output by the output layer which also contains functional neurons. Backpropagation is mainly related to model training. In this embodiment, the error backpropagation algorithm is selected, i.e., based on the gradient descent strategy, which needs to change the parameters in the direction of the negative gradient of the target. First, for a training example (x k ,y k ), x k is [x11, x12, x13, x14, x21, x22, x23, x24, f1, f2], y k is [0, 1, 2, 3, 4], where x11, x12, x13, x14 represent the coordinates of the workers; x21, x22, x23, x24 represent the coordinates of the abnormal behavior markers; f1 belongs to R_d, which is a human feature with a dimension of d aligned on the FCOS feature map according to the coordinates; f2 belongs to R_d, which is an object feature with a dimension of d aligned on the FCOS feature map according to the coordinates; y k label belongs to R_1, 0 represents no abnormal behavior, and 1 to 4 represent different abnormal behaviors.

[0062] The output expression can be expressed as:

[0063]

[0064] where β j is the input of the jth neuron of the output layer, and θ j is the bias of the jth neuron of the output layer. Then the mean square error on this training example is:

[0065]

[0066] where, represents the output of the jth neuron, represents the output of the jth training example. For the above error, the concept of learning rate η is introduced:

[0067]

[0068] where △ω hjdenotes the update value of the connection weight between the hth neuron of the hidden layer and the jth neuron of the output layer, ω hj is the connection weight between the hth neuron of the hidden layer and the jth neuron of the output layer. It can be noted that there is a certain chain relationship between the related parameters, and the chain rule is as follows:

[0069]

[0070] Further, the gradient term of the output layer neuron can be obtained:

[0071]

[0072] Similarly, the mathematical formula of the gradient term of the hidden layer neuron, the update of the connection weight and the bias can be derived according to the above formula, so as to realize the BP algorithm.

[0073] Step 5, taking the FCOS deep learning network model as the first-level network and the MLP multilayer perceptron as the second-level network, an abnormal behavior detection model is obtained, as shown in Figure 4 The test image is input into the abnormal behavior detection model to obtain a detection result. The generated detection result output table is output, 0 indicates that there is no abnormal behavior relationship between the two corresponding cells, 1 indicates crossing the safety fence, 2 indicates moving the safety fence, 3 indicates moving the cable support frame, and 4 indicates moving the fire hydrant; and a detection result output image, a red bounding box is an operator, a yellow bounding box is an abnormal behavior marker, and a green bounding box is an abnormal behavior. Wherein, if there is an abnormal behavior of the operator, the coordinates of the abnormal behavior recognition box are (max(x1, x3)-10, max(y1, y3)+10, max(x2, x4)+10, max(y2, y4)-10).

[0074] Finally, the test set is used to test the abnormal behavior detection model. In the specific implementation process, a total of 12977 images are obtained, 8307 of which are randomly classified as the first training data set in step 1, the FCOS deep learning network model is trained, 2075 are used as a cross-validation set, and 2595 are used as a test set.

[0075] Before the image used for training is input into the model, it is uniformly scaled to 1024x800 size, and normalized according to the mean and variance of the color distribution of the training set. Random gradient descent (SGD) is used for training, the initial learning rate is 0.0025, the momentum term is 0.9, and the weight decay coefficient is 1x10-4. The pre-trained weights of ResNet50 on ImageNet are used to initialize the backbone network, and the default method is used to normally and randomly initialize the weights of the newly added network layer.

[0076] According to the experimental results, through the two-level network structure, the detection of four typical abnormal behaviors of workers, i.e., crossing the safety fence, moving the safety fence, moving the cable support frame, and moving the fire hydrant, is realized, as shown in the following table. Figure 5 The experimental results also show that the detection results of the improved FCOS target detection algorithm have obvious advantages compared with the traditional FCOS target detection algorithm, as shown in Table 2.

[0077] Table 2

[0078]

[0079]

[0080] In one embodiment, a construction site abnormal behavior detection system based on FCOS target detection is also disclosed, which corresponds to the above method, as shown in Figure 7 , comprising:

[0081] A first training data set acquisition module is configured to establish a first training data set, wherein the first training data set includes construction site abnormal behavior sample images and corresponding sample label files;

[0082] An FCOS deep learning network model training module is configured to train an FCOS deep learning network model using the first training data set;

[0083] A second training data set acquisition module is configured to input the first training data set into the trained FCOS deep learning network model again to obtain the recognition result, positioning result, and feature of the workers and abnormal behavior markers in the construction site abnormal behavior sample images; calibrate the relationship between the workers and abnormal behavior markers in the construction site abnormal behavior sample images according to the recognition result to obtain a calibration result; and use the calibration result, positioning result, and feature as the second training data set;

[0084] An MLP multilayer perceptron training module is configured to train an MLP multilayer perceptron using the second training data set;

[0085] A detection module is configured to combine the trained FCOS deep learning network model and the trained MLP multilayer perceptron as an abnormal behavior detection model; input a to-be-detected image into the abnormal behavior detection model to obtain a detection result.

[0086] The various embodiments described in this specification are implemented in a progressive manner, each embodiment highlighting the differences from other embodiments, and the same or similar parts between embodiments can be mutually referred to. For the system modules disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0087] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those of ordinary skill in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A construction site abnormal behavior detection method based on FCOS target detection, characterized in that, The method comprises the following steps: establishing a first training data set comprising construction site abnormal behavior sample images and corresponding sample label files; training an FCOS deep learning network model using the first training data set; inputting the first training data set again into the trained FCOS deep learning network model to obtain recognition results, positioning results and features of workers and abnormal behavior markers in the construction site abnormal behavior sample images; calibrating the relationship between the workers and the abnormal behavior markers in the construction site abnormal behavior sample images according to the recognition results to obtain calibration results; and taking the calibration results, the positioning results and the features as a second training data set; training an MLP multilayer perceptron using the second training data set; combining the trained FCOS deep learning network model and the trained MLP multilayer perceptron as an abnormal behavior detection model; and inputting a to-be-detected image into the abnormal behavior detection model to obtain a detection result.

2. The construction site abnormal behavior detection method based on FCOS target detection according to claim 1, characterized in that, The construction site abnormal behavior sample images include two large categories of crossing and moving, and four small categories of crossing safety fences, moving safety fences, moving cable support frames and moving fire hydrants; The abnormal behavior markers include safety fences, cable support frames and fire hydrants; The abnormal behaviors include crossing safety fences, moving safety fences, moving cable support frames and moving fire hydrants.

3. The construction site abnormal behavior detection method based on FCOS target detection according to claim 1, characterized in that, The sample label files meet the xml label file standard of PascalVOC format, and the content includes image name, image path, image height and width and real recognition box coordinates.

4. The construction site abnormal behavior detection method based on FCOS target detection according to claim 1, characterized in that, The FCOS deep learning network model is composed of three parts: a backbone network, a feature pyramid and a detection module; the backbone network extracts features from the input image to obtain three layers of feature maps, the three layers of feature maps are sent to the feature pyramid for feature fusion, and the obtained five layers of fused feature maps are sent to the detection module for pixel-by-pixel regression in turn to obtain recognition results and positioning results.

5. The construction site abnormal behavior detection method based on FCOS target detection according to claim 4, characterized in that, ResNet50 is selected as the backbone network of the FCOS deep learning network model.

6. The construction site abnormal behavior detection method based on FCOS target detection according to claim 4, characterized in that, In the detection module, each feature map has two branches, classification and regression, which respectively output predicted classification results, center degrees and regression results through convolution operations; The classification results are the recognition results; the regression results are the positioning results, which are represented by a vector t*=(l*,t*,r*,b*), and the parameters l*, t*, r* and b* represent the distances of the four edges of the recognition box to the four edges of the image, respectively; the center degree prediction process is located in the classification branch and is used to reduce the weight of the recognition box far away from the target center, and the calculation formula is:

7. The construction site abnormal behavior detection method based on FCOS target detection according to claim 2, characterized in that, The method for calibrating the relationship between the workers and the abnormal behavior markers in the construction site abnormal behavior sample images is that 0 represents no abnormal behavior relationship between the two, 1 represents crossing safety fences, 2 represents moving safety fences, 3 represents moving cable support frames, and 4 represents moving fire hydrants.

8. The construction site abnormal behavior detection method based on FCOS target detection according to claim 1, characterized in that, The MLP multilayer perceptron is composed of an input layer, a plurality of hidden layers and an output layer, and each layer of neurons is fully interconnected with the next layer of neurons, but there is no neuron interconnection between the same layer.

9. The construction site abnormal behavior detection method based on FCOS target detection according to claim 1, characterized in that, The detection result includes a work personnel bounding box and coordinates, an abnormal behavior marker bounding box and coordinates, an abnormal behavior bounding box and coordinates, and a relationship calibration result between the work personnel and the abnormal behavior marker.

10. A construction site abnormal behavior detection system based on FCOS target detection, characterized in that, Comprise: A first training data set acquisition module is configured to establish a first training data set, wherein the first training data set comprises construction site abnormal behavior sample images and corresponding sample label files; An FCOS deep learning network model training module is configured to train an FCOS deep learning network model using the first training data set; A second training data set acquisition module is configured to input the first training data set into the trained FCOS deep learning network model again to obtain identification results, positioning results, and features of work personnel and abnormal behavior markers in the construction site abnormal behavior sample images; calibrate the relationship between the work personnel and the abnormal behavior markers in the construction site abnormal behavior sample images according to the identification results to obtain calibration results; and use the calibration results, the positioning results, and the features as a second training data set; An MLP multilayer perceptron training module is configured to train an MLP multilayer perceptron using the second training data set; A detection module is configured to combine the trained FCOS deep learning network model and the trained MLP multilayer perceptron as an abnormal behavior detection model; input a to-be-detected image into the abnormal behavior detection model to obtain a detection result.

Images