A method and system for detecting worker climbing over fence behavior based on computer vision

Abstract

The present invention relates to a computer vision-based method and system for detecting workers climbing over fences. The method involves first using an instance segmentation algorithm to determine whether workers in the video are touching the fence, based on video footage captured by a construction site surveillance camera. Secondly, a 3D skeleton recognition algorithm is used to determine the worker's current posture in the video. This posture is then compared against thresholds defined by a pre-established database of unsafe behaviors to determine whether the worker is performing a fence-climbing action. Finally, the detection results of the two algorithms are combined to comprehensively determine the probability of the worker engaging in the unsafe fence-climbing behavior. Compared to existing technologies, this method can improve the efficiency of detecting unsafe fence-climbing behaviors at construction sites, thereby enhancing construction site safety management. The system can also be applied to detecting various other unsafe worker behaviors at construction sites.

Description

Technical Field

[0001] The present invention relates to a field of computer vision and in particular to a method and system for detecting workers' fence-climbing behavior. Background Art

[0002] Falls from heights account for over 50% of construction safety accidents, far exceeding other types of accidents. Therefore, reducing and preventing falls from heights is crucial to lowering casualties in the construction industry. Safety equipment, such as tool-type protective fencing in hazardous areas like the "four entrances" and "five edges" of construction sites, is the most fundamental safeguard against falls from height. However, due to a lack of safety awareness or a tendency to take chances, workers often resort to unsafe behaviors such as climbing over tool-type protective fencing to take shortcuts or rest. Traditionally, this approach relies on on-site inspections by management personnel. However, as construction scale increases, this management efficiency declines, and it is impossible to provide 24 / 7 supervision of construction sites. Therefore, there is an urgent need for comprehensive, all-day monitoring of workers' unsafe behavior of climbing over tool-type protective fencing.

[0003] In order to improve the efficiency of unsafe behavior management at construction sites, the industry has introduced information technology to inspect unsafe behaviors at construction sites, including technologies such as the Internet of Things, sensors, and wearable devices, which provide new ideas for detecting unsafe behaviors at construction sites. For example, the invention with publication number CN115853300A discloses an intelligent edge protection warning system for a construction site, comprising a fence (5), wherein the fence (5) is composed of a plurality of guardrails (51) connected in sequence, and the outer surface of the fence (5) is fixedly connected with a plurality of intelligent edge protection detection devices (1) in sequence. The intelligent edge protection detection devices (1) further comprise an intelligent safety helmet (2), a front-end APP (3), and a back-end cloud platform database (4). The outer surface of the intelligent edge protection detection device (1) is fixedly installed with an infrared radiation detection device (11), an infrared pyroelectric sensor (12), a Hall sensor (13), and a first LED sound and light alarm (14).

[0004] Although this solution uses sensors to detect unsafe behaviors on construction sites, it is costly, inflexible in layout, and can easily affect workers' normal work. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies, such as high cost, inflexible layout, and easy impact on the normal work of workers, and to provide a computer vision-based method and system for detecting workers climbing over fences. The layout cost is low and flexible, and all-weather automated monitoring of unsafe behaviors of workers climbing over fences at construction sites is achieved.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A method for detecting a worker climbing over a fence based on computer vision includes the following steps:

[0008] Through simulation experiments, we collected videos of workers climbing over fences in an unsafe manner and developed action standards for workers climbing over fences as judgment rule data.

[0009] Obtain construction video streams from the construction site;

[0010] Using a pre-built and trained instance segmentation algorithm module, environmental information is extracted from the construction video stream to identify workers and fences, and spatial positional relationship data between the workers and the fence is obtained to determine whether the workers are in contact with the fence.

[0011] Using a pre-built and trained 3D skeleton detection algorithm module, the worker's posture and movement data are collected from the construction video stream, the worker's 3D skeleton information is generated, and the angle data of the skeleton key points during the worker's movement are read;

[0012] Comparing the angle data of the key points of the skeleton during the worker's movement with the judgment rule data to obtain the posture and movement analysis data of the worker's unsafe behavior of climbing over the fence, so as to determine whether the worker is performing the unsafe behavior of climbing over the fence;

[0013] The spatial position relationship data between the worker and the fence and the posture and motion analysis data of the worker's unsafe behavior of climbing over the fence are subjected to data fusion analysis. If it is determined that the worker is in contact with the fence and is performing an unsafe behavior of climbing over the fence, it is determined that the worker is performing an unsafe behavior of climbing over the fence, and the detection result is output.

[0014] Furthermore, during the simulation experiment, the unsafe behavior of workers climbing over the fence was divided into the following stages:

[0015] Approach phase: The worker walks towards the fence, assuming a standing position;

[0016] Preparation stage: The worker comes into contact with the fence, places one or both hands on the upper part of the fence, and supports the center of gravity of the body on the fence;

[0017] Initial stage: The worker lifts one leg to support the fence and shifts the body weight to the fence;

[0018] Climbing stage: The worker lifts his other foot, leaves the ground, and climbs over the top of the fence, completing the entire climbing action. At this point, the worker's limbs are in contact with the upper part of the fence, and the upper body is bent;

[0019] Ending stage: The worker jumps from the top of the fence and lands on the ground, completing the climbing process. The worker reaches the other side of the fence and resumes a standing position.

[0020] Furthermore, during the simulation experiment, the angle data of key points of the worker's skeleton in the approach phase, preparation phase, start phase and climbing phase are obtained as the judgment rule data.

[0021] Furthermore, the angle data of the skeleton key points include the angle between the upper and lower body, the right knee joint angle, the left knee joint angle, the left elbow angle and the right elbow angle.

[0022] Furthermore, the judgment rule data includes: in the preparation phase, the left elbow angle or the right elbow angle is in the elbow bending judgment interval, and the right knee joint angle or the left knee joint angle is in the knee joint upright judgment interval;

[0023] In the initial stage, the right knee joint angle or the left knee joint angle is in the knee flexion judgment range;

[0024] During the climbing phase, the right knee angle or the left knee angle is in the knee flexion judgment range, or the angle between the upper and lower body is in the body flexion judgment range;

[0025] If the angle data of the skeleton key points obtained during the worker's movement is consistent with the judgment rule data, it is judged that the worker is performing an unsafe action of climbing over the fence; otherwise, it is judged that the worker is not performing an unsafe action of climbing over the fence.

[0026] Furthermore, the instance segmentation algorithm module performs target segmentation on the worker and the fence based on the Mask R-CNN instance segmentation algorithm.

[0027] Furthermore, the process of determining whether the worker has touched the fence is specifically as follows:

[0028] The instance segmentation algorithm module is used to obtain the worker area and the fence area in the construction video stream, thereby calculating the worker overlap ratio. If the worker overlap ratio is within a preset contact position range, it is determined that the worker is in contact with the fence.

[0029] Furthermore, the calculation expression for determining whether the worker touches the fence is:

[0030] IoW=Area of ​​Overlap / Area of ​​Worker

[0031]

[0032] Where IoW is the worker overlap ratio, Area of ​​Overlap is the fence area, Area of ​​Worker is the worker area, and R e is the spatial relationship between the worker and the fence, if R e The output is 1, if the worker touches the fence; if R e The output is 0 if the worker did not touch the fence.

[0033] Furthermore, the 3D skeleton detection algorithm module generates the worker's three-dimensional skeleton posture based on the VideoPose3D algorithm of the time-domain dilated convolution model.

[0034] The present invention also provides a computer vision-based system for detecting workers' behavior of climbing over fences, comprising a surveillance camera, a memory, and a processor arranged at a construction site. The surveillance camera faces the fence and is connected to the processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of the method described above.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] (1) The present invention utilizes computer vision technology and a deep neural network model, and uses the spatial position relationship data between workers and fences and the posture and motion analysis data of workers' unsafe behaviors of climbing over fences for fusion analysis, to identify and detect whether workers have engaged in unsafe behaviors of climbing over fences and at which stage of the behaviors of climbing over fences they are in, thereby realizing all-weather automated monitoring of unsafe behaviors of workers climbing over fences at construction sites.

[0037] (2) Experimental verification shows that the method of the present invention has an average accuracy of 0.82 for identifying unsafe behaviors such as workers climbing over fences, which is superior to the accuracy of relying solely on a single detection algorithm. It also provides better explanations for the various stages of unsafe behaviors such as workers climbing over fences. This invention uses only construction site cameras as a data source, reducing the manpower and financial costs of construction site safety management and providing technical support for the intelligent management of unsafe behaviors on construction sites. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a method for detecting worker climbing over a fence based on computer vision provided by an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a five-stage fence climbing action provided by an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of key nodes of a skeleton for detecting worker movements provided by an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of a data fusion matrix for fence-climbing behavior detection provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0045] Example 1

[0046] like Figure 1 As shown, this embodiment provides a method for detecting a worker's behavior of climbing over a fence based on computer vision, comprising the following steps:

[0047] Through simulation experiments, we collected videos of workers climbing over fences in an unsafe manner and developed action standards for workers climbing over fences as judgment rule data.

[0048] S1: Connect to the construction site surveillance camera to obtain the construction video stream;

[0049] S2: Using a pre-built and trained instance segmentation algorithm module, it extracts environmental information from the construction video stream, identifies workers and fences, and obtains the spatial positional relationship data between the workers and the fence to determine whether the workers are in contact with the fence.

[0050] S3: Using a pre-built and trained 3D skeleton detection algorithm module, it collects worker posture and motion data from the construction video stream, generates the worker's 3D skeleton information, and reads the angle data of the skeleton's key points during the worker's movement;

[0051] S4: Comparing the angle data of the key points of the skeleton during the worker's movement with the judgment rule data to obtain the posture and movement analysis data of the worker's unsafe behavior of climbing over the fence, so as to determine whether the worker is performing the unsafe behavior of climbing over the fence;

[0052] S5: Perform data fusion analysis on the spatial position relationship data between the worker and the fence and the posture and motion analysis data of the worker's unsafe behavior of climbing over the fence. If it is determined that the worker is in contact with the fence and is performing an unsafe behavior of climbing over the fence, it is determined that the worker is performing an unsafe behavior of climbing over the fence, and the detection result is output.

[0053] The specific implementation process of the above method is introduced below with a specific example.

[0054] In this example, the computer vision-based method for detecting workers climbing over fences includes the following steps:

[0055] 1) Digitally analyze workers' unsafe behaviors of climbing over fences to provide quantitative evaluation criteria for subsequent inspection processes, thereby eliminating differences in movement habits, movement amplitude, and movement speed among different workers due to individual differences.

[0056] 2) Use simulation experiments to simulate the unsafe behavior of workers climbing over fences in a safe environment and shoot videos. Through expert decomposition and analysis of the video, the unsafe behavior of workers climbing over fences can be divided into five action stages: ① Approach stage: The tester approaches the fence and stands; ② Preparation stage: The tester contacts the fence, places one or both hands on the upper part of the fence, and supports the center of gravity of the body on the fence. Sometimes, the tester sticks his head out to look at the back of the fence to confirm the foothold; ③ Starting stage: The tester lifts one leg to support the fence, and the tester's body center of gravity is transferred to the fence, marking the official start of the fence climbing process; ④ Climbing stage: The tester lifts the other foot, the body leaves the ground, climbs over the top of the fence, and completes the entire climbing action. At this time, the tester's limbs are in contact with the upper part of the fence, and the upper body is bent; ⑤ Ending stage: The tester jumps from the top of the fence and lands on the ground, completing the climbing process. The tester comes to the other side of the fence and resumes a standing position, and the climbing process officially ends. The five-stage diagram is as follows Figure 2 shown.

[0057] 3) Identify ①, ②, ③, and ④ as the key stages of fence-climbing behavior, determine the interaction between the fence and the worker in these four stages, and analyze the angle data of the key points (i.e., joints) of the worker's skeleton in these four stages. Record these data as judgment data for the computer vision analysis rules of fence-climbing behavior and store them in the unsafe behavior database.

[0058] 4) Collect images and videos of tool guardrails from construction sites, and after manual annotation, establish a sample dataset of construction protection fences for instance segmentation and target detection training.

[0059] 5) Perform instance segmentation detection of fence targets. The Mask R-CNN instance segmentation algorithm is used to train instance segmentation and recognition of fence targets. The ResNet101 network is used as the pre-trained network for transfer learning. After multiple rounds of iteration, the requirements for fence target detection are met.

[0060] 6) Worker detection is performed using the Mask R-CNN algorithm, and instance segmentation of worker targets is performed using the ResNet101 network trained on the open source MS COCO dataset.

[0061] 7) To determine the interaction between workers and fences, based on the instance segmentation detection results, this paper proposes the concept of "worker overlap ratio" to determine the spatial positional relationship between workers and fences. "Intersection over Worker" (IoW) is derived from the classic concept of "Intersection over Union" (IoU) in computer vision and is defined as shown in Equation (1).

[0062] IoW=Area of ​​Overlap / Area of ​​Worker (1)

[0063] 8) Because the body size of the workers is relatively fixed, while the size of the fence varies greatly on the construction site, the present invention uses the worker overlap ratio to judge the spatial position relationship between the workers and the fence. Since the workers and the fence are on the same ground, and the cameras on the construction site are usually shot horizontally at a fixed position, when the workers approach the fence, the area of ​​the workers in the image will become smaller, while the area of ​​the fence will not change. Accordingly, when the workers approach the fence and are about to climb over it, the value of IoW gradually increases, showing an upper limit. Therefore, it can be considered that the value of IoW has a maximum and minimum threshold. Based on the analysis of a large amount of fence climbing behavior data, the spatial position relationship between the workers and the fence (R e ) as shown in formula (2). e When the output is 1, it can be considered that the worker is very close to the fence (that is, the worker has touched the fence), and there is a possibility that the worker may engage in unsafe behavior by climbing over the fence.

[0064]

[0065] 9) In order to judge the worker's current specific action state, the present invention proposes a method for judging the worker's action posture based on the geometric parameters of the key points of the worker's three-dimensional skeleton, that is, using a deep learning algorithm to extract the worker's three-dimensional posture skeleton from the construction site video, calculate the geometric parameters of each key point in the skeleton, and compare them with the thresholds stored in the database of unsafe behaviors of workers climbing over fences, so as to realize the judgment of the worker's current posture action.

[0066] 10) The VideoPose3D algorithm, based on a temporally dilated convolutional model, is used to generate the worker's 3D skeletal pose. This algorithm uses the 2D coordinates of key human skeleton points extracted by existing 2D pose detection algorithms as input. It considers the time span that composes the skeleton coordinate sequence in each frame of the video, processes this sequence information, and predicts the 3D skeletal pose of the detected object using a temporal convolutional network (TCN).

[0067] 11) The database of unsafe behaviors of workers climbing over fences is obtained through “experiment + expert analysis”, that is, videos of unsafe behaviors of climbing over fences are filmed under experimental conditions, and through expert analysis, the geometric parameters of key nodes of the human skeleton in each action stage of the fence climbing behavior are determined, such as Figure 3 As shown in Table 1 and Table 2, the corresponding value range is obtained by statistics, so as to determine the judgment criteria of each action stage of the unsafe behavior of climbing over the fence and store them in the database.

[0068] Table 1 Key points for worker skeleton detection

[0069]

[0070]

[0071] Table 2 Key joint parameters for worker skeleton detection

[0072] Key points describe <![CDATA[θ0]]> Angle between upper and lower body <![CDATA[θ1]]> Right knee joint angle <![CDATA[θ2]]> Left knee angle <![CDATA[θ3]]> Left elbow angle <![CDATA[θ4]]> Right elbow angle

[0073] 12) For the preparation phase ②, the worker's elbow and knee angles are considered key parameters, with the worker's elbow in the bent judgment range and the knee still in the upright judgment range. In contrast, for the start phase ③, the worker's knee becomes the key parameter, with one leg undergoing bending while the other leg remains upright. When both knees of the worker's legs are bent or the worker's body is bent, the worker is judged to be in the climbing phase ④. The key parameters and threshold judgment criteria for each action phase of the fence climbing behavior are shown in Table 3.

[0074] Table 3 Judgment criteria for the five action stages of fence climbing behavior

[0075]

[0076] 13) When detecting the 3D skeleton information parameters of the worker, if the detection results of the geometric parameters of the key points of the human skeleton in stages ②, ③ and ④ are consistent with Table 3, the detection result R is output. p =1, otherwise output R p =0.

[0077] 14) The instance segmentation algorithm can only determine whether the worker is near the fence, while the posture-based behavior detection algorithm can only determine whether the worker has performed certain specific actions such as climbing over the fence, but cannot determine whether there is a fence around them. To this end, the present invention proposes a method for data fusion analysis of the detection and judgment results of the two algorithms, that is, to perform the above R e and R p The output results of the data fusion analysis are as follows: Figure 4 As shown, this is to detect whether workers have engaged in unsafe behavior of climbing over the fence.

[0078] 15) P is the final output of the data fusion model, which can be understood as the probability of a worker climbing over a fence. When P = 1, the fence climbing behavior detection results obtained by the above two algorithms are consistent. When one algorithm detects the existence of a fence climbing behavior and the other algorithm does not detect it, in order to avoid false detection, the output is an intermediate value P = 0.5. In this case, since the output results of the two detection algorithms are inconsistent, this part of the data will be marked as suspicious data in the final model detection output. Combined Figure 4 The data fusion matrix shown in Figure 3 is calculated using the formula P.

[0079]

[0080] 16) When the P output is 1, it is assumed that a worker is performing an unsafe behavior of climbing over the fence, and an early warning is issued to the management. When the P output is 0.5, the suspicious data is recorded, and further analysis of the worker's posture and movement status is required.

[0081] Experimental verification shows that the proposed method achieves an average accuracy of 0.82 for identifying unsafe behaviors such as workers climbing over fences. This method outperforms the accuracy of a single detection algorithm and provides better interpretation of each stage of unsafe behaviors. This invention utilizes only construction site cameras as a data source, reducing the manpower and financial costs of construction site safety management and providing technical support for the intelligent management of unsafe behaviors on construction sites.

[0082] This embodiment also provides a computer vision-based system for detecting workers' behavior of climbing over a fence, including a surveillance camera, a memory, and a processor set up at the construction site. The surveillance camera faces the fence and is connected to the processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of the above-mentioned computer vision-based method for detecting workers' behavior of climbing over a fence.

[0083] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for detecting workers climbing over fences based on computer vision, characterized in that: The following steps are involved: Through simulation experiments, we collected videos of workers climbing over fences in an unsafe manner and developed action standards for workers climbing over fences as judgment rule data. Obtain construction video streams from the construction site; Using a pre-built and trained instance segmentation algorithm module, environmental information is extracted from the construction video stream to identify workers and fences, and spatial positional relationship data between the workers and the fence is obtained to determine whether the workers are in contact with the fence. Using a pre-built and trained 3D skeleton detection algorithm module, the worker's posture and movement data are collected from the construction video stream, the worker's 3D skeleton information is generated, and the angle data of the skeleton key points during the worker's movement are read; Comparing the angle data of the key points of the skeleton during the worker's movement with the judgment rule data to obtain the posture and movement analysis data of the worker's unsafe behavior of climbing over the fence, so as to determine whether the worker is performing the unsafe behavior of climbing over the fence; Performing data fusion analysis on the spatial position relationship data between the worker and the fence and the posture and motion analysis data of the worker's unsafe behavior of climbing over the fence; if it is determined that the worker is in contact with the fence and is performing an unsafe behavior of climbing over the fence, then determining that the worker is performing an unsafe behavior of climbing over the fence and outputting a detection result; The process of determining whether a worker has touched the fence is as follows: The instance segmentation algorithm module is used to obtain the worker area and the fence area in the construction video stream, thereby calculating the worker overlap ratio. If the worker overlap ratio is within a preset contact position range, it is determined that the worker is in contact with the fence; The calculation expression for determining whether the worker has touched the fence is: IoW=Area of ​​Overlap / Area of ​​Worker Where IoW is the worker overlap ratio, Area of ​​Overlap is the fence area, Area of ​​Worker is the worker area, and R e is the spatial relationship between the worker and the fence, if R e The output is 1, if the worker touches the fence; if R e The output is 0 if the worker did not touch the fence.

2. The method for detecting worker climbing over a fence based on computer vision according to claim 1, characterized in that: During the simulation experiment, the unsafe behavior of workers climbing over the fence was divided into the following stages: Approach phase: The worker walks towards the fence, assuming a standing position; Preparation stage: The worker comes into contact with the fence, places one or both hands on the upper part of the fence, and supports the center of gravity of the body on the fence; Initial stage: The worker lifts one leg to support the fence and shifts the body weight to the fence; Climbing stage: The worker lifts his other foot, leaves the ground, and climbs over the top of the fence, completing the entire climbing action. At this point, the worker's limbs are in contact with the upper part of the fence, and the upper body is bent; Ending stage: The worker jumps from the top of the fence and lands on the ground, completing the climbing process. The worker reaches the other side of the fence and resumes a standing position.

3. The method for detecting worker climbing over a fence based on computer vision according to claim 2, characterized in that: During the simulation experiment, the angle data of each key point of the worker's skeleton in the approach phase, preparation phase, start phase and climbing phase are obtained as the judgment rule data.

4. The method for detecting worker climbing over a fence based on computer vision according to claim 3, characterized in that: The angle data of the skeleton key points include the angle between the upper and lower body, the right knee joint angle, the left knee joint angle, the left elbow angle and the right elbow angle.

5. The method for detecting worker climbing over a fence based on computer vision according to claim 4, characterized in that: The judgment rule data includes: in the preparation phase, the left elbow angle or the right elbow angle is in the elbow bending judgment interval, and the right knee joint angle or the left knee joint angle is in the knee joint upright judgment interval; In the initial stage, the right knee joint angle or the left knee joint angle is in the knee flexion judgment range; During the climbing phase, the right knee angle or the left knee angle is in the knee flexion judgment range, or the angle between the upper and lower body is in the body flexion judgment range; If the angle data of the skeleton key points obtained during the worker's movement is consistent with the judgment rule data, it is judged that the worker is performing an unsafe action of climbing over the fence; otherwise, it is judged that the worker is not performing an unsafe action of climbing over the fence.

6. The method for detecting worker climbing over a fence based on computer vision according to claim 1, characterized in that: The instance segmentation algorithm module performs target segmentation on the workers and the fence based on the Mask R-CNN instance segmentation algorithm.

7. The method for detecting worker climbing over a fence based on computer vision according to claim 1, characterized in that: The 3D skeleton detection algorithm module generates the worker's 3D skeleton posture based on the VideoPose3D algorithm of the time-domain dilated convolution model.

8. A computer vision-based system for detecting workers climbing over fences, characterized in that: The method comprises a monitoring camera, a memory and a processor arranged at a construction site, wherein the monitoring camera faces the fence and is connected to the processor, the memory stores a computer program, and the processor calls the computer program to execute the steps of any one of the methods described in claims 1-7.

Images