Safety monitoring system and method based on monocular camera to detect workers approaching lotus crane

Abstract

The present invention relates to a safety monitoring system for detecting workers approaching a lotus crane based on a monocular camera, comprising a video data acquisition module and a video data processing module, wherein the video data acquisition module is used to collect video data information on a construction site and transmit it to the video data processing module, and the video data processing module is used to calibrate the danger range of the incoming video and detect pedestrians entering the danger range. The present invention can determine the danger range of the lotus crane operation according to the movement trajectory of the lotus crane through simple picture calibration, and use a deep algorithm to detect and warn pedestrians entering the danger range, thereby overcoming the defect of the traditional calibration method that on-site calibration is required when adjusting the external parameters of the camera, and effectively reducing the risk to personnel safety caused by the crane operation process.

Description

Technical Field

[0001] The present invention relates to the field of construction safety supervision, and more specifically, to a safety monitoring system and method based on a monocular camera to detect workers approaching a lotus crane. Background Art

[0002] During the operation of the crane, the hoisted objects may become loose or slip, so it is stipulated that "no one is allowed to stand" under the crane arm. However, due to the complex conditions at the construction site, manual management has not been put in place in a timely manner, and information cannot be accurately and effectively transmitted, pedestrians often enter the rotation radius of the crane boom. This situation will pose a great risk to the personal safety of on-site construction personnel, and the form of manual supervision can no longer meet the needs of complex engineering scenarios. It is necessary to introduce video monitoring and intelligent detection methods to effectively warn of such risks, and promptly inform pedestrians when they enter the working range of the crane boom to avoid accidents caused by objects hitting them.

[0003] To implement the above method, by monitoring the information in the video, it is necessary to determine whether there are pedestrians entering the dangerous working range of the crane. It is necessary to estimate the three-dimensional information in the real scene based on the two-dimensional information of the image, that is, the relative three-dimensional relationship between the pedestrian and the lotus crane. The commonly used estimation methods currently include "binocular vision estimation" and "camera calibration estimation". "Binocular vision estimation" uses two cameras to obtain two images of the object to be measured from different positions, and calculates the position deviation between the corresponding points of the image to obtain the three-dimensional information of the object; "Camera calibration estimation" uses the internal and external parameters of the camera, such as the camera focal length, camera attitude angle, etc., to establish a geometric model of camera imaging and obtain the mapping relationship between the two-dimensional information of the picture and the three-dimensional information of the world. For the former, generally speaking, a construction scene within a certain range only contains one surveillance camera, which does not meet the "binocular" prerequisite; for the latter, the external parameters of the surveillance camera are not fixed. When the surveillance image is zoomed in or out, or the surveillance camera is swung, the external parameters of the camera will change. When the parameters change, the camera needs to be recalibrated. The traditional calibration method requires a combination of reality and image information for calibration, and on-site calibration is required. The operation is complicated and time-consuming, and is not feasible in project scenarios. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a safety monitoring system and method based on a monocular camera to detect workers approaching a lotus crane, which overcomes the defect that traditional calibration methods require on-site calibration when adjusting the external parameters of the camera, and effectively reduces the risks to personnel safety caused by the crane operation process.

[0005] The technical solution adopted by the present invention to solve its technical problem is: constructing a safety monitoring system based on a monocular camera to detect workers approaching a lotus crane, including a video data acquisition module and a video data processing module. The video data acquisition module is used to collect video data information on the construction site and transmit it to the video data processing module. The video data processing module is used to calibrate the danger range of the incoming video and detect pedestrians entering the danger range.

[0006] According to the above scheme, the danger range calibration is specifically to capture the moving trajectory of the claw grab bucket through the Yolov3 algorithm, fit the danger range using the ellipse fitting algorithm, and adjust the danger range to the correct plane to complete the calibration.

[0007] According to the above scheme, the YOLOv3 algorithm is specifically as follows: the original image is divided into three grids of different scales: 13*13 grid, 26*26 grid, and 52*52 grid; for the three grids of different scales, each grid is predicted to contain a target, and the most accurate prediction result is taken as the final result; the 13*13 grid is used to predict large-scale targets, the 26*26 grid is used to predict medium-scale targets, and the 52*52 grid is used to predict small-scale targets.

[0008] According to the above scheme, the ellipse fitting algorithm uses the least square method to obtain a set of parameters so as to minimize the distance measure between the set of points to be fitted and the ellipse determined by the set of parameters.

[0009] According to the above scheme, the detection of pedestrians entering the danger zone is specifically to detect the personnel using Yolov3, and obtain the personnel depth information by improving the TCMono fusion video timing information, so as to determine whether the pedestrian enters the danger zone of the lifting claw operation.

[0010] The present invention also provides a method for constructing a safety monitoring system based on a monocular camera to detect workers approaching a lotus crane, comprising the following steps:

[0011] S1, on-site video data collection;

[0012] S2. Calibration of crane body height and rotation center:

[0013] S3, real-time motion trajectory tracking and danger range determination of the lotus grab;

[0014] S4: Pedestrian detection and depth estimation: Pedestrian detection and depth estimation are used to determine whether a pedestrian has entered a dangerous area.

[0015] According to the above scheme, step S2 specifically includes the following steps:

[0016] S201. Determine the rotation center of the crane based on manual observation and experience, and mark it with a mouse;

[0017] S202. Determine the height of the crane based on manual observation and experience, and mark the height of the crane with a mouse.

[0018] According to the above scheme, step S3 specifically includes the following steps:

[0019] S301, using the target detection algorithm YoLov3 to obtain the movement trajectory of the lifting claw;

[0020] S302, fitting the trajectory using an ellipse fitting algorithm;

[0021] S303. Danger range adjustment.

[0022] According to the above scheme, step S4 specifically includes the following steps:

[0023] S401, detecting pedestrians using YoLov3 target detection algorithm;

[0024] S402, using TCMono to perform depth estimation, calculate the depth of the pedestrian in the video, and extract depth information of the video image;

[0025] S403: Combine pedestrian detection and depth information to determine whether the pedestrian has entered a dangerous area.

[0026] The implementation of the safety monitoring system and method of the present invention based on a monocular camera to detect workers approaching a lotus crane has the following beneficial effects:

[0027] 1. The present invention effectively solves the limitation that construction sites are generally not equipped with "binocular cameras" and uses video data obtained by "monocular cameras" to calibrate the danger range, which is more in line with the actual needs of the construction site;

[0028] 2. The present invention adopts simple picture calibration and introduces TCMono depth estimation algorithm. Since the external parameters of the surveillance camera are not fixed, the camera needs to be recalibrated when the parameters change. The traditional calibration method needs to be calibrated in combination with reality and picture information. The present invention effectively improves the work efficiency through simple picture calibration. At the same time, the TCMono depth estimation algorithm has higher computational efficiency and the obtained depth map sequence is more stable and smooth.

[0029] 3. The present invention does not need to select the size information of objects already on site as a reference to calculate the three-dimensional distance between the worker and the lotus grab; the present invention can directly calculate the relative distance between the person and the lotus grab by estimating the depth information in the video, thereby realizing automatic judgment and identification of workers invading the dangerous area of ​​the lotus grab. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0031] Figure 1 It is a flow chart of a safety monitoring system and method for detecting workers approaching a lotus crane based on a monocular camera of the present invention;

[0032] Figure 2 is a structural schematic diagram of crane height and rotation center calibration in an embodiment of the present invention;

[0033] Figure 3 is a flow chart of real-time motion trajectory tracking and dangerous range determination of a lotus grab in an embodiment of the present invention;

[0034] Figure 4 is a flow chart of a pedestrian detection and depth estimation algorithm according to an embodiment of the present invention;

[0035] Figure 5 It is a flow chart of the operation of a safety monitoring system for real-time detection of workers approaching a lotus crane based on a monocular camera in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0037] like Figure 1-5 As shown, the safety monitoring system of the present invention for detecting workers approaching a lotus crane based on a monocular camera includes a video data acquisition module and a video data processing module. The video data acquisition module is used to collect video data information on the construction site and transmit it to the video data processing module. The video data processing module is used to calibrate the danger range of the incoming video and detect pedestrians entering the danger range.

[0038] The specific method of calibrating the danger range is to capture the moving trajectory of the grab bucket through the Yolov3 algorithm, fit the danger range using the ellipse fitting algorithm, and adjust the danger range to the correct plane to complete the calibration. The YOLOv3 algorithm is as follows: the original image is divided into three different scales of 13*13 grid, 26*26 grid, and 52*52 grid; for the three scales of grids, each grid is predicted to contain a target, and the most accurate of all prediction results is taken as the final result; the 13*13 grid is used to predict large-scale targets, the 26*26 grid is used to predict medium-scale targets, and the 52*52 grid is used to predict small-scale targets. The ellipse fitting algorithm uses the least squares method to obtain a set of parameters to minimize the distance measure between the set of points to be fitted and the ellipse determined by this set of parameters.

[0039] The detection of pedestrians entering the danger zone is specifically to detect people using Yolov3, and obtain the depth information of people by improving the TCMono fusion video timing information, so as to determine whether the pedestrians enter the danger zone of the lifting claw operation.

[0040] The present invention also provides a method for constructing a safety monitoring system for real-time detection of workers approaching a lotus crane based on a monocular camera, comprising the following steps:

[0041] S1. Collection of on-site video data.

[0042] S2. Calibration of crane height and rotation center includes the following steps:

[0043] S201, determining the rotation center of the crane based on manual observation and experience, and marking it with a mouse; that is, marking the position of the "crane rotation center" with a mouse;

[0044] S202, determining the height of the crane body according to manual observation and experience, and marking the height of the crane with a mouse, that is, calibrating the height of the crane body on one side of the crane with a mouse.

[0045] Assume that the target crane will not move for a long period of time; the camera will not change its viewing angle or scale for a long period of time. Therefore, the area projected into the screen by the actual danger range will not change during this period of time. As long as the area of ​​the danger range in the monitoring screen is calibrated, there is no need to repeat the detection of the danger range in the future.

[0046] S3, real-time motion trajectory tracking of the lotus grab and determination of the danger range, including the following steps:

[0047] S301, use the target detection algorithm YoLov3 to obtain the movement trajectory of the lifting claw. Use the target detection algorithm YOLOv3 to identify the center of the lifting claw, and record the movement trajectory of the center in a video frame, which is the movement trajectory of the lifting claw during this period of time. Specifically: Use the target detection algorithm YOLOv3 to identify the center of the lifting claw, assuming that the image sequence to be detected (i.e., video) is marked as [x1, x2, ..., x N ], for each frame x i , the position of the lotus grasp detected by YOLOv3 in the image is marked as [x, y, w, h], where x and y are the coordinates of the upper left corner of the detection box, w and y are the width and height of the detection box), and the midpoint coordinates of the detection box is the position coordinate of the lotus gripper in this frame. N ] Repeat the above operation to obtain the position coordinates of the lotus grab between multiple frames, which is the moving trajectory of the lotus grab in this image sequence.

[0048] S302, fit the trajectory using an ellipse fitting algorithm. Use an ellipse fitting algorithm to fit the partial trajectory of the lifting claw selected in the surveillance video. According to the given surveillance video content, Assumption 1: the starting point of the moving trajectory of the lifting claw is aligned with the vehicle body, and the ending point of the moving trajectory is close to 1 / 4 of the circumference; Assumption 2: the lifting claw rotates one circle, and the trajectory of this circle is projected as an ellipse in the picture. Specifically: the algorithm uses the least squares method to obtain a set of parameters so that the distance measure between the set of points to be fitted and the ellipse determined by this set of parameters is minimized. First, any conic section can be expressed as a second-order polynomial (1):

[0049] F(a,x)=a·x=ax 2 +bxy+cy 2 +dx+ey+f=0 (1)

[0050] Where a = [abcdef] T , x=[x 2 xy y 2 xy 1] T , F(a; x i ) is called the algebraic distance between the point (x, y) and the conic section F(a; x) = 0. Fitting any conic section can be achieved by minimizing the quadratic algebraic distance (2):

[0051]

[0052] Among them, [x1, x2, ..., x N ] is the point set to be fitted.

[0053] In order to avoid the feasible solution a=06 and ensure that the fitting result is an elliptic curve, it is necessary to make some constraints on the parameter vector a. T Ca = 1 can well solve the above problem, which can be specifically expressed as (3):

[0054]

[0055] When the parameter vector a is quadratically constrained, the minimization of (2) can be transformed into the solution of (4):

[0056] D T Da=λCa (4)

[0057] where D = [x1, x2, ..., x N ] T , λ is the Lagrange multiplier.

[0058] Finally, by Lagrangian optimization, our optimal parameter vector a is the solution to the joint equation (5):

[0059] 2D T Da-2λCa=0

[0060] a T Ca=1 (5)

[0061] Formula (5) can be rewritten as:

[0062] Sa=λCa (6)

[0063] a T Ca=1 (7)

[0064] Where S is D T D. According to linear algebra knowledge, the solution of formula (6) can be obtained by its eigenvector (λ,u i ) is obtained, and for real numbers μ, (λ i , μu i ) is also a solution to (6); we can obtain the specific value of μ (8), because from the constraint (7) we have

[0065]

[0066] Finally, the optimal parameter vector The resulting ellipse is the rough range of the actual danger range projected into the picture.

[0067] S303, adjust the danger range. The rough danger range obtained by fitting is not in the same plane as the pedestrian movement range. In order to improve accuracy and flexibility, it is necessary to introduce artificial knowledge guidance. According to the actual position of the crane in the monitoring picture and the height of the vehicle body drawn by the operator in the picture, the danger range is readjusted to the correct plane. Specifically: use the crane height marked in step S201 to adjust the rough range obtained in step S202. Specifically, assuming that the crane height is line segment AB, the coordinates of point A and point B are (x A ,y A ) and (x B ,y B ). The center coordinate of the ellipse is (x e ,y e ), then the coordinates of the center of the ellipse on the pedestrian plane are (x e +x B -x A ,y e +y B -y A ), the major and minor axis information of the ellipse remains unchanged. The rough range obtained in step S202 is translated to obtain the accurate range.

[0068] S4, pedestrian detection and depth estimation, judging whether a pedestrian has entered a dangerous area through pedestrian detection and depth estimation, including the following steps:

[0069] S401, using YoLov3 target detection algorithm to detect pedestrians. YoLov3 algorithm divides the video frame image into three grids of different scales: 13*13, 26*26, and 52*52, and detects whether there is a target object in the grid, and takes the most accurate of all prediction results as the final result.

[0070] S402, use TCMono to perform depth estimation, calculate the depth of the pedestrian in the video, and extract the depth information of the video image. The TCMono depth estimation algorithm is introduced to determine whether the depth of the pedestrian meets the set range, so as to determine whether the foot falls within the calibrated danger range. TCMono uses temporal consistency loss, which is used to measure the stability between the depth maps of adjacent frames to avoid the generation of depth image sequences with severe numerical jitter. TCMono uses temporal consistency loss. Specifically, a video with a length of n is defined as X i (i=1,...,n), the corresponding depth map sequence is D i (i=1, ..., n), because the change between frames in the video is very small, for two consecutive frames X i ,X i+1 For example, the corresponding D i ,D i+1 The transformation between them should also be small. Directly calculate D i and D i+1 The difference between the depth maps cannot well reflect the jitter between them, because due to the movement of the target or the camera, D i and D i+1 The pixel positions between the two frames are not one-to-one corresponding. In order to match the pixel positions, optical flow is needed to assist calculation. Optical flow reflects the change information between the two frames. For example, for point A that appears in the first frame and the second frame, the optical flow of point A in the two frames is the vector from the position of point A in the first frame to the position in the second frame. i+1 To X i The optical flow f i+1→i , so we can get and D i The spatial position corresponds to Similarly, we can get X i The spatial position corresponds to Due to occlusion and other reasons, D i and It is impossible for all pixels to have a practical correspondence. In order to obtain the set of truly corresponding pixel positions, we define The pixel set is M i , M iThe pixels that are included are the pixels with corresponding relationships. Finally, the temporal consistency loss can be written as formula (9):

[0071]

[0072] S403: Combine pedestrian detection and depth information to determine whether the pedestrian has entered the danger zone. The algorithm outputs True for "warning" and False for "no warning".

[0073] The operation process of the safety monitoring system for real-time detection of workers approaching a lotus crane based on a monocular camera of the present invention comprises the following steps:

[0074] S1. The user retrieves the video surveillance of the construction site and establishes a transmission path with the security monitoring system;

[0075] S2. The user marks the rotation center and vehicle height of the crane in the on-site video surveillance on the system platform;

[0076] S3. The system determines whether pedestrians have entered the dangerous working range of the crane based on video surveillance information and manual operation information.

[0077] The specific operation process and involved formulas in each module will not be repeated here.

[0078] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for constructing a safety monitoring system based on a monocular camera to detect workers approaching a lotus crane, characterized in that: The safety monitoring system based on the monocular camera for detecting workers approaching the lotus crane comprises a video data acquisition module and a video data processing module. The video data acquisition module is used to collect video data information on the construction site and transmit it to the video data processing module. The video data processing module is used to calibrate the danger range of the incoming video and detect pedestrians entering the danger range. The construction method includes the following steps: S1, on-site video data collection; S2. Calibration of crane body height and rotation center: specifically including the following steps: S201. Determine the rotation center of the crane based on manual observation and experience, and mark it with a mouse; S202, determining the height of the crane body according to manual observation and experience, and marking the height of the crane body with a mouse; S3, real-time motion trajectory tracking of the lotus grab and determination of the danger range; specifically includes the following steps: S301, using the target detection algorithm YoLov3 to obtain the movement trajectory of the lifting claw; using the target detection algorithm YOLOv3 to identify the center of the lifting claw, and record the movement trajectory of the center in a video frame, which is the movement trajectory of the lifting claw during this period of time; S302, fitting the trajectory using an ellipse fitting algorithm; using an ellipse fitting algorithm to fit a portion of the trajectory of the lifting claw selected from the surveillance video, according to the given surveillance video content, assumption 1, the starting point of the moving trajectory of the lifting claw is aligned with the vehicle body, and the ending point of the moving trajectory is close to 1 / 4 of the circumference; assumption 2, the lifting claw rotates one circle, and the trajectory of this circle is projected as an ellipse in the picture, specifically: the algorithm uses the least squares method to obtain a set of parameters, so that the distance measure between the set of points to be fitted and the ellipse determined by this set of parameters is minimized, and the obtained ellipse is the rough range of the real danger range projected in the picture; S303, adjust the danger range. The rough danger range obtained by fitting is not in the same plane as the pedestrian movement range. In order to improve accuracy and flexibility, it is necessary to introduce artificial knowledge guidance. According to the actual position of the crane in the monitoring picture and the height of the vehicle body drawn by the operator in the picture, the danger range is readjusted to the correct plane. Specifically, the rough range obtained in step S202 is adjusted using the crane height marked in step S201. Specifically, assuming that the crane height is line segment AB, the coordinates of point A and point B are and , the center coordinates of the ellipse are , then the coordinates of the center of the ellipse on the pedestrian plane are , the major and minor axis information of the ellipse remains unchanged, and the rough range obtained in step S202 is translated to obtain the accurate range; S4, pedestrian detection and depth estimation, judging whether a pedestrian has entered a dangerous area through pedestrian detection and depth estimation, specifically including the following steps: S401, using YoLov3 target detection algorithm to detect pedestrians; YoLov3 algorithm divides the video frame image into three grids of different scales: 13*13, 26*26, and 52*52, and detects whether there is a target object in the grid, and takes the most accurate prediction result as the final result; S402, using TCMono to perform depth estimation, calculate the depth of the pedestrian in the video, and extract the depth information of the video image; introducing the TCMono depth estimation algorithm to determine whether the depth of the pedestrian meets the set range, so as to determine whether the foot falls within the calibrated danger range; S403: Combine pedestrian detection and depth information to determine whether the pedestrian has entered a dangerous area.

Images