A method, system, equipment and medium for monitoring the safe operation of tower cranes

By combining YOLOv8 and ByteTrack with optical flow tracing, the problem of real-time monitoring of tower crane tilt and settlement was solved, achieving efficient, accurate and safe monitoring in complex environments and reducing the impact of lighting and weather conditions.

CN118229737BActive Publication Date: 2026-11-10SHAANXI UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410336583.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-22
Publication Date
2026-11-10
Estimated Expiration
2044-03-22

AI Technical Summary

Technical Problem

Existing technologies cannot monitor the tilt and settlement of tower cranes in real time, and their real-time performance and accuracy are insufficient in complex construction environments, making it impossible to effectively monitor the overall safety of tower cranes.

Method used

The YOLOv8 model and ByteTrack algorithm are combined with optical flow tracing to monitor the tilt and settlement of tower cranes through visual information segmentation and real-time tracking. MobileSAM is used for image segmentation, and optical flow tracing is used for safety monitoring.

Benefits of technology

It enables real-time and accurate monitoring of tower cranes in complex environments, providing timely warnings of tilting and settlement, improving the system's robustness and accuracy, and reducing the impact of lighting and weather conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118229737B_ABST
    Figure CN118229737B_ABST
Patent Text Reader

Abstract

This invention provides a method, system, device, and medium for monitoring the safe operation of tower cranes, comprising the following steps: collecting visual information of the target tower crane; separating and marking the tower crane region in the video frame of the visual information to obtain tower crane segmentation information; using the upper left and lower right coordinates of the initial bounding box information in the tower crane segmentation information as input to the YOLOv8 model, and the YOLOv8 model outputting the target position information of the target tower crane; inputting the target position information into ByteTrack to achieve real-time tracking of the tower crane; and performing safety monitoring of the tower crane based on the real-time tracking results, combined with optical flow tracing. This application is not affected by factors such as lighting, shadows, and different weather conditions, has high accuracy and robustness, can meet the requirements of real-time performance, and can provide early warning of safety events such as tower crane tilting and settlement.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of tower crane monitoring technology, and specifically relates to a method, system, equipment and medium for monitoring the safe operation of tower cranes. Background Technology

[0002] The complexity and dynamic changes of construction site environments significantly impact the safe operation of construction equipment. In recent years, the number of construction site accidents has continued to rise. According to the Ministry of Housing and Urban-Rural Development's "Notification on Production Safety Accidents in Housing and Municipal Engineering," 773 production safety accidents occurred nationwide in 2017, resulting in 904 deaths. Among these, injuries related to lifting equipment accounted for 10.40% of all accidents. In the construction industry, tower cranes, due to their complex structure and harsh working environment, are considered high-risk special machinery. Accidents involving them can cause enormous losses. Statistical analysis of the 2013-2018 China Tower Crane Accident Investigation Report shows that collapse accidents are the most common type of tower crane accident, accounting for 52% of all safety accidents. Therefore, effective monitoring of tower crane tilt and settlement is of great significance. Such monitoring systems can promptly detect potential tilt or settlement problems, providing early warnings for construction site management and reducing the probability of accidents.

[0003] Existing methods face challenges in terms of real-time performance and accuracy, especially in complex construction site environments where real-time performance is crucial for tower crane safety. Faster and more efficient algorithms are needed to ensure real-time monitoring and response to tower crane status. Furthermore, many existing algorithms require the annotation of large amounts of video data, leading to high costs and susceptibility to subjectivity and annotation errors. Simultaneously, complex construction environments present challenges for computer vision systems in handling lighting, shadows, and varying weather conditions, affecting the system's accuracy and robustness. While existing vision-based methods can monitor the rotational motion of the boom, the tower crane hook, and workers below the tower crane separately, no method exists for monitoring the overall tilt and settlement of the tower crane. Therefore, this invention aims to provide a method for monitoring the safe operation of tower cranes. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention provides a method, system, equipment and medium for monitoring the safe operation of tower cranes, which is used to monitor the operating status of tower cranes in real time and provide a data basis for early warning of safety events such as tower crane tilting and settlement, so as to solve the problems that the prior art cannot monitor in real time and lacks safety monitoring of the overall tilting and settlement of tower cranes.

[0005] This invention is achieved through the following technical solution:

[0006] A method for monitoring the safe operation of tower cranes includes the following steps:

[0007] S1: Collect visual information of the target tower crane, determine the video frames in the visual information, separate and mark the tower crane area in the video frames to obtain tower crane segmentation information;

[0008] S2: Determine the initial bounding box information in the tower crane segmentation information, and use the upper left and lower right coordinates of the initial bounding box information as input to the YOLOv8 model. The YOLOv8 model outputs the target position information of the target tower crane.

[0009] S3: Input the target location information into ByteTrack, and ByteTrack outputs the real-time tracking results of the tower crane;

[0010] S4: Based on the real-time tracking results, combine optical flow tracing method to perform safety monitoring of tower cranes.

[0011] Furthermore, in step S1, the MobileSAM method is used to separate and mark the tower crane area in the visual information video frame to obtain tower crane segmentation information.

[0012] Furthermore, in the process of obtaining tower crane segmentation information through separation and labeling using the MobileSAM method, the following methods are employed: point-hint segmentation, box-hint segmentation, or segmentation of all.

[0013] The point-based segmentation method involves segmenting the entire tower crane by inputting the coordinates of a point in the image.

[0014] The box prompt segmentation involves inputting the coordinate information of the tower crane in the image and segmenting the corresponding part accordingly;

[0015] The "segmentation of all" refers to segmenting the entire input image to obtain a mask representation for each category.

[0016] Furthermore, the point suggestion segmentation process is as follows:

[0017] The resolution of an image I of arbitrary size in the video frame of the acquired visual information of the target tower crane is scaled and filled to 1024×1024;

[0018] After feeding the scaled and filled image I into ViT-H, we get:

[0019] f i =E i (I)

[0020] In the formula, E i (I) represents the ViT-H encoding process for image I; f i Image embedding;

[0021] The sequence of image I is obtained through the cue encoder:

[0022] f p =E p (P)

[0023] Where P is the prompt, E p (P) indicates the encoding process for image I;

[0024] M=D m (f i f p f o )

[0025] Among them, f o M and the output token and final mask are respectively; f p For prompt embedding; D m This is the decoding process to obtain the final mask for image I.

[0026] Furthermore, the target location information includes location coordinates and category labels.

[0027] Furthermore, the safety monitoring of tower cranes based on real-time tracking results and combined with optical flow tracing includes: monitoring the tower crane's rotation under non-operational conditions and monitoring its tilt and settlement under operating conditions.

[0028] Furthermore, the monitoring process for the tower crane's rotation under non-operational conditions is as follows:

[0029] By segmenting and then tracking the visual information, a predicted bounding box A1 for the previous frame and a predicted bounding box A2 for the next frame are obtained. The overlap between the predicted bounding boxes A1 and A2 is then calculated.

[0030]

[0031] Specifically, the closer the IoU is to 0, the less overlap there is between the two frames; the closer the IoU is to 1, the higher the overlap between the two frames; when the IoU is equal to 1, the two frames completely overlap. A preset IoU threshold condition is used as the boundary for judging the stability of the tower crane under non-operational conditions. If the IoU value of the predicted frames of the preceding and following frames is greater than or equal to the set IoU threshold condition, the tower crane is in a stable state and there is no abnormality; if the IoU is less than the IoU threshold condition, it indicates that the tower crane has been detected to be rotating, at which point an early warning mechanism is triggered and a warning signal is issued.

[0032] The process of monitoring tilt and settlement during operation is as follows:

[0033] After obtaining the real-time tower crane segmentation and tracking results, a base point (x) is selected in the first frame of the input tower crane video stream. b y bAs the corner point for sparse optical flow tracing, the sparse feature point set of the previous frame image is obtained using the Lucas-Kanade algorithm. Then, using the previous frame image, the sparse feature point set of the previous frame image, and the next frame image, a point set with the same features in the next frame image is obtained. This set is used to determine and obtain the optical flow between the sparse point sets of the two frames, thereby obtaining the motion trajectory of the corner point, i.e., the reference point of the tilted tower crane (x). b +Δx, y b The trajectory of motion +Δy);

[0034] Assume two consecutive image frames, the first frame being I(x) b y b ,t), second frame I(x) b +Δx, y b +Δy,t+Δt), where (x b y b Let be the pixel coordinates of the reference point in the image, and t be time. Then the equation for grayscale change is:

[0035] I x (p)Δx+I y (p)Δy+I t (p)=0

[0036] Among them, I x (p) and I y (p) is the spatial gradient at point p, Δx and Δy are the components of the motion vector, i t (p) is the temporal gradient at point p, and the spatial and temporal gradients are calculated using a local window around each pixel. The local averaging equation is as follows:

[0037] ∑ p∈w I x (p) 2 Δx+∑ p∈w I x (p)I y (p)Δy=-∑ p∈W I x (p)I t (p)

[0038] W is a local window around the pixel, and the motion parameters Δx and Δy are finally solved by using local averaging to solve the linear equations:

[0039]

[0040] For tilt detection, a fixed point in the video is selected as the reference point (x). r y r Let the reference point (x) r yr ) to the initial reference point (x) in the first frame b y b The vector V1 is formed from the tilted tower crane reference point (x). b +Δx, y b +Δy) forms vector V2. The angle between the two vectors V1 and V2 is calculated to detect the tilt of the tower crane. Based on the tower crane's verticality requirements, a tilt angle threshold is set, and an alarm is triggered when the detected tilt angle exceeds the tilt angle threshold.

[0041]

[0042]

[0043]

[0044] θ = arccos(cos(θ))

[0045] In the formula, θ is the tilt angle of the tower crane, · represents the inner product of vectors, and ||v1|| and ||v2|| represent the norms of the vectors, respectively.

[0046] A tower crane safety operation monitoring system, comprising:

[0047] The acquisition and segmentation module is configured to acquire visual information of the target tower crane, determine the video frames in the visual information, separate and mark the tower crane area in the video frames, and obtain tower crane segmentation information.

[0048] The processing module is configured to determine the initial bounding box information in the tower crane segmentation information, and use the upper left and lower right coordinates of the initial bounding box information as input to the YOLOv8 model. The YOLOv8 model outputs the target position information of the target tower crane.

[0049] The tracking module is configured to input the target location information into ByteTrack, and ByteTrack outputs the real-time tracking results of the tower crane;

[0050] The safety monitoring module is configured to monitor the tower crane's safety based on real-time tracking results, combined with optical flow tracing.

[0051] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a tower crane safety operation monitoring system.

[0052] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a tower crane safety operation monitoring system.

[0053] Compared with the prior art, the present invention has the following beneficial technical effects:

[0054] This invention provides a method, system, device, and medium for monitoring the safe operation of tower cranes, comprising the following steps: collecting visual information of the target tower crane; separating and marking the tower crane region in the video frame of the visual information to obtain tower crane segmentation information; using the upper left and lower right coordinates of the initial bounding box information in the tower crane segmentation information as input to the YOLOv8 model, and the YOLOv8 model outputting the target position information of the target tower crane; inputting the target position information into ByteTrack to achieve real-time tracking of the tower crane; and performing safety monitoring of the tower crane based on the real-time tracking results, combined with optical flow tracing. This application is not affected by factors such as lighting, shadows, and different weather conditions, has high accuracy and robustness, can meet the requirements of real-time performance, and can provide early warning of safety events such as tower crane tilting and settlement. Attached Figure Description

[0055] Figure 1 This is a flowchart of a tower crane safety operation monitoring method according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the tower crane's rotation in an embodiment of the present invention;

[0057] Figure 3 This is a front view comparison diagram of the overall rotation of the tower crane boom in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of the tower crane tilting in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of tower crane settlement in an embodiment of the present invention;

[0060] Figure 6 This is a comparison chart of the segmentation results of the target tower crane using the SAM model and MobileSAM in an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram illustrating point-based segmentation of tower cranes in different scenarios using the MobileSAM model in an embodiment of the present invention;

[0062] Figure 8 These are the segmentation and tracking results of different frames in the embodiments of the present invention;

[0063] Figure 9 The images show the segmentation and tracking results of some video frames in this embodiment of the invention.

[0064] Figure 10 This refers to the reference point trajectory and its corresponding tilt angle in the embodiments of the present invention, as well as the corresponding tilt angle for each frame. Detailed Implementation

[0065] The present invention will be further described in detail below with reference to specific embodiments. These descriptions are for explanation purposes only and are not intended to limit the scope of the invention.

[0066] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0067] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0068] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for monitoring the safe operation of a tower crane, comprising the following steps:

[0069] S1: Collect visual information of the target tower crane, determine the video frames in the visual information, separate and mark the tower crane area in the video frames to obtain tower crane segmentation information;

[0070] S2: Determine the initial bounding box information in the tower crane segmentation information, and use the upper left and lower right coordinates of the initial bounding box information as input to the YOLOv8 model. The YOLOv8 model outputs the target position information of the target tower crane.

[0071] S3: Input the target location information into ByteTrack, and ByteTrack outputs the real-time tracking results of the tower crane;

[0072] It should be noted that the operating system used in this embodiment is Windows 11, the CPU is an i9-13900kf, the GPU is an RTX A6000 with 48GB of video memory, and the RAM is 64GB. All models used are based on PyTorch 1.9.1, CUDA version 11.7, and cuDNN version 7.6.5.

[0073] Furthermore, the ByteTrack described in this application is a real-time target tracking model. ByteTrack is a simple, efficient, and universal multi-target tracking algorithm. By making full use of the information from low-scoring detection boxes, it effectively solves the problems of missed detections and fragmented trajectories in multi-target tracking, thereby improving the accuracy and robustness of tracking.

[0074] Preferably, in step S1, the MobileSAM method is used to separate and label the tower crane region in the visual information video frame to obtain tower crane segmentation information. Specifically, MobileSAM is a lightweight image segmentation model that performs well in terms of performance and versatility, making it particularly suitable for mobile applications. Training MobileSAM can be completed on a single GPU in less than a day, resulting in high training efficiency. In terms of inference speed, MobileSAM runs approximately 10ms per image, with the image encoder running 8ms and the Mask decoder running 2ms, giving it extremely fast image processing speed. It should be noted that MobileSAM can effectively segment tower cranes without requiring training and can meet real-time requirements.

[0075] It should be noted that in the YOLOv8 model, a bounding box is a rectangular box used to identify and locate detected objects in an image. This rectangular box tightly surrounds the object, helping the algorithm determine the object's location and extent in the image. In object detection tasks, the YOLOv8 model predicts a series of bounding boxes, each containing information about its location (e.g., the coordinates of its center point and its width and height) and confidence (indicating the probability that an object actually exists within the bounding box). In addition, for each bounding box, YOLOv8 also predicts one or more class labels, indicating the category of the object within that bounding box.

[0076] Furthermore, in the process of obtaining tower crane segmentation information by separating and marking using the MobileSAM method, the following methods are employed: point-hint segmentation, box-hint segmentation, or segmentation of all.

[0077] The point-based segmentation method involves segmenting the entire tower crane by inputting the coordinates of a point in the image.

[0078] The box prompt segmentation involves inputting the coordinate information of the tower crane in the image and segmenting the corresponding part accordingly;

[0079] The "segmentation of all" refers to segmenting the entire input image to obtain a mask representation for each category.

[0080] Specifically, the point-based segmentation process is as follows:

[0081] The resolution of an image I of arbitrary size in the video frame of the acquired visual information of the target tower crane is scaled and filled to 1024×1024;

[0082] After feeding the scaled and filled image I into ViT-H, we get:

[0083] f i =E i (I)

[0084] In the formula, E i (I) represents the ViT-H encoding process for image I; f i Image embedding;

[0085] The sequence of image I is obtained through the cue encoder:

[0086] f p =E p (P)

[0087] Where P is the prompt, E p (P) indicates the encoding process for image I;

[0088] M=D m (f i f p f o )

[0089] Among them, f o M and the output token and final mask are respectively; f p For prompt embedding; D m This is the decoding process to obtain the final mask for image I.

[0090] It should be noted that the target location information described in this embodiment includes location coordinates and category labels.

[0091] Preferably, the safety monitoring of the tower crane based on the real-time tracking results and combined with the optical flow tracing method includes: monitoring the tower crane's rotation under non-operational conditions and monitoring its tilt and settlement under operating conditions.

[0092] Specifically, the monitoring process for the tower crane's rotation under non-operational conditions is as follows:

[0093] By segmenting and then tracking the real-time video stream, the predicted bounding boxes A1 and A2 of the preceding and following frames are compared, and the overlap of the predicted bounding boxes is evaluated by comparing their IoU.

[0094]

[0095] Specifically, the closer the IoU is to 0, the less the two frames overlap; the closer the IoU is to 1, the greater the overlap between the two frames; when the IoU equals 1, the two frames completely overlap. Based on specific needs and actual conditions, an IoU threshold of 0.95 is set as the boundary for judging the stability of the tower crane under non-operational conditions. If the IoU value of the predicted frames in the preceding and following frames is greater than or equal to the set IoU threshold, it indicates that the tower crane is in a stable state without any abnormalities, and no warning is needed. However, if the IoU is less than the IoU threshold, it indicates that the tower crane has been detected to be rotating, triggering the warning mechanism and issuing a warning signal through an alarm or other means to remind staff to conduct an inspection. Other warning signals include a buzzer alarm, flashing warning lights, and fault indicator lights.

[0096] Furthermore, the tower crane rotation described in this embodiment is as follows: Figure 2 As shown, its horizontal swaying may be caused by external factors such as wind, or it may be due to the contactor contacts sticking together or the rotating brake mechanism of the boom being loose. A frontal view comparison of the tower crane boom after its rotation under non-moving conditions is shown below. Figure 3 As shown, Figure 3 The image at the top is the front view of the tower crane boom in its non-parallel configuration. Figure 3 The figure at the bottom is a front view of the tower crane boom after its overall rotation. As can be seen from the figure, the front view changes when the tower crane boom swings, and this application detects the state of the tower crane boom based on this change.

[0097] The tower crane tilt described in this embodiment is as follows: Figure 4 As shown, the tower crane settlement is as follows: Figure 5 As shown, the process of monitoring tilt and settlement during operation is as follows:

[0098] After obtaining the real-time tower crane segmentation and tracking results, a base point (x) is selected in the first frame of the input tower crane video stream. b y b As the corner point for sparse optical flow tracing, the sparse feature point set of the previous frame image is obtained using the Lucas-Kanade algorithm. Then, using the previous frame image, the sparse feature point set of the previous frame image, and the next frame image, a point set with the same features in the next frame image is obtained. This set is used to determine and obtain the optical flow between the sparse point sets of the two frames, thereby obtaining the motion trajectory of the corner point, i.e., the reference point of the tilted tower crane (x).b +Δx, y b The trajectory of the motion of +Δy).

[0099] Assume two consecutive image frames, the first frame being I(x) b y b ,t), second frame I(x) b +Δx, y b +Δy,t+Δt), where (x b y b Let be the pixel coordinates of the reference point in the image, and t be time. Then the equation for grayscale change is:

[0100] I x (p)Δx+I y (p)Δy+I t (p)=0

[0101] Among them, I x (p) and I y (p) is the spatial gradient at point p, Δx and Δy are the components of the motion vector, I t (p) is the temporal gradient at point p. Then, the spatial and temporal gradients are calculated using a local window around each pixel, with the local averaging equation as follows:

[0102]

[0103] W is a local window around the pixel, and the motion parameters Δx and Δy are finally solved by using local averaging to solve the linear equations:

[0104]

[0105] For tilt detection, a fixed point in the video is selected as the reference point (x). r y r For example, the base of a tower crane, let the reference point (x) r y r ) to the initial reference point (x) in the first frame b y b The vector V1 is formed from the tilted tower crane reference point (x). b +Δx, y b +Δy) forms vector V2. The angle between the two vectors V1 and V2 is calculated to detect the tilt of the tower crane. Based on the tower crane's verticality requirements, a tilt angle threshold is set, and an alarm is triggered when the detected tilt angle exceeds the tilt angle threshold.

[0106]

[0107]

[0108]

[0109] θ = arccos(cos(θ))

[0110] In the formula, θ is the tilt angle of the tower crane, · represents the inner product of vectors, and ||v1|| and ||v2|| represent the norms of the vectors, respectively.

[0111] Meanwhile, when the tower crane's tilt angle is within a threshold range, the distance difference Δh between vectors V1 and V2 is calculated to detect whether the tower crane has settled.

[0112] Δh=||v1||-||v2||

[0113] This process further enhances the monitoring of the tower crane's status.

[0114] The present invention provides an embodiment as follows:

[0115] like Figure 6 As shown in Table 1, the input image of the target tower crane was segmented using point-cue segmentation, bounding box segmentation, and all-inclusive segmentation, and the segmentation results were compared with those of the original SAM model. It can be seen that the segmentation results of MobileSAM are comparable to those of the original SAM, but its encoder module has fewer parameters. In addition, MobileSAM is faster because it performs inference on a single GPU for a single image.

[0116] Table 1 Comparison of MobileSAM and SAM parameter sizes and inference speed of a single image on a single GPU

[0117]

[0118] After comparing the segmentation results with the original SAM model, the MobileSAM model was used to perform point-cue segmentation of tower cranes in different scenarios, such as... Figure 7 As shown in the figure; the experimental results show that MobileSAM can perform fine segmentation of the tower crane edge in any scenario; the MobileSAM model performs well in the segmentation of tower crane edge, which indicates that the model has strong generalization ability and can effectively segment the tower crane structure in different scenarios.

[0119] Although SAM was not designed for tracking tasks, we adapted it for a tracking workflow to perform this important task. First, we created a dataset of tower cranes to test the ability to track tower crane motion and evaluate the performance of the tracking function.

[0120] The initial bounding box information (x1, y1, x2, y2) extracted by MobileSAM is input into the YOLOv8 algorithm, which classifies objects within the bounding box as tower cranes. Then, the output of YOLOv8 and the updated bounding box information are input into the ByteTrack algorithm, which tracks the movement of the tower cranes in the scene. Using YOLOv8 and ByteTrack in this workflow enables the system to accurately track the movement of tower cranes in various positions and directions within complex scenes.

[0121] A 15-second video clip of a tower crane operation was analyzed to extract 375 frames at a frame rate of 25.00 frames per second. Figure 8 The results of segmentation and tracking of different frames are shown in the figure. It can be clearly seen that the method of segmenting first and then tracking can more clearly observe the state of the tower crane jib and body, and obtain real-time data of the edges of the tower crane during operation, thus monitoring the operation of the tower crane. Therefore, the method of the present invention is very effective for tracking tower cranes for safety monitoring.

[0122] Self-rotation detection under non-operational conditions:

[0123] The method of this invention segments and tracks the tower crane video transmitted from the camera, and then compares the previous and next frames. If the IoU of the predicted bounding boxes of the previous and next frames is greater than 0.95, the tower crane is determined to be in a stable state. If it is less than 0.95, an alarm is issued to prevent accidents from occurring.

[0124] Experimental analysis was conducted on real video data of a tower crane in operation, such as tower crane tracking. Assuming the tower crane was not in operation, the 15-second video contained 375 frames. The IoU of the predicted bounding boxes of the preceding and following frames was determined, and then the precision and recall were calculated to verify the accuracy of the system and reduce false alarms.

[0125]

[0126]

[0127] In this system, TP represents true positives, FP represents false positives, and FN represents false negatives. TP and FP samples are categorized based on whether the Intersection over Union (IoU) between the predicted result and the true value is greater than a threshold. The accuracy and recall of the system were verified using 375 frames of video data, comprising 374 data sets. The results show an accuracy of 0.89 and a recall of 0.92. Therefore, this system is highly effective for providing settlement warnings for tower cranes operating outside of normal conditions.

[0128] Actual tower crane tilting accident detection:

[0129] This invention was used to detect tower crane collapse accidents at actual construction sites. The actual accident video was 20 seconds long, with a frame rate of 30 frames per second, containing a total of 503 frames. Figure 9 The segmentation and tracking results of some video frames in this embodiment are shown. Observing the tracking results in the figure, it can be clearly seen that between the 1st frame and the 96th frame, the tower crane gradually tilts to the left. At the 153rd frame, the tower crane returns to the vicinity of the starting position and then continues to tilt to the right in the following frames, eventually leading to collapse. This process occurs between the 154th frame and the 503rd frame. For the tracking of the reference point, the position of the reference point is manually determined in the 1st frame, and the position of the reference point in each frame is tracked by the Lucas-Kanade algorithm, and its trajectory diagram is drawn. At the same time, the vector angle between each two frames is calculated by formula (19) to reflect the change of the tilt angle of the tower crane.

[0130] Select a reference point (x) r y r The image shows the lower position of the tower crane in the video. The origin of the video coordinate system is located in the lower left corner of the video, and the starting point of the ordinate is located in the upper left corner of the video. Figure 10 In the left-hand graph, the black dots forming a dotted line represent the Lucas-Kanade algorithm's tracking of the reference point (x). b y b The movement trajectory of the tower crane is shown in the figure, while the pink dotted line graph represents the change in the angle between vectors V1 and V2 as the reference point of the tower crane changes, i.e., the tilt angle θ of the tower crane; the figure on the right shows the tilt angle θ corresponding to each frame in the 503 frames of images.

[0131] By setting a threshold for the tilt angle of the tower crane, the system can monitor whether the tilt degree exceeds the set threshold. When the tilt degree of the tower crane exceeds the set threshold, the system will issue an alarm to provide timely warning and prevent potential accident risks.

[0132] In this embodiment, the MobileSAM model is applied to a real construction site to segment the tower crane to verify its segmentation performance on the tower crane. MobileSAM, combined with YOLOv8+ByteTrack, enables more accurate real-time tracking of the tower crane. Based on segmentation and tracking, the model is used to monitor the tower crane's self-rotation, tilt, and settlement under non-operational conditions.

[0133] This invention provides a tower crane safety operation monitoring system, comprising:

[0134] The acquisition and segmentation module is configured to acquire visual information of the target tower crane, determine the video frames in the visual information, separate and mark the tower crane area in the video frames, and obtain tower crane segmentation information.

[0135] The processing module is configured to determine the initial bounding box information in the tower crane segmentation information, and use the upper left and lower right coordinates of the initial bounding box information as input to the YOLOv8 model. The YOLOv8 model outputs the target position information of the target tower crane.

[0136] The tracking module is configured to input the target location information into ByteTrack, and ByteTrack outputs the real-time tracking results of the tower crane;

[0137] The safety monitoring module is configured to monitor the tower crane's safety based on real-time tracking results, combined with optical flow tracing.

[0138] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used in the operation of a tower crane safety operation monitoring method.

[0139] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device within a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium of the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space containing the terminal's operating system. Furthermore, this storage space also contains one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the tower crane safety operation monitoring method described in the above embodiments.

[0140] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring the safe operation of a tower crane, characterized in that, Includes the following steps: S1: Collect visual information of the target tower crane, determine the video frames in the visual information, separate and mark the tower crane area in the video frames to obtain tower crane segmentation information; S2: Determine the initial bounding box information in the tower crane segmentation information, and use the upper left and lower right coordinates of the initial bounding box information as input to the YOLOv8 model. The YOLOv8 model outputs the target position information of the target tower crane. S3: Input the target location information into ByteTrack, and ByteTrack outputs the real-time tracking results of the tower crane; S4: Based on the real-time tracking results, combine optical flow tracing method to perform safety monitoring of tower cranes; The safety monitoring of tower cranes based on real-time tracking results and combined with optical flow tracing includes: monitoring the tower crane's rotation under non-operating conditions and monitoring its tilt and settlement under operating conditions. The monitoring process for the tower crane's rotation under non-operational conditions is as follows: By segmenting and then tracking visual information, the predicted bounding box from the previous frame is obtained. , subsequent frame prediction box The previous frame prediction box , subsequent frame prediction box By comparing the results, the predicted bounding boxes from the previous frame are obtained. , subsequent frame prediction box degree of overlap: Specifically, the closer the IoU is to 0, the less overlap there is between the two frames; the closer the IoU is to 1, the higher the overlap between the two frames; when the IoU is equal to 1, the two frames completely overlap. A preset IoU threshold condition is used as the boundary for judging the stability of the tower crane under non-operational conditions. If the IoU value of the predicted frames of the preceding and following frames is greater than or equal to the set IoU threshold condition, the tower crane is in a stable state and there is no abnormality; if the IoU is less than the IoU threshold condition, it indicates that the tower crane has been detected to be rotating, at which point an early warning mechanism is triggered and a warning signal is issued. The process of monitoring tilt and settlement during operation is as follows: After obtaining the real-time tower crane segmentation and tracking results, a base point is selected in the first frame of the input tower crane video stream. As corner points for sparse optical flow tracing, the sparse feature point set of the previous frame image is obtained using the Lucas-Kanade algorithm. Then, using the previous frame image, the sparse feature point set of the previous frame image, and the next frame image, a point set with the same features in the next frame image is obtained. This set is used to determine and obtain the optical flow between the sparse point sets of the two frames, thereby obtaining the motion trajectory of the corner point, i.e., the reference point of the tilted tower crane. The trajectory of movement; Assuming two consecutive image frames, the first frame is... The second frame ,in These are the pixel coordinates of a reference point in the image. Let time be the time factor, then the grayscale change equation is: in, and It is a point Spatial gradient at that location and These are the components of the motion vector. It is a point The temporal gradient is calculated at a given location, and the spatial and temporal gradients are computed using a local window around each pixel. The local averaging equation is as follows: It is a local window around the pixel, and finally the motion parameters are solved by using local averaging to solve the system of linear equations. and : For tilt detection, a fixed point in the video is selected as the reference point. Let the reference point To the initial reference point in the first frame Constructing vectors to the tower crane reference point after tilting Constructing vectors Calculate two vectors and The angle between the two sides is used to detect the tilt of the tower crane. A tilt angle threshold is set according to the tower crane's verticality requirements, and an alarm is triggered when the detected tilt angle exceeds the threshold. In the formula, For the tilt angle of the tower crane, Represents the dot product of vectors. and Let represent the norms of the vectors.

2. The tower crane safety operation monitoring method according to claim 1, characterized in that, In step S1, the MobileSAM method is used to separate and mark the tower crane area in the visual information video frame to obtain tower crane segmentation information.

3. The tower crane safety operation monitoring method according to claim 2, characterized in that, In the process of obtaining tower crane segmentation information through separation and labeling using the MobileSAM method, the following methods are employed: point-hint segmentation, box-hint segmentation, or segmentation of all. The point-based segmentation method involves segmenting the entire tower crane by inputting the coordinates of a point in the image. The box prompt segmentation involves inputting the coordinate information of the tower crane in the image and segmenting the corresponding part accordingly; The "segmentation of all" refers to segmenting the entire input image to obtain a mask representation for each category.

4. The tower crane safety operation monitoring method according to claim 3, characterized in that, The point-based segmentation process is as follows: The resolution of an image I of arbitrary size in the video frame of the acquired visual information of the target tower crane is scaled and filled to 1024×1024; After feeding the scaled and filled image I into ViT-H, we get: In the formula, The ViT-H encoding process for image I; Image embedding; The sequence of image I is obtained through the cue encoder: in, For the prompt, Provide a hint about the encoding process for image I; in, and These are the output token and the final mask, respectively. For the purpose of embedding; This is the decoding process to obtain the final mask for image I.

5. The tower crane safety operation monitoring method according to claim 1, characterized in that, The target location information includes location coordinates and category labels.

6. A tower crane safety operation monitoring system, characterized in that, A tower crane safety operation monitoring method according to any one of claims 1-5 includes: The acquisition and segmentation module is configured to acquire visual information of the target tower crane, determine the video frames in the visual information, separate and mark the tower crane area in the video frames, and obtain tower crane segmentation information. The processing module is configured to determine the initial bounding box information in the tower crane segmentation information, and use the upper left and lower right coordinates of the initial bounding box information as input to the YOLOv8 model. The YOLOv8 model outputs the target position information of the target tower crane. The tracking module is configured to input the target location information into ByteTrack, and ByteTrack outputs the real-time tracking results of the tower crane; The safety monitoring module is configured to monitor the tower crane's safety based on real-time tracking results, combined with optical flow tracing.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the tower crane safety operation monitoring method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the tower crane safety operation monitoring method as described in any one of claims 1-5.