A method for real-time detection and early warning of lightweight hoisting operation violation behavior towards the edge end of a tower crane

Abstract

This invention discloses a lightweight real-time detection and early warning method for violations in lifting operations at the edge of tower cranes, belonging to the field of computer vision and edge safety monitoring. The invention first acquires images of the hook area using an industrial camera located beneath the luffing trolley; then, it utilizes a TinyBackbone backbone network built based on depthwise separable convolution and coordinate attention for lightweight feature extraction, compressing the computational cost to 1 / 8 to 1 / 9 of standard convolution; after PANet+EMA multi-scale fusion, a multi-task decoupled detection head outputs personnel bounding boxes, rope deflection angles, and hook disengagement probabilities in parallel; finally, it triggers audible and visual alarms based on hierarchical early warning logic, and achieves low-latency operation on Jetson Orin edge devices through TensorRT FP16 inference and a four-stage pipeline. The model size is only 7.8MB, the inference latency is reduced by approximately 70% compared to YOLOv8n, and the mAP reaches 91.6%, meeting the real-time multi-violation detection and early warning requirements for scenarios such as tower cranes at construction sites and quay cranes in ports.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, edge computing, and industrial safety monitoring technology. Specifically, it relates to a visual detection and early warning method for violations that can be run in real time on an edge computing device for tower crane hoisting operations. Background Technology

[0002] Tower cranes are core equipment in modern construction, port loading and unloading, and large-scale logistics warehousing. Their operational safety is directly related to the lives and property of personnel on site. Industry statistics show that a significant proportion of tower crane safety accidents stem from three typical violations: first, personnel staying or walking in the area below the load; second, the sway angle of the hoisting rope exceeding the safety limit during hoisting; and third, abnormal hook-rope connection or accidental rope detachment.

[0003] Currently, monitoring of the above three types of behavior mainly relies on the following two technical approaches:

[0004] (a) Manual inspection and video review. This method relies on on-site inspections by safety officers or video patrols in the monitoring room. Due to the limited field of vision during tower crane operations, blind spots for the operator in the cab, and the difficulty in maintaining a high level of concentration for extended periods, this method has inherent drawbacks such as a high rate of missed inspections and a delayed response (usually tens of seconds to several minutes).

[0005] (II) Physical Sensors and Behavioral Analysis Technology. Some existing tower cranes are equipped with angle sensors and torque limiters to monitor the hoisting rope attitude, or use depth cameras for hook derailment detection. For example, Chinese patent CN114368691A discloses a hook deflection angle detection device, but it requires a specific guiding light source. Other hook derailment detection solutions are based on depth image differential, which is sensitive to light and weather conditions and cannot simultaneously perform location analysis of personnel entering the area. These solutions either rely on the fusion of multiple sensor types, leading to complex wiring and high calibration costs, or they have a single detection dimension, making it difficult to form a comprehensive assessment of the three types of risks: "person-rope-hook."

[0006] (III) The Challenges of Edge Deployment for Deep Learning-Based Visual Inspection. In recent years, visual inspection methods based on the YOLO series of models have begun to be explored and applied in the field of industrial safety. However, if existing methods are directly applied to tower crane scenarios, they face at least three unresolved difficulties:

[0007] 1. The contradiction between model size and edge computing power is prominent. High-precision models (such as YOLOv8s) typically have more than 10M parameters, and the inference time per frame on edge devices such as Jetson exceeds 200ms, which cannot meet the real-time warning requirement of less than 100ms.

[0008] 2. Lightweighting can lead to missed detections of slender or small targets. A sling appears as a thin, elongated line in an image, occupying only a few hundred pixels; the visual difference of a hook detaching from the sling is subtle. While conventional depthwise separable convolutional compression can reduce the model size, it often weakens its ability to represent features of such structures.

[0009] 3. There is currently no efficient framework for joint detection of multiple violations. Existing technologies often treat personnel, perspectives, and decoupling detection as independent models or serial modules, resulting in repeated use of memory and computing power on edge devices, or the need for frequent switching of model weights.

[0010] In summary, how to achieve low-latency, high-reliability real-time early warning under the constraints of monocular visual input, edge computing power limitations, and joint detection of three types of violations is a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0011] (a) Purpose of the invention

[0012] The present invention aims to provide a real-time detection and early warning method for violations in lightweight hoisting operations at the edge of tower cranes, in order to solve the technical problems of excessive latency of edge models, decreased detection accuracy after lightweighting, and difficulty in concurrent detection of multiple violations in the prior art.

[0013] (II) Technical Solution

[0014] This invention adopts an architecture of "lightweight backbone network - direction awareness enhancement - multi-task decoupling - edge inference acceleration", and the core implementation steps are as follows.

[0015] Step S1: On-site image acquisition and preprocessing

[0016] An industrial camera with wide dynamic range and basic image stabilization is fixedly installed below the tower crane's luffing trolley, with the lens's main optical axis aligned with the hook's potential movement area. Raw images are acquired in real-time (1920×1080 resolution, 30fps), and the following preprocessing pipeline is performed upon receipt:

[0017] Remove invalid areas such as the sky and tower background, and retain the hook and the ROI area within 5 to 8 meters below it;

[0018] The ROI region is normalized to a fixed size of 640×640 pixels using bilinear interpolation;

[0019] During the training phase, a random motion blur kernel with a 50% probability (simulating hook swaying), random HSV offset (20% brightness perturbation), and random occlusion (simulating boom shadows) are applied to enhance the model's adaptability to various working conditions. The overall architecture of the above image acquisition and preprocessing workflow is as follows: Figure 1 As shown.

[0020] Step S2: Construct the lightweight backbone network TinyBackbone

[0021] This invention proposes a backbone network based on depthwise separable convolution and incorporating a coordinate attention mechanism, hereinafter referred to as TinyBackbone.

[0022] Let the standard convolution kernel size be... Input channel number Number of output channels The output feature map size is Then the computational complexity is:

[0023] (1)

[0024] Depthwise separable convolution breaks down the above process into channel-wise convolution and pointwise convolution, resulting in the following total computational cost:

[0025] (2)

[0026] The ratio of the two:

[0027] (3)

[0028] Pick , At that time, the theoretical compression ratio is approximately The operations of standard convolution and depthwise separable convolution are compared as follows: Figure 3 As shown.

[0029] To further compensate for the reduced spatial awareness after compression, this invention embeds coordinate attention modules in each of the three downsampling stages of TinyBackbone. For the input feature map... Coordinate attention aggregates features along the height and width directions respectively:

[0030] (4)

[0031] After concatenating the features from the two directions, the data undergoes 1×1 convolution dimensionality reduction, batch normalization, and nonlinear activation, and is then decomposed back into attention vectors in the two directions. and The final output of the first passage, location The eigenvalues ​​are:

[0032] (5)

[0033] The internal structure of the TinyBackbone backbone network and its embedded coordinate attention module constructed in this step is as follows: Figure 2 As shown.

[0034] This design makes the network more responsive to vertically oriented rope structures while retaining sensitivity to horizontal personnel movement.

[0035] Step S3: Multi-scale feature fusion and enhancement

[0036] The three feature maps at different scales (scale factors of 8, 16, and 32) output by the backbone network are fed into the improved PANet path aggregation network. Based on the original top-down and bottom-up fusion paths, this invention introduces an efficient multi-scale attention module (EMA) at each cross-scale connection.

[0037] The EMA module uses three convolutional branches in parallel: 3×3, 5×5, and 7×7, and employs adaptive weights. Combine the outputs of each branch:

[0038] (6)

[0039] in A small subnetwork dynamically generates the receptive field based on input features, allowing the network to automatically select the appropriate receptive field according to the current field of view (distance of the hook, length of the rope). The multi-scale feature fusion path and the embedding position of the EMA module are as follows: Figure 4 As shown.

[0040] Step S4: Multi-task decoupling detection head

[0041] After PANet outputs the feature map, this invention designs a multi-task decoupled detection head, which includes three parallel branches:

[0042] (a) Personnel Detection Branch: Outputs a 2D bounding box of the area below the suspended object. and the corresponding personnel confidence level The regression loss uses CIoU Loss:

[0043] (7)

[0044] in , , This is the length of the diagonal of the minimum bounding box.

[0045] (b) Rope Angle Detection Branch: Regressing to two key points: the connection point between the rope and the hook. And the intersection of the hoisting rope and the lower edge of the luffing trolley. The deflection angle of the suspension rope is calculated using the following formula:

[0046] (8)

[0047] Keypoint regression uses mean squared error loss:

[0048] (9)

[0049] (c) Decoupling Detection Branch: Extracts higher resolution features from the hook region, performs global average pooling and two fully connected layers, and outputs the decoupling probability. The classification loss is binary cross-entropy:

[0050] (10)

[0051] The overall loss function is a weighted sum of three parts:

[0052] (11)

[0053] The three parallel branches and their output signals of the multi-task decoupling detection head of this invention are as follows: Figure 5 As shown.

[0054] Step S5: Graded Early Warning and Anti-shake Logic

[0055] Let the projection point of the hook be in the image plane. The corresponding ground safety zone is a radius of circular area ( Based on the tower crane model and the size of the lifted object, the default distance is 3 meters. When the pixel distance between the center of the personnel boundary frame and the center of this circle is less than the threshold, a yellow warning for "person standing under the lifted object" is triggered.

[0056] Determination of abnormal rope angle: continuous Time window ( Within seconds (approximately 15 frames), if This triggers a yellow alert. To prevent false alarms caused by brief wind disturbances, the alert status uses a counter accumulation logic: the counter increments by 1 for each frame that meets the condition, and decrements by 1 (not less than 0) until the counter reaches a certain value. Confirm the alarm in time.

[0057] Hook detachment warning: Required Only when the conditions are met for three consecutive frames or more will a red emergency alarm be triggered. This signal can simultaneously alert the driver via an audible and visual alarm. If necessary, a normally closed contact signal can be output via a digital interface to link the tower crane control system to restrict lifting or slewing actions.

[0058] Step S6: Edge Engineering Optimization

[0059] The target edge platform is either an NVIDIA Jetson Orin NX (8GB memory, 70 TOPS computing power) or a Rockchip RK3588. The following optimizations will be performed during deployment:

[0060] Export the trained PyTorch model to ONNX format, then build the inference engine using TensorRT version 8.5 or higher, enable FP16 half-precision, and turn on layer fusion (merge convolution + BN + ReLU into one kernel).

[0061] A four-stage pipelined parallel architecture is adopted: acquisition thread (reading camera frames), preprocessing thread (scaling, normalizing, and channel conversion), inference thread (executing the TensorRT engine), and post-processing thread (parsing bounding boxes, calculating angles and probabilities, and triggering alarms). Data is exchanged between the four threads using a lock-free circular queue.

[0062] The final end-to-end processing latency was measured to be ≤50ms, meeting the design target of less than 100ms. The hardware connectivity of the edge computing devices and the communication topology of each system module are as follows: Figure 6 As shown.

[0063] (III) Beneficial Effects

[0064] Compared with the prior art, the present invention has the following outstanding advantages:

[0065] 1. Significantly lightweight model performance: The TinyBackbone backbone network has only 0.37M parameters, which is about 12.2% of YOLOv8n; the model file (FP16) is 7.8MB, which can be compressed to 3.2MB after INT8 quantization, and can run stably on Jetson or RK3588 platforms.

[0066] 2. High efficiency of joint detection of multiple violations: A single forward propagation can simultaneously output personnel position, rope deflection angle and decoupling probability. Compared with running three independent models for personnel detection, angle measurement and decoupling classification, the engineering deployment cost is reduced by about 70% and the inference latency is reduced by more than 60%.

[0067] 3. Edge real-time performance meets standards: Through TensorRT FP16 inference and pipelined parallelism, an average latency of 47ms is achieved on the Jetson Orin NX platform, which is far lower than the 158ms required by existing solutions running YOLOv8n on the same device, meeting the hard real-time requirement of <100ms in tower crane field.

[0068] 4. Strong detection capability for slender / small targets: The coordinate attention module improves the average accuracy of rope angle detection from 87.4% to 89.1% with only a minimal increase in computation (+6ms), and improves the recall rate of decoupling detection by about 4 percentage points, verifying the effectiveness of the attention mechanism in tower crane scenarios. Attached Figure Description

[0069] Figure 1: A flowchart of the overall method of the present invention;

[0070] Figure 2 : Internal structure diagram of TinyBackbone backbone network and coordinate attention module;

[0071] Figure 3 A diagram comparing the operations of depthwise separable convolution and standard convolution;

[0072] Figure 4 PANet+EMA multi-scale feature fusion path diagram;

[0073] Figure 5 : Output branch structure diagram of the multi-task decoupling detection head;

[0074] Figure 6 Hardware connection topology diagram of the tower crane edge intelligent safety monitoring system. Detailed Implementation

[0075] To make the technical solution and effects of the present invention clearer, the present invention will be further described in detail below with reference to embodiments. It should be noted in advance that the embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0076] Example 1 (Complete Implementation Scheme)

[0077] 1. Dataset Construction and Labeling

[0078] A self-built TCD-Violation (Tower Crane Violation) dataset was created. Data collection locations included three construction sites and two inland ports, covering sunny, cloudy, dusky, and light rainy weather. Camera angles were: direct downward view from the luffing vehicle (approximately 70%) and auxiliary monitoring view from the driver's cab (30%). 12.5 hours of valid video footage were collected, and approximately 45,000 raw images were obtained by extracting frames at 1fps. After cleaning, 43,000 images were retained. The annotation types and quantities are shown in Table 1 below.

[0079] Table 1. Label Types and Quantities

[0080] People standing below the suspended object 13,200 rectangular frame The angle of the suspension rope is abnormal (>15°). 9,800 Two key points Hook and rope 6,400 Two-category labels No non-compliant negative samples 13,600 –

[0081] The dataset was randomly divided into a training set (34,400 images), a validation set (4,300 images), and a test set (4,300 images) in an 8:1:1 ratio.

[0082] 2. Model training parameters

[0083] Hardware: Dual NVIDIA RTX 4090 graphics cards (24GB x 2)

[0084] Framework: PyTorch 2.0 + CUDA 11.8

[0085] Input size: 640×640

[0086] Batch size: 64 (32 per calorie)

[0087] Optimizer: SGD (initial learning rate 0.01, momentum 0.937, weight decay 0.0005)

[0088] Learning rate scheduling: cosine annealing, minimum learning rate 0.0001

[0089] Total rounds: 300

[0090] Data augmentation: Mosaic (4 images stitched together), MixUp ( ), random motion blur (kernel size 5, angle random), HSV offset factor (H:0.015, S:0.7, V:0.4).

[0091] 3. Edge deployment and testing

[0092] Edge computing device: NVIDIA Jetson Orin NX 16GB, JetPack 5.1.2, TensorRT 8.6, deployed using C++ API.

[0093] The four-stage pipeline is implemented using "std::thread" and "boost::lockfree::spsc_queue". The preprocessing includes (1) "cv::resize" to 640×640; (2) normalize by dividing by 255; (3) BGR2RGB and send to GPU memory using "cudaMemcpyAsync".

[0094] 4. Experimental Results

[0095] The overall mAP@0.5 on the test set is 91.6%, and the various performance metrics are shown in Table 2.

[0096] Table 2. Performance Statistics Table

[0097] People standing below the suspended object 93.2% 90.8% 92.7% 43ms Abnormal angle of the suspension rope 89.7% 87.2% 89.4% 47ms Hook and rope 90.4% 88.9% 91.0% 45ms

[0098] Table 3 shows a comparison with the benchmark YOLOv8n in the same hardware environment:

[0099] Table 3. Comparison with YOLOv8n in the same hardware environment

[0100] YOLOv8n 3.01M 11.8MB 93.2% 158ms This invention 0.37M 7.8MB 91.6% 47ms

[0101] After 720 hours of operation at the construction site, a total of 287 violations were detected. These included 175 instances of people standing under suspended loads, 78 instances of abnormal rope angles, and 34 instances of hook detachment. Based on manual recording and comparison, the system's overall warning accuracy rate was 93.4%, with an average warning delay of 82ms, a false alarm rate of 3.8%, and a missed alarm rate of 2.8%.

[0102] 5. Ablation test

[0103] To verify the effectiveness of each innovative module, the following four ablation methods (all based on Jetson Orin) were designed, as shown in Table 4:

[0104] Table 4. Ablation Test

[0105] A (Benchmark: YOLOv8n) no no no no 93.2% 158ms B yes no no no 87.4% 62ms C yes yes no no 89.1% 68ms D yes yes yes no 90.3% 52ms E (This invention) yes yes yes yes 91.6% 47ms

[0106] The ablation results show that depthwise separable convolution significantly reduces latency, but at the cost of 5.8 percentage points of mAP; coordinate attention gains 1.7 percentage points of accuracy with a small amount of latency (+6ms); multitasking heads further reduce latency by avoiding redundant computations; and the EMA module brings the accuracy back to 91.6%, approaching the baseline model level.

[0107] Example 2 (System Configuration Example)

[0108] A tower crane edge intelligent safety monitoring system, the hardware connection relationship is as follows: Figure 6 As shown, it includes the following modules:

[0109] Image acquisition module: Hikvision MV-CS020 industrial camera, mounted on a dedicated shock-absorbing bracket under the luffing carriage, connected to Jetson Orin NX via a 15-meter MIPI extension cable.

[0110] Edge computing module: The Orin NX motherboard is installed in a waterproof enclosure (IP54 protection rating), which is fixed to the shaded side of the cab and communicates with the tower crane programmable controller (PLC) via CAN bus.

[0111] Warning module: An audible and visual alarm and a voice module are placed in the driver's cab. When a yellow warning is issued, the alarm sounds briefly at a frequency of 1Hz. When a red warning is issued, the alarm sounds continuously and announces "Emergency, stop immediately".

[0112] Remote management module: The system pushes a summary of the violation event (time, type, confidence level, screenshot URL) to the cloud monitoring platform via a 4G dongle, and the administrators receive real-time alarms via the web and mini-program.

[0113] Example 3 (Scenario Transfer: Port Quay Crane)

[0114] This invention can also be used for safety monitoring of spreader equipment in port quay cranes (ship-to-shore cranes). During migration, only minor data adjustments are needed: approximately 8,000 images of quay crane operations are collected, and the relative positional deviations between container lock holes and spreader equipment are re-annotated. The warning logic is revised: instead of using "person standing under the load," a warning is added for "spreader equipment and container alignment deviation greater than 10 cm." After 50 rounds of fine-tuning training, the final alignment accuracy in quay crane scenarios reaches 94.2%, with an average latency of 51 ms, meeting operational requirements.

[0115] Industrial applicability

[0116] The core components of this invention (industrial camera, Jetson module, alarm) are all commercially available standardized parts, with simple manufacturing and assembly processes, allowing for batch production. The entire system has been piloted in three construction companies, accumulating over 2000 hours of operation, with positive user feedback. This invention can be embedded in newly constructed tower cranes or added to existing tower cranes as an add-on safety kit, demonstrating promising commercial prospects.

Claims

1. A method for real-time detection and early warning of violations in lightweight hoisting operations targeting the edge of tower cranes, characterized in that, Includes the following steps: Step S1: Real-time image sequence of the hook working area is acquired by a monocular industrial camera fixed to the lower part of the tower crane luffing trolley, and the image is cropped and normalized in size. Step S2: Construct a lightweight backbone network TinyBackbone based on depthwise separable convolution and coordinate attention mechanism to extract multi-scale feature maps of the input image; Among them, depthwise separable convolution decomposes standard convolution into channel-wise convolution and point-wise convolution, and coordinate attention aggregates global information in the horizontal and vertical directions and generates bidirectional attention weights. Step S3: The multi-scale feature maps are fused using an improved PANet structure, and an efficient multi-scale attention module is embedded in the feature fusion node; Step S4: Design a multi-task decoupled detection head to output three types of violation-related signals in parallel: the personnel detection branch outputs a rectangular box with confidence level, the rope angle detection branch outputs key points at both ends of the rope and calculates the angle of deviation from the vertical plane. , Disengagement detection branch output hook disengagement probability ; Step S5: Construct a time-series cumulative early warning rule engine: When the IoU between the personnel rectangle and the ground projection area of ​​the hook is >0.3, or When the angle exceeds 15° for more than 0.5 seconds consecutively, or When three consecutive frames are above 0.7, an alarm of the corresponding level is triggered. Step S6: Utilize the TensorRT inference engine to perform FP16 half-precision inference and operator fusion optimization on the model, and execute it on the edge computing device using a four-stage pipeline architecture of "acquisition-preprocessing-inference-postprocessing" to make the end-to-end latency from image acquisition to alarm output less than 100ms.

2. The method according to claim 1, characterized in that, The coordinate attention module records the input features as follows: Global pooling is performed along the height and width directions respectively to obtain and After splicing, Highly oriented attention is generated after convolution, nonlinear activation, and further decomposition. and width direction attention The final output is .

3. The method according to claim 1, characterized in that, In the multi-task decoupling detection head, the angle of the suspension rope Calculated by the following formula: in The network regression coordinates of the connection point between the lifting rope and the hook are given. The network regression coordinates are the intersection points of the suspension rope and the lower edge of the luffing trolley.

4. The method according to claim 1, characterized in that, The warning rule engine described in step S5 uses a counter accumulation anti-shake logic: if the warning condition is met for each frame, the counter is incremented by 1; otherwise, the counter is decremented by 1 but not lower than 0, and the decrement only occurs when the counter value exceeds a preset threshold. The alarm is confirmed and output in real time.

5. The method according to claim 1, characterized in that, The edge computing device uses NVIDIA Jetson Orin series or Rockchip RK3588 chip; in the four-stage pipeline architecture, the acquisition thread, preprocessing thread, inference thread and postprocessing thread exchange data through a lock-free circular queue, wherein the preprocessing includes scaling to 640×640, pixel value normalization and BGR to RGB color space conversion.

6. A tower crane edge intelligent safety monitoring system implementing the method of any one of claims 1 to 5, characterized in that, include: Image acquisition module: Image-stabilized industrial camera, mounted below the luffing trolley, with its optical axis aligned with the moving area of ​​the tower crane hook; Edge computing module: An embedded motherboard with TinyBackbone and a multi-task decoupling detection head model, which is connected to the tower crane main controller via CAN or RS485 bus; Warning output module: an audible and visual alarm and voice broadcast unit located in the driver's cab; Remote management module: Uploads the type of violation, time of occurrence, and on-site screenshots to the cloud management platform via wireless network, and supports remote parameter configuration.

7. The system according to claim 6, characterized in that, The edge computing module has offline self-sustaining capability: when network communication is interrupted, it still maintains local real-time detection and early warning, and stores violation event logs on the local SD card, which will be automatically resumed after the network is restored.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor of the edge computing device, it implements all the steps of the method according to any one of claims 1 to 5.

Images