Tower crane operation behavior monitoring and protection system based on computer vision

The tower crane operation behavior monitoring and protection system, which integrates edge computing and multi-source data fusion, solves the safety hazards of existing systems being prone to failure in complex environments and hard interlock control, and achieves safe and reliable operation of tower cranes.

CN123105487APending Publication Date: 2026-11-17GUANGDONG DAFENG MECHANICAL ENG CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611355191.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-03
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing tower crane safety monitoring systems are prone to failure in complex environments, leading to the risk of missed alarms. Furthermore, hard interlock control methods are prone to causing load swaying and structural instability under heavy load or high-speed conditions. The lack of emergency takeover channels also poses safety hazards.

Method used

An edge computing control unit based on computer vision is adopted, which combines a visual acquisition module, a mechanical acquisition module and an environmental acquisition module. The edge computing control unit performs data fusion and risk assessment, dynamically adjusts intervention strategies, introduces non-visual logic compensation mechanisms and flexible reconfiguration control, and sets up bypass switches to ensure system reliability and safety.

Benefits of technology

In complex environments, it eliminates monitoring blind spots and potential reporting omissions, avoids violent swaying of the load and structural instability, provides an emergency takeover channel, and ensures the safe operation of tower cranes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123105487A_ABST
    Figure CN123105487A_ABST
Patent Text Reader

Abstract

This application relates to the field of safety control technology for lifting machinery, and discloses a computer vision-based monitoring and protection system for tower crane operation behavior. The system includes an edge computing control unit, a vision acquisition module, a mechanical acquisition module, an environmental acquisition module, an intervention execution module, and a bypass switch. The edge computing control unit receives multi-source synchronously acquired data for risk assessment and outputs tiered intervention control commands to the intervention execution module. The intervention execution module is connected in series between the output of the control handle and the lower-level programmable logic controller (PLC), and performs reconstructed output or message forwarding on the original output signal according to the command. The bypass switch is connected in parallel with the intervention execution module; when closed, it restores the direct connection between the handle and the lower-level PLC. This invention solves the problems of missed detection of visual failures and secondary hazards caused by traditional hard interlocks through visual and non-visual logic compensation, flexible speed reduction, and unidirectional interlock control, and provides a low-level physical emergency takeover channel, ensuring safety under extreme working conditions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of safety control technology for lifting machinery, specifically a computer vision-based monitoring and protection system for tower crane operation behavior. Background Technology

[0002] Tower cranes are widely used in construction, and their safe operation is highly dependent on the condition and behavior of the operators. In order to reduce the safety risks caused by human error, the industry has begun to introduce computer vision technology in recent years, using cameras installed in the driver's cab to monitor operator fatigue or violations.

[0003] Existing visual monitoring systems have significant limitations in practical engineering applications. Current monitoring solutions mostly rely on single-dimensional image recognition results to determine risks. Due to the complex operating environment of tower cranes, the reliability of image data drops sharply when faced with conditions such as direct sunlight at high altitudes, cameras obstructed by dust, or partial obstruction by the operator's limbs. This causes the system to easily fail in risk assessment under complex environments and generate missed detections.

[0004] Upon detecting a dangerous situation, existing systems typically employ a hard interlock mechanism that directly cuts off the control loop. Tower cranes possess significant operational inertia during heavy-load lifting or high-speed rotation. If the system abruptly cuts off the drive signal under such conditions, it can cause emergency braking of the mechanical mechanism, leading to violent swaying of the load and sudden changes in stress on the tower structure. This abrupt intervention not only fails to smoothly mitigate operational risks but also greatly increases the risk of secondary accidents such as falling loads or structural instability.

[0005] Furthermore, since existing protection and control modules are mostly directly connected in series in the original vehicle's control link, they lack an independent physical emergency takeover channel. Once the monitoring system itself experiences an internal program failure, bus communication interruption, or self-locking of the control state due to misjudgment, the entire underlying control loop will be paralyzed. In the face of sudden emergencies, operators cannot directly control the underlying equipment by bypassing the malfunctioning monitoring system. This makes the added safety protection system often a hidden danger to the functional safety of the crane's underlying structure, and it is difficult to meet the electrical emergency needs in actual construction operations. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a computer vision-based tower crane operation behavior monitoring and protection system. This system solves the problems of existing tower crane safety monitoring systems, which mainly rely on a single type of sensor. Visual monitoring is prone to failure and missed alarms under strong light or obstructed conditions. At the same time, the intervention mechanism of existing systems mostly adopts a hard interlocking method that directly cuts off the control loop. Under heavy load or high speed conditions, emergency power cut-off can easily cause the load to sway or even the structure to become unstable.

[0007] The first aspect of this invention provides a computer vision-based tower crane operation behavior monitoring and protection system, comprising: an edge computing control unit, and a vision acquisition module, a mechanical acquisition module, an environmental acquisition module, and an intervention execution module, which are respectively communicatively connected to the edge computing control unit; the system further comprises a bypass switch connected in parallel with the intervention execution module; The edge computing control unit is located in the tower crane operator's cab. It is used to receive data synchronously collected by the vision acquisition module, the mechanical acquisition module and the environmental acquisition module, perform timestamp alignment, status identification and risk assessment on the data, and output hierarchical intervention control commands to the intervention execution module. The intervention execution module is connected in series between the output end of the control handle and the lower-level programmable logic controller, and is used to reconstruct the original output signal of the control handle or control the message forwarding according to the hierarchical intervention control command. The bypass switch is used to open the bypass signal channel when it is triggered to close, so that the direct connection between the output of the control handle and the lower programmable logic controller is restored.

[0008] Furthermore, the edge computing control unit calculates the operational hazard coefficient based on the lifting torque utilization rate, boom slewing angular acceleration, and hook lifting speed obtained by the mechanical acquisition module; and dynamically updates the permissible time window for visual anomaly determination based on the operational hazard coefficient; when the duration of the anomaly corresponding to the operator status feature extracted from the visual acquisition module exceeds the current permissible time window for visual anomaly determination, a visual anomaly confirmation signal is generated.

[0009] Furthermore, the edge computing control unit calculates the visual data reliability corresponding to the current sampling period based on image features; when the visual data reliability is lower than a preset reliability threshold, the edge computing control unit calculates the rate of change of the control command or the second-order rate of change of the control command based on the original output signal of the control handle, and obtains the torque response delay time of the actuator. When the rate of change of the manipulation command or the second-order rate of change of the manipulation command exceeds the corresponding threshold, and the torque response delay of the actuator exceeds the preset response time range, a non-visual compensation abnormal signal is generated by combining the visual abnormality flags cached within the preset time window before the reliability of the visual data decreases.

[0010] Furthermore, the edge computing control unit is used to determine the personnel risk value based on the recognition results of the visual acquisition module, calculate the lifting torque utilization rate based on the data of the mechanical acquisition module, and calculate the environmental risk value based on the wind speed exceedance ratio and rainfall impact value obtained by the environmental acquisition module. The edge computing control unit dynamically adjusts the personnel risk weight, equipment risk weight, and environmental risk weight based on the reliability of visual data. Under the constraint that the sum of each weight is 1, it performs a weighted calculation on the personnel risk value, lifting torque utilization rate, and environmental risk value to obtain a comprehensive risk value. The edge computing control unit compares the comprehensive risk value with preset first risk threshold, second risk threshold, and third risk threshold to classify the system safety status into normal level, early warning level, restriction control level, or emergency interlock level.

[0011] In a preferred embodiment of the present invention, when the system safety state is at the restricted control level, the edge computing control unit generates a smooth deceleration control command based on feedforward compensation calculation. The feedforward compensation is determined at least based on the estimated load swing angle, swing angle change rate and system delay time under the current operating state. After receiving the smooth deceleration control command, the intervention execution module converts it into a signal that matches the interface of the lower-level programmable logic controller, and performs a corresponding deceleration reconstruction output on the original output signal of the control handle.

[0012] In another preferred embodiment of the present invention, when the system safety status is at the emergency interlock level, and the edge computing control unit determines that a certain direction of motion will cause the lifting torque utilization rate to increase and exceed the preset interlock threshold, the intervention execution module implements directional control interlock for the control channel corresponding to the direction of motion, outputs a zero-value control signal or disconnects the control signal transmission loop for that direction; for directions of motion that can reduce the lifting torque utilization rate or reduce the risk, the intervention execution module keeps the corresponding control channel available.

[0013] Furthermore, the intervention execution module includes an input sampling circuit, an isolation circuit, an output reconstruction circuit, and a relay interlock circuit; When the original output signal of the control handle is a bus control message, the intervention execution module, as a communication gateway, accesses the communication link and parses, rewrites, and forwards the bus control message according to the instructions issued by the edge computing control unit, the pre-stored message identifier, the data field position, and the control quantity conversion relationship.

[0014] Furthermore, the intervention execution module continuously monitors the status of the bypass switch; when the bypass switch is detected to be closed, the intervention execution module switches its output to a high-impedance state, or disconnects the connection between the output reconstruction circuit and the lower-level programmable logic controller through the relay interlock circuit, so as to avoid electrical conflict with the original output signal of the control handle directly transmitted through the bypass signal channel.

[0015] Furthermore, the memory of the edge computing control unit stores camera calibration parameters and driver's cab coordinate system transformation parameters; The edge computing control unit converts the coordinates of human body key points in the image coordinate system or camera coordinate system into the coordinates of human body key points in the driver's cab coordinate system according to the camera calibration parameters and the driver's cab coordinate system transformation parameters, so as to obtain the spatial positional relationship of the operator's face, torso and hands relative to the control handle.

[0016] Furthermore, the system also includes a cloud server; After the system triggers the emergency interlock level, the edge computing control unit initiates a control state self-lock and remains in a waiting-for-authorization-unlock state during the control state self-lock period. Only when the comprehensive risk value is lower than the preset release threshold and continues to reach the preset release time, and the edge computing control unit receives the verified authorization unlock instruction issued by the cloud server, will the edge computing control unit issue a reset instruction to the intervention execution module to release the control state self-lock.

[0017] A second aspect of this invention provides a method for monitoring and protecting the operational behavior of a tower crane based on computer vision, the method comprising the following steps: The edge computing control unit receives synchronously collected visual data, mechanical operation data, and environmental parameters, and acquires the raw output signal of the control handle. The permissible time window for visual anomaly detection is dynamically updated based on lifting torque utilization rate, boom slewing angular acceleration, and hook lifting speed. When the reliability of visual data is lower than the preset reliability threshold, a non-visual compensation abnormal signal is generated by combining the rate of change of manipulation commands, the torque response delay time of the actuator, and the visual abnormality flags cached within the preset time window. The comprehensive risk value is calculated based on the weighted result of personnel risk value, lifting torque utilization rate and environmental risk value, and the system safety status level is classified. Based on the safety status level, the control intervention execution module performs reconstruction output or direction control interlock on the original output signal of the control handle; When the bypass switch connected in parallel with the intervention execution module is closed, the bypass signal channel is activated, restoring the direct connection between the control handle and the lower-level programmable logic controller.

[0018] This invention provides a computer vision-based monitoring and protection system for the operation of tower cranes. It offers the following advantages: 1. This invention addresses the issue of susceptibility to failure in strong light or obstructed environments caused by purely visual monitoring by introducing a non-visual logic compensation mechanism. When the system determines that the reliability of visual data is below a threshold, the edge computing control unit extracts the rate of change of manipulation commands and the torque response delay time of the actuator, and combines this with the abnormal flags cached before the visual failure to generate a compensation abnormal signal. This mechanism eliminates the absolute dependence on a single visual sensor and removes monitoring blind spots and the risk of missed reports under complex environmental interference.

[0019] 2. This invention addresses the problem that direct power cut-off using traditional hard interlocks easily leads to severe swaying of the hoisted load. Instead, it employs an intervention strategy combining flexible reconfiguration and unidirectional interlocking. Under the limited control level, the intervention execution module outputs a smooth deceleration signal based on feedforward compensation to replace the emergency stop command, preventing structural instability. Under the emergency interlock level, the system only cuts off signals in the dangerous direction that increases the utilization rate of the lifting torque, while retaining control loops in the direction that reduces risk. This asymmetric control interception method ensures that the crane still has the operational conditions to safely unload heavy objects even under extreme working conditions.

[0020] 3. To address the risk of uncontrolled lower-level devices due to sudden failures in the monitoring system, this invention incorporates a parallel bypass and anti-electrical conflict mechanism at the hardware level. In the event of an abnormal lockout or communication failure, the operator can quickly restore the direct electrical connection between the control handle and the lower-level controller by closing the bypass switch. Simultaneously, the intervention execution module physically isolates the original signal from the intervention signal by switching to a high-impedance state at the output or disconnecting a relay, preventing command conflicts and short circuits caused by simultaneous input of dual signals to the lower-level controller, thus ensuring hardware safety during manual emergency takeover. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the architecture of the tower crane operation behavior proactive safety intervention system based on deep fusion of multi-source data according to the present invention; Figure 2 The flowchart is a process for the proactive safety intervention method for tower crane operation behavior based on deep fusion of multi-source data according to the present invention. Figure 3 This is a time-series diagram of the multimodal heterogeneous data synchronous sensing and feature extraction of the present invention; Figure 4 This is a block diagram of the dynamic threshold modulation and non-visual feature verification logic of the present invention; Figure 5 This is a flowchart of the comprehensive risk index assessment and level mapping process of the present invention; Figure 6 This is a schematic diagram of the feedforward compensation intervention and logic interlock principle of the present invention; Figure 7 This is a flowchart illustrating the permission decoupling and unauthorized access tracing management process of the present invention; Figure 8This is a schematic diagram of the logic for calculating environmental and visual parameters and determining visual anomalies in this invention. Figure 9 The graphs are comparison curves of the feedforward compensation intervention effect of the present invention, wherein (a) is a comparison curve of the control signal output command, and (b) is a comparison curve of the load swing angle response. Figure 10 This is a bar chart comparing the performance of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] See attached document Figure 1 This invention provides a computer vision-based tower crane operation behavior monitoring and protection system. In this embodiment, the system is used to synchronously collect the tower crane operator's operating status, the tower crane's operating status, and the external environment status, and to perform graded intervention on the control handle output signals based on the comprehensive risk assessment results. The system includes an edge computing control unit, and a vision acquisition module, a mechanical acquisition module, an environmental acquisition module, an intervention execution module, and a cloud server, all of which are communicatively connected to the edge computing control unit; the system also includes a bypass switch connected in parallel with the intervention execution module.

[0024] The edge computing control unit is located in the tower crane operator's cab and includes at least a processor, memory, communication interface, and control output interface. The memory stores camera calibration parameters, operator's cab coordinate system transformation parameters, control handle interface parameters, control signal mapping parameters of the lower-level programmable logic controller, and risk assessment threshold parameters. The edge computing control unit receives data collected by field sensors from various acquisition modules, performs timestamp alignment, status identification, and risk assessment on the data, and outputs corresponding control commands to the intervention execution module.

[0025] The visual acquisition module is located inside the driver's cab and is used to acquire images of the operator's face, head posture, and upper limb posture. The visual acquisition module includes at least one of a calibrated binocular camera or a depth camera; it may also include a calibrated monocular camera and combine it with a monocular depth estimation module, a continuous frame human posture estimation module, or pre-calibrated geometric constraint parameters of the driver's cab to form a visual acquisition component.

[0026] The camera calibration parameters include camera intrinsic parameters and camera extrinsic parameters; the driver's cab coordinate system transformation parameters represent the transformation relationship between the camera coordinate system and the driver's cab coordinate system. The edge computing control unit converts the coordinates of human key points in the image coordinate system or camera coordinate system to the coordinates of human key points in the driver's cab coordinate system based on the camera calibration parameters and the driver's cab coordinate system transformation parameters, thereby obtaining the spatial positional relationship of the operator's face, torso, and hands relative to the control handle.

[0027] The mechanical acquisition module includes a bus interface for connecting to the tower crane controller's local area network (LAN) bus, and a handle signal acquisition interface for acquiring the output signals of the control handle. The mechanical acquisition module is used to acquire tower crane operating status data and the raw output signals of the control handle. The tower crane operating status data includes at least one of hoisting load, luffing position, hoisting height, and slewing angle; the raw output signals of the control handle include analog voltage signals, pulse width modulation signals, or bus control messages.

[0028] An environmental data acquisition module is installed outside the tower crane to acquire environmental parameters. These environmental parameters include at least wind speed and may also include rainfall.

[0029] An intervention execution module is connected in series between the output of the control handle and the lower-level programmable logic controller (PLC). It is used to acquire, isolate, reconstruct, or forward control the original output signal of the control handle according to control commands issued by the edge computing control unit. The intervention execution module includes an input sampling circuit, an isolation circuit, an output reconstruction circuit, and a relay interlock circuit.

[0030] When the original output signal of the control handle is an analog voltage signal or a pulse width modulation signal, the input sampling circuit acquires the original output signal of the control handle, the isolation circuit electrically isolates the acquired signal from the output control signal, and the output reconstruction circuit generates the corresponding analog voltage signal or pulse width modulation signal according to the smooth deceleration control command issued by the edge computing control unit, and outputs it to the lower-level programmable logic controller.

[0031] When the original output signal of the control handle is a bus control message, the intervention execution module acts as a communication gateway, connecting to the communication link between the control handle and the lower-level programmable logic controller (PLC). It receives, parses, rewrites, and forwards the bus control messages sent by the control handle. When the edge computing control unit issues a smooth deceleration control command, the intervention execution module generates a bus control message that matches the interface of the lower-level PLC based on pre-stored message identifiers, data field positions, and control quantity conversion relationships.

[0032] A bypass switch is connected in parallel between the input and output terminals of the intervention execution module. When triggered to close, it opens the bypass signal channel, restoring the direct electrical connection or original communication connection between the control handle output and the lower-level programmable logic controller (PLC). Upon detecting the bypass switch closure, the intervention execution module switches its output to a high-impedance state or disconnects the output reconstruction circuit from the PLC via a relay isolation unit to prevent electrical interference between the control signal generated by the output reconstruction circuit and the original output signal of the control handle.

[0033] The cloud server connects to the edge computing control unit via a wireless communication network. It receives and stores event logs, sensor data, and intervention records uploaded by the edge computing control unit, and issues authorization unlock commands to the edge computing control unit. These authorization unlock commands provide remote authorization for unlocking the edge computing control unit after the system enters a locked state.

[0034] See attached document Figure 2 This invention provides a computer vision-based method for monitoring and protecting the operation behavior of tower cranes. In this embodiment, the method can be executed by the aforementioned tower crane operation behavior safety monitoring and graded intervention system based on multi-source data fusion. For ease of explanation, the following example illustrates the processing procedure of this method by using an edge computing control unit receiving visual data, tower crane operating status data, raw output signals from the control handle, and environmental parameters within a sampling period. The method includes the following steps.

[0035] S10, the edge computing control unit receives image data input from the vision acquisition module based on a unified system timestamp, and extracts operator state features from the image data. The operator state features include at least one of eyelid closure rate, gaze deflection angle, head posture, torso posture, and hand key point positions.

[0036] The edge computing control unit also calculates the reliability of visual data corresponding to the current sampling period based on image contrast, facial key point occlusion, or human key point detection completeness.

[0037] Under the same system timestamp reference, the edge computing control unit synchronously acquires the lifting load, luffing position, lifting height, slewing angle, and raw output signal of the control handle input from the mechanical acquisition module, and acquires the wind speed and rainfall parameters input from the environmental acquisition module. The raw output signal of the control handle includes analog voltage signals, pulse width modulation signals, or bus control messages.

[0038] S20, the edge computing control unit calculates the operational hazard factor based on the lifting torque utilization rate, boom slewing angular acceleration and hook lifting speed.

[0039] Among them, the lifting torque utilization rate is calculated based on the lifting load, luffing position, and rated lifting torque parameters at the corresponding amplitude; the boom slewing angular acceleration is calculated based on continuously sampled slewing angle data; and the hook lifting speed is calculated based on continuously sampled lifting height data.

[0040] The edge computing control unit updates the permissible time window for visual anomaly detection based on the operational hazard coefficient. When the duration of a visual anomaly corresponding to an operator's status characteristic exceeds the current permissible time window for visual anomaly detection, the edge computing control unit generates a visual anomaly confirmation signal.

[0041] When the reliability of visual data is lower than a preset reliability threshold, the edge computing control unit enables non-visual compensation judgment. The non-visual compensation judgment includes: calculating the rate of change of the control command or the second-order rate of change of the control command based on the original output signal of the control handle, and obtaining the torque response delay time of the actuator; when the rate of change of the control command or the second-order rate of change of the control command exceeds the corresponding threshold, and the torque response delay time of the actuator exceeds the preset response time range, a non-visual compensation abnormal signal is generated by combining the visual abnormality flags cached within the preset time window before the decrease in visual data reliability.

[0042] S30, the edge computing control unit determines the personnel risk value based on the visual anomaly confirmation signal or the non-visual compensation anomaly signal, and simultaneously obtains the current lifting torque utilization rate and environmental risk value.

[0043] The environmental risk value is calculated based on the wind speed exceedance ratio and the rainfall impact value. The wind speed exceedance ratio indicates the degree to which the actual wind speed exceeds the preset safe operating wind speed threshold; the rainfall impact value indicates the degree of impact of the current rainfall relative to the preset rainfall threshold or preset rainfall level.

[0044] The edge computing control unit adjusts the personnel risk weight, equipment risk weight, and environmental risk weight based on the reliability of visual data. Under the constraint that the sum of each weight is 1, it performs a weighted calculation on the personnel risk value, lifting torque utilization rate, and environmental risk value to obtain a comprehensive risk value.

[0045] Subsequently, the edge computing control unit compares the comprehensive risk value with the preset first risk threshold, second risk threshold, and third risk threshold, and classifies the system security status into normal level, early warning level, restriction control level, or emergency interlock level.

[0046] S40, the intervention execution module performs graded intervention on the original output signal of the control handle according to the safety status level.

[0047] When the system is in a restricted control level, the edge computing control unit generates a smooth deceleration control command based on feedforward compensation. The feedforward compensation is determined at least based on the estimated load swing angle, swing angle change rate, and system delay time under the current operating state. After receiving the smooth deceleration control command, the intervention execution module converts it into an analog quantity, pulse width modulation signal, or bus control message that matches the interface of the lower-level programmable logic controller, and reconstructs or rewrites the original output signal of the control handle before outputting or forwarding the message.

[0048] When the system is in an emergency interlock level, and the edge computing control unit determines that a certain direction of movement will cause the lifting torque utilization rate to increase and exceed the preset interlock threshold, the intervention execution module implements directional control interlock for the control channel corresponding to that direction of movement, outputs a zero-value control signal, or disconnects the control signal transmission loop for that direction. For directions of movement that can reduce the lifting torque utilization rate or reduce risk, the intervention execution module keeps the corresponding control channel available.

[0049] The S50 edge computing control unit initiates self-locking of the control state after the system triggers the emergency interlock level, and packages and uploads visual data, tower crane operating status data, environmental parameters, comprehensive risk value, safety status level and intervention action records within a preset time period before and after the triggering time to the cloud server.

[0050] During the control state self-locking period, the edge computing control unit remains in a waiting-for-authorization-unlocking state. Only when the overall risk value is lower than the preset release threshold and continues to reach the preset release time, and the edge computing control unit receives a verified authorization-unlocking command, will the edge computing control unit issue a reset command to the intervention execution module to release the control state self-locking.

[0051] During the self-locking period of the control state, if the edge computing control unit detects that the operator has triggered the bypass switch, it records the bypass trigger action, generates an emergency bypass operation identifier, and uploads the emergency bypass operation identifier, the original output signal of the control handle, the safety status level, the comprehensive risk value, and the bypass switch status for the corresponding time period to the cloud server.

[0052] See attached document Figure 3 In this embodiment, the vision acquisition module acquires video stream data of the operator in the tower crane operator's cab and sends the video stream data to the edge computing control unit. The edge computing control unit extracts operator state features based on the video stream data. The operator state features include at least one of eyelid closure rate, gaze deflection angle, head posture, torso posture, and key hand position.

[0053] To simultaneously obtain the aforementioned multiple operator status features, a multi-task convolutional neural network model is deployed in the edge computing control unit. In specific implementation, the multi-task convolutional neural network model can use a ResNet residual network as a shared backbone network to extract basic image features; multiple independent output branches are set at the output end of the shared backbone network to output eyelid closure rate, gaze deflection angle, and coordinates of key human body points, respectively.

[0054] The input data of the multi-task convolutional neural network model includes a sequence of continuous RGB image frames, a sequence of binocular image frames, or a sequence of RGB-D image frames after size normalization. The output of the multi-task convolutional neural network model includes: eyelid closure rate parameters output by the first regression output branch; yaw and pitch angles of the gaze direction relative to a preset forward reference direction output by the second pose estimation output branch; and three-dimensional coordinates of the operator's torso key points and hand key points output by the third key point detection output branch.

[0055] The edge computing control unit pre-stores the camera intrinsic and extrinsic parameters of the vision acquisition module, as well as the coordinate transformation matrix between the camera coordinate system and the driver's cab coordinate system. Based on the camera intrinsic and extrinsic parameters and the coordinate transformation matrix, the edge computing control unit converts the three-dimensional coordinates of the key points of both hands in the camera coordinate system into the spatial position of the hands in the driver's cab coordinate system.

[0056] When the vision acquisition module uses a monocular camera, the edge computing control unit combines the monocular depth estimation results, continuous frame human pose estimation results, or pre-calibrated driver's cab geometric constraint parameters to determine the spatial positional relationship of the key points of both hands relative to the three-dimensional enclosed area of ​​the control handle.

[0057] In the case of a monocular depth estimation module, the module takes continuous monocular image frames and camera calibration parameters as input and outputs the depth values ​​of the hand keypoints relative to the camera coordinate system. The edge computing control unit calculates the spatial position of the hands in the driver's cab coordinate system based on these depth values ​​and the coordinate transformation matrix between the camera coordinate system and the driver's cab coordinate system.

[0058] The three-dimensional enclosing area of ​​the control handle is a pre-calibrated spatial region. The edge computing control unit compares the positions of the key points of both hands in the driver's cab coordinate system with the three-dimensional enclosing area of ​​the control handle, and determines whether the operator is in a state of single-handed disengagement from the control handle based on the comparison result.

[0059] For the aforementioned multi-task convolutional neural network model, images of the operator in the driver's cab under different lighting conditions, occlusion conditions, and operating postures can be collected offline to construct a training sample set. The training sample set is labeled with at least one ground truth label from eyelid state, head angle, gaze direction, and coordinates of key human body points.

[0060] During the model training phase, a joint loss function is used to update the network parameters. The joint loss function is obtained by weighted summation of the mean squared error loss of eyelid closure rate, the L1 smoothing loss of angle regression, and the cross-entropy loss of key point heatmap. The weight coefficients of each loss term are obtained through offline validation set calibration.

[0061] The edge computing control unit simultaneously calculates visual data reliability while processing image frames. This visual data reliability indicates the usability of the current image data for operator status recognition. The edge computing control unit calculates the image contrast parameters of the current image frame using the image grayscale histogram and obtains the number of unoccluded facial landmarks or the facial landmark occlusion rate using the facial landmark detection module.

[0062] The edge computing control unit calculates the reliability of visual data based on image contrast parameters and facial key point occlusion. The reliability of the visual data is positively correlated with the image contrast parameters and negatively correlated with the facial key point occlusion rate.

[0063] In one implementation, visual feature confidence satisfy: ) in The normalized image contrast parameter. For facial key point occlusion rate, and For the preset weighting coefficients, and , The value range is from 0 to 1. and It can also be calibrated based on the historical light interference and shading occurrences at the construction site.

[0064] The mechanical data acquisition module is used to acquire the tower crane's operating status data and the raw output signals of the control handle. Specifically, the mechanical data acquisition module connects to the tower crane's controller area network (SAR) bus via the SAR bus interface and reads bus messages at a preset sampling frequency, parsing the lifting load, trolley luffing position, hook lifting height, and boom slewing angle from the bus messages.

[0065] The mechanical acquisition module also acquires analog voltage signals, pulse width modulation signals, or bus control messages output by the control handle through the handle signal acquisition interface, and sends them to the edge computing control unit as the raw output signals of the control handle.

[0066] The environmental acquisition module is used to acquire environmental parameters outside the tower crane. Specifically, the environmental acquisition module collects the output signals of the wind speed sensor and the rain gauge, generates wind speed parameters and rainfall parameters, and sends the wind speed parameters and rainfall parameters to the edge computing control unit.

[0067] The edge computing control unit performs time alignment on visual data, tower crane operating status data, raw output signals from the control handle, and environmental parameters based on a unified system timestamp. Since the image sampling frame rate of the visual acquisition module differs from the bus sampling frequency of the mechanical acquisition module, the edge computing control unit uses the sampling period of the mechanical acquisition module as a benchmark and employs a zero-order hold-and-resample method to align low-frequency updated feature items in the visual data, generating a multi-source data fusion vector corresponding to the same sampling time.

[0068] When the edge computing control unit receives image feature data, bus message data, or environmental parameters, it adds an arrival timestamp to the corresponding data. For data with communication delays, the edge computing control unit performs time correction based on the arrival timestamp and preset communication delay compensation parameters to reduce the sampling time deviation between different acquisition modules.

[0069] During the resampling process, if the edge computing control unit does not receive the data packet transmitted by the vision acquisition module within a preset timeout period, the reliability of the vision data corresponding to the current sampling time will be adjusted. The value is assigned to 0, and interpolation processing of the visual features at that sampling time is stopped. After time alignment is completed, the edge computing control unit encapsulates the operator status characteristics, visual data reliability, tower crane operating status data, raw output signals of the control handle, and environmental parameters into a multi-source data fusion vector for use in subsequent risk assessment and tiered intervention steps.

[0070] See attached document Figure 4 In this embodiment, the edge computing control unit calculates the operational hazard coefficient D at the current sampling time based on the tower crane's operating status data synchronously uploaded by the mechanical acquisition module. The operational hazard coefficient D represents the degree of operational risk of the tower crane under the current load, slewing, and hoisting conditions, and serves as the basis for subsequent adjustments to the visual anomaly detection time.

[0071] The edge computing control unit acquires the lifting load, trolley luffing position, hook lifting height, and boom slewing angle at the current sampling moment.

[0072] Among them, lifting torque utilization rate It is calculated based on the current lifting load, the trolley luffing position, and the rated lifting torque parameters at the corresponding luffing position, and is used to represent the proportion of the current lifting torque relative to the rated lifting torque.

[0073] The boom slewing angular acceleration is obtained by differential and filtering of continuously sampled boom slewing angle data; the hook lifting speed is obtained by differential and filtering of continuously sampled hook lifting height data. To ensure that parameters of different dimensions can participate in the same weighted calculation, the edge computing control unit normalizes the absolute values ​​of the boom slewing angular acceleration and the hook lifting speed.

[0074] In one implementation, the dynamic hazard factor satisfy: ; in To improve the utilization rate of the lifting torque of the equipment, It is the angular acceleration of rotation. For lifting speed, , and Let be the operating condition weight constant, and .

[0075] Specifically, It can be determined based on the ratio of the absolute value of the boom slewing angular acceleration to the preset reference value of the slewing angular acceleration. The lifting speed can be determined based on the ratio of the absolute value of the hook lifting speed to the preset lifting speed benchmark value. The aforementioned benchmark value and working condition weight coefficient can be calibrated during the equipment installation and commissioning phase based on the rated load curve, operation test data, and on-site safety regulations of the same model of tower crane, and stored in the parameter configuration area of ​​the edge computing control unit.

[0076] The edge computing control unit adjusts the permissible time window for visual anomaly detection based on the operational hazard factor D. The permissible time window for visual anomaly detection. This indicates the maximum time that the system allows an operator's status characteristic to persist in an abnormal state.

[0077] In one implementation, a visual judgment tolerance time window satisfy: ; in Based on the determination of the delay threshold, For the minimum tolerance time window, This is the sensitivity adjustment coefficient. This represents the dynamic hazard coefficient. The higher the dynamic hazard coefficient, the shorter the visual judgment tolerance time window for the calculated output. The edge computing unit integrates and accumulates the duration of the extracted visual anomaly features. When the duration of the anomaly exceeds the dynamically generated visual judgment tolerance time window at the current moment, the edge computing unit outputs a confirmation status anomaly signal.

[0078] The edge computing control unit performs anomaly detection on the operator status characteristics output by the vision acquisition module. The anomaly detection includes at least one of the following: whether the eyelid closure rate exceeds the eyelid closure threshold, whether the gaze deflection angle exceeds the gaze deflection threshold, and whether the key points of the hand leave the three-dimensional enclosure area of ​​the control handle.

[0079] When any visual anomaly persists within a continuous sampling period, the edge computing control unit accumulates the duration of the visual anomaly; when the accumulated duration exceeds the current allowable time window for visual anomaly determination... At that time, the edge computing control unit generates a visual anomaly confirmation signal. If the visual anomaly occurs within the current permissible time window for visual anomaly determination... If the previous time disappeared, the edge computing control unit will clear or decay the corresponding cumulative time.

[0080] Reliability of visual data When the reliability falls below a preset reliability threshold, the edge computing control unit enables non-visual compensation judgment. This non-visual compensation judgment is used to assist in determining abnormal operating conditions by utilizing the original output signal of the control handle and the feedback signal of the actuator when image data is affected by strong light, occlusion, or data loss.

[0081] Specifically, the edge computing control unit calculates the rate of change of the control command based on the raw output signal of the control handle acquired by the mechanical acquisition module, or further calculates the second-order rate of change of the control command. For analog voltage signals or pulse width modulation signals, the edge computing control unit can calculate based on the signal amplitude changes at adjacent sampling times; for bus control messages, the edge computing control unit can calculate based on the changes in the control quantity field in the message.

[0082] The edge computing control unit also acquires the actuator torque response delay time. The actuator torque response delay time is the time difference between the moment when the original output signal of the control handle undergoes a valid change and the moment when the actuator output torque or load state responds and changes.

[0083] The output torque or load state of the actuator changes in response, which can be determined based on at least one of the following: inverter torque feedback, motor current feedback, lifting torque sensor output, and feedback state of the lower-level programmable logic controller; or it can be estimated based on changes in lifting load, luffing position, hook lifting speed, or boom slewing angular velocity.

[0084] The parameter configuration area of ​​the edge computing control unit stores the threshold for the rate of change of manipulation commands, the threshold for the second-order rate of change of manipulation commands, and the preset response time range.

[0085] When the rate of change of the manipulation command or the second-order rate of change of the manipulation command exceeds the corresponding threshold, and the torque response delay of the actuator exceeds the upper limit of the preset response time range, the edge computing control unit reads the visual data reliability. Visual anomaly markers cached within a preset time window before descent.

[0086] The visual anomaly flag is a binary judgment result generated by the edge computing control unit based on abnormal eyelid closure rate, abnormal gaze deflection, or abnormal single-hand detachment from the control handle. The edge computing control unit performs a logical OR operation on the visual anomaly flag and the control command anomaly judgment result; when the operation result is true, the edge computing control unit generates a non-visual compensation anomaly signal.

[0087] Therefore, even when the reliability of visual data decreases, the edge computing control unit can still continuously judge the abnormal state of the operator based on the original output signal of the control handle, the feedback signal of the actuator, and the visual abnormality flags buffered before descent.

[0088] See attached document Figure 5 In this embodiment, the edge computing control unit calculates the comprehensive risk value R at the current sampling time based on the aforementioned visual anomaly confirmation signal, non-visual compensation anomaly signal, lifting torque utilization rate, and environmental parameters, and determines the system safety status level based on the comprehensive risk value R.

[0089] The edge computing control unit first determines the personnel risk value P. Specifically, the edge computing control unit reads the visual anomaly confirmation signal or the non-visual compensation anomaly signal, and determines the personnel risk value P corresponding to the current anomaly type according to the risk value lookup table pre-stored in the parameter configuration area.

[0090] The risk value lookup table records the basic risk values ​​corresponding to different abnormality types. These abnormality types include at least one of the following: abnormal eyelid closure rate, abnormal gaze deflection, abnormal single-handed detachment from the control handle, and abnormal changes in control commands. If more than two abnormality types exist at the same sampling time, the edge computing control unit can take the maximum value among the risk values ​​corresponding to each abnormality type as the personnel risk value P, or it can synthesize multiple risk values ​​according to a preset superposition rule.

[0091] The edge computing control unit synchronously acquires the lifting torque utilization rate at the current sampling moment. The lifting torque utilization rate It is used to indicate the proportion of the current lifting torque relative to the rated lifting torque. Its calculation is based on the lifting load, the luffing position of the trolley, and the rated lifting torque parameters at the corresponding luffing position.

[0092] The edge computing control unit also calculates environmental risk values ​​based on wind speed and rainfall parameters uploaded by the environmental acquisition module. .

[0093] In one implementation, the environmental wind speed exceedance index satisfy: ; in, This refers to the actual wind speed. Critical wind speed for safe operation.

[0094] The edge computing control unit calculates the rainfall impact value based on rainfall parameters. When the rainfall does not exceed the preset rainfall threshold, Set to 0; when the rainfall exceeds the preset rainfall threshold, It can be calculated based on the ratio of the difference between the rainfall and the preset rainfall threshold, or it can be determined based on the classification table corresponding to light rain, moderate rain, heavy rain, and rainstorm.

[0095] The edge computing control unit will determine the wind speed over-limit ratio. Impact value of rainfall The environmental risk value is obtained by performing a weighted calculation. In one implementation, the environmental risk value... satisfy: ; in, and This is the environmental weighting coefficient, and .

[0096] Edge computing control unit based on visual data reliability Adjust personnel risk weights, equipment risk weights, and environmental risk weights. The reliability of the visual data. The higher the value, the greater the risk weight for personnel; the reliability of the visual data. The lower the value, the smaller the personnel risk weight, and the correspondingly larger the equipment risk weight and environmental risk weight.

[0097] In one implementation, personnel risk weighting satisfy: ; in, As the lower limit of personnel risk weight, This is the upper limit of personnel risk weight, and .

[0098] Equipment risk weight and environmental risk weights The remaining weights are allocated according to a preset ratio, and the following conditions are met: ; After the weights are determined, the edge computing control unit calculates the personnel risk value P, the lifting torque utilization rate Um, and the environmental risk value. Calculate the overall risk value R.

[0099] In one implementation, the overall risk value R satisfies: ; Where P is the personnel risk value, To improve the utilization rate of lifting torque, Environmental risk value, For personnel risk weights, For equipment risk weights, Environmental risk weights.

[0100] The edge computing control unit compares the comprehensive risk value R with preset first, second, and third risk thresholds, and determines the system security status level accordingly. The first, second, and third risk thresholds increase progressively.

[0101] When the comprehensive risk value R is less than the first risk threshold, the system security status level is normal.

[0102] When the comprehensive risk value R is greater than or equal to the first risk threshold and less than the second risk threshold, the system security status level is the warning level.

[0103] When the comprehensive risk value R is greater than or equal to the second risk threshold and less than the third risk threshold, the system security status level is the restricted control level.

[0104] When the comprehensive risk value R is greater than or equal to the third risk threshold, the system security status level is the emergency interlock level.

[0105] To reduce frequent level switching caused by fluctuations in the overall risk value R around the threshold, the edge computing control unit sets a preset duration of judgment or a hysteresis threshold when switching security status levels. The edge computing control unit can only perform a security status level switch when the overall risk value R continuously meets the threshold conditions corresponding to the target security status level and reaches the preset duration of judgment, or meets the corresponding hysteresis threshold conditions.

[0106] The aforementioned first risk threshold, second risk threshold, third risk threshold, preset continuous judgment time, hysteresis threshold, risk value comparison table, and various weight coefficients can be calibrated during the equipment installation and commissioning phase based on the rated load curve, historical operating data, fault records, and on-site safety regulations of the same model of tower crane, and stored in the parameter configuration area of ​​the edge computing control unit.

[0107] See attached document Figure 6 In this embodiment, the edge computing control unit estimates the load swing angle based on the tower crane's operating status data and generates a smooth deceleration control command by combining the system delay time. This smooth deceleration control command is used to reconstruct the original output signal of the control handle when the system is in a limited control level, thereby reducing load swing and mechanical impact.

[0108] The edge computing control unit is equipped with a swing angle state estimation module. The swing angle state estimation module estimates the hoisting swing angle state at the current sampling moment based on the current hoisting rope length, trolley luffing position, luffing speed, luffing acceleration, boom slewing angular velocity, boom slewing angular acceleration, and the swing angle state value of the previous sampling period.

[0109] The swing angle state quantities include the load swing angle θ and the load swing angular velocity. When the equipment is started, the load is stationary, or the system is reset, the edge computing control unit will adjust the load swing angle. and the angular velocity of the hoisting swing Initialize to zero.

[0110] During tower crane operation, the edge computing control unit recursively updates the load swing angle state quantity based on continuously sampled luffing, slewing, and hoisting motion parameters. The edge computing control unit can also correct the load swing angle state quantity based on hook image recognition results, load inertial measurement unit data, or historical swing angle estimation deviations.

[0111] In one implementation, the edge computing control unit performs recursive calculations using a simplified pendulum dynamics estimation relation, which is:

[0112] In one implementation, the edge computing unit uses a simplified pendulum state equation as follows: ; Perform recursive calculations, where L is the current hoisting rope length. The damping coefficient is... This is the equivalent lateral excitation acceleration derived from the amplitude acceleration and angular acceleration. The edge computing unit calculates the acceleration based on the current moment... , And the inherent delay time τ, obtained by forward integration and This data was used to predict the swing angle and the rate of change of the swing angle. This calculation process provides advanced data support for subsequent feedforward interventions.

[0113] The intervention execution module reconstructs and issues unauthorized deceleration commands under the restricted level. When the system safety status level is determined to be restricted, the intervention execution module intercepts the original control signal issued by the control handle between the control handle output and the underlying programmable logic controller. To suppress residual load sway and avoid mechanical shock caused by hard braking, the edge computing unit calculates the phase lead compensation gain based on the predicted sway angle and sway angle change rate calculated above. The edge computing unit superimposes this phase lead compensation gain onto its internal preset deceleration reference curve to generate the reconstructed flexible deceleration command.

[0114] In one implementation, a flexible deceleration command satisfy: ; in, The original control commands for the control handle. To predict the swing angle, To predict the rate of change of the pendulum angle, and For compensation coefficient, This is a limiting function. The edge computing unit also... After slope limiting or S-curve smoothing, the flexible deceleration command is sent to the intervention execution module. This flexible deceleration command exhibits a smooth decay characteristic in the control time domain to generate a damping force opposite to the load swing direction. Subsequently, the intervention execution module bypasses the normal operating permissions, converts the flexible deceleration command into an analog quantity, pulse width modulation signal, or bus message that matches the execution interface of the underlying programmable logic controller (PLC), and overwrites the output to the PLC's execution interface, causing the crane to enter a deceleration convergence state.

[0115] When the original output signal of the control handle is an analog voltage signal or a pulse width modulation signal, the intervention execution module generates the corresponding analog voltage signal or pulse width modulation signal through the output reconstruction circuit.

[0116] When the original output signal of the control handle is a bus control message, the intervention execution module rewrites the original control message of the control handle and forwards it according to the pre-stored message identifier, control quantity field position and control quantity conversion relationship, or generates a new bus control message and sends it to the lower programmable logic controller.

[0117] When the system safety status level is at the emergency interlock level, the edge computing control unit parses the control commands for each motion channel. The motion channel includes at least one of the trolley luffing channel, hoisting channel, and slewing channel. Based on the current trolley luffing position, hoisting load, hook hoisting speed, boom slewing rate, and lifting torque utilization rate, the edge computing control unit determines whether the control commands for each motion channel will lead to an increase in lifting torque utilization rate, an increase in load swing amplitude, or a decrease in stability margin.

[0118] When a control command for a certain motion channel increases the lifting torque utilization rate, and the increased lifting torque utilization rate exceeds a preset interlock threshold, the edge computing control unit determines the motion direction corresponding to that motion channel as the direction of increased risk.

[0119] For directions that increase risk, the intervention execution module disconnects the control signal transmission loop in the corresponding direction via a relay interlock circuit, or outputs a zero-value control signal to the lower-level programmable logic controller. For directions of motion that can reduce the utilization rate of lifting torque or reduce the risk of load swaying, the intervention execution module keeps the corresponding control channel available.

[0120] For example, if the outward luffing of the trolley causes the lifting torque utilization rate to exceed the preset interlock threshold, the intervention execution module implements a directional control interlock in the outward luffing direction of the trolley while keeping the control channel available in the inward luffing direction. As another example, if accelerated lifting of the hook under high load conditions leads to increased lifting torque utilization or increased risk of load swaying, the intervention execution module implements a directional control interlock in the corresponding lifting direction.

[0121] Until the overall risk value R falls back to the preset release threshold, the intervention execution module maintains the directional control interlock state.

[0122] When the overall risk value R falls below the preset release threshold and remains below the preset release time, the edge computing control unit remains in a waiting-for-authorization-unlock state. The edge computing control unit only issues a reset command to the intervention execution module to release the directional control interlock state after receiving an authorized unlock command that has been verified by digital signature.

[0123] See attached document Figure 7 In this embodiment, when the system security status level enters the emergency interlock level and the intervention execution module implements directional control interlock, the edge computing control unit maintains the interlock state and initiates the intervention event recording and remote authorization unlocking process.

[0124] The edge computing control unit has a ring-shaped buffer inside. The ring-shaped buffer is used to continuously cache visual data, tower crane operating status data, environmental parameters, operating hazard coefficient D, comprehensive risk value R, safety status level, raw output signal of the control handle, and output status of the intervention execution module.

[0125] When the direction control interlock is triggered, the edge computing control unit extracts data from the ring buffer for a preset duration before and after the trigger moment, and encapsulates it into an intervention event snapshot data packet according to a preset data frame format. The intervention event snapshot data packet includes at least the device identification number, operator registration code, trigger timestamp, visual anomaly confirmation signal or non-visual compensation anomaly signal, operational hazard coefficient D, comprehensive risk value R, safety status level, interlock direction, original output signal of the control handle, and output status of the intervention execution module.

[0126] The edge computing control unit uploads the intervention event snapshot data packet to the cloud server via the wireless communication module. Before the direction control interlock is released, the intervention execution module keeps the control channel for the direction that increases risk in a disconnected state or a zero-value output state, and keeps the movement direction that can reduce the utilization rate of lifting torque or reduce the risk of hoisting sway available.

[0127] After receiving and parsing the intervention event snapshot data packet, the cloud server sends an intervention alarm message to the security administrator's terminal with unlocking authority. Based on the intervention event snapshot data packet and the on-site confirmation results, the security administrator generates an authorization unlocking command through the security administrator terminal.

[0128] The authorized unlock command includes a device identifier, event number, unlock validity period, administrator identity identifier, and unlock action identifier. The security administrator terminal digitally signs the authorized unlock command using a pre-assigned private key and sends the digitally signed authorized unlock command to the cloud server. The cloud server then distributes the authorized unlock command to the edge computing control unit.

[0129] The edge computing control unit pre-stores the public key or digital certificate corresponding to the private key. After receiving the authorization unlock command, the edge computing control unit verifies the authenticity and integrity of the digital signature; the digital signature can use one of the SM2, RSA, or ECDSA signature algorithms.

[0130] After successful signature verification, the edge computing control unit continues to determine whether the overall risk value R is lower than the preset release threshold and whether this state continues for the preset release time. If the signature verification is successful and the overall risk value R meets the release conditions, the edge computing control unit sends a reset command to the intervention execution module to release the directional control interlock state.

[0131] If the digital signature verification fails, or the comprehensive risk value R does not meet the release conditions, the edge computing control unit maintains the direction control interlock state and continues to upload the current risk status and interlock status to the cloud server.

[0132] In this embodiment, the system also includes a bypass switch to create an emergency bypass when it is necessary to temporarily restore the direct control link of the control handle. The bypass switch is connected in parallel between the input and output terminals of the intervention execution module.

[0133] When the original output signal of the control handle is an analog voltage signal or a pulse width modulation signal, after the bypass switch is closed, a direct electrical connection is formed between the output terminal of the control handle and the lower-level programmable logic controller; when the original output signal of the control handle is a bus control message, after the bypass switch is closed, an original communication link is formed between the control handle and the lower-level programmable logic controller, bypassing the intervention execution module.

[0134] When the operator closes the bypass switch, the intervention execution module stops reconstructing the output and switches the output to a high-impedance state, or disconnects the output reconstruction circuit from the lower-level programmable logic controller via a relay isolation unit. Closing the bypass switch only creates a temporary emergency bypass channel and does not change the emergency interlock level and control state self-locking records already recorded by the edge computing control unit.

[0135] The edge computing control unit acquires the voltage level of the bypass switch through a monitoring port. When the bypass switch is detected to be at a closed active level, the edge computing control unit determines that the system has entered the bypass operation time window.

[0136] At the start of the bypass operation time window, the edge computing control unit extracts the device identifier, current timestamp, operator registration code, and bypass trigger sequence number, and performs hash operations on the above data to generate an emergency bypass operation identifier. The operator registration code is generated from at least one of the following: operator login account, RFID card, employee ID input module, or facial recognition result.

[0137] The edge computing control unit writes the emergency bypass operation identifier to the local append-to-log storage area or to the WORM storage area for one-time write-multiple reads, in order to prevent or reduce the risk of the identifier being overwritten, deleted or tampered with.

[0138] During the bypass operation time window, the edge computing control unit increases the data sampling frequency of the original output signal of the control handle to a preset multiple of the original sampling frequency, and records the original output signal of the control handle, the bypass switch status, and the lifting torque utilization rate during this time window. The overall risk value R, safety status level, and output changes of each motion channel.

[0139] When the edge computing control unit detects that the bypass switch is open and the bypass switch level returns to its initial state, it determines that the bypass operation time window has ended.

[0140] After the bypass switch is turned off, if the overall risk value R is not lower than the preset release threshold and continues to reach the preset release time, or if the edge computing control unit does not receive an authorized unlocking command verified by digital signature, the intervention execution module restores the directional control interlock state before the bypass trigger.

[0141] The edge computing control unit will link the emergency bypass operation flag with the raw output signal of the control handle, bypass switch status, and lifting torque utilization rate recorded within the bypass operation time window. The comprehensive risk value R, safety status level, and output changes of each motion channel are associated and stored, and then packaged and uploaded to the cloud server for archiving.

[0142] The aforementioned preset duration, preset release threshold, preset release time, sampling frequency boost factor, risk threshold, weighting coefficient, compensation coefficient, system delay time, and hysteresis threshold can be calibrated during the equipment installation and commissioning phase based on the rated load curve of the same model of tower crane, controller response characteristics, historical operating data, and on-site safety regulations, and stored in the parameter configuration area of ​​the edge computing control unit.

[0143] Specific application examples: Scene setting: Assuming the current time is July 2026, a tower crane is performing heavy-duty lifting operations at a construction site. Because it is afternoon, the setting sun creates strong localized glare in the operator's cab, causing fatigue from prolonged work and resulting in the operator closing their eyes and dozing off. A light breeze is also present in the external environment.

[0144] Step S10: Data Synchronization and Reliability Calculation See attached document Figure 8 Environmental and visual parameters: The contrast parameter of the current image is calculated using the image histogram. Due to the strong light and the influence of sunglasses, the occlusion rate of key facial features was reduced. .

[0145] Visual data reliability : Set weighting coefficients , .

[0146] =0.4×0.8+0.6×(1−0.3)=0.32+0.42=0.74 (At this point, the reliability is acceptable, and full non-visual compensation has not been triggered).

[0147] Step S20: Calculation of operational hazard and dynamic time window Mechanical condition: Lifting torque utilization rate (Under heavy load); Normalized angular acceleration Normalized lift speed .

[0148] Operational risk factor Set operating condition weights , , .

[0149] ; Allowable time for visual anomaly determination : Set basic decision delay Minimum tolerance Sensitivity coefficient .

[0150] ; See attached document Figure 9 Logical analysis: In an unloaded, safe state, a 3-second drowsiness period is required for intervention; however, in the current heavy-load, high-risk state, a 1.52-second drowsiness will trigger an anomaly confirmation signal. Assuming the operator's eyes are closed for 1.6 seconds, the edge computing unit generates a "visual anomaly confirmation signal," corresponding to the personnel risk value. .

[0151] Step S30: Comprehensive Risk Assessment and Level Mapping Environmental risks Current wind speed Safe critical wind speed No rainfall ( ).

[0152] ; ; Dynamic weight allocation: Set upper and lower limits for personnel weights. , Personnel risk weights Residual weights Allocate to equipment and environment at an 8:2 ratio: , .

[0153] Overall Risk Value : 0.781; Security level classification: setting thresholds (Warning) (Restriction and control) (Emergency interlock). Current It is under restricted control level ( ).

[0154] Step S40: Feedforward compensation intervention under restricted control: Because it is at a restricted control level, the intervention execution module intercepts the original handle signal.

[0155] Assuming the operator handle's original output analog quantity (Requires the trolley to change amplitude rapidly).

[0156] The swing angle state estimation module predicts the current leading swing angle. rate of change of swing angle .

[0157] Let the compensation coefficient be set .

[0158] ; Intervention results: The system reconfigured the control signal, smoothly limiting the drive voltage from 8.0V to 5.3V, effectively preventing the load from swinging violently due to the operator suddenly pushing the handle after being startled awake.

[0159] To verify the effectiveness of the present invention, a comparative experimental platform was built in a standard test field.

[0160] Comparison objects: Traditional system: Single-source fatigue monitoring system based on fixed threshold (the control loop is directly cut off after the eye is closed and fixed for 3 seconds).

[0161] The system of this invention is an active safety intervention system based on deep fusion of multi-source data (supporting dynamic time windows and feedforward deceleration reconstruction).

[0162] Experimental Scenario 1: Fatigue-induced operational errors under heavy load: See attached document Figure 10 Setting: The load reaches 85% of the rated load, and the operator simulates dozing off with their eyes closed, accompanied by unconsciously pressing down on the handle.

[0163] result: Traditional system: The action is triggered only 3 seconds after the drowsiness. At this time, the handle has been pressed down and generated a huge acceleration, which directly cuts off the circuit and causes mechanical hard braking. The maximum load swing angle surges to 8.4°, and the mechanical shock is extremely high.

[0164] The system of this invention, due to its high D-value, triggers limiting control and generates a flexible deceleration command within 1.5 seconds of slumber. This not only intervenes early but also, by suppressing the swing angle reconstruction signal, controls the maximum load swing angle to within 3.1°, reducing the impact load by 63%.

[0165] Experimental Scenario 2: Aggressive Operations Under Backlighting and Occlusion: Setting: Dusk backlighting directly hitting the camera (visual reliability) When the value drops to 0.2, non-visual compensation is triggered. The operator pulls the handle sharply (the second-order rate of change of the control command exceeds the limit), but the crane has not yet responded significantly due to inertia (within the delay time).

[0166] result: Traditional system: Due to camera blindness, visual judgment fails, the system classifies it as normal, and no intervention occurs. False alarm rate: 100%.

[0167] The system of this invention combines the visual anomaly flags cached before degradation (a deviation in line of sight was captured 1 second before backlighting) with the current extremely high second-order change rate of the instructions. The non-visual compensation module quickly identifies it as an abnormal operation, the comprehensive risk value jumps, the system smoothly enters the emergency interlock level and cuts off the direction that increases the risk.

[0168] Comparison of Results Data (50 consecutive tests):

[0169] The macroeconomic improvement trend corresponding to the above statistical results can be found in the appendix. Figure 10 The performance comparison bar chart, by comparing the differences between the traditional system and the present invention in key indicators such as average response time, maximum swing angle, and missed intervention rate, intuitively reflects the significant comprehensive improvement of the present invention system in eliminating missed detections, reducing mechanical shocks, and reducing false alarm disturbances.

[0170] In summary, this invention overcomes the limitations of traditional single fatigue monitoring devices. By introducing a dynamic tolerance time window modulation mechanism, the system can flexibly adjust the intervention threshold according to the current mechanical load and operating status, effectively preventing "late intervention" in high-risk conditions and "frequent false alarms" in no-load conditions. By introducing non-visual compensation verification logic, the reliability of the system in harsh visual environments such as strong light and obstruction is greatly improved, avoiding safety hazards caused by monitoring blindness. Furthermore, the application of feedforward compensation smooth deceleration reconstruction technology completely solves the risk of huge mechanical impact and load swing caused by forced power failure in traditional interlocking systems. In addition, the combination of multi-level authorized unlocking and bypass operation traceability mechanism takes into account both the flexibility of emergency operation at the construction site and the traceability of post-event safety responsibility, forming a closed-loop proactive safety management system.

[0171] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A computer vision-based tower crane operation behavior monitoring and protection system, including: The system includes an edge computing control unit, and a vision acquisition module, a mechanical acquisition module, an environmental acquisition module, and an intervention execution module, which are respectively communicatively connected to the edge computing control unit; the system also includes a bypass switch connected in parallel with the intervention execution module. The edge computing control unit is located in the tower crane operator's cab. It is used to receive data synchronously collected by the vision acquisition module, the mechanical acquisition module and the environmental acquisition module, perform timestamp alignment, status identification and risk assessment on the data, and output hierarchical intervention control commands to the intervention execution module. The intervention execution module is connected in series between the output end of the control handle and the lower-level programmable logic controller, and is used to reconstruct the original output signal of the control handle or control the message forwarding according to the hierarchical intervention control command. The bypass switch is used to open the bypass signal channel when it is triggered to close, so that the direct connection between the output of the control handle and the lower programmable logic controller is restored.

2. The tower crane operation behavior monitoring and protection system based on computer vision according to claim 1, characterized in that, The edge computing control unit is equipped with dynamic risk assessment logic: The edge computing control unit calculates the operational hazard coefficient based on the lifting torque utilization rate, boom slewing angular acceleration, and hook lifting speed obtained by the mechanical acquisition module; and dynamically updates the permissible time window for visual anomaly determination based on the operational hazard coefficient; when the duration of the anomaly corresponding to the operator status feature extracted from the visual acquisition module exceeds the current permissible time window for visual anomaly determination, a visual anomaly confirmation signal is generated.

3. The tower crane operation behavior monitoring and protection system based on computer vision according to claim 1, characterized in that, The edge computing control unit is equipped with non-visual compensation judgment logic: The edge computing control unit calculates the visual data reliability corresponding to the current sampling period based on image features; when the visual data reliability is lower than a preset reliability threshold, the edge computing control unit calculates the rate of change of the control command or the second-order rate of change of the control command based on the original output signal of the control handle, and obtains the torque response delay time of the actuator. When the rate of change of the manipulation command or the second-order rate of change of the manipulation command exceeds the corresponding threshold, and the torque response delay of the actuator exceeds the preset response time range, a non-visual compensation abnormal signal is generated by combining the visual abnormality flags cached within the preset time window before the reliability of the visual data decreases.

4. The tower crane operation behavior monitoring and protection system based on computer vision according to claim 1, characterized in that, The edge computing control unit is used to determine the personnel risk value based on the recognition results of the vision acquisition module, calculate the lifting torque utilization rate based on the data of the mechanical acquisition module, and calculate the environmental risk value based on the wind speed exceedance ratio and rainfall impact value obtained by the environmental acquisition module. The edge computing control unit dynamically adjusts the personnel risk weight, equipment risk weight, and environmental risk weight based on the reliability of visual data. Under the constraint that the sum of each weight is 1, it performs a weighted calculation on the personnel risk value, lifting torque utilization rate, and environmental risk value to obtain a comprehensive risk value. The edge computing control unit compares the comprehensive risk value with preset first risk threshold, second risk threshold and third risk threshold, and classifies the system security status into normal level, early warning level, restriction control level or emergency interlock level.

5. The tower crane operation behavior monitoring and protection system based on computer vision according to claim 1, characterized in that, When the system safety status is at the limit control level, the edge computing control unit generates a smooth deceleration control command based on feedforward compensation calculation. The feedforward compensation is determined at least based on the estimated load swing angle, swing angle change rate and system delay time under the current operating state. After receiving the smooth deceleration control command, the intervention execution module converts it into a signal that matches the interface of the lower-level programmable logic controller, and performs a corresponding deceleration reconstruction output on the original output signal of the control handle.

6. The tower crane operation behavior monitoring and protection system based on computer vision according to claim 4, characterized in that, When the system safety status is at the emergency interlock level, and the edge computing control unit determines that a certain direction of movement will cause the lifting torque utilization rate to increase and exceed the preset interlock threshold, the intervention execution module implements directional control interlock for the control channel corresponding to the direction of movement, outputs a zero-value control signal or disconnects the control signal transmission loop for that direction; for directions of movement that can reduce the lifting torque utilization rate or reduce risk, the intervention execution module keeps the corresponding control channel available.

7. The tower crane operation behavior monitoring and protection system based on computer vision according to claim 1, characterized in that, The intervention execution module includes an input sampling circuit, an isolation circuit, an output reconstruction circuit, and a relay interlock circuit; When the original output signal of the control handle is a bus control message, the intervention execution module, as a communication gateway, accesses the communication link and parses, rewrites, and forwards the bus control message according to the instructions issued by the edge computing control unit, the pre-stored message identifier, the data field position, and the control quantity conversion relationship.

8. The tower crane operation behavior monitoring and protection system based on computer vision according to claim 7, characterized in that, The intervention execution module continuously monitors the status of the bypass switch; when the bypass switch is detected to be closed, the intervention execution module switches its output to a high-impedance state, or disconnects the connection between the output reconstruction circuit and the lower-level programmable logic controller through the relay interlock circuit, so as to avoid electrical conflict with the original output signal of the control handle directly transmitted through the bypass signal channel.

9. The tower crane operation behavior monitoring and protection system based on computer vision according to claim 1, characterized in that, The edge computing control unit stores camera calibration parameters and driver's cab coordinate system transformation parameters in its memory; The edge computing control unit converts the coordinates of human body key points in the image coordinate system or camera coordinate system into the coordinates of human body key points in the driver's cab coordinate system according to the camera calibration parameters and the driver's cab coordinate system transformation parameters, so as to obtain the spatial positional relationship of the operator's face, torso and hands relative to the control handle.

10. The tower crane operation behavior monitoring and protection system based on computer vision according to claim 4, characterized in that, The system also includes a cloud server; After the system triggers the emergency interlock level, the edge computing control unit initiates a control state self-lock and remains in a waiting-for-authorization-unlock state during the control state self-lock period. Only when the comprehensive risk value is lower than the preset release threshold and continues to reach the preset release time, and the edge computing control unit receives the verified authorization unlock instruction issued by the cloud server, will the edge computing control unit issue a reset instruction to the intervention execution module to release the control state self-lock.