Early warning method and early warning system for lower part of portal crane

Abstract

The invention relates to the technical field of image recognition, in particular to a method and system for early warning below a portal crane, and the method comprises the steps: obtaining site environment sensing data, constructing a space map, and recognizing the position and motion state of equipment in real time; predicting an equipment motion track and a load swing range, and generating a predicted dangerous area; image acquisition devices are installed at a plurality of fixed positions, field images are acquired, an image recognition model is used for detecting a target, and the spatial position and the posture state of the target are obtained; tracking a target trajectory based on a motion estimation model and predicting a future position of the target trajectory; dynamically switching the image acquisition devices according to the motion state of the equipment; and comparing the future position of the target with the predicted dangerous area in real time, when the future position and the predicted dangerous area coincide, carrying out potential interference prediction, calculating predicted interference time, and carrying out grading processing in combination with the attitude state of the target. According to the invention, equipment dynamic perception and personnel behavior prediction are realized, and the intelligent protection capability of the operation area is improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method and a warning system for warning below a gantry crane. Background Art

[0002] During the operation of large equipment, dynamic targets such as workers and material vehicles are often present in the work area, making accidents easily caused by the slightest carelessness. Existing early warning systems for operational safety often use fixed cameras to capture images of the work area and incorporate image recognition technology to detect people or obstacles, monitoring their positions in real time and providing feedback when targets are detected approaching the work area. Some solutions also incorporate deep learning-based personnel detection models to improve detection accuracy and speed. Some systems support projected displays or audio and visual prompts of detected target locations to assist operators in making timely avoidance decisions.

[0003] However, existing image recognition-based feedback methods have the following major shortcomings: First, traditional methods are typically based on two-dimensional image information from fixed cameras, lacking real-time perception and integration of the device's own motion state, making it difficult to dynamically adjust the feedback area, resulting in feedback lag or misjudgment. Second, existing methods are mostly static detection, lacking prediction of personnel movement trajectories, and unable to perceive potential dangers in advance. Third, in complex environments, due to occlusion and blind spots, the coverage capability of fixed image acquisition equipment is limited, affecting the overall perception integrity and pre-feedback accuracy of the system. Therefore, an improved technology that can integrate the device's motion state, dynamically predict dangerous areas, and achieve flexible perception and active warning based on multi-point image recognition is urgently needed to further improve the safety level of operations. Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for warning below a gantry crane, which combines the equipment's motion state, dynamically predicts dangerous areas, and achieves flexible perception and active warning based on multi-point image recognition, thereby improving the safety level of crane operations. Specifically, the method includes: constructing a spatial map by acquiring environmental perception data of the site and identifying the equipment's position and motion state in real time; predicting the equipment's motion trajectory and load swing range, and generating a predicted dangerous area; installing an image acquisition device at a key position of the equipment to capture the site image, and detecting the site image using an image recognition model; if a target is detected in the site image, obtaining the target's spatial position and posture state; using a motion estimation model to track the target's motion trajectory and predict the target's future position; comparing the target's future position in the site image with the predicted dangerous area in real time; when the future position coincides with the predicted dangerous area, predicting potential interference and calculating the expected interference time; and performing graded processing based on different expected interference times and the target's posture state.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for early warning below a gantry crane, comprising:

[0007] Build a spatial map by acquiring environmental perception data of the site and identify the location and movement status of the equipment in real time;

[0008] Predict equipment motion trajectory and load swing range based on the equipment's real-time operating parameters, location, and site static object information, and generate predicted danger zones.

[0009] Image acquisition devices are installed at the first, second, third, and fourth positions of the device to capture images of the site, and the site images are detected using an image recognition model; if a target is detected in the site image, the spatial position and posture state of the target are obtained; a motion estimation model is used to track the motion trajectory of the target and predict the future position of the target; and activation rules of the image acquisition devices are dynamically switched according to the motion state;

[0010] The future position of the target in the site image and the predicted danger area are compared in real time; when the future position coincides with the predicted danger area, potential interference prediction is performed and the estimated interference time is calculated; and graded processing is performed according to different estimated interference times and the target's posture state.

[0011] Preferably, constructing a spatial map and identifying the position and motion status of a device in real time specifically includes:

[0012] Using visual simultaneous positioning and mapping technology to process environmental perception data collected by multiple visual sensors, generate a three-dimensional point cloud map of the site, and obtain the spatial map;

[0013] In the spatial map, the position and posture of the device body are continuously tracked and updated by matching the environmental perception data with map features in real time;

[0014] Based on the position and device posture, and in combination with the encoder readings and / or control instructions provided by the device control system, the real-time motion status of the device is obtained, including forward, reverse and load operation.

[0015] Preferably, the generation of the predicted dangerous area specifically includes: obtaining real-time operating parameters of the equipment and control instructions of the equipment control system; the real-time operating parameters include speed, angular velocity, acceleration and direction;

[0016] Based on the kinematic model of the device, the motion trajectory envelope of the device is predicted to obtain the device motion trajectory;

[0017] If the motion state of the device is load operation, then combining the weight and size information of the load and the load physical model, predicting the maximum swing range of the load during the motion process to obtain the load swing range;

[0018] The predicted motion trajectory envelope and load swing range are comprehensively combined with the geometric dimensions of the equipment itself and the static obstacle information extracted from the spatial map to generate a three-dimensional spatial area that changes with time as the predicted danger zone.

[0019] Preferably, the dynamic switching of the activation rules of the image acquisition device specifically includes:

[0020] If the motion state is forward, the first sensor 1 at the first position and the second sensor 2 at the second position are automatically activated;

[0021] If the movement state is backward, the third sensor 3 at the third position and the fourth sensor 4 at the fourth position are automatically activated;

[0022] If the movement state is load operation, the first sensor 1 , the second sensor 2 , the third sensor 3 and the fourth sensor 4 are started simultaneously.

[0023] Preferably, the image recognition model includes: a feature extraction unit, a target detection unit and a posture estimation unit;

[0024] The feature extraction unit extracts hierarchical visual features from low-level to high-level from the input site image layer by layer through a series of convolutional layers, pooling layers and activation functions based on CNN;

[0025] The target detection unit detects the target from the scene image based on the hierarchical visual features and obtains the spatial position of the target;

[0026] The posture estimation unit identifies the key skeleton points of the target and determines the posture state of the target, including falling, squatting, standing, walking and running.

[0027] Preferably, the motion estimation model comprises: a data association unit, a state filtering unit and a future position prediction unit;

[0028] The data association unit matches the spatial position and posture state of the target in the current frame with the tracked target trajectory in the historical frame, and establishes associations between the spatial position and posture state of the same target at different times;

[0029] The state filtering unit receives the associated spatial position sequence and attitude state sequence, and uses a state estimation algorithm to estimate the optimal state of the target at the current moment, wherein the optimal state includes at least the position and speed of the target;

[0030] The future position prediction unit predicts the spatial position of the target in the next N time steps based on the optimal state using the complex motion pattern learned based on LSTM to obtain the future position.

[0031] Preferably, the process of the grading treatment specifically includes:

[0032] By performing geometric collision detection in three-dimensional space, determining whether the future position predicted by the motion estimation model overlaps with the predicted danger area, and identifying potential interference;

[0033] If potential interference is identified, a predicted time point at which the future position and the predicted danger zone first overlap is determined, and the time difference between the predicted time point and the current moment is calculated to obtain an estimated interference time;

[0034] The calculated estimated interference time is compared with a preset time threshold, and combined with the posture state obtained by the image recognition model, hierarchical processing is performed according to predetermined rules.

[0035] Preferably, a warning system for the bottom of a gantry crane comprises:

[0036] The environmental perception and self-positioning module is used to build a spatial map by acquiring the environmental perception data of the site and identify the location and motion status of the device in real time;

[0037] The danger zone prediction module is used to predict the equipment's motion trajectory and load swing range based on the equipment's real-time operating parameters, location, and static object information on the site, and generate a predicted danger zone.

[0038] A dynamic human body detection and prediction module is configured to install image acquisition devices at the first, second, third, and fourth locations of the device to capture images of the site, and detect the site images using an image recognition model; if a target is detected in the site image, the spatial position and posture state of the target are obtained; the motion trajectory of the target is tracked using a motion estimation model, and the future position of the target is predicted; and the activation rules of the image acquisition devices are dynamically switched according to the motion state;

[0039] The intrusion detection and hierarchical processing module is used to compare the future position of the target in the site image with the predicted dangerous area in real time; when the future position coincides with the predicted dangerous area, potential interference prediction is performed, interference levels are classified and expected interference time is calculated; hierarchical processing is performed according to different expected interference times and the target's posture state.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. This invention constructs a spatial map and identifies the position and motion of gantry cranes in real time. Combining kinematic modeling with load swing range estimation, it generates a three-dimensional, time-varying prediction of hazardous areas. Compared to traditional approaches that rely on static rules or simple area demarcation, this invention dynamically and finely reflects potential risk areas for current and future equipment operations. This effectively addresses real-world scenarios with complex equipment movement and frequently changing environments, thereby improving operational safety and adaptability in the lower area.

[0042] 2. This invention deploys image acquisition devices at various locations on the equipment, dynamically switches activation strategies based on real-time motion status, and integrates deep learning image recognition models to accurately detect people or other dynamic targets within the venue. Furthermore, a motion estimation model that integrates data association and LSTM is used to track the trajectory of detected targets and predict their future positions. Compared to existing methods based solely on static image analysis, this invention significantly improves the accuracy of target detection and future motion trend assessment, providing a reliable data foundation for subsequent interference analysis.

[0043] 3. This invention identifies potential interference events in advance by performing real-time geometric overlap detection between the predicted future target position and the three-dimensional predicted danger zone. It then intelligently implements graded warnings based on the estimated interference time and target posture. Compared to existing methods that trigger simple warnings based solely on fixed distances, this invention integrates temporal, spatial, and behavioral factors to refine risk management, promptly distinguishing interference risks of varying urgency and guiding subsequent response measures. This effectively reduces false alarm and missed alarm rates, further enhancing the overall system's security capabilities and intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart of a method for warning below a gantry crane provided by an embodiment of the present invention;

[0045] Figure 2 A schematic structural diagram of an early warning system for a gantry crane provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the arrangement of the image acquisition device provided by an embodiment of the present invention in a gantry crane;

[0047] Figure 4 A schematic diagram of the structure of a dynamic human body detection and prediction module provided in an embodiment of the present invention.

[0048] In the figure: 1. First sensor; 2. Second sensor; 3. Third sensor; 4. Fourth sensor; 5. Beam; 6. Bracket 1; 7. Bracket 2; 8. Track. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0050] When a gantry crane is in operation, it's prohibited to stand beneath it or in front of the tracks. This is a major cause of injuries caused by people standing illegally beneath it. Gantry cranes are tall, and their four track supports create blind spots for operators. This makes it difficult to visually detect people standing beneath the crane or in front of the tracks, especially during nighttime or rainy weather, or when the operator is focused on other tasks. Furthermore, since operators may be operating the crane from the ground using a wired remote control, accidental injuries due to illegal standing positions are common, and urgent warnings and alerts are needed.

[0051] This invention proposes a method and system for warning below gantry cranes. This method combines the equipment's motion status to dynamically predict dangerous areas, and uses multi-point image recognition to achieve flexible perception and proactive warnings, thereby improving crane operation safety. To demonstrate the effectiveness of this method in improving crane operation safety, the following two examples will illustrate its effectiveness.

[0052] Example 1

[0053] In the embodiments of the present application, the process of improving the safety level of crane operations is described in detail using the method and system proposed in the present invention. Figure 1 The specific flow chart of the method proposed in the present invention includes: constructing a spatial map by acquiring environmental perception data of the site and identifying the position and motion state of the equipment in real time; predicting the motion trajectory of the equipment and the swing range of the load, and generating a predicted danger zone; installing an image acquisition device at a key position of the equipment to acquire the site image, and detecting the site image using an image recognition model; if a target is detected in the site image, acquiring the spatial position and posture state of the target; tracking the target motion trajectory using a motion estimation model, and predicting the future position of the target; comparing the future position of the target in the site image with the predicted danger zone in real time; when the future position coincides with the predicted danger zone, performing potential interference prediction, dividing the interference level and calculating the expected interference time; performing graded processing according to different expected interference times and the posture state of the target. Combined with Figure 1 and Figure 2 The following describes the contents:

[0054] A method for early warning below a gantry crane, comprising:

[0055] Acquiring environmental perception data from the site to construct a spatial map and identify the location and motion status of the device in real time; specifically, using visual simultaneous positioning and mapping technology to process environmental perception data collected by multiple visual sensors to generate a three-dimensional point cloud map of the site to obtain the spatial map;

[0056] In the spatial map, the position and posture of the device body are continuously tracked and updated by matching the environmental perception data with map features in real time;

[0057] Based on the position and device posture, and in combination with the encoder readings and / or control instructions provided by the device control system, the real-time motion status of the device is obtained, including forward, reverse and load operation.

[0058] Specifically, the multiple visual sensors include two RGB-D cameras and a laser radar;

[0059] The SLAM algorithm processes the images and depth data captured by the sensor in real time to dynamically generate a three-dimensional point cloud map of the site;

[0060] During the positioning process, a method based on feature point matching is used to continuously compare the new environmental perception data collected by the sensor with map features, realizing real-time updates of the device body in the map;

[0061] The state judgment logic includes: when the device driving instruction is forward and there is no load movement, it is judged to be forward; when the instruction is backward and there is no load movement, it is judged to be backward; when the instruction is to lift a heavy object and the hook load movement is obvious or the hook swings greatly, it is judged to be in load operation.

[0062] Visual SLAM combines multi-sensor fusion to build spatial maps and perform precise self-positioning, overcoming the shortcomings of traditional positioning methods that rely on GPS (stable signals or ineffective indoors) or simple odometers (accumulated errors). It can provide high-precision real-time position and motion status of equipment in complex operating environments, laying a solid foundation for the subsequent accurate definition of dynamic danger zones and precise calculation of the relative position of targets. This significantly improves the accuracy and reliability of the entire early warning system's environmental perception and is a prerequisite for achieving precise early warnings.

[0063] Preferably, the equipment motion trajectory and load swing range are predicted based on the equipment's real-time operating parameters, location, and site static object information, and a predicted danger zone is generated. The generation of the predicted danger zone specifically includes:

[0064] Acquire real-time operating parameters of the device and control instructions of the device control system; the real-time operating parameters include speed, angular velocity, acceleration and direction;

[0065] Based on the kinematic model of the device, the motion trajectory envelope of the device is predicted to obtain the device motion trajectory;

[0066] If the motion state of the device is load operation, then combining the weight and size information of the load and the load physical model, predicting the maximum swing range of the load during the motion process to obtain the load swing range;

[0067] The predicted motion trajectory envelope and load swing range are comprehensively combined with the geometric dimensions of the equipment itself and the static obstacle information extracted from the spatial map to generate a three-dimensional spatial area that changes with time as the predicted danger zone.

[0068] Specifically, the kinematic model adopts a differential drive model or a rigid body motion model;

[0069] If the motion state of the device is load operation, the weight and declared size information of the load are obtained from the device information system, and the double pendulum physical model is applied, combined with the acceleration disturbance generated by the device's predicted motion trajectory, to predict the maximum three-dimensional swing range of the load during the motion process, thereby obtaining the load swing range;

[0070] The predicted motion trajectory envelope (taking into account the geometric dimensions of the device itself and adding a 0.5-meter safety margin) and the predicted load swing range are voxelized and superimposed in the three-dimensional spatial map constructed by SLAM. At the same time, the space occupied by known static obstacles (such as walls and columns) in the map is excluded to generate a three-dimensional danger zone volume sequence that changes with time and includes the device body and the load activity range, which is used as the predicted danger zone.

[0071] Table 1 is a comparison table of the accuracy of equipment dynamic perception and dangerous area prediction of the traditional method and the method of the present invention.

[0072] Table 1 Comparison of the accuracy of equipment dynamic perception and dangerous area prediction

[0073] Test Number Accuracy of traditional methods (based only on current status) (%) Accuracy of the method of the present invention (%) 1 72.3 92.7 2 68.5 93.2 3 70.1 91.5 4 69.8 94.0 5 71.4 93.3

[0074] Compared to risk zones based solely on current conditions, this method generates predicted risk zones that evolve over time and more closely reflect actual risk conditions by predicting the future dynamics of equipment and loads and incorporating actual environmental obstacles. This dynamic and precise definition of risk zones significantly improves the relevance of early warnings, avoids missed or false alarms caused by inaccurate risk zone delineation, increases safety margins and operational efficiency, and effectively prevents collisions caused by equipment movement or load swings.

[0075] Preferably, image acquisition devices are installed at the first position, the second position, the third position, and the fourth position of the device to capture images of the site, and the image recognition model is used to detect the images of the site; if an object is detected in the image of the site, the spatial position and posture state of the object are obtained;

[0076] The image recognition model includes: a feature extraction unit, a target detection unit and a posture estimation unit;

[0077] The feature extraction unit extracts hierarchical visual features from low-level to high-level from the input site image layer by layer through a series of convolutional layers, pooling layers and activation functions based on CNN;

[0078] The target detection unit detects the target from the scene image based on the hierarchical visual features and obtains the spatial position of the target;

[0079] The posture estimation unit identifies the key skeleton points of the target and determines the posture state of the target, including falling, squatting, standing, walking and running.

[0080] Specifically, if Figure 3 The image acquisition device shown is arranged schematically, including a first sensor 1, a second sensor 2, a third sensor 3, a fourth sensor 4, a crossbeam 5, a bracket 1 6, a bracket 2 7 and a track 8;

[0081] In this embodiment, the gantry crane is composed of a crossbeam 5, a support 1 6 and a support 2 7, and moves on a track 8;

[0082] The first sensor 1, the second sensor 2, the third sensor 3, and the fourth sensor 4 are image acquisition devices with wide-angle lenses and night vision functions;

[0083] Install the first sensor 1 at the first position, which is the front connection between the beam 5 and the second bracket 7;

[0084] Install the second sensor 2 at the second position, which is the front connection between the beam 5 and the bracket 1 6;

[0085] Install the third sensor 3 at the third position, which is the rear connection between the crossbeam 5 and the second bracket 7;

[0086] Install a fourth sensor 4 at the fourth position, which is the rear connection between the crossbeam 5 and the bracket 1 6;

[0087] The feature extraction unit is based on CNN (using the lightweight MobileNetV3 or the more accurate ResNet-50 as the backbone network), and extracts multi-scale hierarchical visual features containing rich semantic information from the input site image layer by layer through a series of convolutional layers, pooling layers and activation functions (such as ReLU or SiLU);

[0088] The target detection unit (using the detection head of YOLOv5 or Faster R-CNN) detects people in the site image based on the hierarchical visual features using preset anchor boxes, outputs the bounding box coordinates and confidence score of the target, and obtains the 3D spatial position of the target in the world coordinate system through coordinate transformation (combining camera internal and external parameters and SLAM positioning results);

[0089] The posture estimation unit (using HRNet) identifies the key skeleton points (25 key points defined by COCO) within the detected target bounding box, determines the target's posture state, and clearly distinguishes it from falling, squatting, standing, walking, and running.

[0090] By deploying image acquisition devices at four key locations and applying advanced deep learning image recognition models, this method achieves high-precision detection, positioning, and posture recognition of people in the area below and around the equipment. This far surpasses the limitations of traditional infrared and ultrasonic sensors, which can only sense presence or absence. It can distinguish between people and objects and identify specific postures (especially dangerous falls and squats). This provides rich and accurate input for subsequent risk assessment and graded early warning. It is a key step in achieving intelligent and refined early warning, effectively addressing safety hazards caused by visual blind spots or human negligence.

[0091] Preferably, a motion estimation model is used to track the target's motion trajectory and predict the target's future position;

[0092] The motion estimation model includes: a data association unit, a state filtering unit and a future position prediction unit;

[0093] The data association unit matches the spatial position and posture state of the target in the current frame with the tracked target trajectory in the historical frame, and establishes associations between the spatial position and posture state of the same target at different times;

[0094] The state filtering unit receives the associated spatial position sequence and attitude state sequence, and uses a state estimation algorithm to estimate the optimal state of the target at the current moment, wherein the optimal state includes at least the position and speed of the target;

[0095] The future position prediction unit predicts the spatial position of the target in the next N time steps based on the optimal state using the complex motion pattern learned based on LSTM to obtain the future position.

[0096] Specifically, the data association unit uses the Hungarian algorithm to optimally match the target (spatial position and posture state) detected in the current frame with the target trajectory that has been successfully tracked in the previous frame, establishes a temporal association between the spatial position and posture state of the same target at different times, and handles the appearance, disappearance, and occlusion of the target;

[0097] The state filtering unit uses an extended Kalman filter or an unscented Kalman filter to receive the associated spatial position sequence and attitude state sequence, fuses the observed position (with noise) of the target and the uniform acceleration model, and recursively estimates the optimal state of the target at the current moment, wherein the optimal state includes the three-dimensional position and three-dimensional velocity of the target;

[0098] The LSTM is trained on a large number of pedestrian trajectory datasets and can capture nonlinear motion characteristics;

[0099] The future position prediction unit uses the complex motion pattern learned based on the LSTM to predict the spatial position sequence of the target in the next N (N=10) time steps to obtain the future position.

[0100] Table 2 is a comparison table of the dynamic target detection and future position prediction effects of the traditional method (based on static image analysis) and the method of the present invention.

[0101] Table 2 Comparison of dynamic target detection and future position prediction results

[0102]

[0103] By combining data association, state filtering, and a motion estimation model based on LSTM-based prediction technology, the system not only stably tracks people in its field of view but also relatively accurately predicts their future movement trends based on their historical movement patterns and current state. This predictive capability is the core of early warning. It elevates safety assessments from the current "collision" to the future "impending collision," creating a valuable window of time for implementing risk avoidance measures and significantly enhancing the foresight and proactive safety of the early warning system.

[0104] Preferably, the activation rule of the image acquisition device is dynamically switched according to the motion state;

[0105] The dynamic switching of the activation rules of the image acquisition device specifically includes:

[0106] If the motion state is forward, the first sensor 1 at the first position and the second sensor 2 at the second position are automatically activated;

[0107] If the movement state is backward, the third sensor 3 at the third position and the fourth sensor 4 at the fourth position are automatically activated;

[0108] If the movement state is load operation, the first sensor 1 , the second sensor 2 , the third sensor 3 and the fourth sensor 4 are started simultaneously.

[0109] Dynamically switching image data processing priorities and key monitoring areas based on the equipment's operating status is an intelligent strategy for achieving efficient and reliable monitoring with limited computing resources. This ensures that resources are focused on the front or rear areas, where danger is most likely, while the equipment is in motion. During the highest-risk and most complex load operation (hoisting) phase, comprehensive and comprehensive monitoring is provided. This adaptive strategy ensures safety at critical moments while avoiding unnecessary resource waste, improving the system's overall operational efficiency and practicality. It represents an intelligent advancement over the simple switching approach used in the original solution.

[0110] Preferably, the future position of the target in the site image and the predicted dangerous area are compared in real time; when the future position coincides with the predicted dangerous area, potential interference prediction is performed and the expected interference time is calculated; and graded processing is performed according to different expected interference times and the target's posture state.

[0111] The process of the hierarchical treatment specifically includes:

[0112] By performing geometric collision detection in three-dimensional space, determining whether the future position predicted by the motion estimation model overlaps with the predicted danger area, and identifying potential interference;

[0113] If potential interference is identified, a predicted time point at which the future position and the predicted danger zone first overlap is determined, and the time difference between the predicted time point and the current moment is calculated to obtain an estimated interference time;

[0114] The calculated estimated interference time is compared with a preset time threshold, and combined with the posture state obtained by the image recognition model, hierarchical processing is performed according to predetermined rules.

[0115] Specifically, the estimated interference time is accurate to millisecond level;

[0116] Geometric collision detection can use AABB bounding box collision detection or GJK algorithm;

[0117] The calculated estimated time of intervention (TTI) is compared with two preset time thresholds (e.g., T1 = 3.0 seconds, T2 = 1.0 seconds). Combined with the target's real-time posture (falling, squatting, standing, walking, and running) acquired by the image recognition model, classification is performed according to pre-defined rules. The preset time thresholds and risk classification rules can be adjusted and configured based on the safety requirements of the specific operating environment.

[0118] The reservation rules are as follows:

[0119] When TTI≥T1 and the posture is normal (standing, walking), it is of low level, triggering a prompt warning. The target and / or predicted dangerous area are marked in a specific color (such as yellow) in the ground projection, accompanied by low-frequency sound and light prompts;

[0120] When T2≤TTI<T1 or the posture is squatting but TTI is long, it is of medium level, triggering a warning warning. It is marked in a more prominent way (such as red flashing) in the A projection, and a clearer and higher-frequency sound and light alarm is issued, and a deceleration suggestion instruction is sent to the control system;

[0121] When TTI<T2 or the posture is a falling posture, it is of the highest level, triggering an emergency alarm, emitting the highest-level sound and light alarm (such as a rapid siren and a high-brightness flashing visual prompt), and immediately sending a forced stop or braking instruction to the equipment control system.

[0122] By comparing the future position of the target in the working area of the gantry crane with the predicted dangerous area in real time and performing hierarchical processing based on the predicted interference time and the target posture state, it is possible to achieve millisecond-level early identification and dynamic response of potential interference risks. Through a multi-dimensional hierarchical warning mechanism, the system automatically triggers prompt, warning or emergency braking measures according to different risk levels, improving the timeliness and accuracy of the warning, significantly enhancing the overall safety and intelligent level of the working area, and at the same time having good adaptability and configurability, which can effectively reduce the need for manual intervention and safety management costs.

[0123] Through technical means such as spatial map construction, real-time motion recognition of equipment, load dynamic characteristic modeling, site target detection and future position prediction, dynamic perception switching and multi-dimensional risk hierarchical management, the present invention forms an intelligent warning system for the area below the gantry crane with a full process and high precision. Compared with the existing solutions that only rely on static division, fixed rules or single perception methods, the present invention can dynamically and accurately depict the future operation risk area of the equipment and the load in a complex working environment, real-time perceive the change trend of dynamic targets such as personnel, intelligently judge potential interference, and take hierarchical responses according to the actual degree of danger. Overall, the present invention not only greatly improves the accuracy and timeliness of the warning, reduces the risk of false alarms and missed alarms, but also provides targeted intervention suggestions (such as deceleration, stop) for the equipment control system, significantly enhancing the safety protection level, system intelligence level and environmental adaptability in the working environment of the gantry crane, and having significant engineering application value and promotion prospects.

[0124] Embodiment 2

[0125] In Example 1, the method proposed in this invention successfully combined the equipment's motion status, dynamically predicted hazardous areas, and implemented flexible perception and proactive warnings based on multi-point image recognition, thereby improving crane operation safety. To further validate the effectiveness of this invention, the accuracy of the warnings provided by gantry crane B during operation was also verified in this example.

[0126] An early warning system for a gantry crane, comprising:

[0127] The environmental perception and self-positioning module is used to build a spatial map by acquiring the environmental perception data of the site and identify the location and motion status of the device in real time;

[0128] Building a spatial map and identifying the location and motion status of devices in real time specifically involves:

[0129] The environmental perception data collected by multiple visual sensors are processed using visual synchronous positioning and mapping technology to generate a three-dimensional point cloud map of the site to obtain the spatial map; in the spatial map, the position and posture of the device body are continuously tracked and updated by matching the environmental perception data with map features in real time; based on the position and posture of the device, and in combination with the encoder readings and / or control instructions provided by the device control system, the real-time motion status of the device is obtained, including forward, backward and load operation.

[0130] Preferably, the danger zone prediction module is used to predict the equipment motion trajectory and load swing range based on the equipment's real-time operating parameters, location, and static object information at the site, and generate a predicted danger zone;

[0131] The generation of the predicted dangerous area specifically includes:

[0132] Acquire the real-time operating parameters of the device and the control instructions of the device control system; the real-time operating parameters include speed, angular velocity, acceleration and direction; based on the kinematic model of the device, predict the motion trajectory envelope of the device to obtain the motion trajectory of the device; if the motion state of the device is load operation, then combine the weight and size information of the load and the load physical model to predict the maximum swing range of the load during the movement to obtain the load swing range; comprehensively combine the predicted motion trajectory envelope and load swing range, and combine the geometric dimensions of the device itself and the static obstacle information extracted from the spatial map to generate a three-dimensional spatial area that changes with time as the predicted danger zone.

[0133] Preferably, the dynamic human body detection and prediction module, see Figure 4, used to install image acquisition devices at the first position, the second position, the third position, and the fourth position of the device to capture site images, and use the image recognition model to detect the site images; if a target is detected in the site image, the spatial position and posture state of the target are obtained;

[0134] The image recognition model includes: a feature extraction unit, a target detection unit and a posture estimation unit;

[0135] The feature extraction unit extracts hierarchical visual features from low-level to high-level from the input site image layer by layer based on CNN through a series of convolutional layers, pooling layers and activation functions; the target detection unit detects the target from the site image based on the hierarchical visual features and obtains the spatial position of the target; the posture estimation unit identifies the key skeleton points of the target and determines the posture state of the target, including falling, squatting, standing, walking and running.

[0136] Preferably, a motion estimation model is used to track the target's motion trajectory and predict the target's future position;

[0137] The motion estimation model includes: a data association unit, a state filtering unit and a future position prediction unit;

[0138] The data association unit matches the spatial position and posture state of the target in the current frame with the tracked target trajectory in the historical frame, and establishes an association between the spatial position and posture state of the same target at different times; the state filtering unit receives the associated spatial position sequence and posture state sequence, and uses a state estimation algorithm to estimate the optimal state of the target at the current moment, and the optimal state includes at least the position and speed of the target; the future position prediction unit predicts the spatial position of the target in the next N time steps based on the optimal state using the complex motion pattern learned based on LSTM to obtain the future position.

[0139] Preferably, the activation rule of the image acquisition device is dynamically switched according to the motion state; specifically including:

[0140] If the movement state is forward, the first sensor 1 at the first position and the second sensor 2 at the second position are automatically started; if the movement state is backward, the third sensor 3 at the third position and the fourth sensor 4 at the fourth position are automatically started; if the movement state is load running, the first sensor 1, the second sensor 2, the third sensor 3 and the fourth sensor 4 are started at the same time.

[0141] Preferably, the intrusion detection and hierarchical processing module is used to compare the future position of the target in the site image with the predicted dangerous area in real time; when the future position coincides with the predicted dangerous area, potential interference prediction is performed and the expected interference time is calculated; and hierarchical processing is performed according to different expected interference times and the target's posture state.

[0142] The process of the hierarchical treatment specifically includes:

[0143] By performing geometric collision detection in three-dimensional space, it is determined whether there is spatial overlap between the future position predicted by the motion estimation model and the predicted danger area, and potential interference is identified; if potential interference is identified, the predicted time point when the future position and the predicted danger area overlap for the first time is determined, and the time difference between the predicted time point and the current moment is calculated to obtain the estimated interference time; the calculated estimated interference time is compared with a preset time threshold, and combined with the posture state obtained by the image recognition model, hierarchical processing is performed according to predetermined rules.

[0144] Table 3 provides a comparison of the potential interference intelligent identification and graded processing effects of the traditional method (which only triggers a simple warning based on a fixed distance) and the method of the present invention.

[0145] Table 3 Comparison of potential interference intelligent identification and hierarchical processing effects

[0146]

[0147] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for early warning below a gantry crane, characterized in that: include: Build a spatial map by acquiring environmental perception data of the site and identify the location and movement status of the equipment in real time; Predict equipment motion trajectory and load swing range based on the equipment's real-time operating parameters, location, and site static object information, and generate predicted danger zones. Image acquisition devices are installed at the first, second, third, and fourth positions of the device to capture images of the site, and the site images are detected using an image recognition model; if a target is detected in the site image, the spatial position and posture state of the target are obtained; a motion estimation model is used to track the motion trajectory of the target and predict the future position of the target; and activation rules of the image acquisition devices are dynamically switched according to the motion state; The future position of the target in the site image and the predicted danger area are compared in real time; when the future position coincides with the predicted danger area, potential interference prediction is performed and the estimated interference time is calculated; and graded processing is performed according to different estimated interference times and the target's posture state.

2. The method for early warning below a gantry crane according to claim 1, characterized in that: Constructing a spatial map and identifying the position and motion status of the device in real time specifically includes: using visual synchronous positioning and mapping technology to process environmental perception data collected by multiple visual sensors, generating a three-dimensional point cloud map of the site, and obtaining the spatial map; in the spatial map, continuously tracking and updating the position and device posture of the device body by real-time matching the environmental perception data with map features; based on the position and device posture, and in combination with the encoder readings and / or control instructions provided by the device control system, obtaining the real-time motion status of the device, including forward, backward and load operation.

3. The method for early warning below a gantry crane according to claim 1, characterized in that: The generation of the predicted danger zone specifically includes: obtaining the real-time operating parameters of the equipment and the control instructions of the equipment control system; the real-time operating parameters include speed, angular velocity, acceleration and direction; based on the kinematic model of the equipment, predicting the motion trajectory envelope of the equipment to obtain the motion trajectory of the equipment; if the motion state of the equipment is load operation, then combining the weight, size information and physical model of the load to predict the maximum swing range of the load during the movement to obtain the load swing range; comprehensively predicting the motion trajectory envelope and load swing range, and combining the geometric dimensions of the equipment itself and the static obstacle information extracted from the spatial map, to generate a three-dimensional spatial area that changes with time as the predicted danger zone.

4. The method for early warning below a gantry crane according to claim 1, characterized in that: The dynamic switching of the activation rules of the image acquisition device specifically includes: if the motion state is forward, automatically starting the first sensor (1) at the first position and the second sensor (2) at the second position; if the motion state is backward, automatically starting the third sensor (3) at the third position and the fourth sensor (4) at the fourth position; if the motion state is load operation, simultaneously starting the first sensor (1), the second sensor (2), the third sensor (3) and the fourth sensor (4).

5. The method for early warning below a gantry crane according to claim 1, characterized in that: The image recognition model includes: a feature extraction unit, a target detection unit and a posture estimation unit; the feature extraction unit extracts hierarchical visual features from low-level to high-level from the input site image layer by layer based on CNN through a series of convolutional layers, pooling layers and activation functions; the target detection unit detects the target from the site image based on the hierarchical visual features and obtains the spatial position of the target; the posture estimation unit identifies the key skeletal points of the target and determines the target's posture state, including falling, squatting, standing, walking and running.

6. The method for early warning below a gantry crane according to claim 1, characterized in that: The motion estimation model includes: a data association unit, a state filtering unit and a future position prediction unit; the data association unit matches the spatial position and posture state of the target in the current frame with the tracked target trajectory in the historical frame, and establishes an association between the spatial position and posture state of the same target at different times; the state filtering unit receives the associated spatial position sequence and posture state sequence, and uses a state estimation algorithm to estimate the optimal state of the target at the current moment, and the optimal state includes at least the position and speed of the target; the future position prediction unit predicts the spatial position of the target in the next N time steps based on the optimal state using the complex motion pattern learned based on LSTM to obtain the future position.

7. The method for early warning below a gantry crane according to claim 1, characterized in that: The hierarchical processing process specifically includes: performing geometric collision detection in three-dimensional space to determine whether there is spatial overlap between the future position predicted by the motion estimation model and the predicted dangerous area, and identifying potential interference; if potential interference is identified, determining the predicted time point when the future position and the predicted dangerous area overlap for the first time, and calculating the time difference between the predicted time point and the current moment to obtain the expected interference time; comparing the calculated expected interference time with a preset time threshold, and combining it with the posture state obtained by the image recognition model, hierarchical processing is performed according to predetermined rules.

8. A warning system for the bottom of a gantry crane, characterized in that: include: The environmental perception and self-positioning module is used to build a spatial map by acquiring the environmental perception data of the site and identify the location and motion status of the device in real time; The danger zone prediction module is used to predict the equipment's motion trajectory and load swing range based on the equipment's real-time operating parameters, location, and static object information on the site, and generate a predicted danger zone. A dynamic human body detection and prediction module is configured to install image acquisition devices at the first, second, third, and fourth locations of the device to capture images of the site, and detect the site images using an image recognition model; if a target is detected in the site image, the spatial position and posture state of the target are obtained; the motion trajectory of the target is tracked using a motion estimation model, and the future position of the target is predicted; and the activation rules of the image acquisition devices are dynamically switched according to the motion state; The intrusion detection and hierarchical processing module is used to compare the future position of the target in the site image with the predicted dangerous area in real time; when the future position coincides with the predicted dangerous area, potential interference prediction is performed and the expected interference time is calculated; and hierarchical processing is performed according to different expected interference times and the target's posture state.

Images