Crane sling deflection detection method based on artificial intelligence

Abstract

The present application relates to the technical field of crane sling deflection detection. The present application relates to a crane sling deflection detection method based on artificial intelligence. It includes the following steps: S1, build a multi-modal perception array, synchronously collect multi-source perception data, the multi-modal perception array contains an industrial camera, an IMU inertial measurement unit, a cart trolley displacement encoder and a wind speed sensor, which respectively obtain real-time image data of the sling, inertial attitude data of the sling, crane walking motion data and on-site wind field environment data; The present application builds a multi-modal perception array composed of an industrial camera, an IMU inertial measurement unit, a displacement encoder and a wind speed sensor, realizes the synchronous collection of multi-source data of the sling image, inertial attitude, crane motion state and on-site wind field environment, breaks through the limitation of single sensor detection in the prior art, and multiple sensors play their respective roles and complement each other's shortcomings, laying a foundation for subsequent high-precision detection from the data collection level.

Description

Technical Field

[0001] This invention relates to the field of crane spreader sway detection technology, and more specifically, to a crane spreader sway detection method based on artificial intelligence. Background Technology

[0002] As a core supporting technology for crane operations, spreader sway detection plays a crucial role in acquiring key parameters such as the sway angle and displacement of the spreader in the horizontal direction in real time. The purpose is to provide data support for the anti-sway control and precise positioning of the crane, ultimately improving the crane's operating efficiency, preventing collisions between the spreader and surrounding equipment, and ensuring the safety of operators and goods.

[0003] Existing detection technologies mainly revolve around single-sensor detection or simple multi-sensor combinations. These technologies have significant technical limitations. Single-vision detection methods are susceptible to environmental factors such as strong light, dust, and obstruction by the lifting equipment, resulting in blurred images, target loss, data interruption, and a sharp drop in accuracy. This makes it impossible to provide continuous and reliable data for anti-sway control. Meanwhile, although single-IMU detection methods have a fast response speed, they are prone to drift errors over long-term operation. As the operation time increases, the deviation accumulates, eventually leading to distorted detection results. Therefore, an artificial intelligence-based crane lifting equipment sway detection method is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide an artificial intelligence-based method for detecting the sway of crane spreaders, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, an artificial intelligence-based method for detecting crane spreader sway is provided, comprising the following steps: S1. Construct a multimodal sensing array to synchronously collect multi-source sensing data. The multimodal sensing array includes an industrial camera, an IMU inertial measurement unit, a trolley displacement encoder, and a wind speed sensor to acquire real-time image data of the lifting device, inertial attitude data of the lifting device, travel motion data of the crane, and on-site wind field environmental data, respectively. S2. Input the image data acquired by the industrial camera into the lightweight target detection model to select the lifting target. Then, input the selected lifting area into the key point detection network based on the attention mechanism to perform sub-pixel level positioning of the four corner points, the center lifting point and the lock head of the lifting. Based on the positioning of the key point coordinates, combined with the camera intrinsic and extrinsic parameter calibration data and the actual physical size of the lifting, the initial visual sway parameters are obtained through perspective geometry calculation. S3. Perform spatiotemporal synchronization calibration on multi-source sensing data, unify the time reference and spatial coordinate system, and complete the spatiotemporal alignment of visual data, IMU data, encoder data, and wind speed data. S4. A hierarchical fusion strategy of visual reference, IMU compensation, encoder constraint, and wind disturbance removal is adopted to carry out multi-source data fusion. The initial visual sway parameters are used as the global reference. The attitude angle, angular velocity and linear acceleration of the IMU are used to perform short-term compensation for the high-frequency micro-amplitude sway between adjacent visual frames. The kinematic constraints of the sway are established based on the displacement and acceleration of the trolley and the carriage collected by the encoder. The wind-induced sway component is calculated and removed according to the wind speed and wind direction data to obtain accurate fused sway data. S5. Based on the visual detection confidence and IMU drift error value, construct the mode adaptive switching logic. When the visual detection confidence is lower than the preset threshold, switch to the inertial and kinematic estimation mode of IMU and encoder. When the IMU drift error exceeds the allowable range and the visual recognition recovers to stability, use the features of the fixed anchor point of the lifting device to perform zero bias correction and error compensation on the IMU attitude data. S6. Accurately integrate sway data, on-site wind field environmental data, and current operating parameters into a hybrid time-series prediction model combining bidirectional LSTM and Transformer. Current operating parameters include rope length, load, wind speed, trolley travel speed, trolley travel speed, and start-stop acceleration. Then, local time-series features are extracted through bidirectional LSTM, and long-term time-series dependencies are captured through Transformer. The model outputs the sway angle change curve, sway displacement prediction value, and extreme peak value of the spreader within a preset time period.

[0006] As a further improvement to this technical solution, in S1, when building the multimodal sensing array, a binocular high-definition industrial camera is selected and is fixed inverted on the lower end of the crane trolley frame to collect front and side view images from the top surface of the lifting device. The IMU (Inertial Measurement Unit) uses a high-precision MEMS sensor, which is rigidly fixed to the center of the spreader to collect the spreader's three-axis attitude angle, angular velocity, and linear acceleration data in real time. Displacement encoders are installed on the drive motor shafts of the trolley and the carriage respectively, and collect data on the trolley's travel displacement, carriage's travel displacement, start-stop acceleration and running speed in real time. The wind speed sensor is fixed at an unobstructed position at the top of the crane boom to continuously collect data on instantaneous wind speed, average wind speed, and wind direction angle. Four types of sensors constitute a multi-source collaborative sensing network, and high-precision synchronous sampling is achieved through a unified hardware clock triggering mechanism.

[0007] As a further improvement to this technical solution, in S2, when performing artificial intelligence feature recognition, a lightweight target detection model is first used to select the target frame of the lifting device in the image and remove background interference areas. Then, the key point detection network is used to perform sub-pixel level positioning of the four corner points, the center lifting point, and the lock head of the spreader; Among them, the target detection model is a deep learning-based target recognition model, and the key point detection network is a key point localization model based on the attention mechanism; Based on the coordinates of the feature points after positioning, combined with the camera's intrinsic and extrinsic calibration data and the actual physical dimensions of the lifting device, the two-dimensional yaw displacement and three-dimensional yaw angle of the lifting device are obtained through perspective geometry calculation, forming the initial visual yaw parameters.

[0008] As a further improvement to this technical solution, in step S3, the sampling frequency of the multi-source sensing data is first unified to 50Hz, linear interpolation is performed on the low sampling rate data, and downsampling filtering is performed on the high sampling rate data to achieve complete timestamp alignment. Then, by calibrating the sensor installation position, the image pixel coordinate system, IMU carrier coordinate system, encoder motion coordinate system, and wind speed effect coordinate system are uniformly mapped to the crane's global geodetic coordinate system to eliminate spatial position deviation. A unified multi-coordinate system mapping model is constructed to eliminate errors caused by spatial installation deviations and form a highly consistent temporal multi-source feature dataset. Finally, the visual data, IMU data, encoder data, and wind speed data are combined in chronological order to form a continuous, equally spaced multi-source fusion feature sequence.

[0009] As a further improvement to this technical solution, in step S4, the yaw parameter calculated by visual calculation is used as a global absolute reference to eliminate accumulated errors. By utilizing the high-frequency response characteristics of the IMU, the dynamic lag caused by the low visual frame rate is compensated for, and short-time precision compensation is performed on the high-frequency micro-amplitude swing component of the lifting device. Based on the walking displacement and acceleration data collected by the encoder, the kinematic equation of the swaying of the spreader is established to constrain the rationality of the sway trajectory and filter out abnormal fluctuation values. A wind disturbance calculation model is constructed based on wind speed sensor data. The wind-induced sway component is calculated and removed from the total sway data to obtain undisturbed and accurate fused sway data. The hierarchical fusion strategy achieves dynamic consistency correction and disturbance component separation of the yaw state through multi-level collaborative constraints.

[0010] As a further improvement to this technical solution, in step S5, two indicators are calculated in real time: visual detection confidence and IMU drift error value. Construct a multimodal state evaluation index system and make mode switching decisions based on the evaluation results; Set the confidence threshold and the allowable range of IMU drift; When the visual detection confidence level is lower than the confidence threshold, it is determined that the image has occlusion, blurring, or strong light interference failure. The visual main test mode is immediately turned off and switched to the inertial and kinematic estimation mode of IMU and encoder to maintain the continuity of yaw detection. When the IMU drift error exceeds the allowable range and visual recovery stabilizes the recognition, the features of the fixed anchor point of the lifting device are extracted as the calibration benchmark. The IMU attitude data is then subjected to zero bias correction and error compensation to suppress drift recurrence and form a closed-loop error correction mechanism.

[0011] As a further improvement to this technical solution, in S6, an artificial intelligence time-series prediction model is established through a hybrid time-series network structure combining bidirectional LSTM and Transformer, using fused sway characteristics and on-site wind field environmental data from multiple consecutive moments as input; during the model training phase, historical data of the spreader sway under different rope lengths, different loads, and different wind speeds are incorporated to learn the sway law of the spreader under various working conditions, and output the spreader sway angle change curve, sway displacement prediction value, and extreme peak value in the next few seconds, thereby achieving adaptive prediction under working conditions; The operating conditions include rope length, load, instantaneous wind speed, average wind speed, trolley travel speed, and start-stop acceleration parameters.

[0012] As a further improvement to this technical solution, in S6, when the artificial intelligence time-series prediction model performs prediction, it adaptively adjusts the time-series window length in combination with real-time operating conditions. The historical time-series input length is extended for windy conditions and high-speed walking conditions, while the time-series window is shortened for stable conditions, thereby improving the prediction response speed and accuracy.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This AI-based crane spreader sway detection method utilizes a multimodal sensing array consisting of an industrial camera, an IMU (Inertial Measurement Unit), a displacement encoder, and a wind speed sensor. This array enables the synchronous acquisition of multi-source data, including spreader images, inertial attitude, crane motion status, and on-site wind field conditions. This overcomes the limitations of existing single-sensor detection technologies. The multiple sensors each perform their respective functions and complement each other, laying the foundation for subsequent high-precision detection from the data acquisition perspective. Simultaneously, a unified hardware clock triggering mechanism ensures the time synchronization of multi-source data, avoiding detection errors caused by asynchronous data acquisition and allowing the detection data to accurately reflect the actual sway state of the spreader.

[0014] 2. In this AI-based crane spreader sway detection method, spatiotemporal synchronization calibration of multi-source sensing data is performed to unify sensor data with different sampling frequencies to a standard sampling frequency of 50Hz. Time stamp alignment is achieved through linear interpolation and downsampling filtering. Simultaneously, multiple coordinate systems, such as image pixel coordinates and IMU carrier coordinates, are mapped to the crane's global geodetic coordinate system, constructing a unified multi-coordinate system mapping model. This completely solves the core problem of inconsistent temporal and spatial data in existing multi-sensor combined detection, eliminating detection errors caused by sensor installation deviations and sampling frequency differences. The resulting highly consistent time-series multi-source feature dataset provides a standardized and regulated data foundation for subsequent data fusion, significantly improving the reliability of the detection data.

[0015] 3. In this AI-based crane spreader sway detection method, a hierarchical fusion strategy is adopted to carry out multi-source data fusion. The visual detection results are used as the global absolute benchmark to eliminate accumulated errors. The high-frequency response characteristics of the IMU are used to compensate for the dynamic lag caused by the low visual frame rate. Kinematic equations are established based on encoder data to constrain the rationality of the sway trajectory. A wind disturbance model is constructed by combining wind speed sensor data to eliminate wind-induced sway components. The multi-level collaborative constraints realize the dynamic consistency correction of the sway state and the separation of disturbance components. Compared with the existing simple sensor data stitching method, this fusion strategy gives full play to the technical advantages of each sensor, effectively filters the detection deviation caused by environmental interference and equipment errors, and finally obtains uninterrupted and accurate fused sway data, which is far superior to the existing technology in terms of detection accuracy and dynamic adaptability. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the artificial intelligence-based crane spreader sway detection method of the present invention. Figure 2 This is a flowchart of S1 of the present invention; Figure 3 This is a flowchart of S2 of the present invention; Figure 4 This is a flowchart of S3 of the present invention; Figure 5 This is a flowchart of S4 of the present invention; Figure 6 This is a flowchart of S5 of the present invention; Figure 7 This is a flowchart of S6 of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figures 1-7 As shown, the purpose of this embodiment is to provide an artificial intelligence-based method for detecting crane spreader sway, including the following steps: S1. Construct a multimodal sensing array to synchronously collect multi-source sensing data. The multimodal sensing array includes an industrial camera, an IMU inertial measurement unit, a trolley displacement encoder, and a wind speed sensor to acquire real-time image data of the lifting device, inertial attitude data of the lifting device, travel motion data of the crane, and on-site wind field environmental data, respectively. In S1, when building a multimodal sensing array, a binocular high-definition industrial camera is selected and is fixed upside down on the lower end of the crane trolley frame. It collects frontal and side views of the top surface of the spreader. After completing the upside-down fixed installation of the binocular high-definition industrial camera, the camera is placed on the lower end of the crane trolley frame to ensure that the lens is facing the top surface of the spreader, so as to achieve coverage of frontal and side views. The IMU (Inertial Measurement Unit) uses a high-precision MEMS sensor, rigidly fixed to the center of the spreader, to collect the spreader's three-axis attitude angles, angular velocities, and linear acceleration data in real time. The rigid installation of the high-precision MEMS IMU is completed, and the IMU is positioned at the geometric center of the spreader to ensure that attitude acquisition is completely synchronized with the spreader's movement. Displacement encoders are installed on the drive motor shafts of the trolley and the carriage respectively to collect real-time data on trolley travel displacement, carriage travel displacement, start-stop acceleration and running speed; after completing the shaft installation of the displacement encoders, the encoders are fixed to the drive motor shafts of the trolley and the carriage respectively to ensure that the pulse acquisition is synchronized with the movement of the traveling mechanism; The wind speed sensor is fixed at an unobstructed position at the top of the crane boom to continuously collect instantaneous wind speed, average wind speed, and wind direction angle data. To ensure that the wind field data is not affected by structural obstruction, the wind speed sensor is installed in an unobstructed position at the top of the crane boom. Four types of sensors constitute a multi-source collaborative sensing network, and high-precision synchronous sampling is achieved through a unified hardware clock triggering mechanism.

[0019] It iterates through all sensors, connects to a unified hardware clock trigger module, configures the same sampling trigger signal, extracts the original sampling frequency parameters of each sensor, performs linear interpolation completion on low-frequency sensors, performs downsampling filtering on high-frequency sensors, unifies the sampling frequency to the target value, and simultaneously starts synchronous sampling. The industrial camera outputs dual-view image data with timestamps, and the IMU outputs three-axis attitude angle, angular velocity and linear acceleration data with timestamps. The displacement encoder outputs a pulse signal with timestamps, which is used to calculate the displacement, speed and acceleration data of the large vehicle / small vehicle. The wind speed sensor outputs instantaneous wind speed, average wind speed and wind direction angle data with timestamps. At the same time, the sampled data of all sensors are accumulated and timestamp alignment verification is performed to form a high-precision synchronous multi-source sensing dataset.

[0020] S2. Input the image data acquired by the industrial camera into the lightweight target detection model to select the lifting target. Then, input the selected lifting area into the key point detection network based on the attention mechanism to perform sub-pixel level positioning of the four corner points, the center lifting point and the lock head of the lifting. Based on the positioning of the key point coordinates, combined with the camera intrinsic and extrinsic parameter calibration data and the actual physical size of the lifting, the initial visual sway parameters are obtained through perspective geometry calculation. In S2, when performing artificial intelligence feature recognition, a lightweight target detection model is first used to select the target bounding box of the lifting device in the image and remove background interference areas. The system takes raw images from both frontal and side views captured by a binocular high-definition industrial camera as input, and reads the corresponding sampling timestamps simultaneously to ensure temporal alignment with multi-source sensing data. It also performs grayscale processing on the raw images, converting the RGB three-channel images into a single-channel grayscale image to reduce subsequent computational complexity. Gaussian filtering is then used to remove image noise and suppress interference from environmental dust and light fluctuations. Finally, distortion coefficients obtained during camera calibration are used to correct radial and tangential distortion, eliminating coordinate shifts caused by lens optical distortion. An adaptive histogram equalization algorithm is used to adjust the brightness and contrast of the image, enhance the visual difference between the rigging and the background, improve the stability of target detection and key point localization, and output a clear dual-view image after preprocessing, with timestamp added, before entering the lightweight target detection stage. The preprocessed image is input into a lightweight deep learning object detection model (YOLO-Nano or MobileNet-YOLO structure). After loading the pre-trained weights, the model performs forward inference and extracts features from the lifting target in the image. The model outputs the bounding box coordinates, confidence score and class label of the lifting area, and only retains the detection results with a confidence score higher than the preset threshold. The image is cropped based on the effective bounding box coordinates, retaining only the core area of ​​the lifting device and removing interference areas such as the crane's metal structure, background environment, and operators to reduce the scope of subsequent processing. Then, the cropped image of the lifting device area is normalized to unify the input resolution to the standard size required by the key point detection network, and the image of the lifting device target area and the corresponding bounding box coordinates are output. Then, the key point detection network is used to perform sub-pixel level positioning of the four corner points, the center lifting point, and the lock head of the spreader; Among them, the target detection model is a deep learning-based target recognition model, and the key point detection network is a key point localization model based on the attention mechanism; The image of the target area of ​​the lifting device is input into a keypoint detection network based on an attention mechanism (HRNet-Attention or Transformer-based KeypointDetector). The network extracts local fine features of the lifting device through the encoder. Then, the attention mechanism module automatically focuses on key areas such as the four corner points, the center lifting point, and the lock head of the lifting device, suppresses interference from irrelevant features, and generates a keypoint heatmap. Peak extraction and sub-pixel level optimization are performed on the keypoint heatmap. Through quadratic curve fitting, the accuracy of the keypoint coordinates is improved from the pixel level to the sub-pixel level (accuracy can reach 0.1 pixels). The detection of key points is validated, and outliers with confidence levels below the threshold are removed. At least four stable feature points are retained for subsequent calculations. The sub-pixel coordinate set of each key point is output, including the pixel coordinates of the four corner points of the lifting device, the center lifting point, and the lock head. The process then proceeds to the coordinate transformation and sway calculation stage. Based on the coordinates of the feature points after positioning, combined with the camera's intrinsic and extrinsic calibration data and the actual physical dimensions of the lifting device, the two-dimensional yaw displacement and three-dimensional yaw angle of the lifting device are obtained through perspective geometry calculation, forming the initial visual yaw parameters.

[0021] Read the intrinsic parameter matrix, distortion coefficients, stereo camera extrinsic transformation matrix (including rotation matrix and translation vector) stored during the camera calibration stage, as well as the actual physical size parameters of the scaffold. Then, substitute the pixel coordinates of the key points into the camera projection model, combine them with the distortion correction results, and convert them into three-dimensional coordinates in the camera coordinate system. By using the extrinsic matrix of the binocular camera, the coordinates of the monocular camera coordinate system are converted into coordinates of the global unified coordinate system, eliminating the spatial deviation caused by the camera installation position. Combined with the actual physical dimensions of the lifting device (such as the side length of the lifting device and the distance from the center lifting point to the four corners), the coordinates are scaled and normalized, mapping the pixel coordinates to the real physical space coordinates, thereby outputting a set of feature point coordinates of the global coordinate system with a unified scale. A reference coordinate system is established based on the standard posture of the spreader (no sway state). The reference coordinates of the four corner points and the center lifting point of the spreader are extracted as comparison references. The horizontal offset (trolley direction, trolley direction) and vertical height difference between the coordinates of the current posture feature points and the reference coordinates are calculated. Based on the lifting height of the spreader, the trolley sway angle, the gantry sway angle, and the torsion angle around the vertical axis are calculated through spatial geometric relationships. The two-dimensional sway displacement of the center lifting point of the spreader is calculated, and the rationality of the calculation results is verified. Abnormal sway values ​​that exceed physical limits are eliminated. Then, the sway angle, sway displacement, feature point coordinates and other data are summarized and timestamps are added to form the initial visual sway parameters.

[0022] S3. Perform spatiotemporal synchronization calibration on multi-source sensing data, unify the time reference and spatial coordinate system, and complete the spatiotemporal alignment of visual data, IMU data, encoder data, and wind speed data. In S3, the sampling frequency of multi-source sensing data is first unified to 50Hz, linear interpolation is performed on low sampling rate data, and downsampling filtering is performed on high sampling rate data to achieve complete timestamp alignment. The raw data from the four types of sensors are traversed, and their respective sampling frequency parameters are extracted. The target unified sampling frequency is determined to be 50Hz. For sensor data with sampling frequencies lower than 50Hz, linear interpolation is performed to complete the missing data points according to the 50Hz time node. The formula is as follows: ; in, For interpolated data at the target timestamp t, , For adjacent known timestamps , The original data at the location; For sensor data with a sampling frequency higher than 50Hz, a sliding window mean downsampling filter is performed, and the sampled data is aggregated according to the 50Hz time node, as shown in the following formula: ; in, target timestamp Downsampling data at the location, This represents the number of sampling points within the window. This is the original sampling interval; Add a uniform timestamp to all interpolated or downsampled data to align the time base. Then, by calibrating the sensor installation position, the image pixel coordinate system, IMU carrier coordinate system, encoder motion coordinate system, and wind speed effect coordinate system are uniformly mapped to the crane's global geodetic coordinate system to eliminate spatial position deviation. Based on the sensor installation location, the transformation relationship between each sensor coordinate system and the crane's global geodetic coordinate system is obtained through hand-eye calibration and structural measurement. The camera intrinsic parameter matrix and extrinsic parameter transformation matrix are read simultaneously to establish the mapping relationship from the image pixel coordinate system to the camera coordinate system and then to the global geodetic coordinate system. Read the IMU installation attitude parameters and establish the rotation and translation transformation relationship from the IMU carrier coordinate system to the global geodetic coordinate system; Read the encoder installation reference position and establish the displacement mapping relationship from the encoder motion coordinate system to the global geodetic coordinate system; Read the installation height and orientation of the wind speed sensor, and establish the direction and position mapping relationship from the wind speed coordinate system to the global geodetic coordinate system; A unified multi-coordinate system mapping model is constructed to eliminate errors caused by spatial installation deviations and form a highly consistent temporal multi-source feature dataset. The transformation parameters from the coordinate systems of each sensor to the global geodetic coordinate system are summarized, and a unified coordinate transformation matrix is ​​constructed. At the same time, the original data coordinates of each sensor are substituted into the transformation matrix to complete the coordinate system transformation, eliminate the error caused by spatial installation deviation, and then the coordinate rationality of the transformed data is verified, and abnormal coordinate points that are outside the physical range are removed to form a highly consistent time-series multi-source feature dataset. All data are under the same global geodetic coordinate system. Finally, the visual data, IMU data, encoder data, and wind speed data are combined in chronological order to form a continuous, equally spaced multi-source fusion feature sequence.

[0023] The system iterates through all sensor data in a unified timestamp order, matching visual data, IMU data, encoder data, and wind speed data frame by frame. It then stitches features from multiple sources at the same timestamp to form a single-frame multi-source fusion feature vector. Finally, it arranges all single-frame fusion feature vectors in chronological order to generate a continuous, equally spaced multi-source fusion feature sequence. The sequence integrity is then checked to ensure there are no missing timestamps, data misalignments, or format incompatibility issues. The output is a multi-source fusion feature sequence that can be used for subsequent hierarchical fusion and temporal prediction.

[0024] S4. A hierarchical fusion strategy of visual reference, IMU compensation, encoder constraint, and wind disturbance removal is adopted to carry out multi-source data fusion. The initial visual sway parameters are used as the global reference. The attitude angle, angular velocity and linear acceleration of the IMU are used to perform short-term compensation for the high-frequency micro-amplitude sway between adjacent visual frames. The kinematic constraints of the sway are established based on the displacement and acceleration of the trolley and the carriage collected by the encoder. The wind-induced sway component is calculated and removed according to the wind speed and wind direction data to obtain accurate fused sway data. In S4, the yaw parameter calculated by vision is used as the global absolute reference to eliminate cumulative error; The yaw parameters obtained from visual calculation are set as the global absolute reference benchmark for overall detection. Subsequent data fusion is carried out based on this benchmark, effectively avoiding the cumulative errors generated by various sensors during long-term operation. Visual yaw displacement and yaw angle data are read, and the validity of the data is determined. Once the validity is confirmed, it is used as the initial value for fusion, providing a unified reference standard for subsequent multi-source data correction. By utilizing the high-frequency response characteristics of the IMU, the dynamic lag caused by the low visual frame rate is compensated for, and short-time precision compensation is performed on the high-frequency micro-amplitude swing component of the lifting device. The system reads the attitude angle, angular velocity, and linear acceleration data of the lifting device acquired by the IMU. Leveraging the high-frequency response of the IMU, it compensates for the dynamic detection lag caused by the low frame rate of the industrial camera. Within the visual data sampling interval, it tracks the high-frequency micro-amplitude swaying of the lifting device in real time, integrating the subtle attitude changes detected by the IMU into the baseline sway data to complete short-term accuracy compensation and improve the dynamic fineness of sway detection. Based on the walking displacement and acceleration data collected by the encoder, the kinematic equation of the swaying of the spreader is established to constrain the rationality of the sway trajectory and filter out abnormal fluctuation values. Based on the trolley travel displacement, trolley travel displacement, running speed, and start / stop acceleration data collected by the encoder, a kinematic relationship of the spreader swing that conforms to the crane's operating law is established. Theoretically, a reasonable swing trajectory for the spreader is derived based on this kinematic relationship. The actual measured swing data is compared with the theoretical swing trajectory, and abnormal fluctuation values ​​deviating from the reasonable range are filtered out to constrain the physical rationality of the swing trajectory. A wind disturbance calculation model is constructed based on wind speed sensor data. The wind-induced sway component is calculated and removed from the total sway data to obtain undisturbed and accurate fused sway data. Based on the instantaneous wind speed, average wind speed, and wind direction angle data collected by the wind speed sensor, a calculation model of the influence of the wind field on the sway of the spreader is constructed. The model calculates the wind-induced sway component generated by the wind field, and this wind disturbance component is separated and removed from the total sway data to eliminate the interference of the environmental wind field on the detection results, thus obtaining sway data generated only by the movement of the spreader itself. The hierarchical fusion strategy achieves dynamic consistency correction and disturbance component separation of the yaw state through multi-level collaborative constraints. It integrates visual reference data, IMU compensated data, encoder constraint data, and wind disturbance-removed data, applying multi-level collaborative constraints based on the reliability of each data set. Dynamic consistency correction is performed on multiple sets of data to ensure temporal and numerical uniformity, thus separating the disturbance component from the true yaw component and ultimately outputting interference-free, accurately fused yaw data.

[0025] S5. Based on the visual detection confidence and IMU drift error value, construct the mode adaptive switching logic. When the visual detection confidence is lower than the preset threshold, switch to the inertial and kinematic estimation mode of IMU and encoder. When the IMU drift error exceeds the allowable range and the visual recognition recovers to stability, use the features of the fixed anchor point of the lifting device to perform zero bias correction and error compensation on the IMU attitude data. In S5, two metrics are calculated in real time: visual detection confidence and IMU drift error value. The system continuously collects visual inspection results and IMU operation data, calculates two indicators in real time: visual inspection confidence index and IMU drift error index, and transmits the index data synchronously to the status evaluation unit to provide a quantitative basis for subsequent system status determination. Construct a multimodal state evaluation index system and make mode switching decisions based on the evaluation results; Establish a multimodal state evaluation index system that includes visual effective state, IMU stable state, and system operating state. Input the confidence level and drift error value calculated in real time into the evaluation system to complete the comprehensive judgment of the working state of multiple sensors and output standardized state evaluation results. Set the confidence threshold and the allowable range of IMU drift; The system presets the visual detection confidence threshold and the allowable range of IMU drift error, and writes the threshold parameters into the decision unit as the judgment criteria for system mode switching and error correction, so as to ensure the accuracy and stability of system action execution. When the visual detection confidence level is lower than the confidence threshold, it is determined that the image has occlusion, blurring, or strong light interference failure. The visual main test mode is immediately turned off and switched to the inertial and kinematic estimation mode of IMU and encoder to maintain the continuity of yaw detection. The system compares the real-time visual detection confidence level with a preset threshold. When the confidence level is lower than the threshold, the industrial camera is deemed to be in a malfunctioning state. The criteria for this judgment include image occlusion, motion blur, strong light interference, and target loss. The system immediately shuts down the main visual detection mode and automatically switches to the IMU and encoder joint working mode. It uses a joint estimation method of inertial and kinematics to solve the yaw state, ensuring that the yaw detection process is uninterrupted and maintaining detection continuity. When the IMU drift error exceeds the allowable range and visual recovery stabilizes the recognition, the features of the fixed anchor point of the lifting device are extracted as the calibration benchmark. The IMU attitude data is then subjected to zero bias correction and error compensation to suppress drift recurrence and form a closed-loop error correction mechanism.

[0026] The real-time IMU drift error is compared with the allowable range. When the drift error exceeds the allowable range and visual recognition stabilizes, the closed-loop correction process is initiated. The features of the fixed anchor point of the spreader are extracted as the calibration benchmark. The IMU attitude data is corrected for zero bias and compensated for error using the benchmark data as a reference. This eliminates the detection deviation caused by drift, suppresses the recurrence of drift, and forms a complete closed-loop error correction mechanism. Once the visual signal returns to normal and IMU drift correction is complete, the system automatically exits the inertial and kinematic estimation mode and switches back to the vision-based measurement mode. It updates sensor status parameters and baseline data to ensure consistency in multi-source detection results, restores the multi-source hierarchical fusion operation, and outputs stable and reliable yaw detection data.

[0027] S6. Accurately integrate sway data, on-site wind field environmental data, and current operating parameters into a hybrid time-series prediction model combining bidirectional LSTM and Transformer. Current operating parameters include rope length, load, wind speed, trolley travel speed, trolley travel speed, and start-stop acceleration. Then, local time-series features are extracted through bidirectional LSTM, and long-term time-series dependencies are captured through Transformer. The model outputs the sway angle change curve, sway displacement prediction value, and extreme peak value of the spreader within a preset time period.

[0028] In S6, an artificial intelligence time-series prediction model is established through a hybrid temporal network structure combining bidirectional LSTM and Transformer. The model takes the fused sway characteristics and on-site wind field environmental data from multiple consecutive time points as input. During the model training phase, historical data of spreader sway under different rope lengths, loads, and wind speeds are incorporated to learn the sway patterns of spreader under various working conditions. The model outputs the spreader sway angle change curve, sway displacement prediction value, and extreme peak value in the next few seconds, thus achieving adaptive prediction under working conditions.

[0029] The operating conditions include rope length, load, instantaneous wind speed, average wind speed, trolley travel speed, and start-stop acceleration parameters. A hybrid architecture combining bidirectional LSTM and Transformer is adopted to construct an artificial intelligence time series prediction model. First, the network weights are initialized, the activation function and optimizer parameters are configured, and a model framework with the ability to extract local time series features and capture long time series dependencies is built. The model input dimension, output time series length and prediction step size are determined. Import historical working condition datasets with different rope lengths, loads, and wind speeds. Use continuous time-series fused sway characteristics and wind field environment data as training inputs, and use real sway data as labels to carry out iterative training of the model, optimize network parameters, and make the model fit the swing law of the spreader under all working conditions until the loss function converges and the optimal training model is saved. Real-time acquisition of current operating parameters, determination of operating condition type, and dynamic matching of time sequence input window length; automatic extension of time sequence window for strong wind and high-speed walking conditions, and automatic shortening of time sequence window for stable and windless conditions; filtering of historical fusion sway data and wind field data of corresponding length to construct standardized real-time input sequence. In S6, when the AI-powered time-series prediction model performs predictions, it adaptively adjusts the time-series window length based on real-time operating conditions. The historical time-series input length is extended for windy and high-speed walking conditions, while the time-series window is shortened for stable operating conditions, thereby improving prediction response speed and accuracy.

[0030] The standardized time-series input sequence is imported into the trained hybrid model, forward inference calculation is performed, time-series features are extracted and future yaw change trends are inferred, and the yaw angle, yaw displacement and extreme peak value within the future preset time period are output to complete the advance prediction. The physical rationality of the model output is verified, outliers are removed, and the yaw angle change curve, quantitative prediction value and extreme peak value are generated and output to the control system to provide predictive data for active anti-sway.

[0031] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting crane spreader sway based on artificial intelligence, characterized in that: Includes the following steps: S1. Construct a multimodal sensing array to synchronously collect multi-source sensing data. The multimodal sensing array includes an industrial camera, an IMU inertial measurement unit, a trolley displacement encoder, and a wind speed sensor to acquire real-time image data of the lifting device, inertial attitude data of the lifting device, travel motion data of the crane, and on-site wind field environmental data, respectively. S2. Input the image data acquired by the industrial camera into the lightweight target detection model to select the lifting target. Then, input the selected lifting area into the key point detection network based on the attention mechanism to perform sub-pixel level positioning of the four corner points, the center lifting point and the lock head of the lifting. Based on the positioning of the key point coordinates, combined with the camera intrinsic and extrinsic parameter calibration data and the actual physical size of the lifting, the initial visual sway parameters are obtained through perspective geometry calculation. S3. Perform spatiotemporal synchronization calibration on multi-source sensing data, unify the time reference and spatial coordinate system, and complete the spatiotemporal alignment of visual data, IMU data, encoder data, and wind speed data. S4. A hierarchical fusion strategy of visual reference, IMU compensation, encoder constraint, and wind disturbance removal is adopted to carry out multi-source data fusion. The initial visual sway parameters are used as the global reference. The attitude angle, angular velocity and linear acceleration of the IMU are used to perform short-term compensation for the high-frequency micro-amplitude sway between adjacent visual frames. The kinematic constraints of the sway are established based on the displacement and acceleration of the trolley and the carriage collected by the encoder. The wind-induced sway component is calculated and removed according to the wind speed and wind direction data to obtain accurate fused sway data. S5. Based on the visual detection confidence and IMU drift error value, construct the mode adaptive switching logic. When the visual detection confidence is lower than the preset threshold, switch to the inertial and kinematic estimation mode of IMU and encoder. When the IMU drift error exceeds the allowable range and the visual recognition recovers to stability, use the features of the fixed anchor point of the lifting device to perform zero bias correction and error compensation on the IMU attitude data. S6. Accurately integrate sway data, on-site wind field environmental data, and current operating parameters into a hybrid time-series prediction model combining bidirectional LSTM and Transformer. Current operating parameters include rope length, load, wind speed, trolley travel speed, trolley travel speed, and start-stop acceleration. Then, local time-series features are extracted through bidirectional LSTM, and long-term time-series dependencies are captured through Transformer. The model outputs the sway angle change curve, sway displacement prediction value, and extreme peak value of the spreader within a preset time period.

2. The crane spreader sway detection method based on artificial intelligence according to claim 1, characterized in that: In S1, when building the multimodal sensing array, a binocular high-definition industrial camera is selected and is fixed inverted on the lower end of the crane trolley frame to collect front and side view images from the top surface of the lifting device. The IMU (Inertial Measurement Unit) uses a high-precision MEMS sensor, which is rigidly fixed to the center of the spreader to collect the spreader's three-axis attitude angle, angular velocity, and linear acceleration data in real time. Displacement encoders are installed on the drive motor shafts of the trolley and the carriage respectively, and collect data on the trolley's travel displacement, carriage's travel displacement, start-stop acceleration and running speed in real time. The wind speed sensor is fixed at an unobstructed position at the top of the crane boom to continuously collect data on instantaneous wind speed, average wind speed, and wind direction angle. Four types of sensors constitute a multi-source collaborative sensing network, and high-precision synchronous sampling is achieved through a unified hardware clock triggering mechanism.

3. The crane spreader sway detection method based on artificial intelligence according to claim 1, characterized in that: In S2, when performing artificial intelligence feature recognition, a lightweight target detection model is first used to select the target frame of the lifting device in the image and remove background interference areas. Then, the key point detection network is used to perform sub-pixel level positioning of the four corner points, the center lifting point, and the lock head of the spreader; Among them, the target detection model is a deep learning-based target recognition model, and the key point detection network is a key point localization model based on the attention mechanism; Based on the coordinates of the feature points after positioning, combined with the camera's intrinsic and extrinsic calibration data and the actual physical dimensions of the lifting device, the two-dimensional yaw displacement and three-dimensional yaw angle of the lifting device are obtained through perspective geometry calculation, forming the initial visual yaw parameters.

4. The artificial intelligence-based crane spreader sway detection method according to claim 1, characterized in that: In step S3, the sampling frequency of the multi-source sensing data is first unified to 50Hz, linear interpolation is performed on the low sampling rate data, and downsampling filtering is performed on the high sampling rate data to achieve complete timestamp alignment. Then, by calibrating the sensor installation position, the image pixel coordinate system, IMU carrier coordinate system, encoder motion coordinate system, and wind speed effect coordinate system are uniformly mapped to the crane's global ground coordinate system to eliminate spatial position deviation. A unified multi-coordinate system mapping model is constructed to eliminate errors caused by spatial installation deviations and form a highly consistent temporal multi-source feature dataset. Finally, the visual data, IMU data, encoder data, and wind speed data are combined in chronological order to form a continuous, equally spaced multi-source fusion feature sequence.

5. The artificial intelligence-based crane spreader sway detection method according to claim 1, characterized in that: In S4, the yaw parameter calculated by visual analysis is used as a global absolute reference to eliminate cumulative error. By utilizing the high-frequency response characteristics of the IMU, the dynamic lag caused by the low visual frame rate is compensated for, and short-time precision compensation is performed on the high-frequency micro-amplitude swing component of the lifting device. Based on the walking displacement and acceleration data collected by the encoder, the kinematic equation of the swaying of the spreader is established to constrain the rationality of the sway trajectory and filter out abnormal fluctuation values. A wind disturbance calculation model is constructed based on wind speed sensor data. The wind-induced sway component is calculated and removed from the total sway data to obtain undisturbed and accurate fused sway data. The hierarchical fusion strategy achieves dynamic consistency correction and disturbance component separation of the yaw state through multi-level collaborative constraints.

6. The crane spreader sway detection method based on artificial intelligence according to claim 1, characterized in that: In S5, two indicators are calculated in real time: visual detection confidence and IMU drift error value. Construct a multimodal state evaluation index system and make mode switching decisions based on the evaluation results; Set the confidence threshold and the allowable range of IMU drift; When the visual detection confidence level is lower than the confidence threshold, it is determined that the image has occlusion, blurring, or strong light interference failure. The visual main test mode is immediately turned off and switched to the inertial and kinematic estimation mode jointly used by the IMU and encoder to maintain the continuity of yaw detection. When the IMU drift error exceeds the allowable range and visual recovery stabilizes the recognition, the features of the fixed anchor point of the lifting device are extracted as the calibration benchmark. The IMU attitude data is then subjected to zero bias correction and error compensation to suppress drift recurrence and form a closed-loop error correction mechanism.

7. The artificial intelligence-based crane spreader sway detection method according to claim 1, characterized in that: In S6, an artificial intelligence time-series prediction model is established through a hybrid time-series network structure combining bidirectional LSTM and Transformer, with the fused sway characteristics and on-site wind field environmental data at multiple consecutive time points as input. During the model training phase, historical data of spreader swaying under different rope lengths, loads, and wind speeds are incorporated to learn the swaying patterns of spreaders under various working conditions. The model outputs the curve of the spreader sway angle change, the predicted value of the sway displacement, and the extreme peak value in the next few seconds, thus achieving adaptive prediction of working conditions. The operating conditions include rope length, load, instantaneous wind speed, average wind speed, trolley travel speed, and start-stop acceleration parameters.

8. The artificial intelligence-based crane spreader sway detection method according to claim 7, characterized in that: In S6, when the artificial intelligence time-series prediction model performs prediction, it adaptively adjusts the time-series window length based on real-time operating conditions. The historical time-series input length is extended for windy and high-speed walking conditions, while the time-series window is shortened for stable operating conditions, thereby improving the prediction response speed and accuracy.

Images