A gantry crane grab bucket control system and method based on multi-sensor fusion

Abstract

The present invention discloses a gantry crane grab bucket control system and method based on multi-sensor fusion, and specifically relates to the technical field of gantry crane grab bucket control. By deploying image acquisition equipment and multiple types of sensors in the gantry crane operating area, a multi-dimensional input data set under a unified time sequence is constructed, and an optimized visual detection model and attention mechanism are combined to achieve high-precision recognition and abnormal judgment of the grab bucket state. The system can not only automatically plan the grabbing path and posture adjustment instructions, but also dynamically optimize the control parameters according to the grabbing effect and posture feedback, which significantly improves the grab bucket recognition accuracy, control stability and system intelligence level, especially showing higher robustness and reliability in processing small-size, complex posture or multi-target operating environments.

Description

Technical Field

[0001] The present invention relates to the technical field of gantry crane grab bucket control, and in particular to a gantry crane grab bucket control system and method based on multi-sensor fusion. Background Art

[0002] Multi-sensor fusion-based gantry crane grab control involves integrating data collected by multiple sensors (such as position sensors, force sensors, image sensors, and accelerometers) within the gantry crane (gantry crane) grab control system to comprehensively analyze and process the grab's operating status, material location, and environmental information, thereby achieving precise control and optimization of the grab's movements. This approach improves grasping accuracy, operational efficiency, and system intelligence, and is particularly suitable for complex or dynamically changing operations, such as port loading and unloading or bulk material handling.

[0003] The existing technology has the following shortcomings:

[0004] While introducing dilated convolutions in grab detection based on the improved YOLOv3-tiny network enhances high-level semantic information, it can easily lead to feature alignment errors. Because YOLOv3-tiny's inherent feature maps are relatively shallow, dilated convolutions, while expanding the receptive field, can cause spatial coordinate deviations between high-level features and low-level details during the multi-scale fusion process. This leads to inaccurate positioning of the grab edge, particularly when dealing with small grabs or those in unusual postures (such as an open state). Furthermore, in multi-target scenarios, this can cause boundary overlap and positioning offsets, impacting detection accuracy and system stability. Summary of the Invention

[0005] The purpose of the present invention is to provide a gantry crane grab bucket control system and method based on multi-sensor fusion to solve the shortcomings in the background technology.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a gantry crane grab bucket control method based on multi-sensor fusion, comprising:

[0007] Image acquisition equipment and multiple types of sensors are deployed in the gantry crane operating area to obtain the grab bucket's video images, posture information, position coordinates, and rotation angle data in real time;

[0008] Enhance and normalize the acquired video images and synchronize them with the sensor data at a unified timestamp to construct a multi-dimensional input dataset.

[0009] A dilated convolutional layer is introduced into the original YOLOv3-tiny architecture. Before the high-level and low-level features are fused, the feature pyramid alignment mechanism is used to achieve spatial alignment between features of different scales. The fused feature map is then convolved to generate the grab position and bounding box information.

[0010] Based on image detection results and multi-source sensor position data, the attention mechanism is used to continuously track the grab state and detect abnormalities.

[0011] The detection and tracking results are input into the grab control system to automatically generate accurate grab paths and posture adjustment instructions, and control the grab movement in real time;

[0012] Based on the grasping results and the posture deviation sent back by the sensor, the control model parameters are dynamically adjusted to continuously optimize the detection accuracy and control strategy.

[0013] Preferably, the multi-type sensors include a posture sensor, a position coordinate sensor, a rotation angle sensor, and a laser ranging and obstacle avoidance sensor.

[0014] Preferably, the step of enhancing and normalizing the acquired video image and synchronously fusing it with the sensor data at a unified timestamp includes:

[0015] Perform histogram equalization, gamma correction, filtering noise reduction and sharpening enhancement on the image;

[0016] Normalize the image pixel values ​​to the range [0, 1] and unify the image size to the YOLOv3-tiny network input requirements;

[0017] Normalize and unify the dimensions of all sensor data and encapsulate them into a standardized tensor format;

[0018] Align image frames and sensor data timestamps based on a unified clock system;

[0019] Construct a multimodal dataset of image-sensor joint input through splicing or network fusion.

[0020] Preferably, a dilated convolutional layer is introduced into the original YOLOv3-tiny structure and spatial alignment is performed based on a feature pyramid alignment mechanism, including:

[0021] Introducing dilated convolution after deep feature extraction;

[0022] Use upsampling operations to upsample high-level feature maps to the same spatial resolution as shallow feature maps;

[0023] Perform 1×1 convolution on the shallow feature map to match the number of channels;

[0024] Before feature fusion, a deformable convolution or spatial transformation network module is introduced to spatially align the feature maps;

[0025] The fused feature map is convolved using a 3×3 standard convolution to output a feature map of uniform scale for bounding box detection.

[0026] Preferably, the use of the attention mechanism to achieve continuous tracking and abnormal judgment of the grab state includes: constructing a fusion feature vector, inputting a channel attention mechanism, introducing a spatial attention mechanism, using LSTM combined with a Self-Attention mechanism to construct a state time series model to judge the dynamic stability trend of the grab; and identifying abnormal states according to preset abnormal judgment rules.

[0027] Preferably, the automatic generation of accurate grasping paths and posture adjustment instructions includes:

[0028] Based on the grab bucket's current position, target position, and obstacle distribution in the work area, a path planning algorithm is used to generate the optimal drop trajectory.

[0029] Establish critical control points along the route, including grab points, closing points, and transfer points;

[0030] Combine IMU attitude and visual detection results to calculate the required attitude adjustment angle;

[0031] Output motor control voltage or hydraulic valve opening instruction to achieve real-time adjustment of grab bucket posture;

[0032] The control period is set to no more than 100 milliseconds to meet high-frequency real-time control requirements.

[0033] Preferably, the lateral drift rate of the grab bucket's motion trajectory is generated by analyzing the trajectory deviation amplitude and trend in the horizontal direction during the grab bucket's movement from the starting point to the target position. The acquisition method is:

[0034] The point sequences of the ideal motion trajectory and the actual motion trajectory are respectively represented as two-dimensional coordinate point sets sampled in time sequence, where the ideal trajectory point sequence is , N is the total number of ideal trajectory points, and the actual trajectory point sequence is , M is the total number of actual trajectory points, where each point contains the coordinate values ​​of the X-axis and Y-axis; for each pair of trajectory points, the absolute distance difference in the Y-axis direction is calculated, that is, a two-dimensional distance matrix is ​​constructed to represent the lateral offset between any pair of ideal points and actual points, and each element represents the distance error between two points in the Y direction;

[0035] The minimum cumulative cost path algorithm in the DTW algorithm is used. Starting from the starting point, the minimum cumulative cost value from the starting point to any point pair is recursively calculated through dynamic programming. The cumulative cost is based on the previous state. The minimum cost path of the adjacent points is selected and superimposed on the current Y-direction deviation until all point pairs are traversed to form a complete cost matrix.

[0036] After the cost matrix is ​​calculated, the optimal alignment path is extracted by backtracking from the end point of the cost matrix. This involves pairing the actual trajectory points with the ideal trajectory points at the optimal time. The Y-direction offset of each point pair in the entire trajectory is extracted through the path. The Y-direction deviations of all matching point pairs are averaged to obtain the lateral drift rate of the grab motion.

[0037] Preferably, the terminal posture stabilization time is generated by analyzing the time taken for the posture of the grab bucket to converge from the yaw state to the tolerable range before it descends and closes. The acquisition method is:

[0038] Collect continuous attitude data of the grab bucket from the IMU or attitude solution system: pitch angle sequence: , roll angle sequence: ; Set a sliding time window length , corresponding to several frames of data, at any time t, calculate the average absolute error of the posture in the past time window ; The average absolute error of the posture in the current window satisfies: < and < , that is, judging that the grab bucket posture enters a stable state, is the angle error range, and the time ts from the start of descent to the first time the condition is met is recorded, then: ; Among them: t0 is the time point when the grab bucket starts to descend or start to swing; Tstable is the end posture stabilization time.

[0039] Preferably, the lateral drift rate and attitude stabilization time are used as inputs of the fuzzy logic system; the control error level is used as the output of the fuzzy logic system, and graded parameter adjustment is performed according to the error level: the current control parameters are maintained in the low error stage; the proportional control gain Kp is increased and the waiting stabilization time is increased in the medium error stage; the adaptive compensation mechanism is triggered in the high error stage to improve the response rate of the PID controller or activate the trajectory correction strategy.

[0040] The present invention also provides a gantry crane grab control system based on multi-sensor fusion, which includes an image acquisition module, an image preprocessing module, a grab visual detection module, a grab state tracking module, a path planning and motion control module, and an adaptive control optimization module;

[0041] Image acquisition module: Image acquisition equipment and multiple types of sensors are deployed in the gantry crane operating area to obtain the grab bucket's video images, posture information, position coordinates, and rotation angle data in real time;

[0042] Image preprocessing module: Enhances and normalizes the acquired video images, and synchronizes them with the sensor data at a unified timestamp to construct a multi-dimensional input dataset;

[0043] Grab bucket visual detection module: This module introduces a dilated convolutional layer into the original YOLOv3-tiny architecture. Before fusing high- and low-level features, it uses a feature pyramid alignment mechanism to achieve spatial alignment between features of different scales. It then performs convolution processing on the fused feature map to generate the grab bucket's position and bounding box information.

[0044] Grab bucket state tracking module: Based on image detection results and multi-source sensor position data, it uses an attention mechanism to continuously track the grab bucket state and detect anomalies.

[0045] Path planning and motion control module: Inputs detection and tracking results into the grab control system, automatically generates precise grab paths and posture adjustment instructions, and controls the grab motion in real time;

[0046] Adaptive control optimization module: Dynamically adjusts control model parameters based on the grasping results and the posture deviation sent back by the sensor, and continuously optimizes detection accuracy and control strategies.

[0047] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0048] 1. This invention overcomes the limitations of traditional grab bucket recognition and control based on single visual information. By integrating video images with multiple sensor data, this system builds a unified multidimensional input model for image preprocessing, time synchronization, and data normalization. By introducing dilated convolution and feature alignment mechanisms through an improved YOLOv3-tiny architecture, it achieves precise detection and bounding box regression for grab buckets of varying scales. By integrating an attention mechanism with a temporal network, the system continuously tracks grab bucket status and intelligently identifies anomalies during operation, such as sway, jamming, and collision risks. This effectively improves grab bucket recognition accuracy and system stability.

[0049] 2. This invention uses fuzzy logic analysis on key control feedback variables, such as lateral drift rate and terminal attitude stabilization time, to achieve real-time assessment of control error levels. Based on this error level, it dynamically adjusts control model parameters, including PID gain, adaptive compensation strategy, and gripping wait time, thereby achieving closed-loop optimization and adaptive enhancement of the control system. This system is particularly suitable for large-scale material handling scenarios such as ports and mining areas. It maintains high gripping accuracy and system robustness even under small grab buckets, multiple target interference, or complex dynamic operating conditions, significantly improving the intelligence and efficiency of gantry crane operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0051] Figure 1 This is a mind map of the method of the present invention.

[0052] Figure 2 This is the system mind map of the present invention. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0054] Example 1, please refer to Figure 1 As shown, the gantry crane grab bucket control method based on multi-sensor fusion described in this embodiment includes:

[0055] Image acquisition equipment and multiple types of sensors are deployed in the gantry crane operating area to obtain real-time video images, posture information, position coordinates, and rotation angle data of the grab bucket;

[0056] Enhance and normalize the acquired video images and synchronize them with the sensor data at a unified timestamp to construct a multi-dimensional input dataset.

[0057] A dilated convolutional layer is introduced into the original YOLOv3-tiny architecture. Before the high-level and low-level features are fused, the feature pyramid alignment mechanism is used to achieve spatial alignment between features of different scales. The fused feature map is then convolved to generate the grab position and bounding box information.

[0058] Based on image detection results and multi-source sensor position data, the attention mechanism is used to continuously track the grab state and detect abnormalities.

[0059] The detection and tracking results are input into the grab control system to automatically generate accurate grab paths and posture adjustment instructions, and control the grab movement in real time;

[0060] Based on the grasping results and the posture deviation sent back by the sensor, the control model parameters are dynamically adjusted to continuously optimize the detection accuracy and control strategy.

[0061] In the intelligent control system of a gantry crane grab bucket, to achieve accurate perception and dynamic control of the grab bucket's operating status, it is necessary to deploy image acquisition equipment and multiple types of sensors within the gantry crane's operating area to perform multi-dimensional real-time monitoring of the grab bucket. The following is a detailed description of the deployment process and the data obtained:

[0062] Camera type selection: Use an industrial-grade high-definition camera (such as a CMOS camera with 2 megapixels or above) with wide dynamic range and low latency. Optional infrared night vision cameras or thermal imaging cameras ensure image acquisition capabilities at night or in smoky environments. If depth information is required, an RGB-D camera (such as Intel RealSense) or a stereo camera system can be integrated.

[0063] Installation location layout principles: The camera is installed above the gantry crane trolley beam, grab bucket bracket or spreader, with a downward angle of approximately 45°, covering the up and down movement area of ​​the grab bucket; if the working environment is large (such as a port bulk warehouse), multiple fixed cameras are deployed in key areas for regional monitoring, and PTZ (pan-tilt-zoom) cameras are combined for dynamic tracking when necessary; the camera is connected to industrial Ethernet or an edge computing unit to support real-time transmission of video streams to the control host or AI detection module.

[0064] Collected image content: Real-time capture of grab bucket shape (closed, open), posture changes, and cargo status in the operating area; cooperates with the AI ​​detection module to identify grab bucket contours, edges, shadows, cargo type, etc.

[0065] Attitude sensor (IMU – Inertial Measurement Unit): Installed on the grab frame or spreader connection, it collects the grab's acceleration, angular velocity, inclination, and yaw angle in real time; it senses the grab's motion in three-dimensional space (including swing amplitude and direction); and uploads data to the central processing module via the CAN or RS485 interface.

[0066] Position coordinate sensor: Utilizes a GNSS-RTK high-precision positioning module (centimeter-level accuracy) installed on the gantry body and grab bucket. High-precision indoor positioning is achieved with the aid of a UWB positioning base station and tag system in indoor or obstructed scenarios. This sensor obtains the grab bucket's absolute or relative position coordinates in XYZ space in real time.

[0067] Rotation angle sensor (encoder): Installed on the grab bucket's rotating shaft, motor, or spreader connection; obtains real-time information on the grab bucket's rotation angle in the vertical and horizontal planes (such as the spreader's rotation angle and the grab bucket's flip angle); can use an absolute encoder or incremental encoder, and cooperate with the controller for feedback control.

[0068] Laser ranging and obstacle avoidance sensors: Installed at the four corners of the grab or on the boom, they measure the relative distance between the grab and the ground, cargo, or obstacles. LiDAR or ToF sensors are used to build a contour model of the work area to assist in path planning and collision avoidance.

[0069] Synchronous acquisition and fusion mechanism: All sensors and image acquisition devices synchronize data through timestamps and unify sampling frequencies;

[0070] Use multi-sensor fusion algorithms (such as Kalman filter, extended Kalman filter EKF or deep fusion network) to jointly process image, pose and position data.

[0071] Edge computing and real-time transmission: Deploy some detection models and data processing logic on the door machine's edge computing device (such as the NVIDIA Jetson series); high-frequency data is transmitted to the central control platform via industrial buses (CAN, EtherCAT) or wireless communications (5G, Wi-Fi 6).

[0072] The acquired video images are enhanced and normalized, and then synchronously fused with the sensor data at a unified timestamp to construct a multidimensional input dataset. The detailed process is as follows:

[0073] Image enhancement processing: To improve the recognizability of images in complex environments (such as backlight, low illumination, rain, fog, dust, etc.), the collected video frames are enhanced, including: Histogram equalization: improve image contrast and highlight the edges and contours of the grab; Gamma correction: adjust the brightness range to avoid overexposure or underexposure; Gaussian filtering or bilateral filtering: reduce noise while retaining edge information; Image sharpening processing: enhance the lines of the grab structure and improve the detection model's sensitivity to edges; Gamma enhancement + color channel reconstruction: used to improve feature visibility at night or in complex color temperature environments.

[0074] Image normalization: Normalize the image pixel values ​​to the range of [0, 1] or [-1, 1]. Unify the image size (e.g., 416×416 or 320×320) according to the input requirements of the YOLOv3-tiny model. Normalize the image channel dimension (RGB) to ensure that the network input has good distribution characteristics.

[0075] Raw sensor data acquisition: including the grab's three-axis acceleration, angular velocity (IMU), GPS / UWB coordinates, attitude angles (pitch, roll, yaw), grab angle (encoder), etc.; data is transmitted to the main control system via a high-speed serial port or CAN bus.

[0076] Data normalization and format unification: All physical quantities (angle, position, acceleration) are dimensionalized and standardized (such as unit conversion and normalization to the range [-1, 1]). All data are encapsulated as structured arrays or tensors to facilitate subsequent neural network processing.

[0077] Timestamp alignment: All image frames and sensor data acquisition modules are connected to a unified system clock (NTP server synchronization or PPS pulse synchronization can be used). The image acquisition frame rate (e.g., 30 Hz) and the sensor data sampling frequency (e.g., 100 Hz) are time-aligned through interpolation, downsampling, or Kalman filtering. This ensures that each image frame is accompanied by a set of multi-sensor data with a consistent timestamp.

[0078] Data fusion: Using synchronized timestamps, image features and sensor features are concatenated to construct a multi-dimensional input tensor. The input structure can be multi-channel input (image + numerical channel), or fused after parallel processing of branched neural networks.

[0079] Build multimodal input data that can be used for training or real-time inference; support the fusion of deep features and sensor features (such as intermediate fusion or post-fusion) in the improved YOLOv3-tiny model; and improve the robustness of the grab detection system in complex environments and the response accuracy of the control system.

[0080] A dilated convolutional layer is introduced into the original YOLOv3-tiny structure. Before the high-level and low-level features are fused, spatial alignment between features of different scales is achieved based on the feature pyramid alignment mechanism. The fused feature map is then convolved to generate the grab position and bounding box information. Specifically, the input image size is 416×416×3; it undergoes preprocessing (enhancement and normalization) and is input into the system together with the synchronized sensor data (see the aforementioned multimodal input construction).

[0081] Use the YOLOv3-tiny backbone network (a simplified version of Darknet-19) to extract shallow and deep features.

[0082] Shallow feature map (C3): from the lower convolutional layer, size is about 52×52×256;

[0083] Deep feature map (C5): comes from the end of the network, with a size of about 13×13×512.

[0084] Insert a dilated convolutional layer after the deep feature extraction module, such as dilated conv3×3 with a dilation rate of 2 or 4. This layer expands the receptive field without increasing parameters or significantly reducing resolution. It also enhances the model's ability to recognize large-scale objects (such as an expanded grab or a covered cargo pile). The output is an expanded perceptual feature map, with a size of 13×13×512.

[0085] In order to alleviate the spatial alignment error caused by dilated convolution, an FPN-style alignment mechanism is introduced before multi-scale feature fusion:

[0086] Use upsampling to upsample the deep feature map from 13×13 to 26×26 or 52×52;

[0087] At the same time, 1×1 convolution is performed on the shallow feature map to compress the number of channels and maintain channel consistency.

[0088] Use one of the following techniques to perform spatial alignment:

[0089] Deformable convolution: Automatically learn displacements to enhance spatial adaptation between different scales;

[0090] Attention mechanism (such as SE or CBAM): guides shallow layers to focus on high-level semantic regions;

[0091] Spatial Transformer Module (STN): transforms the spatial coordinates of shallow feature maps before merging;

[0092] Purpose: Ensure that feature maps of different scales are aligned in geometric coordinates to solve the deviation problem caused by void convolution.

[0093] The processed shallow and deep feature maps are fused by the following methods:

[0094] Element-wise addition or concatenation;

[0095] Followed by 3×3 standard convolution for fusion noise reduction;

[0096] A unified scale and information-rich fusion feature map is obtained, with a size of, for example, 52×52×512.

[0097] Use the YOLO detection head to further convolve the fused feature map:

[0098] The number of convolution kernel output channels is B × (5 + C), where B is the number of predicted boxes for each grid (e.g., 3), and C is the number of grab categories (if only grabs are identified, C = 1).

[0099] Outputs the position (x, y, w, h), confidence score, and category probability of each candidate box. x is the relative coordinate of the center point of the candidate box in the image width direction (in units of image width). It indicates the horizontal position of the center of the box. y is the relative coordinate of the center point of the candidate box in the image height direction (in units of image height). It indicates the vertical position of the center of the box. w is the width of the candidate box, which represents the width of the grab or target box relative to the width of the entire image. It is usually the normalized value of the candidate box width. h is the height of the candidate box, which represents the height of the grab or target box relative to the height of the entire image. It is usually the normalized value of the candidate box height.

[0100] The prediction results are then processed using non-maximum suppression (NMS) to remove overlapping boxes and a confidence threshold (e.g., 0.5). Three-dimensional position mapping can be performed in conjunction with sensor spatial information (e.g., GPS, IMU). Finally, the grab's two-dimensional image bounding box coordinates and spatial position estimate are output for use by the control system.

[0101] The improved YOLOv3-tiny model obtains the 2D bounding box coordinates (x, y, w, h) and detection confidence of the grab in each frame. If the pose estimation module is integrated, the grab's orientation and opening and closing state can also be extracted.

[0102] Multi-source sensor data acquisition: IMU sensors capture acceleration, angular velocity, and tilt angle data; GPS / UWB positioning obtains the grab bucket's spatial coordinates (X, Y, Z); encoders measure the grab bucket's rotation angle and spreader's swing angle; and laser ranging measures the distance between the grab bucket and the cargo / ground. Feature vector fusion synchronizes all of this data with a unified timestamp to construct a fused feature vector.

[0103] Input attention network module: Use the channel attention mechanism (SE module) to highlight key physical quantities, such as grab height, angular velocity changes, etc.; use the spatial attention mechanism to strengthen attention to the changed areas in the grab image of the current frame (such as tilt, stretch or offset).

[0104] State trend modeling: Input multi-frame feature vector sequences into a time series network with attention (such as LSTM + Self-Attention); learn the dynamic change trend of the grab state and identify irregular movement behaviors.

[0105] In a grab bucket state tracking system based on image recognition and multi-source sensor fusion, a set of systematic abnormality judgment rules must be established to monitor the grab bucket's operating status in real time and trigger control intervention in a timely manner when an abnormality is detected. Specifically, the following typical abnormal states and their judgment conditions are included:

[0106] Excessive bucket sway: If the bucket swings significantly left and right or forward and backward during lifting, it may cause problems such as bucket collision, misalignment, or structural fatigue. This type of abnormality can be determined by the tilt angle obtained by the attitude sensor (such as IMU):

[0107] When the roll angle (Roll) or pitch angle (Pitch) of the grab exceeds ±15° and lasts for more than 1 second, it is considered as abnormal grab swing; the system will trigger an early warning and slow down the operation speed or perform anti-swing action.

[0108] Grab not fully closed or stuck: If the grab is not fully closed or stuck during closing due to foreign objects, mechanical failure, or no load, this will result in a grab failure. This is determined by a significant decrease in the confidence level of the grab's closed state during image detection; and a simultaneous change in the grab's rotation angle (obtained by the encoder) below a set threshold, indicating that the expected mechanical movement did not occur. This combination of conditions is considered a stuck state, prompting manual inspection or a re-closing command.

[0109] Positioning drift anomaly: During dynamic operations, if the grab bucket position detected by imagery differs significantly from the coordinates reported by positioning systems such as GNSS or UWB, this may indicate a sensor failure or interference with the positioning signal. The judgment logic calculates the spatial distance between the predicted coordinates of the image center point and the GNSS positioning coordinates. If the error value continuously exceeds 30 cm (or other set threshold), a "positioning drift anomaly" is triggered. The system can temporarily disable some positioning signals and use backup data sources instead.

[0110] Abnormal posture fluctuation: If abnormal vibration or rotation is detected while the grab is stationary (such as hovering or awaiting instructions), it may indicate structural looseness, control errors, or external interference. This can be monitored using the IMU: If a sudden acceleration or angular velocity jump occurs within a short period of time while stationary (such as exceeding the set thresholds Δa or Δω), it is identified as "abnormal posture fluctuation." Such anomalies will trigger the dynamic compensation mechanism or suspend the current operation command.

[0111] Grab Failure Exception: If the grab bucket completes a full grab action but fails to grasp the material, this must be identified and corrected promptly. This is determined by the load cell or the change in the hoisted load after the grab action is complete. If the load mass is approximately zero after the grab bucket closes, indicating that the material was not grasped, a "Grab Failure" exception is triggered. The system can then re-identify the target position and initiate a re-grab command.

[0112] Abnormal collision risk: To prevent the grab from colliding with the ground, equipment or materials during its descent, the system needs to evaluate the distance and motion status between the grab and the obstacle in real time: if the distance measurement data at the bottom of the grab is lower than the set minimum safety distance (e.g. <30cm) and the current descent speed exceeds the threshold (e.g. >0.5m / s), it is considered a "collision risk"; at this time, the system should immediately perform emergency braking, alarm or path correction operations.

[0113] The detection and tracking results are input into the grab control system, which automatically generates accurate grab paths and posture adjustment instructions, and realizes the real-time control of the grab action process. Specifically:

[0114] Inputs include: the grab's bounding box coordinates (x, y, w, h) obtained from the improved YOLOv3-tiny network; the grab's dynamic trajectory in the image sequence (which can be tracked using Kalman filtering or an attention mechanism); spatial position information from multiple sensors (such as GNSS / UWB coordinates, IMU attitude angles, and encoder angles); and the location and status of the target material (such as the loading area outline and the target center point).

[0115] Target state analysis: Determine whether the current position of the grab is aligned with the center of the target material; analyze whether the grab's posture is suitable for grabbing (whether the angle is horizontal, whether it is swinging); if multiple targets are identified, select the optimal target area as the current grab object.

[0116] Target path planning inputs: Starting point: the current grab bucket's 3D coordinates and posture; Target point: the material surface or center (using image detection and laser ranging modeling); Constraints: obstacle avoidance zones, maximum speed / acceleration limits, and anti-sway control boundaries. Path generation algorithm: Utilizes common path planning methods (such as A*, RRT*, and DWA) combined with the grab bucket's operating conditions to construct a motion trajectory; Key Control Points (KPs) are inserted into the path, including descending segments, closing points, and ascending segments; and velocity curve fitting is performed on the path to achieve smooth starts and precise braking.

[0117] Based on the target direction, combined with the IMU attitude and visual correction results, the required attitude correction angles (Pitch, Roll) are calculated; the corresponding motor / cylinder action instructions are generated to achieve automatic horizontal or vertical adjustment of the grab.

[0118] The control system is based on a closed-loop structure. After receiving path and posture instructions, it generates control quantities (such as motor voltage and valve opening);

[0119] Use PID control, adaptive control or model predictive control (MPC) algorithms to adjust the execution unit to achieve precise control of position, speed, angle, etc.

[0120] All control commands are transmitted to the door operator execution unit via the CAN bus, EtherCAT or PLC instruction system. The control cycle is generally 50ms to 100ms, meeting the requirements of high-frequency dynamic response. The system has built-in buffering and emergency stop mechanisms to ensure immediate cessation of operation under abnormal circumstances.

[0121] The execution actions include: the grab moves to the target area; automatically leveling and falling vertically; the grab closes; and after completion, it automatically rises and transports to the target unloading position.

[0122] Feedback signal acquisition: Real-time monitoring of the grab bucket's position, posture, speed, and load; if the error deviation is found to be greater than the set tolerance, the trajectory correction mechanism is triggered; after the operation is completed, the status record is updated and fed back to the dispatching system or upper-level management platform.

[0123] Based on the grasping results and the posture deviation sent back by the sensor, the control model parameters are dynamically adjusted to continuously optimize the detection accuracy and control strategy. Specifically:

[0124] The lateral drift rate of the grab's motion trajectory is generated by analyzing the trajectory deviation amplitude and trend in the horizontal direction (especially the Y axis) when the grab moves from the starting point to the target position. The acquisition method is as follows:

[0125] The point sequences of the ideal motion trajectory and the actual motion trajectory are respectively represented as two-dimensional coordinate point sets sampled in time sequence. The ideal trajectory point sequence is , N is the total number of ideal trajectory points, and the actual trajectory point sequence is , M is the total number of actual trajectory points, where each point contains the coordinate values ​​of the X-axis and Y-axis.

[0126] For each pair of trajectory points, the absolute distance difference in the Y direction is calculated. This constructs a two-dimensional distance matrix that represents the lateral offset between any pair of ideal points and actual points. Each element represents the distance error between two points in the Y direction.

[0127] Using the minimum cumulative cost path algorithm from the DTW algorithm, dynamic programming is used to recursively calculate the minimum cumulative cost from the starting point (the first set of trajectory points) to any point pair. This cumulative cost is based on the previous state, and the minimum cost path of adjacent points is superimposed on the current Y-direction deviation until all point pairs are traversed to form a complete cost matrix.

[0128] After the cost matrix is ​​calculated, the optimal alignment path is extracted by backtracking from the end point of the cost matrix (i.e., the last point pair). This path is used to optimally match the actual trajectory points with the ideal trajectory points. This path can be used to extract the Y-direction offset of each point pair throughout the entire trajectory. The Y-direction deviations of all matching point pairs are averaged to obtain the lateral drift rate of the grab movement.

[0129] The terminal posture stabilization time is generated by analyzing the time it takes for the grab bucket to converge from the yaw state to the tolerance range (such as ±2°) before it descends and closes. The method for obtaining the time is as follows:

[0130] Collect continuous attitude data of the grab bucket from the IMU or attitude solution system: Pitch angle sequence (Pitch): , roll angle sequence (Roll): ; Define the angle error range allowed for the grab to stabilize, for example: , which means that when the attitude error is continuously within ±2°, it is considered stable. Set a sliding time window length , in seconds (e.g. = 1s), corresponding to several frames of data (such as 50 frames). At any time t, calculate the average absolute error of the posture in the past time window: Where, The average absolute error of the posture in the current window satisfies: < and < , that is, to judge that the grab bucket posture enters a stable state, and record the time ts from the beginning of descent to the first time the above conditions are met, then: ; Among them: t0 is the time point when the grab bucket starts to descend or start to swing; Tstable is the end posture stabilization time.

[0131] The lateral drift rate of the grab bucket's motion trajectory and the terminal attitude stabilization time are used as the input items of the fuzzy logic, and the error level of the gantry crane grab bucket control is used as the output item of the fuzzy logic.

[0132] Input variable 1: Lateral drift rate, which represents the average deviation of the grab bucket from the ideal trajectory in the Y-axis direction during movement. Its fuzzy language value is set as "small", "medium", and "large".

[0133] Input variable 2: End-position stabilization time, which refers to the time it takes for the grab bucket to converge from its yaw state to a set tolerance range (e.g., ±2°) after descending. Its fuzzy language value is set to "fast," "normal," or "slow."

[0134] Output variable: Control error level, which quantifies the control system's performance during the current process. Output values ​​range from 0 to 1, with fuzzy language values ​​of "low error," "medium error," and "high error."

[0135] Based on control experience and experimental data, a fuzzy rule base is constructed. The rule form is "if...then...", for example:

[0136] If the drift rate is "small" and the settling time is "fast", the control error level is "low error";

[0137] If the drift rate is "medium" and the settling time is "slow", the control error level is "high error";

[0138] If the drift rate is "large" and the settling time is "normal", the control error level is "high error".

[0139] By combining different input situations, a two-dimensional fuzzy reasoning table containing 9 core rules is established.

[0140] Fuzzy processing: The actually measured drift rate and stabilization time are input and mapped into the membership degree of the corresponding language value through the membership function.

[0141] Rule activation and weight calculation: Calculate the activation strength of all rules, usually using the "minimum value" or "multiplication method" for fuzzy intersection processing.

[0142] Fuzzy output aggregation: superimpose the output parts of all activated rules to synthesize the final fuzzy output set.

[0143] Defuzzification: The center of gravity method is used to defuzzify the fuzzy output set to obtain a clear value that represents the control error level.

[0144] According to the error level of the fuzzy system output, a hierarchical adjustment strategy is implemented to dynamically optimize the control model parameters:

[0145] Low error (output value ∈ [0, 0.3]): The control system is in good condition and maintains the current parameter configuration without any adjustment.

[0146] Mean error (output value ∈ (0.3, 0.7]): Appropriately increase the controller response gain, such as increasing the proportional term Kp in the path tracking PID; at the same time, extend the stable waiting time before the grab closing action to improve posture stability.

[0147] High error (output value ∈ (0.7, 1.0]): This indicates significant system deviation and requires stronger compensation mechanisms. For example, activate the trajectory offset adaptive compensation module, increase the derivative gain of the angle closed-loop controller, limit the maximum movement speed of the grab, and, if necessary, issue a system abnormality warning or perform recalibration.

[0148] The above fuzzy reasoning and parameter adjustment process can be embedded in the real-time scheduling process of the grab control system to form a closed-loop adaptive control mechanism of perception, evaluation, adjustment and feedback, and realize the optimization of dynamic control performance.

[0149] Example 2, please refer to Figure 2 As shown, the gantry crane grab control system based on multi-sensor fusion described in this embodiment includes an image acquisition module, an image preprocessing module, a grab visual detection module, a grab state tracking module, a path planning and motion control module, and an adaptive control optimization module;

[0150] Image acquisition module: Image acquisition equipment and multiple types of sensors are deployed in the gantry crane operating area to obtain the grab bucket's video images, posture information, position coordinates, and rotation angle data in real time;

[0151] Image preprocessing module: Enhances and normalizes the acquired video images, and synchronizes them with the sensor data at a unified timestamp to construct a multi-dimensional input dataset;

[0152] Grab bucket visual detection module: This module introduces a dilated convolutional layer into the original YOLOv3-tiny architecture. Before fusing high- and low-level features, it uses a feature pyramid alignment mechanism to achieve spatial alignment between features of different scales. It then performs convolution processing on the fused feature map to generate the grab bucket's position and bounding box information.

[0153] Grab bucket state tracking module: Based on image detection results and multi-source sensor position data, it uses an attention mechanism to continuously track the grab bucket state and detect anomalies.

[0154] Path planning and motion control module: Inputs detection and tracking results into the grab control system, automatically generates precise grab paths and posture adjustment instructions, and controls the grab motion in real time;

[0155] Adaptive control optimization module: Dynamically adjusts control model parameters based on the grasping results and the posture deviation sent back by the sensor, and continuously optimizes detection accuracy and control strategies.

[0156] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0157] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0158] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A gantry crane grab bucket control method based on multi-sensor fusion, characterized by: include: Image acquisition equipment and multiple types of sensors are deployed in the gantry crane operating area to obtain real-time video images, posture information, position coordinates, and rotation angle data of the grab bucket; Enhance and normalize the acquired video images and synchronize them with the sensor data at a unified timestamp to construct a multi-dimensional input dataset. A dilated convolutional layer is introduced into the original YOLOv3-tiny architecture. Before the high-level and low-level features are fused, the feature pyramid alignment mechanism is used to achieve spatial alignment between features of different scales. The fused feature map is then convolved to generate the grab position and bounding box information. The input received by the YOLOv3-tiny structure is a multi-dimensional input data set consisting of video images and sensor data after image enhancement and normalization. Based on image detection results and multi-source sensor position data, the attention mechanism is used to continuously track the grab state and detect abnormalities. The detection and tracking results are input into the grab control system to automatically generate accurate grab paths and posture adjustment instructions, and control the grab movement in real time; Based on the grasping results and the posture deviation sent back by the sensor, the control model parameters are dynamically adjusted to continuously optimize the detection accuracy and control strategy.

2. The method for controlling a gantry crane grab bucket based on multi-sensor fusion according to claim 1, characterized in that: The multi-type sensors include a posture sensor, a position coordinate sensor, a rotation angle sensor, and a laser ranging and obstacle avoidance sensor.

3. The method for controlling a gantry crane grab bucket based on multi-sensor fusion according to claim 1, characterized in that: The step of enhancing and normalizing the acquired video image and synchronously fusing it with the sensor data at a unified timestamp includes: Perform histogram equalization, gamma correction, filtering noise reduction and sharpening enhancement on the image; Normalize the image pixel values ​​to the range [0, 1] and unify the image size to the YOLOv3-tiny structure input requirements; Normalize and unify the dimensions of all sensor data and encapsulate them into a standardized tensor format; Align image frames and sensor data timestamps based on a unified clock system; Construct a multi-dimensional input dataset of joint image-sensor input by splicing or network fusion.

4. The method for controlling a gantry crane grab bucket based on multi-sensor fusion according to claim 1, characterized in that: The dilated convolutional layer is introduced into the original YOLOv3-tiny structure and spatial alignment is performed based on the feature pyramid alignment mechanism, including: Introducing dilated convolution after deep feature extraction; Use upsampling operations to upsample high-level feature maps to the same spatial resolution as shallow feature maps; Perform 1×1 convolution on the shallow feature map to match the number of channels; Before feature fusion, a deformable convolution or spatial transformation network module is introduced to spatially align the feature maps; The fused feature map is convolved using a 3×3 standard convolution to output a feature map of uniform scale for bounding box detection.

5. The method for controlling a gantry crane grab bucket based on multi-sensor fusion according to claim 4, characterized in that: The use of the attention mechanism to achieve continuous tracking of the grab bucket status and abnormality judgment includes: constructing a fusion feature vector, inputting a channel attention mechanism, introducing a spatial attention mechanism, using LSTM combined with a self-attention mechanism to build a state time series model to judge the dynamic stability trend of the grab bucket; and identifying abnormal states according to preset abnormality judgment rules.

6. The method for controlling a gantry crane grab bucket based on multi-sensor fusion according to claim 5, characterized in that: The automatic generation of accurate grasping paths and posture adjustment instructions includes: Based on the grab bucket's current position, target position, and obstacle distribution in the work area, a path planning algorithm is used to generate the optimal drop trajectory. Establish critical control points along the route, including grab points, closing points, and transfer points; Combine IMU attitude and visual detection results to calculate the required attitude adjustment angle; Output motor control voltage or hydraulic valve opening instruction to achieve real-time adjustment of grab bucket posture; The control period is set to no more than 100 milliseconds to meet high-frequency real-time control requirements.

7. The method for controlling a gantry crane grab bucket based on multi-sensor fusion according to claim 6, characterized in that: The horizontal drift rate of the grab's motion trajectory is generated by analyzing the trajectory deviation amplitude and trend in the horizontal direction during the grab's movement from the starting point to the target position. The acquisition method is as follows: The point sequences of the ideal motion trajectory and the actual motion trajectory are respectively represented as two-dimensional coordinate point sets sampled in time sequence, where the ideal trajectory point sequence is , N is the total number of ideal trajectory points, and the actual trajectory point sequence is , M is the total number of actual trajectory points, where each point contains the coordinate values ​​of the X-axis and Y-axis; for each pair of trajectory points, the absolute distance difference in the Y-axis direction is calculated, that is, a two-dimensional distance matrix is ​​constructed to represent the lateral offset between any pair of ideal points and actual points, and each element represents the distance error between two points in the Y direction; The minimum cumulative cost path algorithm in the DTW algorithm is used. Starting from the starting point, the minimum cumulative cost value from the starting point to any point pair is recursively calculated through dynamic programming. The cumulative cost is based on the previous state. The minimum cost path of the adjacent points is selected and superimposed on the current Y-direction deviation until all point pairs are traversed to form a complete cost matrix. After the cost matrix is ​​calculated, the optimal alignment path is extracted by backtracking from the end point of the cost matrix. This involves pairing the actual trajectory points with the ideal trajectory points at the optimal time. The Y-direction offset of each point pair in the entire trajectory is extracted through the path. The Y-direction deviations of all matching point pairs are averaged to obtain the lateral drift rate of the grab motion.

8. The method for controlling a gantry crane grab bucket based on multi-sensor fusion according to claim 7, characterized in that: The terminal posture stabilization time is generated by analyzing the time it takes for the grab bucket to converge from the yaw state to the tolerable range before it descends and closes. The method for obtaining the time is as follows: Collect continuous attitude data of the grab bucket from the IMU or attitude solution system: pitch angle sequence: , roll angle sequence: ; Set a sliding time window length , corresponding to several frames of data, at any time t, calculate the average absolute error of the posture in the past time window ; The average absolute error of the posture in the current window satisfies: and , that is, judging that the grab bucket posture enters a stable state, is the angle error range, and the time ts from the start of descent to the first time the condition is met is recorded, then: ; Among them: t0 is the time point when the grab bucket starts to descend or start to swing; Tstable is the end posture stabilization time.

9. The method for controlling a gantry crane grab bucket based on multi-sensor fusion according to claim 8, characterized in that: The lateral drift rate and attitude stabilization time are used as inputs of the fuzzy logic system; the control error level is used as the output of the fuzzy logic system, and graded parameter adjustments are performed according to the error level: in the low-error stage, the current control parameters are maintained; in the medium-error stage, the proportional control gain Kp is increased and the waiting time for stabilization is increased; in the high-error stage, the adaptive compensation mechanism is triggered to improve the response rate of the PID controller or activate the trajectory correction strategy.

10. A gantry crane grab bucket control system based on multi-sensor fusion, used to implement the gantry crane grab bucket control method based on multi-sensor fusion according to any one of claims 1 to 9, characterized in that: It includes image acquisition module, image preprocessing module, grab visual detection module, grab state tracking module, path planning and motion control module and adaptive control optimization module; Image acquisition module: Image acquisition equipment and multiple types of sensors are deployed in the gantry crane operating area to obtain the grab bucket's video images, posture information, position coordinates, and rotation angle data in real time; Image preprocessing module: Enhances and normalizes the acquired video images, and synchronizes them with the sensor data at a unified timestamp to construct a multi-dimensional input dataset; Grab bucket visual detection module: This module introduces a dilated convolutional layer into the original YOLOv3-tiny architecture. Before fusing high- and low-level features, it uses a feature pyramid alignment mechanism to achieve spatial alignment between features of different scales. It then performs convolution processing on the fused feature map to generate the grab bucket's position and bounding box information. Grab bucket state tracking module: Based on image detection results and multi-source sensor position data, it uses an attention mechanism to continuously track the grab bucket state and detect anomalies. Path planning and motion control module: Inputs detection and tracking results into the grab control system, automatically generates precise grab paths and posture adjustment instructions, and controls the grab motion in real time; Adaptive control optimization module: Dynamically adjusts control model parameters based on the grasping results and the posture deviation sent back by the sensor, and continuously optimizes detection accuracy and control strategies.

Images