Grab posture recognition method and grab anti-swing control method

Abstract

The present application relates to the technical field of bridge type grab ship unloader, more particularly to a grab posture recognition method and a grab anti-swing control method, the grab posture recognition method comprising: collecting grab operation image samples multiple times and forming an image dataset, the grab operation image samples comprising multiple frames of continuous grab operation images; labeling each grab operation image sample in the image dataset; constructing a grab posture recognition model, the grab posture recognition model comprising a MobileNetV3 backbone network, a time sequence feature fusion module, a coordinate solving module and a state classification module; training the constructed grab posture recognition model to obtain a trained grab posture recognition model; collecting multiple frames of continuous grab operation images to be predicted, inputting the preprocessed multiple frames of continuous grab operation images into the trained grab posture recognition model, and outputting a grab posture recognition result. The present application can overcome the limitation that single-frame recognition cannot determine the movement trend of the grab.

Description

Technical Field

[0001] This invention relates to the technical field of bridge-type grab unloaders, and more specifically, to a grab attitude recognition method and a grab anti-sway control method. Background Technology

[0002] With the continuous growth of global economic integration and international trade, ports, as key hubs in the logistics system, directly impact the competitiveness of regional economies through their operational efficiency and service capabilities. Bulk cargo terminals, especially those handling bulk commodities such as ore and coal, are facing multiple real-world challenges: First, the trend towards larger international shipping vessels demands higher unloading efficiency from terminals, while traditional manual operations, constrained by skill differences and fatigue cycles, struggle to maintain optimal pace. Second, equipment operation and maintenance costs remain high, especially in environments with high salt spray and high dust levels, leading to year-on-year increases in mechanical wear and energy consumption. Third, harsh working conditions such as obstructed visibility, ship swaying, and pervasive dust persist, resulting in frequent safety accidents such as grab bucket collisions with bulkheads, hopper blockages, and even personnel injuries. Fourth, the long training period for skilled drivers and continuously rising labor costs, coupled with the difficulty in recruiting port workers, further exacerbate the structural labor shortage.

[0003] Against this backdrop, the automation and intelligent transformation of port machinery has become an inevitable trend and core focus of the industry's development. As the main loading and unloading equipment at bulk cargo terminals, the traditional operation mode of the bridge grab unloader heavily relies on the operator's experience and skill level. The operator must be in a high-altitude cab, visually judging the distribution of materials in the hold, the grab's posture, and the ship's swaying, manually controlling multiple mechanisms such as the trolley, hoisting, and opening / closing to complete a series of high-intensity, highly repetitive, and complex actions, including material handling, lifting, shifting, and unloading. This mode has significant limitations: firstly, the efficiency of manual operation is constrained by driver fatigue and skill differences, making it difficult to maintain an optimal work cycle; secondly, the operating environment presents unfavorable factors such as obstructed vision, dust, and ship swaying, resulting in significant safety hazards and a high risk of accidents such as grab collisions with hold walls, hopper blockages, and even personal injury; thirdly, it is highly dependent on skilled operators, facing the practical problems of labor shortages and long training cycles.

[0004] In recent years, intelligent technologies, represented by artificial intelligence, machine vision, high-precision sensing, and advanced control theory, have provided a powerful impetus for the transformation of port machinery. In the field of automated ship unloaders, the focus of research has shifted from basic remote control and single-mechanism program control to fully automated, unmanned operation systems with environmental perception, autonomous decision-making, and intelligent execution capabilities. Among these, accurate attitude recognition of the grab bucket and anti-sway control during operation are two key technological bottlenecks for achieving efficient, stable, and safe automated ship unloading. Grab bucket attitude recognition is the core of the automated system's perception of the operational site. Traditional methods often rely on single-frame images for attitude recognition, which presents a problem in determining the grab bucket's movement trend.

[0005] Furthermore, anti-sway control aims to suppress the swaying of the grab bucket during horizontal movement and lifting. It is a prerequisite for achieving "flying unloading" (i.e., the grab bucket is precisely positioned above the hopper during movement) and "flying loading" (precisely loading into a designated point in the ship's hold), directly affecting the operation cycle time and positioning accuracy. Classical anti-sway control is mostly based on open-loop or simple closed-loop control using a pendulum mathematical model. However, during ship unloader operations, factors such as sudden changes in grab load, real-time changes in wire rope length, coupled motion of multiple mechanisms, and external wind loads and ship sway cause the system to exhibit strong nonlinearity, time-varying characteristics, and sensitivity to disturbances, limiting the adaptability of traditional methods. Summary of the Invention

[0006] The purpose of this invention is to overcome the problems of difficulty in judging the movement trend of the grab bucket in the existing grab bucket posture recognition method and the limited adaptability of the anti-sway control method, and to provide a grab bucket posture recognition method and a grab bucket anti-sway control method.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for recognizing the attitude of a grab bucket is provided, the method comprising the following steps: S1. Collect grab bucket operation image samples multiple times and form an image dataset, wherein the grab bucket operation image samples include multiple consecutive frames of grab bucket operation images; S2. Label each grab bucket operation image sample in the image dataset to obtain image labeling samples corresponding to each grab bucket operation image sample; S3. Preprocess the image annotation samples; S4. Construct a grab bucket attitude recognition model, which includes a MobileNetV3 backbone network, a temporal feature fusion module, a coordinate calculation module, and a state classification module connected in sequence. S5. Divide the image annotation samples into a training set, a validation set and a test set. Use the training set to train the grab posture recognition model, then use the validation set to evaluate the grab posture recognition model during the training process, and use the test set to test the effectiveness of the grab posture recognition model, so as to obtain the trained grab posture recognition model. S6. Acquire multiple consecutive frames of grab bucket operation images to be predicted, preprocess the multiple consecutive frames of grab bucket operation images to be predicted, input the preprocessed multiple consecutive frames of grab bucket operation images into the trained grab bucket posture recognition model, and output the grab bucket posture recognition result.

[0008] In the technical solution of this application, the grab bucket attitude recognition model includes a MobileNetV3 backbone network and a temporal feature fusion module. The grab bucket operation image samples include multiple consecutive frames of grab bucket operation images. After the multiple consecutive frames of grab bucket operation images are input into the grab bucket attitude recognition model, the grab bucket attitude recognition result can be output. Since multiple consecutive frames of grab bucket operation images are used as input, the limitation of single-frame recognition being unable to determine the grab bucket movement trend can be overcome.

[0009] Preferably, step S4 includes the following steps: S41. Input multiple consecutive grab bucket operation images from the image annotation samples into the MobileNetV3 backbone network. The MobileNetV3 backbone network extracts the spatial features of each grab bucket operation image and outputs the first feature map of each grab bucket operation image. S42. The output first feature maps are input into the temporal feature fusion module, and are calculated sequentially by the bidirectional LSTM network in the temporal feature fusion module along the forward and reverse time dimensions. The motion dependency between multiple consecutive frames of images is fused to obtain the corresponding second feature map. The second feature maps are spliced ​​together according to time steps and output to obtain the third feature map containing dynamic evolution information. S43. The obtained third feature map is input into the coordinate calculation module. The coordinate calculation module combines the depth constraints provided by the laser scanner to perform coordinate system conversion and geometric solution, and calculates the attitude angle of the grab and the three-dimensional coordinates of the center of mass in the world coordinate system in the third feature map in real time. The attitude angle of the grab includes at least the opening angle, horizontal yaw angle, pitch angle and roll angle. S44. Input the three-dimensional coordinates and attitude angles into the state classification module for processing, and output the state category of the grab bucket in the current "grab", "lift", "swing" or "release" operation stage.

[0010] Preferably, in step S44, the state classification module further outputs the confidence probability of the corresponding state category, wherein image labeled samples with a confidence probability greater than a positive threshold are defined as attitude-stable samples, and grab image samples with a confidence probability less than a negative threshold are defined as attitude-abnormal samples.

[0011] Preferably, in step S41, a MobileNetV3 backbone network is used to extract spatial features from each frame of the grab bucket operation image. Let the input grab bucket operation image be... The CNN network outputs the first feature tensor. ,in Includes the bounding box coordinates of the grab Key point set and semantic segmentation mask .

[0012] Preferably, in step S42, the CNN feature sequences corresponding to the consecutive frames of the grab operation images are... The input is fed into a bidirectional LSTM network, which contains two hidden layers, forward and backward, to capture past and future context information, respectively. Its hidden state update can be represented as:

[0013] The final output timing features are .

[0014] Another aspect of the present invention provides a grab bucket anti-sway control method, which applies the grab bucket posture recognition method described above. The grab bucket anti-sway control method includes the following steps: S100. Real-time sensing of the swing angle and motion state of the grab bucket; S200. Based on the swing angle and motion state of the grab bucket, establish a nonlinear dynamic model of the grab bucket-trolley. S300. Predictive control rolling optimization of the grab bucket-trolley nonlinear dynamics model: In each control cycle, using the grab bucket-trolley nonlinear dynamics model established in step S200, a series of optimal control commands are calculated by rolling the solution of the optimal problem in the future finite time domain. S400, a combined feedforward and feedback control, based on the control command calculated in step S300, suppresses the swing of the grab bucket within a safe range and makes it smoothly reach above the target unloading point. S500, Flight Unloading: After the grab reaches the target point, at the instant when the swing angle has been controlled within a very small range in step S400, the grab opens to unload the material.

[0015] In the technical solution of this application, the grab anti-sway control method includes real-time sensing of the grab's swing angle and motion state, establishing a grab-trolley nonlinear dynamic model based on the grab's swing angle and motion state, predictive control rolling optimization of the grab-trolley nonlinear dynamic model, feedforward and feedback composite control, and flight unloading and safety redundancy. This invention can achieve rapid anti-sway and precise positioning of the grab under complex working conditions.

[0016] Preferably, in step S100, the grab attitude recognition method outputs the grab's yaw angle, pitch angle, roll angle, and three-dimensional coordinates of its center of mass; the laser scanner serves as a supplement when dust is severe or there are sudden changes in lighting, providing distance constraints between the grab and the bulkhead / stockpile; the encoder and tilt sensor synchronously acquire the trolley displacement. Horizontal speed of the car The lifting height h and the wire rope length l(t) are used to finally fuse and output the swing angle vector. The motion state of the mechanism serves as the basic input for anti-sway control.

[0017] Preferably, in step S200, the system is simplified to a "cart-variable length pendulum" model, the grab bucket is considered a point mass, the wire rope is considered rigid and its mass is negligible, and the displacement of the cart is taken as... and grabber angle For generalized coordinates, the length of the wire rope The system kinetic energy is given by the time-varying parameter:

[0018] Substituting into the Lagrange equation, we obtain the nonlinear dynamic equation:

[0019]

[0020] Online identification of damping coefficient using visual swing angle data The equation is modified to include the grab bucket load mass m, and is further modified to include damping, where M represents the mass of the trolley. To accelerate the car, Let ω be the angular velocity of the pendulum. For the angular acceleration of the pendulum, This represents the rate of change of the wire rope length.

[0021] Preferably, in step S300, a state vector is defined. Control input Construct the cost function:

[0022] in This is the predicted value of the swing angle at time k. This is the control input at time k. The magnitude of the change in the control quantity at adjacent time points. , , , These are the weight coefficients of each penalty term in the cost function; Acceleration limit and swing angle safety boundary Under constraints, the optimal control sequence is solved by quadratic programming to execute the instruction at the current time.

[0023] Preferably, in step S400, a theoretical S-shaped velocity curve is generated as a feedforward. and visual angle With predicted swing angle deviation Input to PID controller, output The total control quantity is formed by superimposing it onto the MPC instructions. ,in It is the proportional coefficient of the PID controller. It is the integral coefficient of the PID controller. These are the derivative coefficients of the PID controller. It is the current time instruction in the optimal control instruction solved by MPC.

[0024] Compared with existing technologies, the beneficial effects of this invention are as follows: In the grab bucket attitude recognition method of this invention, the grab bucket attitude recognition model includes a MobileNetV3 backbone network and a temporal feature fusion module. The grab bucket operation image samples include multiple consecutive frames of grab bucket operation images. After the multiple consecutive frames of grab bucket operation images are input into the grab bucket attitude recognition model, the grab bucket attitude recognition result can be output. Since multiple consecutive frames of grab bucket operation images are used as input, the limitation that single-frame recognition cannot determine the grab bucket movement trend can be overcome. The grab bucket anti-sway control method of this invention breaks through the technical limitations of traditional grab bucket control of "low speed, stop, and passive correction". It has achieved significant improvements in four core dimensions: control accuracy, response speed, operation efficiency, and safety and reliability. It can realize rapid anti-swaying and precise positioning of the grab bucket under complex working conditions. It can be widely used in various grab bucket operation scenarios such as port ship unloaders and metallurgical casting cranes, and has strong practical value and broad industrialization prospects. Attached Figure Description

[0025] Figure 1 This is a flowchart of the grab bucket posture recognition method in this invention; Figure 2 This is an architecture diagram of the grab bucket attitude recognition model in the grab bucket attitude recognition method of the present invention; Figure 3 This is a flowchart of the grab bucket anti-sway control method in this invention. Detailed Implementation

[0026] The present invention will be further described below with reference to specific embodiments.

[0027] Example 1 like Figure 1 As shown, a grab bucket posture recognition method includes the following steps: S1. Collect grab bucket operation image samples multiple times and form an image dataset, wherein the grab bucket operation image samples include multiple consecutive frames of grab bucket operation images; S2. Label each grab bucket operation image sample in the image dataset to obtain image labeling samples corresponding to each grab bucket operation image sample; S3. Preprocess the image annotation samples; S4. Construct a grab bucket attitude recognition model, which includes a MobileNetV3 backbone network, a temporal feature fusion module, a coordinate calculation module, and a state classification module connected in sequence. S5. Divide the image annotation samples into a training set, a validation set and a test set. Use the training set to train the grab posture recognition model, then use the validation set to evaluate the grab posture recognition model during the training process, and use the test set to test the effectiveness of the grab posture recognition model, so as to obtain the trained grab posture recognition model. S6. Acquire multiple consecutive frames of grab bucket operation images to be predicted, preprocess the multiple consecutive frames of grab bucket operation images to be predicted, input the preprocessed multiple consecutive frames of grab bucket operation images into the trained grab bucket posture recognition model, and output the grab bucket posture recognition result.

[0028] In this embodiment, the architecture diagram of the grab attitude recognition model is as follows: Figure 2 As shown, the grab bucket attitude recognition model includes a MobileNetV3 backbone network and a temporal feature fusion module. The grab bucket operation image samples include multiple consecutive frames of grab bucket operation images. After the multiple consecutive frames of grab bucket operation images are input into the grab bucket attitude recognition model, the grab bucket attitude recognition result can be output. Since multiple consecutive frames of grab bucket operation images are used as input, the limitation of single-frame recognition in determining the grab bucket movement trend can be overcome.

[0029] It should also be noted that the core innovation of this embodiment in step S5 lies in constructing a multi-task collaborative training framework. The ground truth depth value provided by the laser scanner is used as an auxiliary supervision signal, and optimized synchronously with the main tasks of pose regression and keypoint detection. This allows the visual backbone network to consciously capture geometric texture information related to spatial depth while learning image appearance features, thereby effectively mitigating the adverse effects of port lighting changes and dust interference on the accuracy of monocular visual perception. Furthermore, by utilizing a large amount of readily available unlabeled monitoring video from the port, and employing the temporal prior of continuous pose changes between adjacent frames for self-supervised comparative pre-training, the model can achieve good feature representation capabilities even with limited labeled data, significantly reducing the dependence on the number of manually labeled samples.

[0030] It should be noted that the grab operation image sample includes multiple consecutive frames of grab operation images. This means that a single grab operation image sample comprises multiple consecutive frames of grab operation images. Inputting multiple consecutive frames overcomes the limitation of single-frame recognition in determining the grab's movement trend. To comprehensively capture the grab's position (X, Y, Z coordinates) and attitude angles (yaw, pitch, roll) in three-dimensional space during operation—that is, the complete six-degree-of-freedom pose—and also the grab's opening and closing state, while overcoming interference from changes in lighting, dust, and water mist in the port environment, a corresponding visual perception system is installed. This visual perception system can also be used to acquire grab operation image samples. Specifically, the visual perception system includes a core dustproof and waterproof wide-angle industrial camera and a two-dimensional laser scanner arranged above the crossbeam of the gantry grab crane. The former provides a global top-down view with a resolution of 3840×2160 and a frame rate of 30fps, primarily responsible for real-time monitoring of the grab's two-dimensional position (XY coordinates), opening and closing status, and the relative distance between the grab and the ship's bulkhead. The latter obtains the surface contour of the material below the grab and the height information of the grab's lower edge through cross-sectional scanning, providing depth constraints for pose calculation in the vertical direction (Z-axis). Two narrow-angle high dynamic range (HDR) cameras are symmetrically arranged on movable trolleys on both sides of the ship's bulkhead, observing the grab from a side viewpoint. They focus on capturing the grab's pitch angle (forward and backward tilt) and roll angle (lateral twist) in space, complementing the top-down information from the beam camera to jointly form a complete perception of the grab's six degrees of freedom attitude. At the same time, each camera is equipped with a high-brightness 850nm infrared LED supplementary light and integrates an automatic cleaning and defogging device as well as automatic exposure and white balance adjustment functions to cope with the harsh working conditions commonly encountered in ports, such as dust, fog, and insufficient light. By combining global beam perception with local attitude capture of the cabin side, and multimodal fusion of visual cameras and lidar, this system achieves comprehensive and high-precision monitoring of grab bucket operations, providing reliable perception support for automated and unmanned operations of port loading and unloading.

[0031] In this embodiment, the visual perception system can continuously acquire images of the grab bucket operation at a frame rate of 30fps and transmit them to the industrial control computer in real time via a gigabit Ethernet interface.

[0032] Step S4 includes the following steps: S41. Input multiple consecutive grab bucket operation images from the image annotation samples into the MobileNetV3 backbone network. The MobileNetV3 backbone network extracts the spatial features of each grab bucket operation image and outputs the first feature map of each grab bucket operation image. S42. The output first feature maps are input into the temporal feature fusion module, and are calculated sequentially by the bidirectional LSTM network in the temporal feature fusion module along the forward and reverse time dimensions. The motion dependency between multiple consecutive frames of images is fused to obtain the corresponding second feature map. The second feature maps are spliced ​​together according to time steps and output to obtain the third feature map containing dynamic evolution information. S43. The obtained third feature map is input into the coordinate calculation module. The coordinate calculation module combines the depth constraints provided by the laser scanner to perform coordinate system conversion and geometric solution, and calculates the attitude angle of the grab and the three-dimensional coordinates of the center of mass in the world coordinate system in the third feature map in real time. The attitude angle of the grab includes at least the opening angle (0°~180°), horizontal yaw angle, pitch angle and roll angle. S44. Input the three-dimensional coordinates and attitude angles into the state classification module for processing, and output the state category of the grab bucket in the current "grab", "lift", "swing" or "release" operation stage.

[0033] In addition, in step S44, the state classification module also outputs the confidence probability of the corresponding state category, wherein image labeled samples with a confidence probability greater than a positive threshold are defined as attitude-stable samples, and grab image samples with a confidence probability less than a negative threshold are defined as attitude-abnormal samples.

[0034] In step S41, the MobileNetV3 backbone network is used to extract spatial features from each frame of the grab operation image. Let the input grab operation image be... The CNN network outputs the first feature tensor. ,in Includes the bounding box coordinates of the grab Key point set and semantic segmentation mask It should be noted that key points can be the grab's hinge point, the tip of the teeth, etc. The role of the CNN network here is to compress high-dimensional image information into low-dimensional, semantically rich spatial features, providing a foundation for subsequent temporal modeling.

[0035] Additionally, in step S42, the CNN feature sequences corresponding to the consecutive frames of the grab operation images are... The input is fed into a bidirectional LSTM network, which contains two hidden layers, forward and backward, to capture past and future context information, respectively. Its hidden state update can be represented as:

[0036] The final output timing features are The role of LSTM here is to model the temporal dependencies in the movement of the grab, effectively distinguishing dynamic processes such as "grabbing," "lifting," "swinging," and "releasing," overcoming the limitation of single-frame recognition in being unable to determine the movement trend.

[0037] It should also be noted that the model training and update module is based on a self-built dataset of 100,000 labeled images (covering different weather, lighting, and dust conditions). It uses transfer learning and data augmentation strategies (random cropping, color jittering, and simulated dust noise) to update the model weights regularly. The recognition accuracy on the test set reaches 98.2%, ensuring the model's generalization ability in the harsh environment of the port. Finally, the main controller module continuously monitors the operating status and inference latency of the above modules. When the recognition confidence is lower than the threshold or the single frame processing exceeds the limit, it automatically switches to the laser ranging assisted mode or triggers system self-check and alarm.

[0038] Example 2 This embodiment 2 is similar to embodiment 1, except that in this embodiment 2, the grab bucket attitude recognition model also includes an anomaly warning module. The anomaly warning module judges the risk level in real time based on the grab bucket boundary box, hinge point coordinates, bucket tooth tip coordinates and semantic segmentation mask data, combined with a preset rule engine (bucket buried: opening angle <15° and sinking exceeds the threshold; bucket tipping: roll angle >20° or pitch angle >25°; proximity collision: distance to bulkhead <1.5m), and issues an audible and visual alarm or sends a deceleration / emergency stop command to the control layer within 20ms when an anomaly is triggered.

[0039] Example 3 like Figure 3 As shown, a grab bucket anti-sway control method, applying the grab bucket attitude recognition method of embodiments 1 and 2 above, includes the following steps: S100. Real-time sensing of the swing angle and motion state of the grab bucket; S200. Based on the swing angle and motion state of the grab bucket, establish a nonlinear dynamic model of the grab bucket-trolley. S300. Predictive control rolling optimization of the grab bucket-trolley nonlinear dynamics model: In each control cycle, using the grab bucket-trolley nonlinear dynamics model established in step S200, a series of optimal control commands are calculated by rolling the solution of the optimal problem in the future finite time domain. S400, a combined feedforward and feedback control, based on the control command calculated in step S300, suppresses the swing of the grab bucket within a safe range and makes it smoothly reach above the target unloading point. S500, Flight Unloading: After the grab reaches the target point, at the instant when the swing angle has been controlled within a very small range in step S400, the grab opens to unload the material.

[0040] In this embodiment, the grab anti-sway control method includes real-time sensing of the grab's swing angle and motion state, establishing a grab-trolley nonlinear dynamic model based on the grab's swing angle and motion state, predictive control and rolling optimization of the grab-trolley nonlinear dynamic model, feedforward and feedback composite control, and flight unloading and safety redundancy. This invention can achieve rapid anti-sway and precise positioning of the grab under complex working conditions.

[0041] It should be noted that in step S300, within each control cycle, the dynamic model established in step S200 is used to solve the optimal problem in the future finite time domain by rolling the solution, and a series of optimal control commands are calculated. The control commands include control increments or control quantity sequences, such as the target acceleration of the vehicle.

[0042] In step S400, the optimal control command output from step S300 is directly received and used as a core component of the "feedforward control quantity". The main task of step S400 is to suppress the swing of the grab bucket within a safe range and make it smoothly reach the target unloading point by adjusting the driving force in real time during the grab bucket movement (i.e., the trolley travel stage).

[0043] In step S500, after the grab reaches the target point, and taking advantage of the fact that the swing angle has been controlled within a very small range in step S400 (i.e., "stable hook" is successful), the grab opens to unload the material. If step S400 does not effectively suppress the swing (the swing angle is too large), the "flying unloading" in step S500 cannot be accurately positioned, and the material will scatter outside the target area. The control effect of step S400 directly determines whether S500 can be started smoothly.

[0044] In step S100, the grab attitude recognition method outputs the grab's yaw angle, pitch angle, roll angle, and three-dimensional coordinates of its center of mass; the laser scanner serves as a supplement in cases of severe dust or sudden changes in lighting, providing distance constraints between the grab and the bulkhead / stockpile; the encoder and tilt sensor synchronously acquire the trolley displacement. Horizontal speed of the car The lifting height h and the wire rope length l(t) are used to finally fuse and output the swing angle vector. The motion state of the mechanism serves as the basic input for anti-sway control.

[0045] Furthermore, in step S200, the system is simplified to a "cart-variable length pendulum" model, with the grab bucket considered as a point mass, the wire rope considered rigid and its mass negligible, and the displacement of the cart taken as... and grabber angle For generalized coordinates, the length of the wire rope The system kinetic energy is given by the time-varying parameter:

[0046] Substituting into the Lagrange equation, we obtain the nonlinear dynamic equation:

[0047]

[0048] Online identification of damping coefficient using visual swing angle data The equation is modified to include the grab bucket load mass m, and is further modified to include damping, where M represents the mass of the trolley. To accelerate the car, Let ω be the angular velocity of the pendulum. Let angular acceleration be the pendulum angle. This represents the rate of change of the wire rope length.

[0049] In step S300, a state vector is defined. Control input Construct the cost function:

[0050] in This is the predicted value of the swing angle at time k. This is the control input at time k. The magnitude of the change in the control quantity at adjacent time points. , , , These are the weight coefficients of each penalty term in the cost function; Acceleration limit and swing angle safety boundary Under constraints, the optimal control sequence is solved by quadratic programming to execute the instruction at the current time.

[0051] In addition, in step S400, a theoretical S-shaped velocity curve is generated as a feedforward. and visual angle With predicted swing angle deviation Input to PID controller, output The total control quantity is formed by superimposing it onto the MPC instructions. ,in It is the proportional coefficient of the PID controller. It is the integral coefficient of the PID controller. These are the derivative coefficients of the PID controller. It is the current time instruction in the optimal control instruction solved by MPC.

[0052] In step S500, during the flight unloading process, a safety redundancy design is also included. When the swing angle... If the delay exceeds 1 second, an unloading command will be issued without decelerating to zero. If a tipping bucket (roll angle > 20°) or a buried bucket is detected, an emergency stop will be executed; if vision fails, the system will automatically switch to the backup mode of laser + encoder.

[0053] Through steps S100 to S500 above, field tests showed that the maximum swing angle was reduced by 65%, the positioning time was shortened by 30%, and the operation cycle time was reduced by 22%, verifying the effectiveness of the anti-sway strategy.

[0054] To verify the effectiveness of the method, the intelligent control method was deployed on a bridge grab unloader at a port terminal and a three-month field industrial trial was conducted.

[0055] The experimental setup is as follows: The equipment uses a bridge-type grab unloader (rated load 40t, span 30m), and the sensors are a wide-angle industrial camera (1920×1080 resolution), a laser scanner (range measurement accuracy ±5mm), an tilt sensor, and an encoder. The control platform is an industrial PC (Intel i7, 16GB RAM, RTX3090 GPU), and the software environment is Ubuntu 18.04, ROSMelodic, and PyTorch 1.9. Detailed configuration is shown in the table below:

[0056] The experimental results are as follows Regarding attitude recognition accuracy: Under various conditions including sunny days, cloudy days, nighttime, and dusty environments, the visual recognition module achieved a 98.2% accuracy rate in recognizing the opening, tilting, and burying of the grab bucket, with a false alarm rate of less than 1.5%. Regarding anti-sway control effectiveness: Compared to manual operation and traditional PID control, this system reduced the maximum grab bucket swing angle by approximately 65%, shortened positioning time by approximately 30%, and reduced the average work cycle time by 22% during the "flying unloading" process. Regarding system stability: During 72 hours of continuous unattended operation, the system did not experience any major malfunctions or errors, demonstrating good industrial applicability. Specific experimental data are shown in the table below:

[0057] In the specific implementation of the above embodiments, the technical features can be combined in any non-contradictory way. For the sake of brevity, not all possible combinations of the above technical features are described. However, as long as the combination of these technical features is not contradictory, it should be considered to be within the scope of this specification.

[0058] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for recognizing the posture of a grab bucket, characterized in that, The grab attitude recognition method includes the following steps: S1. Collect grab bucket operation image samples multiple times and form an image dataset, wherein the grab bucket operation image samples include multiple consecutive frames of grab bucket operation images; S2. Label each grab bucket operation image sample in the image dataset to obtain image labeling samples corresponding to each grab bucket operation image sample; S3. Preprocess the image annotation samples; S4. Construct a grab bucket attitude recognition model, which includes a MobileNetV3 backbone network, a temporal feature fusion module, a coordinate calculation module, and a state classification module connected in sequence. S5. Divide the image annotation samples into a training set, a validation set and a test set. Use the training set to train the grab posture recognition model, then use the validation set to evaluate the grab posture recognition model during the training process, and use the test set to test the effectiveness of the grab posture recognition model, so as to obtain the trained grab posture recognition model. S6. Acquire multiple consecutive frames of grab bucket operation images to be predicted, preprocess the multiple consecutive frames of grab bucket operation images to be predicted, input the preprocessed multiple consecutive frames of grab bucket operation images into the trained grab bucket posture recognition model, and output the grab bucket posture recognition result.

2. The grab bucket posture recognition method according to claim 1, characterized in that, Step S4 includes the following steps: S41. Input multiple consecutive grab bucket operation images from the image annotation samples into the MobileNetV3 backbone network. The MobileNetV3 backbone network extracts the spatial features of each grab bucket operation image and outputs the first feature map of each grab bucket operation image. S42. The output first feature maps are input into the temporal feature fusion module, and are calculated sequentially by the bidirectional LSTM network in the temporal feature fusion module along the forward and reverse time dimensions. The motion dependency between multiple consecutive frames of images is fused to obtain the corresponding second feature map. The second feature maps are spliced ​​together according to time steps and output to obtain the third feature map containing dynamic evolution information. S43. The obtained third feature map is input into the coordinate calculation module. The coordinate calculation module combines the depth constraints provided by the laser scanner to perform coordinate system conversion and geometric solution, and calculates the attitude angle of the grab and the three-dimensional coordinates of the center of mass in the world coordinate system in the third feature map in real time. The attitude angle of the grab includes at least the opening angle, horizontal yaw angle, pitch angle and roll angle. S44. Input the three-dimensional coordinates and attitude angles into the state classification module for processing, and classify and output the state category of the grab bucket currently in the "grab", "lift", "swing" or "release" operation stage.

3. The grab bucket posture recognition method according to claim 2, characterized in that, In step S44, the state classification module also outputs the confidence probability of the corresponding state category, wherein image labeled samples with a confidence probability greater than a positive threshold are defined as attitude-stable samples, and grab image samples with a confidence probability less than a negative threshold are defined as attitude-abnormal samples.

4. The grab bucket posture recognition method according to claim 2, characterized in that, In step S41, the MobileNetV3 backbone network is used to extract spatial features from each frame of the grab operation image. Let the input grab operation image be... The CNN network outputs the first feature tensor. ,in Includes the bounding box coordinates of the grab Key point set and semantic segmentation mask .

5. The grab bucket posture recognition method according to claim 4, characterized in that, In step S42, the CNN feature sequences corresponding to the consecutive frames of the grab operation images are... The input is fed into a bidirectional LSTM network, which contains two hidden layers, forward and backward, to capture past and future context information, respectively. Its hidden state update can be represented as: The final output timing features are .

6. A grab bucket anti-sway control method, characterized in that, Based on the grab bucket posture recognition method according to any one of claims 1 to 5, the grab bucket anti-sway control method includes the following steps: S100. Real-time sensing of the swing angle and motion state of the grab bucket; S200. Based on the swing angle and motion state of the grab bucket, establish a nonlinear dynamic model of the grab bucket-trolley. S300. Predictive control rolling optimization of the grab bucket-trolley nonlinear dynamics model: In each control cycle, using the grab bucket-trolley nonlinear dynamics model established in step S200, the optimal problem in the future finite time domain is solved by rolling optimization to calculate a series of optimal control commands. S400, a combined feedforward and feedback control, based on the control command calculated in step S300, suppresses the swing of the grab bucket within a safe range and makes it smoothly reach above the target unloading point. S500, Flight Unloading: After the grab reaches the target point, at the instant when the swing angle has been controlled within a very small range in step S400, the grab opens to unload the material.

7. The grab bucket anti-sway control method according to claim 6, characterized in that, In step S100, the grab attitude recognition method outputs the grab's yaw angle, pitch angle, roll angle, and three-dimensional coordinates of its center of mass; the laser scanner serves as a supplement in cases of severe dust or sudden changes in lighting, providing distance constraints between the grab and the bulkhead / stockpile; the encoder and tilt sensor synchronously acquire the trolley displacement. Horizontal speed of the car The lifting height h and the wire rope length l(t) are used to finally fuse and output the swing angle vector. The motion state of the mechanism serves as the basic input for anti-sway control.

8. The grab bucket anti-sway control method according to claim 7, characterized in that, In step S200, the system is simplified to a "trolley-variable length pendulum" model, with the grab bucket considered as a point mass, the wire rope considered as rigid and massless, and the displacement of the trolley taken as... and grabber angle For generalized coordinates, the length of the wire rope The system kinetic energy is given by the time-varying parameter: Substituting into the Lagrange equation, we obtain the nonlinear dynamic equation: Online identification of damping coefficient using visual swing angle data The equation is modified to include the grab bucket load mass m, and is further modified to include damping, where M represents the mass of the trolley. To accelerate the car, Let ω be the angular velocity of the pendulum. Let angular acceleration be the pendulum angle. This represents the rate of change of the wire rope length.

9. The grab bucket anti-sway control method according to claim 8, characterized in that, In step S300, a state vector is defined. Control input Construct the cost function: in This is the predicted value of the swing angle at time k. This is the control input at time k. The magnitude of the change in the control quantity at adjacent time points. , , , These are the weight coefficients of each penalty term in the cost function; Acceleration limit and swing angle safety boundary Under constraints, the optimal control command is solved through quadratic programming, and the command at the current moment is executed.

10. The grab bucket anti-sway control method according to claim 9, characterized in that, In step S400, a theoretical S-shaped velocity curve is generated as a feedforward. and visual angle With predicted swing angle deviation Input to PID controller, output The total control quantity is formed by superimposing it onto the MPC instructions. ,in It is the proportional coefficient of the PID controller. It is the integral coefficient of the PID controller. These are the derivative coefficients of the PID controller. It is the current time instruction in the optimal control instruction solved by MPC.

Images