A hook-picking robot control system and method based on multi-source vision fusion

Abstract

The embodiment of the application provides a kind of based on the control method of hooking robot of multi-source vision fusion, it is applied to robot control technical field, and RGB image, depth image, infrared image and IMU data are collected by the multi-source perception system installed at the end of robot;Adopt double branch neural network to carry out feature fusion identification, output the boundary profile of hook and the three-dimensional coordinates of preferred gripping point;Through registration correction mechanism, visual coordinates are unified to robot base coordinate system;Based on vision and inertial signal, construct Transform prediction model, estimate the future pose change of hook;Generate feedforward control trajectory to offset hook swing, and trajectory correction is carried out in combination with visual servo feedback;Through path planning and inverse kinematics solution, joint instruction is generated, and precise hooking operation is completed by driving mechanical arm.The application effectively improves the recognition accuracy, anti-interference ability and operation success rate of hooking operation in complex lighting and dynamic environment.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a control system and method for a hook-removing robot based on multi-source vision fusion. Background Technology

[0002] With the rapid development of automation technology, robots are being used more and more widely in industrial environments, especially in ports, warehousing, and heavy machinery operations. Traditional unhooking operations rely on human experience, which, while capable of completing the task, has limitations in terms of environmental changes, precision requirements, and operational safety. With technological advancements, intelligent and automated unhooking robots are gradually becoming an important means of improving operational efficiency and safety.

[0003] Currently, most robot control systems on the market use a single sensor, employing visual information or inertial measurement signals to control the robot's end effector. However, these systems have certain drawbacks: First, a single vision system is susceptible to interference from changes in lighting, reflections, and occlusions, leading to unstable recognition results; second, inertial measurement units are affected by mechanical vibrations and magnetic interference, resulting in significant noise in the sensor data; finally, existing control systems still have limited capabilities in dynamic compensation and precise positioning, especially in complex environments where the hook target swings significantly, traditional control methods struggle to effectively compensate for the hook's swing and ensure high-precision hook unhooking operations.

[0004] To address the aforementioned issues, a control method for a hook-unhooking robot based on multi-source visual fusion and a feedforward-feedback collaborative control mechanism is proposed. By utilizing information from multiple sources of sensors, including RGB, depth, infrared, and inertial measurement units, and combining advanced deep learning models and predictive control algorithms, high-precision hook identification, prediction, and control are achieved. Summary of the Invention

[0005] The embodiments of this application provide a control system and method for a hook-removing robot based on multi-source vision fusion, realizing a new method for closed-loop adaptive control, driving the hook-removing robot to achieve truly safe, efficient, and reliable autonomous operation. To achieve the above objectives, this application adopts the following technical solution:

[0006] A control method for a hook-unhooking robot based on multi-source vision fusion, applied to automatic hook-unhooking operations in industrial lifting environments, includes the following steps:

[0007] The system acquires RGB images, depth images, infrared images, and inertial measurement unit data of the unhooking operation area to form multi-source sensing data.

[0008] Based on the multi-source sensing data, a dual-branch neural network is constructed;

[0009] The first branch of the dual-branch neural network is used to perform semantic segmentation processing on the RGB image and the depth image to generate a hook region mask and semantic feature vector.

[0010] The material reflectance feature map is extracted from the infrared image through the second branch of the dual-branch neural network, and the center of gravity distribution feature vector of the hook is calculated from the three-dimensional point cloud data derived from the depth image.

[0011] The semantic feature vector and the centroid distribution feature vector are concatenated along the feature dimension, and the concatenated features are dimensionality reduced and fused through a fully connected layer to obtain the fused features.

[0012] The fused features are input into the regression head network for processing, and the boundary contour of the hook and the coordinates of the gripping point are output.

[0013] The end effector posture is calculated based on the robot's kinematics model. Based on the end effector posture, the registration parameters of the vision system that collects the multi-source perception data are uniformly updated to generate a disparity correction matrix.

[0014] The parallax correction matrix is ​​used to uniformly transform the hook coordinates of multi-source visual perception to the robot base coordinate system, thereby obtaining spatially consistent hook coordinate information.

[0015] The visual feature sequence output by the fusion recognition model is fused with the motion signal of the inertial measurement unit across modes to establish a hook motion prediction model;

[0016] Based on the hook pose change trend output by the hook motion prediction model, a feedforward control trajectory is generated to compensate for the swing deviation.

[0017] During the movement of the end effector along the feedforward control trajectory, a continuous image sequence of a local area of ​​the hook is acquired; based on the continuous image sequence, the pixel deviation between adjacent image frames is calculated;

[0018] The pixel deviation is integrated over time to generate a micro-deviation matrix. The micro-deviation matrix is ​​then compared with a preset ideal unhooking trajectory to solve for the pose correction of the end effector.

[0019] Based on the pose correction amount, spatially consistent hook coordinate information, and feedforward control trajectory, the final motion path of the end effector is generated. The final motion path is then used to generate a hook removal operation command to control the end effector to complete the hook removal action.

[0020] Specifically: the calculation of the center-of-gravity distribution feature vector of the hook from the 3D point cloud data derived from the depth image includes:

[0021] The hook region mask is applied to the three-dimensional point cloud data for segmentation to obtain hook point cloud clusters;

[0022] Calculate the centroid coordinates of the hook point cloud cluster;

[0023] A local coordinate system is established with the centroid coordinates as the origin;

[0024] In the local coordinate system, the hook point cloud cluster is divided into multiple three-dimensional voxel meshes;

[0025] Calculate the point cloud density distribution within each voxel grid and calculate the Euclidean distance from the center of each voxel grid to the origin;

[0026] The point cloud density distribution and the Euclidean distance are weighted and summed to generate a multidimensional vector characterizing the hook mass distribution and inertia tensor properties, which serves as the centroid distribution feature vector.

[0027] Specifically: the step of performing cross-modal fusion of the visual feature sequence output by the fusion recognition model and the motion signal from the inertial measurement unit to establish a hook motion prediction model includes:

[0028] The visual feature sequence and the angular velocity and acceleration signals output by the inertial measurement unit are time-stamp aligned and normalized to obtain aligned and normalized multimodal time series data;

[0029] The multimodal temporal data is input into an attention-based Transformer encoder;

[0030] Inside the Transformer encoder, its self-attention module is used to establish the temporal dependencies within the visual feature sequence, and its cross-attention module is used to construct the cross-modal correlation between visual features and inertial signals, outputting a fused temporal representation.

[0031] The fused temporal representation is input into a multilayer perceptron regressor. Through forward propagation calculation of the multilayer perceptron regressor, the pose changes of the hook in six degrees of freedom within a specific future time window are predicted.

[0032] Specifically: The process of uniformly updating the registration parameters of the vision system that acquires the multi-source perception data based on the end effector posture includes:

[0033] Based on the robot's forward kinematics model and joint encoder feedback, the robot's forward kinematics calculation is performed to obtain the current homogeneous transformation matrix of the end effector in the robot's base coordinate system.

[0034] Based on the pre-stored hand-eye calibration results and the current homogeneous transformation matrix, the latest extrinsic parameter matrix of each visual sensor relative to the robot's base coordinate system is calculated one by one through matrix chain multiplication.

[0035] The latest extrinsic parameter matrix is ​​used as the new registration parameter and sent down to update the coordinate transformation module corresponding to the vision system;

[0036] Based on the unified updated registration parameters of the vision system, a disparity correction matrix is ​​generated.

[0037] Specifically, it also includes:

[0038] Detect the attitude of the end effector;

[0039] When the cumulative change in the position or attitude angle of the end effector since the last update is detected to exceed a preset threshold, a trigger signal is generated.

[0040] The trigger signal is sent to the vision system to initiate the unified update process of the vision system's registration parameters.

[0041] Specifically: the step of performing time-series cumulative integration on the pixel deviation to generate a micro-deviation matrix, comparing the micro-deviation matrix with a preset ideal unhooking trajectory, and solving for the pose correction of the end effector includes:

[0042] The motion of hook feature points between adjacent frames in the continuous image sequence is calculated using optical flow to obtain pixel-level displacement vectors.

[0043] The displacement vector is integrated and accumulated over time to generate a micro-deviation matrix characterizing the cumulative error of the system.

[0044] The preset ideal unhooking trajectory is projected onto the image plane based on the camera intrinsic parameters and the current extrinsic parameter matrix to generate the corresponding ideal image trajectory.

[0045] At consecutive time points, the difference between the micro-deviation matrix and the ideal image trajectory is calculated to form a difference sequence;

[0046] The difference sequence is input into a proportional-integral controller, and the controller calculates the initial correction amount in the image coordinate system.

[0047] The initial correction amount is subjected to an inverse coordinate transformation to solve for the pose correction amount of the end effector in the robot base coordinate system.

[0048] Specifically: the step of generating the unhooking operation command using the final motion path includes:

[0049] The spatially consistent hook coordinate information is set as the final target point, and the feedforward control trajectory is solved as the desired path point sequence in Cartesian space;

[0050] The pose correction amount is used as a disturbance compensation value and superimposed on the desired path point sequence to form the corrected end effector motion trajectory.

[0051] The corrected end effector motion trajectory is converted into an angle command sequence for each joint of the robot using the robot inverse kinematics algorithm, generating the final unhooking operation command.

[0052] A control system for a hook-removing robot based on multi-source vision fusion, the system comprising:

[0053] The multi-source sensing data acquisition module is used to acquire RGB images, depth images, infrared images, and inertial measurement unit data of the unhooking operation area to form multi-source sensing data.

[0054] The fusion recognition module is used to extract semantic features and physical attribute features of the hook from the RGB image, depth image and infrared image based on the multi-source perception data through joint optimization, input the semantic features and physical attribute features into the pre-trained fusion recognition model, and output the boundary contour of the hook and the coordinates of the grab point.

[0055] The registration parameter update module is used to calculate the end effector posture based on the robot kinematics model, and to uniformly update the registration parameters of the vision system that collects the multi-source perception data according to the end effector posture, thereby generating a disparity correction matrix.

[0056] The coordinate transformation module is used to uniformly transform the hook coordinates of multi-source visual perception to the robot base coordinate system through the parallax correction matrix, so as to obtain spatially consistent hook coordinate information;

[0057] The motion prediction module is used to perform cross-modal fusion of the visual feature sequence output by the fusion recognition model and the motion signal of the inertial measurement unit to establish a hook motion prediction model.

[0058] The feedforward control module is used to generate a feedforward control trajectory to compensate for swing deviation based on the hook pose change trend output by the hook motion prediction model.

[0059] The visual servo module is used to acquire a continuous image sequence of a local area of ​​the hook during the movement of the end effector along the feedforward control trajectory; and to calculate the pixel deviation between adjacent image frames based on the continuous image sequence.

[0060] The trajectory correction module is used to perform time-series cumulative integration on the pixel deviation to generate a micro-deviation matrix, compare the micro-deviation matrix with the preset ideal unhooking trajectory, and solve for the pose correction amount of the end effector.

[0061] The path planning module is used to generate the final motion path of the end effector based on the pose correction amount, spatially consistent hook coordinate information, and feedforward control trajectory.

[0062] The instruction generation and execution module is used to generate a hook removal operation instruction using the final motion path, and control the end effector to complete the hook removal action.

[0063] Specifically: the fusion recognition module includes a dual-branch neural network structure, wherein:

[0064] The first processing branch is used to perform semantic segmentation processing on the RGB image and the depth image to generate a hook region mask and a semantic feature vector.

[0065] The second processing branch is used to extract the material reflectance feature map from the infrared image and calculate the center of gravity distribution feature vector of the hook from the three-dimensional point cloud data derived from the depth image.

[0066] The feature fusion unit is used to concatenate the semantic feature vector and the centroid distribution feature vector in the feature dimension, and then perform dimensionality reduction and fusion on the concatenated features through a fully connected layer to obtain the fused features;

[0067] The coordinate regression unit is used to input the fused features into the regression head network for processing and output the boundary contour of the hook and the coordinates of the grab point.

[0068] As can be seen from the above technical solution, this application has the following beneficial effects:

[0069] 1. This method utilizes a multi-source sensing system and a dual-branch deep fusion recognition network, comprehensively leveraging texture, geometry, thermal radiation, and inertial dynamic information. A unique adaptive attention weight matrix adjusts the confidence level of each sensor's information based on conditions such as lighting and occlusion; for example, it relies heavily on infrared under strong light and emphasizes RGB under weak light, while also utilizing point cloud computing to identify physical center-of-gravity features. This deep, complementary fusion mechanism enables the system to stably and accurately identify hooks and calculate their 3D coordinates even in the face of extreme interference such as strong light, reflections, shadows, and smoke, solving the problems of interference susceptibility and inaccurate positioning inherent in traditional solutions.

[0070] 2. This method does not passively respond; instead, it actively predicts the future trajectory of the hook using a vision-inertial fusion Transformer prediction model. Based on this, a feedforward control trajectory is generated, allowing the robot to anticipate and wait for the hook. The visual servo feedback loop calculates and integrates pixel deviations using optical flow to generate millisecond-level pose corrections. This collaborative mechanism enables the robot to maintain spatial synchronization with the swinging hook, improving the capture success rate and stability in dynamic scenarios. By optimizing prediction and control performance through online incremental learning, it solves the core pain points of traditional solutions, such as lag and inability to handle swinging motions. Attached Figure Description

[0071] The invention will now be further described with reference to the accompanying drawings.

[0072] Figure 1 A first flowchart provided for an embodiment of this application;

[0073] Figure 2 A second flowchart provided for embodiments of this application;

[0074] Figure 3 A third flowchart provided for embodiments of this application;

[0075] Figure 4 The fourth flowchart provided for the embodiments of this application. Detailed Implementation

[0076] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are for distinguishing different objects, not for specifying a particular order.

[0077] In the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0078] Research has shown that hook removal still heavily relies on manual labor, resulting in low efficiency and high risk. Existing vision-based automated hook removal robot technology has not performed well in practical applications.

[0079] To address the aforementioned issues, this application provides a control system and method for a hook-removing robot based on multi-source vision fusion: Example

[0080] To solve the above problems, such as Figure 1As shown, the control method for a hook-removing robot based on multi-source vision fusion in this embodiment mainly consists of a multi-source perception system, a fusion recognition module, a registration and correction module, and a feedforward control and visual servo module, providing a control system and method implementation scheme for a hook-removing robot based on multi-source vision fusion.

[0081] In this embodiment, the multi-source sensing system is deployed on the robot's end effector and includes a color camera, a depth camera, an infrared thermal imager, and an inertial measurement unit (IMU). Before performing the hook removal operation, the robot control system first activates the multi-source sensing system to scan the removal operation area in real time, acquiring RGB images, depth images, infrared images, and IMU data. The RGB images are used to capture the hook's surface texture, the depth images reflect the spatial depth relationship between the hook and the background, the infrared images detect the temperature difference reflection characteristics of the metal surface, and the IMU data records the robot's end effector's acceleration and angular velocity information to assist in attitude estimation.

[0082] After receiving multi-source sensing data, the system first identifies the hook target using a jointly optimized feature extraction network. This network consists of a dual-branch neural structure composed of an RGB-depth channel and an infrared channel. The RGB-depth channel performs semantic segmentation, generating a semantic mask and semantic feature vector for the hook. The infrared channel extracts material reflectivity feature maps from thermal imaging data and calculates the hook's center of gravity distribution features based on point data derived from the depth image. Subsequently, the system concatenates the semantic feature vector and the center of gravity distribution feature vector along the feature dimension, performs dimensionality reduction and fusion through a fully connected layer to form a fused feature, and outputs the hook's boundary contour and the coordinates of the grab point by the regression head network.

[0083] In this embodiment, the physical characteristics and inertia tensor information of the hook are obtained by weighting point cloud density and Euclidean distance to enhance the stability of the gripping point. Specifically, the system establishes a local coordinate system in the local point cloud cluster of the hook, calculates the density distribution within the voxel grid with the centroid as the origin, and weights and sums the distances from the grid center to the origin to form a multidimensional centroid distribution vector. This vector, together with the semantic segmentation result, is input into the fusion recognition model, significantly reducing recognition errors caused by reflections, partial occlusion, or background interference.

[0084] To ensure that the recognition results are consistent with the robot's coordinate system, this embodiment introduces a registration and correction mechanism driven by the end effector's posture. The robot control unit calculates the homogeneous transformation matrix of the end effector based on the current joint angles and forward kinematics, and calculates the extrinsic parameter matrices of each vision sensor in real time by combining hand-eye calibration data. The registration parameters of the vision system are updated through matrix chain multiplication to generate a unified disparity correction matrix. Using this correction matrix, the hook coordinates detected in RGB, depth, and infrared images can be uniformly transformed to the robot's base coordinate system, forming a spatially consistent hook pose description.

[0085] Based on this, the control system establishes a hook motion prediction model. This model performs time alignment and normalization processing on the visual feature sequence output by the fusion recognition module and the IMU inertial signal, and inputs it into the Transformer encoder based on the attention mechanism. Through the self-attention structure, it learns the visual temporal dependency, and through the cross-attention structure, it captures the dynamic correlation between visual features and inertial signals, thereby predicting the hook's pose change trend within future time windows.

[0086] The system generates a feedforward control trajectory to counteract hook sway based on the prediction results. This trajectory can compensate for the hook's inertial offset in advance, ensuring that the end effector's motion dynamically matches the actual sway trend of the hook in space. During the end effector's execution along the feedforward trajectory, the system continuously acquires a sequence of images of the hook's local area. It calculates the pixel displacement of hook feature points between adjacent frames using an optical flow algorithm and performs temporal integration on the pixel deviation to generate a micro-deviation matrix. The difference between this matrix and the ideal hook unhooking trajectory is calculated, and the correction amount in the image coordinate system is obtained through a proportional-integral controller. Finally, the pose correction amount in the robot's base coordinate system is obtained through inverse coordinate transformation.

[0087] The system inputs the pose correction, spatially consistent hook coordinates, and feedforward trajectory into the path planning module to generate the final end effector motion path. This path is optimized for smoothness and dynamic feasibility to ensure the continuity and stability of the robotic arm in complex spaces. Subsequently, the controller uses an inverse kinematics algorithm to convert this motion path into joint angle commands, generating hook removal commands. The actuator then performs precise alignment, insertion, and removal actions according to these commands, achieving automatic hook removal.

[0088] In the specific operation of this embodiment, if the position or attitude angle of the end effector changes beyond a preset threshold, the system will automatically trigger the registration parameter update process to ensure that the multi-source coordinates of the vision system are always consistent with the motion state of the robotic arm.

[0089] Using the methods described above, the hook-unhooking robot of this application can achieve high-precision identification, dynamic prediction, and adaptive control of hook targets in complex lifting environments. Compared with existing technologies, this solution has significant advantages in terms of resistance to light interference, dynamic response accuracy, and spatial coordinate consistency, which can significantly improve the success rate and safety of automatic hook-unhooking operations. Example

[0090] like Figure 2Specifically, this embodiment details the core algorithm model in the unhooking robot control system. In complex industrial environments such as port hoisting, the visual features of the hook are easily affected by factors such as sudden changes in illumination, partial occlusion, and reflections from metal surfaces, while the inertial measurement unit signal suffers from mechanical vibration noise and magnetic interference. Therefore, this embodiment proposes a joint feature encoding and prediction architecture based on multimodal complementary learning, focusing on the specific implementation methods of the multi-source visual feature fusion mechanism, inertial signal enhancement processing, and high-precision pose prediction model.

[0091] The system employs a dual-branch coding structure. The first branch processes the fusion information of the RGB image and the depth image, while the second branch collaboratively processes the infrared image and inertial measurement data.

[0092] The first branch (RGB-Depth Fusion Branch) employs an improved multi-scale dual-stream convolutional neural network, with its backbone based on the ResNet-50 architecture. This branch features two parallel input streams: an RGB image stream and a depth image stream. In the shallow layers of the network, feature extraction is performed on both streams separately. In the third stage of the network, a cross-modal feature fusion module concatenates and convolves the feature maps of both streams, achieving early fusion of RGB texture information and depth geometric information. This fusion module introduces deformable convolutions at different scales to adapt to the morphological changes of the hook at different distances and angles. Finally, this branch outputs a 512-dimensional semantic feature vector that fuses appearance and geometric information. Specifically, a channel attention mechanism is introduced into the network, enabling the model to adaptively focus on feature channels related to the hook region, effectively suppressing interference from complex backgrounds.

[0093] The second branch (infrared-inertial fusion branch) designs a cooperative network for processing non-visible light information and motion signals. This branch consists of two subsystems:

[0094] Infrared Feature Extraction Subsystem: This subsystem employs a convolutional network incorporating residual connections and channel attention mechanisms to process infrared images. Since infrared images reflect the thermal radiation characteristics of objects, this network places particular emphasis on extracting temperature distribution features. By analyzing the temperature difference pattern between the hook and its surrounding environment, this subsystem can effectively identify hook areas that are difficult to discern in visible light images due to reflections, and output a material reflectivity feature map.

[0095] The inertial signal processing subsystem employs a temporal convolutional network to process the IMU's acceleration and angular velocity data. This network comprises eight temporal convolutional layers, each equipped with a random dropout mechanism to prevent overfitting. To handle high-frequency noise in the IMU signal, an adaptive filter is integrated into the network input, dynamically adjusting the filtering parameters based on signal characteristics. This subsystem outputs a 256-dimensional temporal feature vector to capture the dynamic characteristics of the hook's motion.

[0096] Within the second branch, infrared and inertial features are initially fused. Specifically, the inertial temporal feature vector is copied spatially to match the size of the infrared feature map, and then added element-wise to form an infrared fused feature map that enhances dynamic information.

[0097] In the feature fusion stage, the system introduces an innovative adaptive attention weight matrix to perform the final fusion of features from the two branches. The generation of this matrix is ​​based on the spatiotemporal consistency evaluation of multimodal features, specifically calculated through three dimensions of metrics: feature entropy, temporal stability, and intermodal consistency.

[0098] The feature entropy calculation module analyzes the information richness of each branch feature and assigns higher weights to features with more information. For example, under good lighting conditions, the entropy value of the first branch feature is higher, and it receives a larger weight accordingly; while in low light or smoky environments, the entropy value of the second branch feature increases, and its weight also increases.

[0099] The temporal stability assessment module monitors the consistency of features across different branches over time. This module evaluates the stability of features by calculating the cosine similarity between feature vectors from adjacent frames. For branch features with higher stability, the system assigns a higher confidence weight to ensure the reliability of the fusion result.

[0100] The intermodal consistency detection module analyzes the correlation between features of two branches. When two branches give similar estimates for the same hook state, the system will increase the weight of these features; conversely, when the features of one branch differ significantly from those of the other branch, their weight will be appropriately reduced to eliminate interference from abnormal sensors.

[0101] Based on the above evaluation, the system generates a dynamic attention weight matrix, which weights and fuses the semantic feature vector of the first branch with the infrared-inertial fusion feature vector of the second branch. The fused feature vector is then fed into a cross-modal Transformer encoder containing six attention layers, each equipped with a multi-head attention mechanism. In the encoder, visual features and inertial signals interact deeply through the cross-attention mechanism, enabling the model to establish an intrinsic connection between visual observation and motion dynamics.

[0102] The pose prediction module employs a recursive attention architecture, which includes an LSTM-based temporal encoder and an attention-based predictor. The temporal encoder is responsible for capturing the historical dependencies of the hook's motion, while the predictor infers the future state based on historical information.

[0103] To improve the physical plausibility of the predictions, rigid body dynamics constraints are introduced into the prediction model. These constraints are implemented through a loss function and include energy conservation constraints, momentum conservation constraints, and motion smoothness constraints. The energy conservation constraint ensures that the predicted trajectory conforms to the law of conservation of mechanical energy; the momentum conservation constraint ensures that the momentum change of the hook during motion conforms to physical laws; and the motion smoothness constraint ensures the smoothness of the predicted trajectory through the limitation of the second derivative.

[0104] The predictive model outputs the 6-DOF pose (3D position and 3D attitude) of the hook at each time step over the next 2 seconds. To handle the uncertainty of the prediction, the model also outputs a confidence score for each prediction, which is used in the subsequent control module to adjust the aggressiveness of the control strategy.

[0105] The model training process employs a multi-stage optimization strategy. First, each branch encoder is pre-trained using a large-scale industrial scenario dataset for supervised learning. Then, end-to-end joint training is performed to optimize the parameters of the entire model.

[0106] Various data augmentation techniques were employed during training, including illumination variation simulation, sensor noise injection, and partial occlusion simulation, to enhance the model's robustness. In particular, to address the domain offset problem commonly encountered in real-world applications, the training data encompassed samples from different seasons, weather conditions, and operational scenarios.

[0107] The loss function design comprehensively considers multiple objectives: the accuracy of pose prediction, the degree to which physical constraints are satisfied, and the smoothness of the prediction results. The total loss function consists of three parts: prediction loss, physical constraint loss, and regularization loss. The prediction loss uses a smoothed L1 loss function, calculating the error separately for position and pose; the physical constraint loss uses the Lagrange multiplier method to transform physical laws into optimization constraints; and the regularization loss prevents the model from overfitting.

[0108] The system also includes an online learning module that continuously optimizes model performance during actual operation. This module automatically detects prediction bias by comparing predicted results with actual observations and initiates a fine-tuning process. The online learning employs elastic weight consolidation technology, retaining learned knowledge while adapting to new data, thus avoiding catastrophic forgetting.

[0109] To ensure the security of online learning, the system has implemented a strict quality control mechanism. Model parameters are only updated when the quality score of new data exceeds a threshold and the model's performance on the new data is significantly better than the original model. This cautious update strategy ensures that the system maintains operational stability while continuously improving.

[0110] The algorithm model in this embodiment performed excellently in multiple test scenarios. Test data from actual deployment in ports show that the model significantly reduced the average error in hook pose prediction, while its robustness under extreme weather conditions was significantly improved. These improvements lay a solid foundation for subsequent precise control. Example

[0111] like Figure 3 Specifically, this embodiment provides a concrete implementation scheme for a hook-unhooking robot system based on multi-source vision fusion. The system adopts a layered architecture, consisting of a perception layer, a fusion computing layer, a control execution layer, and a feedback learning layer from top to bottom. The layers interact with each other via high-speed industrial Ethernet and a real-time bus, forming a complete closed loop from environmental perception, intelligent decision-making, precise execution to continuous optimization. This provides a concrete implementation scheme for a hook-unhooking robot control system based on multi-source vision fusion.

[0112] The perception layer, acting as the system's sensory organs, is responsible for acquiring information from the work site in a comprehensive and multi-dimensional manner. Its core hardware is a multi-source vision sensor array, which is precisely mechanically designed and fixed in a rigid structure about 20 centimeters above the robot's end effector to ensure that the centers of the fields of view of each sensor are basically aligned, minimizing subsequent registration errors.

[0113] RGB Camera: A global shutter industrial camera is selected. Its main task is to provide high-resolution 2D texture and color information for initial hook identification and contour extraction. The camera is equipped with an infrared cutoff filter to suppress ambient light interference in specific wavelengths.

[0114] Depth camera: A depth sensor employing the principle of active infrared structured light. It synchronously outputs a depth image aligned with the RGB image and can directly generate dense 3D point cloud data, providing crucial data for the physical spatial positioning and geometric measurement of the hook.

[0115] Infrared thermal imager: An uncooled microbolometer thermal imager is selected. Its purpose is to generate an infrared thermal radiation image by detecting the temperature difference on the surface of the hook. Since the hook is usually made of metal, its emissivity is different from that of its background (such as air or rope), so the thermal outline of the hook can be clearly delineated even in low-light or smoky environments.

[0116] Inertial Measurement Unit (IMU): A nine-axis IMU (three-axis accelerometer, three-axis gyroscope, and three-axis magnetometer) is integrated inside the sensor array and mounted close to the vision sensor. It acquires the three-axis angular velocity and three-axis linear acceleration of the sensor array body at a high frequency to capture the high-frequency, micro-amplitude swaying of the hook caused by crane movement or wind, which is difficult to capture completely at the visual frame rate.

[0117] After the system starts up, the perception layer periodically triggers all sensors at a fixed frequency. The acquired RAW format RGB images, depth maps, infrared thermal images, and raw data streams from the IMU are initially packaged and timestamped via a customized data acquisition board through MIPI CSI-2 and SPI interfaces, and finally transmitted to the fusion computing layer via gigabit Ethernet.

[0118] The converged computing layer is the brain of the system, deployed on a high-performance embedded GPU platform. This layer runs all the core algorithms and employs an efficient dual-threaded concurrency mechanism to ensure real-time performance and reliability.

[0119] Feature Fusion and Recognition: The main thread first receives multi-source data from the perception layer. It starts a pre-trained dual-branch neural network. The first branch (semantic branch) takes RGB and depth images as input, and through an encoder-decoder semantic segmentation network, accurately outputs a pixel-level region mask of the hook, and extracts a high-dimensional semantic feature vector from the end of the encoder. The second branch (physical property branch) works in parallel: on the one hand, it performs convolution processing on the infrared image to extract its material reflectance feature map; on the other hand, using the 3D point cloud generated from the depth image, it first segments the hook point cloud cluster through the mask output by the semantic branch, calculates its centroid, establishes a local coordinate system and divides it into voxel grids, and finally generates a 128-dimensional feature vector representing the hook's center of gravity distribution and inertial characteristics. Subsequently, the semantic feature vector and the center of gravity distribution feature vector are concatenated and dimensionality reduction and depth fusion are performed through a fully connected layer. Finally, a regression head network outputs the accurate boundary contour of the hook and the 3D coordinates of the preferred grab point.

[0120] Coordinate unification and motion prediction: Based on the robot's forward kinematics model, the system calculates the homogeneous transformation matrix of the robotic arm's end effector in the robot's base coordinate system in real time. Combining pre-stored hand-eye calibration parameters, the dynamic calibration unit uses this matrix to calculate the latest extrinsic parameter matrix of each vision sensor relative to the robot's base coordinate system and generates a disparity correction matrix. The coordinate transformation unit applies this matrix to uniformly transform the coordinates of the identified hook gripping points to the robot's base coordinate system, obtaining "spatially consistent hook coordinate information." Next, the prediction control unit begins operation, aligning and normalizing the continuous visual feature sequences (such as changes in gripping point coordinates) from the past two seconds with the high-frequency angular velocity / acceleration signals from the IMU. The processed multimodal temporal data is fed into a Transformer encoder based on an attention mechanism. This encoder uses a self-attention mechanism to capture long-term temporal dependencies within the visual sequence, while simultaneously using a cross-attention mechanism to establish cross-modal correlations between visual features and inertial signals, outputting a powerful fused temporal representation. Finally, a multilayer perceptron (MLP) regressor is used to predict the future pose changes of the hook in six degrees of freedom based on this representation.

[0121] Trajectory Generation: Based on the predicted hook pose change trend, the system generates a feedforward control trajectory to compensate for its swing deviation. This trajectory is not a direct command, but a desired sequence of path points in Cartesian space.

[0122] Secondary thread: System status monitoring and parameter maintenance

[0123] The secondary thread, a high-priority daemon process, is dedicated to monitoring the stability of the IMU signal and the health of the system. It analyzes the variance and peak value of the IMU data in real time. If a sudden drop in the image sensor's signal-to-noise ratio is detected due to abrupt changes in external lighting, or if strong mechanical vibration causes abnormal IMU data, this thread immediately adjusts the fusion weight matrix of the RGB and infrared branches in the fusion recognition model. For example, it reduces the RGB weight and increases the infrared weight under strong light to ensure the robustness of the evidence input. Simultaneously, it monitors the end effector's posture. When the cumulative change in its position or posture angle exceeds a preset threshold, a trigger signal is generated, actively invoking the dynamic calibration unit to initiate a unified update process for the vision system's registration parameters, ensuring continuous accuracy in coordinate transformation.

[0124] The control and execution layer is the "limb" of the system, including a six-degree-of-freedom industrial robotic arm, a two-finger gripper end effector customized for unhooking tasks, and a high-precision servo electronic control system.

[0125] The motion control module receives three key pieces of information from the fusion computing layer: spatially consistent hook coordinates as the final target point, the desired path point sequence of the feedforward control trajectory, and the pose correction amount calculated by the servo correction unit. It first uses the pose correction amount as a dynamic disturbance compensation value, superimposing it in real-time onto the desired path point sequence of the feedforward control trajectory to form a smooth, online-corrected end effector motion trajectory. Then, using a robot inverse kinematics algorithm, this trajectory in Cartesian space is rapidly converted into a sequence of angle and angular velocity commands for the robot's six joints—the final hook-unhooking command. This command is sent via the EtherCAT bus to the servo drivers of each joint, driving the robotic arm to smoothly and quickly approach the hook.

[0126] The feedback learning layer is the "cerebellum" of the system and is the key to achieving long-term autonomy and intelligence.

[0127] Visual servoing and fine-tuning: As the robotic arm moves along the planned trajectory until it contacts the hook, an RGB camera fixed at the end effector captures a continuous sequence of images of the hook's local area at a high frame rate. The optical flow calculation subunit in the servo correction unit calculates the pixel displacement vectors of hook feature points between adjacent frames. The integration and comparison subunit integrates these displacement vectors in the time dimension to generate a micro-deviation matrix representing the system's accumulated error. Simultaneously, it projects the preset ideal hook removal trajectory onto the image plane using the current camera's intrinsic and extrinsic parameters, generating an ideal image trajectory. The controller subunit continuously calculates the difference sequence between the micro-deviation matrix and the ideal image trajectory, outputs a preliminary correction amount in the image coordinate system, and then performs an inverse coordinate transformation to solve for the end effector pose correction amount in the robot's base coordinate system. This correction amount is immediately sent back to the motion control module for dynamic adjustment of the robotic arm's movement, forming millisecond-level closed-loop control, effectively suppressing deviations caused by model prediction errors or external disturbances, and preventing accidental hook removal or collisions.

[0128] Incremental Learning and Model Evolution: During breaks in continuous system operation, the feedback learning layer initiates a low-priority background learning task. This task compares the predicted hook motion output by the predictive control unit with the true hook motion measured by sensors, calculating the prediction error. Using this error, the system fine-tunes the parameters of the cross-attention layer in the Transformer prediction model through an incremental learning algorithm. This process enables the model to gradually learn and memorize specific lifting scenarios, such as the operating habits of different crane operators and the unique dynamic characteristics of the hook under different wind speeds. After several days or weeks of continuous operation, the system's prediction accuracy will significantly improve, thus achieving the long-term autonomous optimization goal of "getting smarter with use."

[0129] This embodiment, through the precise collaboration of the above four layers, fully realizes the automation of the entire process from multi-source perception, fusion recognition, dynamic prediction, feedforward planning, feedback to online learning, ensuring the high reliability, high precision and strong adaptability of the unhooking robot in complex and dynamic industrial lifting environments. Example

[0130] like Figure 4 Specifically, this embodiment proposes a feedforward-feedback cooperative control mechanism to achieve high-precision motion compensation and adaptive stability control of a hook-unhooking robot in complex dynamic environments. The core of this mechanism lies in establishing a feedforward trajectory control channel based on vision-inertial fusion prediction results and constructing a feedback servo loop in conjunction with real-time error observations, thereby forming a bidirectional cooperative control self-correction process. A feedforward-feedback cooperative control mechanism is proposed to achieve high-precision motion compensation for a hook-unhooking robot in complex dynamic environments.

[0131] Before the unhooking robot performs its task, the control system first calls the timing prediction model from Example 2 to predict the hook's motion trend and obtain information on the hook's pose change trend. Based on this, the control system generates a feedforward control path, enabling the end effector to output a compensation signal in advance to ensure dynamic consistency with the hook's future motion in space. The trajectory generation module uses a continuous curve interpolation algorithm during path construction and introduces an inertial coupling constraint matrix to suppress abrupt changes in angular acceleration, ensuring the smoothness and dynamic feasibility of the robotic arm's trajectory.

[0132] During execution, the feedback control channel operates synchronously. The system calculates the displacement deviation of the hook in the image space in real time through the visual servo module and the attitude angle error of the robot's end effector through the inertial servo module. The controller weights and fuses the two error information to obtain a comprehensive error signal. The weights of this signal are automatically adjusted according to the current visual signal-to-noise ratio, inertial measurement stability, and system latency to ensure the real-time performance and robustness of the fusion result.

[0133] The controller employs an adaptive proportional-integral-derivative (PID) architecture, combined with a sliding mode observer for dynamic error correction. The adaptive control module automatically adjusts the gain parameters based on the error change rate and variance, achieving a balance between response speed and system stability. The sliding mode observer identifies external disturbance sources, such as the swing changes of the hook due to wind, mechanical inertial impact, or sudden load changes, and feeds the estimated disturbances back to the control input channel to counteract the effects of uncertainties.

[0134] To ensure timing consistency between the feedforward and feedback control channels, this embodiment introduces a dynamic delay compensation mechanism. This mechanism estimates the delay between visual acquisition, information processing, and actuator actions in real time, and corrects the reference time of the predicted trajectory through a time resampling algorithm, ensuring that the trajectory information output by the feedforward channel and the error information obtained by the feedback channel are fused on the same time scale. The dynamically compensated control signal is then sent to the inverse kinematics solution module to generate angle and velocity commands for each joint, achieving precise control of spatial attitude.

[0135] During continuous operation, the control unit comprehensively monitors energy consumption, error settling time, and trajectory smoothness. When continuous overshoot or oscillation is detected, the adaptive control module reduces the proportional gain and increases the integral gain to suppress oscillation. When system response is delayed, the control module adjusts the feedforward prediction time advance accordingly to achieve dynamic performance balance.

[0136] To further enhance the system's self-learning and self-optimization capabilities, this embodiment embeds an experience-risk learning module into the collaborative control framework. After each unhooking task, this module constructs a performance cost function based on error signals and historical control response data from the task execution process. It then continuously adjusts the control gain parameters through optimization algorithms, resulting in a more stable dynamic response for the next task execution. Through long-term iterative learning, the system gradually develops adaptive control models for different hoisting environments, achieving continuous performance optimization under complex conditions.

[0137] In multiple rounds of experiments and field verification, the feedforward-feedback coordinated control mechanism proposed in this embodiment can still maintain the closed-loop stability of visual recognition and mechanical control even under environments with large hook swings, strong light interference, and multiple background targets. The motion trajectory of the robotic arm is significantly better than the traditional single-channel PID control method in terms of spatial continuity, posture accuracy, and energy consumption control, demonstrating higher dynamic response speed and environmental adaptability.

[0138] The foregoing has shown and described the basic principles, main features, and advantages of this application. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this application. Various changes and modifications can be made to this application without departing from the spirit and scope thereof, and all such changes and modifications fall within the scope of this application as claimed. The scope of protection of this application is defined by the appended claims and their equivalents.

Claims

1. A control method for a hook-unhooking robot based on multi-source vision fusion, applied to automatic hook-unhooking operations in industrial lifting environments, characterized in that, The method includes the following steps: The system acquires RGB images, depth images, infrared images, and inertial measurement unit data of the unhooking operation area to form multi-source sensing data. Based on the multi-source sensing data, a dual-branch neural network is constructed; The first branch of the dual-branch neural network is used to perform semantic segmentation processing on the RGB image and the depth image to generate a hook region mask and a semantic feature vector. The material reflectance feature map is extracted from the infrared image through the second branch of the dual-branch neural network, and the center of gravity distribution feature vector of the hook is calculated from the three-dimensional point cloud data derived from the depth image. The semantic feature vector and the centroid distribution feature vector are concatenated along the feature dimension, and the concatenated features are dimensionality reduced and fused through a fully connected layer to obtain the fused features. The fused features are input into the regression head network for processing, and the boundary contour of the hook and the coordinates of the gripping point are output. The end effector posture is calculated based on the robot's kinematics model. Based on the end effector posture, the registration parameters of the vision system that collects the multi-source perception data are uniformly updated to generate a disparity correction matrix. The parallax correction matrix is ​​used to uniformly transform the hook coordinates of multi-source visual perception to the robot base coordinate system, thereby obtaining spatially consistent hook coordinate information. The visual feature sequence output by the fusion recognition model is fused with the motion signal of the inertial measurement unit across modes to establish a hook motion prediction model; Based on the hook pose change trend output by the hook motion prediction model, a feedforward control trajectory is generated to compensate for the swing deviation. During the movement of the end effector along the feedforward control trajectory, a continuous image sequence of a local area of ​​the hook is acquired; based on the continuous image sequence, the pixel deviation between adjacent image frames is calculated; The pixel deviation is integrated over time to generate a micro-deviation matrix. The micro-deviation matrix is ​​then compared with a preset ideal unhooking trajectory to solve for the pose correction of the end effector. Based on the pose correction amount, spatially consistent hook coordinate information, and feedforward control trajectory, the final motion path of the end effector is generated. The final motion path is then used to generate a hook removal operation command to control the end effector to complete the hook removal action.

2. The method according to claim 1, characterized in that, The calculation of the center-of-gravity distribution feature vector of the hook from the 3D point cloud data derived from the depth image includes: The hook region mask is applied to the three-dimensional point cloud data for segmentation to obtain hook point cloud clusters; Calculate the centroid coordinates of the hook point cloud cluster; A local coordinate system is established with the centroid coordinates as the origin; In the local coordinate system, the hook point cloud cluster is divided into multiple three-dimensional voxel meshes; Calculate the point cloud density distribution within each voxel grid and calculate the Euclidean distance from the center of each voxel grid to the origin; The point cloud density distribution and the Euclidean distance are weighted and summed to generate a multidimensional vector characterizing the hook mass distribution and inertia tensor properties, which serves as the centroid distribution feature vector.

3. The method according to claim 1, characterized in that, The step of fusing the visual feature sequence output by the fusion recognition model with the motion signal from the inertial measurement unit across modes to establish a hook motion prediction model includes: The visual feature sequence and the angular velocity and acceleration signals output by the inertial measurement unit are time-stamp aligned and normalized to obtain aligned and normalized multimodal time series data; The multimodal temporal data is input into an attention-based Transformer encoder; Inside the Transformer encoder, its self-attention module is used to establish the temporal dependencies within the visual feature sequence, and its cross-attention module is used to construct the cross-modal correlation between visual features and inertial signals, outputting a fused temporal representation. The fused temporal representation is input into a multilayer perceptron regressor. Through forward propagation calculation of the multilayer perceptron regressor, the pose changes of the hook in six degrees of freedom within a specific future time window are predicted.

4. The method according to claim 1, characterized in that, The step of uniformly updating the registration parameters of the vision system that acquires the multi-source perception data based on the end effector posture includes: Based on the robot's forward kinematics model and joint encoder feedback, the robot's forward kinematics calculation is performed to obtain the current homogeneous transformation matrix of the end effector in the robot's base coordinate system. Based on the pre-stored hand-eye calibration results and the current homogeneous transformation matrix, the latest extrinsic parameter matrix of each visual sensor relative to the robot's base coordinate system is calculated one by one through matrix chain multiplication. The latest extrinsic parameter matrix is ​​used as the new registration parameter and sent down to update the coordinate transformation module corresponding to the vision system; Based on the unified updated registration parameters of the vision system, a disparity correction matrix is ​​generated.

5. The method according to claim 4, characterized in that, Also includes: Detect the attitude of the end effector; When the cumulative change in the position or attitude angle of the end effector since the last update is detected to exceed a preset threshold, a trigger signal is generated. The trigger signal is sent to the vision system to initiate the unified update process of the vision system's registration parameters.

6. The method according to claim 1, characterized in that, The step of performing time-series cumulative integration on the pixel deviation to generate a micro-deviation matrix, comparing the micro-deviation matrix with a preset ideal unhooking trajectory, and solving for the pose correction of the end effector includes: The motion of hook feature points between adjacent frames in the continuous image sequence is calculated using optical flow to obtain pixel-level displacement vectors. The displacement vector is integrated and accumulated over time to generate a micro-deviation matrix characterizing the cumulative error of the system. The preset ideal unhooking trajectory is projected onto the image plane based on the camera intrinsic parameters and the current extrinsic parameter matrix to generate the corresponding ideal image trajectory. At consecutive time points, the difference between the micro-deviation matrix and the ideal image trajectory is calculated to form a difference sequence; The difference sequence is input into a proportional-integral controller, and the controller calculates the initial correction amount in the image coordinate system. The initial correction amount is subjected to an inverse coordinate transformation to solve for the pose correction amount of the end effector in the robot base coordinate system.

7. The method according to claim 1, characterized in that, The step of generating the unhooking operation command using the final motion path includes: The spatially consistent hook coordinate information is set as the final target point, and the feedforward control trajectory is solved as the desired path point sequence in Cartesian space; The pose correction amount is used as a disturbance compensation value and superimposed on the desired path point sequence to form the corrected end effector motion trajectory. The corrected end effector trajectory is converted into an angle command sequence for each joint of the robot using the robot inverse kinematics algorithm, generating the final unhooking operation command.

8. A control system for a hook-removing robot based on multi-source vision fusion, characterized in that, The system includes: The multi-source sensing data acquisition module is used to acquire RGB images, depth images, infrared images, and inertial measurement unit data of the unhooking operation area to form multi-source sensing data. The fusion recognition module is used to extract the semantic features and physical attribute features of the hook from the RGB image, depth image and infrared image based on the multi-source perception data through joint optimization, input the semantic features and physical attribute features into the pre-trained fusion recognition model, and output the boundary contour of the hook and the coordinates of the grab point. The registration parameter update module is used to calculate the end effector posture based on the robot kinematics model, and to uniformly update the registration parameters of the vision system that collects the multi-source perception data according to the end effector posture, thereby generating a disparity correction matrix. The coordinate transformation module is used to uniformly transform the hook coordinates of multi-source visual perception to the robot base coordinate system through the parallax correction matrix, so as to obtain spatially consistent hook coordinate information; The motion prediction module is used to perform cross-modal fusion of the visual feature sequence output by the fusion recognition model and the motion signal of the inertial measurement unit to establish a hook motion prediction model. The feedforward control module is used to generate a feedforward control trajectory to compensate for swing deviation based on the hook pose change trend output by the hook motion prediction model. The visual servo module is used to acquire a continuous image sequence of a local area of ​​the hook during the movement of the end effector along the feedforward control trajectory; and to calculate the pixel deviation between adjacent image frames based on the continuous image sequence. The trajectory correction module is used to perform time-series cumulative integration on the pixel deviation to generate a micro-deviation matrix, compare the micro-deviation matrix with the preset ideal unhooking trajectory, and solve for the pose correction amount of the end effector. The path planning module is used to generate the final motion path of the end effector based on the pose correction amount, spatially consistent hook coordinate information, and feedforward control trajectory. The instruction generation and execution module is used to generate a hook removal operation instruction using the final motion path, and control the end effector to complete the hook removal action.

9. The system according to claim 8, characterized in that: The fusion recognition module includes a dual-branch neural network structure, wherein: The first processing branch is used to perform semantic segmentation processing on the RGB image and the depth image to generate a hook region mask and a semantic feature vector. The second processing branch is used to extract the material reflectance feature map from the infrared image and calculate the center of gravity distribution feature vector of the hook from the three-dimensional point cloud data derived from the depth image. The feature fusion unit is used to concatenate the semantic feature vector and the centroid distribution feature vector in the feature dimension, and then perform dimensionality reduction and fusion on the concatenated features through a fully connected layer to obtain the fused features; The coordinate regression unit is used to input the fused features into the regression head network for processing and output the boundary contour of the hook and the coordinates of the grab point.

Images