A real-time compensation method and device for material grabbing deviation based on multi-sensor data fusion

Abstract

The present application discloses a method and device for real-time compensation of material grasping offset based on multi-sensor data fusion, which relates to the field of automated production technology. The disclosed method and device for real-time compensation of material grasping offset based on multi-sensor data fusion construct a material space coordinate system by fusing multi-sensor data, and combine a neural network model to generate three-dimensional offset in real time and control the compensation of the robotic arm, thereby solving the problems of large positioning error and insufficient compensation accuracy of a single sensor, improving the accuracy of material grasping, and thereby enhancing the stability of the automated production line.

Description

Technical Field

[0001] The present application relates to the field of automated production technology, and in particular to a method and device for real-time compensation of material grabbing offset based on multi-sensor data fusion. Background Art

[0002] The robotic arm is the core actuator for material grasping in automated production lines. In existing technologies, materials are generally grasped by the robotic arm through a single visual positioning technology. Due to changes in ambient lighting and interference from reflective material surfaces, its positioning error is large, resulting in frequent offsets during the material grasping process. The accuracy of material grasping is low, affecting the stability and efficiency of the automated production line.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a real-time compensation method and device for material grasping deviation based on multi-sensor data fusion, aiming to improve the accuracy of material grasping.

[0005] To achieve the above objectives, the present application proposes a real-time compensation method for material grasping offset based on multi-sensor data fusion, which is applied to a robotic arm. The robotic arm includes a joint module and an end effector connected to the joint module. The joint module is used to adjust the position of the end effector, and the end effector is used to grasp materials. The method includes:

[0006] Collecting characteristic image data of the material surface through a visual sensor and three-dimensional point cloud data of the material surface through a laser radar, and constructing a material space coordinate system based on the image data and the three-dimensional point cloud data;

[0007] Collecting torque data of the end effector through a torque sensor, and collecting posture data of the end effector through an inertial measurement unit;

[0008] Performing spatiotemporal synchronization processing on the material space coordinate system, the torque data, and the posture data to generate corresponding multi-source fusion data;

[0009] Inputting the multi-source fusion data into a pre-trained convolutional neural network model to output a three-dimensional offset of the material center point relative to the grasping point;

[0010] generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset;

[0011] Based on the offset compensation instruction, the joint module is controlled to perform corresponding offset compensation actions to compensate for the offset that occurs during the material grabbing process.

[0012] In one embodiment, the step of constructing a material space coordinate system based on the image data and the three-dimensional point cloud data includes:

[0013] Performing edge detection and feature point extraction on the feature image data to obtain a material contour key point set;

[0014] Registering the three-dimensional point cloud data with the feature point set to establish a transformation matrix between the visual coordinate system and the point cloud coordinate system;

[0015] Performing coordinate transformation on the feature point set in the visual coordinate system based on the transformation matrix to generate a three-dimensional space coordinate sequence that integrates visual and point cloud features;

[0016] The three-dimensional space coordinate sequence is dynamically compensated by timestamp alignment, and the compensated three-dimensional space coordinate sequence is spatially mapped to the robot arm base coordinate system, and the material space coordinate system is constructed with the material center point as the coordinate system origin.

[0017] In one embodiment, the step of performing spatiotemporal synchronization processing on the material space coordinate system, the torque data, and the posture data to generate corresponding multi-source fusion data includes:

[0018] Use timestamp alignment algorithm to synchronize material space coordinate system, moment data and inertia data across modalities;

[0019] The torque data is converted to the material space coordinate system through the space transformation matrix and vector synthesized with the posture data;

[0020] Perform sliding window sampling on the multi-dimensional data stream after spatiotemporal synchronization to establish a spatiotemporal correlation matrix containing spatial position, mechanical state and motion trajectory;

[0021] A Kalman filter is used to perform noise suppression and state estimation on the spatiotemporal correlation matrix to generate corresponding multi-source fusion data.

[0022] In one embodiment, the method further comprises:

[0023] Collect historical offset datasets for various material specifications and gripping postures, and perform normalization and sample balancing on the datasets;

[0024] Performing enhancement processing on the dataset to construct corresponding training sets and validation sets;

[0025] Initializing network parameters using a transfer learning strategy for the training set, and constructing a dual-channel feature extraction network including a three-dimensional convolution kernel and an attention mechanism;

[0026] The network weights are optimized through end-to-end training, and a dynamic learning rate decay strategy is used for iterative optimization until the mean absolute error of the model on the validation set is less than the preset threshold to obtain a pre-trained convolutional neural network model.

[0027] In one embodiment, the step of generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset comprises:

[0028] Performing kinematic decomposition on the three-dimensional offset to obtain a lateral displacement component and an attitude deflection component;

[0029] Converting the lateral displacement component into a compensation vector in a Cartesian coordinate system, and converting the posture deflection component into an angle correction matrix based on a homogeneous transformation matrix;

[0030] The weight ratio of the compensation vector and the angle correction matrix is ​​adjusted according to the real-time load state of the robot arm, and an offset compensation instruction including the compensation vector and the angle correction matrix is ​​generated through a weighted fusion algorithm.

[0031] In one embodiment, the step of controlling the joint module to perform a corresponding offset compensation action based on the offset compensation instruction to compensate for the offset occurring during the material grabbing process includes:

[0032] determining a translation compensation component and a rotation compensation component of the end effector based on the offset compensation instruction;

[0033] The translation compensation component is converted into the displacement increment of each joint axis of the joint module by using an inverse kinematics algorithm, and the rotation compensation component is converted into the angular correction value of each joint axis of the joint module based on the Jacobian matrix;

[0034] Dynamically compensate the displacement increment and the angle correction amount according to the dynamic parameters of the joint module to generate a target posture instruction for each joint axis;

[0035] The target posture instruction is sent to the joint module to drive each joint axis of the joint module to perform corresponding offset compensation actions to compensate for the offset that occurs during the material grasping process.

[0036] In one embodiment, the method further comprises:

[0037] During the process of the joint module performing the offset compensation action, the actual offset of the end effector is monitored in real time;

[0038] If the actual offset exceeds the preset threshold, the process returns to executing the steps of collecting feature image data of the material surface through the visual sensor, collecting three-dimensional point cloud data of the material surface through the lidar, and constructing the material space coordinate system based on the image data and the three-dimensional point cloud data, so as to re-acquire the offset compensation instruction until the actual offset of the end effector meets the preset accuracy requirement.

[0039] In one embodiment, the method further comprises:

[0040] After the end effector grabs the material, collecting pressure distribution data of the material contact surface and calculating the pressure gradient change rate;

[0041] Comparing the pressure gradient change rate with a preset safe grasping threshold, and identifying a slip trend of the contact surface between the material and the end effector when an abnormal pressure distribution is detected;

[0042] Dynamically correcting the compensation priority of the three-dimensional offset according to the slip trend, and generating a secondary compensation instruction including dynamic compensation parameters;

[0043] The clamping force and compensation timing of the end effector are adjusted according to the secondary compensation instruction until the pressure gradient change rate returns to the preset safety threshold range.

[0044] In one embodiment, the method further comprises:

[0045] Build an adaptive grasping strategy library based on material types, which contains grasping parameters and compensation strategies for different material types;

[0046] According to the type of material to be grasped, the corresponding grasping parameters and compensation strategies are matched from the strategy library;

[0047] During the material grabbing process, the changes in material types are monitored in real time, and the grabbing parameters and compensation strategies are dynamically adjusted to adapt to the changes in material types;

[0048] When the material type mutates or cannot be matched to the corresponding grasping strategy, the exception handling mechanism is triggered to re-identify the material and plan the corresponding grasping strategy.

[0049] In addition, to achieve the above-mentioned purpose, the present application also proposes a real-time compensation device for material grabbing offset based on multi-sensor data fusion, and the real-time compensation device for material grabbing offset based on multi-sensor data fusion includes: a memory, a processor, and a real-time compensation program for material grabbing offset based on multi-sensor data fusion stored on the memory and runnable on the processor, and the real-time compensation program for material grabbing offset based on multi-sensor data fusion is configured to implement the steps of the real-time compensation method for material grabbing offset based on multi-sensor data fusion.

[0050] From the above, it can be seen that the present application provides a real-time compensation method and device for material grasping offset based on multi-sensor data fusion. By fusing multi-sensor data to construct a material space coordinate system, combined with a neural network model to generate three-dimensional offset in real time and control the robotic arm compensation, it solves the problems of large positioning error and insufficient compensation accuracy of a single sensor, can improve the accuracy of material grasping, and thus enhance the stability of the automated production line. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0053] Figure 1 A flow chart illustrating an embodiment of a method for real-time compensation of material grabbing offset based on multi-sensor data fusion provided in this application;

[0054] Figure 2 For this application Figure 1 Detailed flow chart of step S100;

[0055] Figure 3 For this application Figure 1 Detailed flow diagram of step S300;

[0056] Figure 4 A flow chart illustrating another embodiment of the method for real-time compensation of material grabbing offset based on multi-sensor data fusion provided in this application;

[0057] Figure 5 For this application Figure 1 Detailed flow chart of step S500;

[0058] Figure 6 For this application Figure 1 Detailed flowchart of step S600;

[0059] Figure 7 A flow chart illustrating another embodiment of the method for real-time compensation of material grabbing offset based on multi-sensor data fusion provided by the present application;

[0060] Figure 8 A flow chart illustrating another embodiment of the method for real-time compensation of material grabbing offset based on multi-sensor data fusion provided by the present application;

[0061] Figure 9 A flow chart illustrating another embodiment of the method for real-time compensation of material grabbing offset based on multi-sensor data fusion provided in this application;

[0062] Figure 10 This is a structural diagram of an embodiment of a real-time compensation device for material grabbing deviation based on multi-sensor data fusion provided in this application.

[0063] Description of Figure Numbers:

[0064] 10. Memory; 20. Processor.

[0065] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0066] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0067] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0068] The main solution of the embodiment of the present application is: collecting feature image data of the material surface through a visual sensor, and collecting three-dimensional point cloud data of the material surface through a laser radar, and constructing a material space coordinate system based on the image data and the three-dimensional point cloud data; and collecting the torque data of the end effector through a torque sensor, and collecting the posture data of the end effector through an inertial measurement unit; then performing spatiotemporal synchronization processing on the material space coordinate system, the torque data and the posture data to generate corresponding multi-source fusion data; and inputting the multi-source fusion data into a pre-trained convolutional neural network model to output the three-dimensional offset of the material center point relative to the grasping point; then generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset; finally, controlling the joint module to perform corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset occurring during the material grasping process.

[0069] In this embodiment, for ease of description, the following description is made by taking the identification of a real-time compensation device for material grabbing deviation based on multi-sensor data fusion as the execution subject.

[0070] Since the existing robotic arm is the core actuator for material grabbing in the automated production line, in the existing technology, the robotic arm generally uses a single visual positioning technology to grab materials. Due to changes in ambient light and interference from reflective material surfaces, its positioning error is large, resulting in frequent offsets during the material grabbing process. The accuracy of material grabbing is low, affecting the stability and efficiency of the automated production line.

[0071] The solution provided in this application constructs a material spatial coordinate system by fusing multi-sensor data, and combines a neural network model to generate three-dimensional offsets in real time and control the robotic arm compensation. It solves the problems of large positioning errors and insufficient compensation accuracy of a single sensor, and can improve material grasping accuracy, thereby enhancing the stability of the automated production line.

[0072] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution capabilities, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of implementing the aforementioned functions, such as a real-time compensation device for material grasping deviation based on multi-sensor data fusion. This embodiment and the following embodiments will be described below using the real-time compensation device for material grasping deviation based on multi-sensor data fusion as an example.

[0073] In existing technologies, material grasping in automated production lines usually relies on robotic arms, and the positioning accuracy of the robotic arms directly affects the success rate of material grasping. Traditional methods mainly use a single visual sensor for positioning, identify the material position through image processing, and control the movement of the robotic arm. However, in complex industrial environments, factors such as changes in lighting conditions, reflections or occlusions on the material surface can easily lead to visual recognition errors, which in turn cause grasping deviations. For example, in metal parts grasping scenarios, strong light reflections may cause the visual sensor to misjudge the edge position of the material, causing the actual grasping point of the end effector to deviate from the target center, ultimately resulting in material falling off or assembly misalignment.

[0074] In order to solve the above problems, it is necessary to consider how to improve the real-time perception and compensation capabilities of the robot arm for offset during the grasping process. The traditional single sensor solution is not robust enough in a dynamic environment, and it is difficult to accurately reflect the comprehensive state of the material position and the robot arm posture. Through analysis, it is found that multimodal data fusion can effectively make up for the limitations of a single sensor, such as combining vision and three-dimensional point cloud data to build a more accurate coordinate system, while introducing mechanical and kinematic parameters to enhance state perception. Furthermore, real-time data processing and dynamic compensation mechanisms become the key to solving grasping offsets. It is necessary to establish a cross-sensor spatiotemporal synchronization framework to ensure the consistency of multi-source data in time and space dimensions. Based on this, it is necessary to design a neural network model that adapts to multi-dimensional data to achieve rapid prediction of offsets and generate composite control instructions including translation and rotation compensation, and finally achieve real-time compensation through collaborative actions of joint modules.

[0075] Based on this, the embodiment of the present application provides a real-time compensation method for material grasping deviation based on multi-sensor data fusion, which is applied to a robotic arm, wherein the robotic arm includes a joint module and an end effector connected to the joint module, wherein the joint module is used to adjust the position of the end effector, and the end effector is used to grasp materials. Figure 1 The method includes steps S100 to S600, wherein:

[0076] Step S100, collecting feature image data of the material surface through a visual sensor, and collecting three-dimensional point cloud data of the material surface through a laser radar, and constructing a material space coordinate system based on the image data and the three-dimensional point cloud data;

[0077] Step S200, collecting torque data of the end effector through a torque sensor, and collecting posture data of the end effector through an inertial measurement unit;

[0078] Step S300, performing spatiotemporal synchronization processing on the material space coordinate system, the moment data, and the posture data to generate corresponding multi-source fusion data;

[0079] Step S400: inputting the multi-source fusion data into a pre-trained convolutional neural network model to output a three-dimensional offset of the material center point relative to the grasping point;

[0080] Step S500, generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset;

[0081] Step S600: Controlling the joint module to perform corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset occurring during the material grabbing process.

[0082] A visual sensor is an image acquisition device used to capture the optical characteristics of a material's surface. Specifically, this can be achieved using an industrial camera coupled with a ring light source. High-contrast images are used to extract the material's contours. A lidar (LiDAR) sensor, such as a line-scan radar using the Time-of-Flight (TOF) ranging principle, acquires the three-dimensional coordinates of an object through laser scanning. This sensor is used to construct a point cloud model of the material's surface. Spatiotemporal synchronization refers to the alignment of timestamps and spatial coordinates of multi-source data. Specifically, a hardware-triggered synchronization mechanism can be used to ensure temporal consistency between visual, laser, and mechanical data, and spatial reference alignment is achieved through a coordinate transformation matrix. Three-dimensional offset refers to the deviation between the actual material position and the intended grasping point along the X, Y, and Z axes. This is obtained by nonlinearly mapping multidimensional input data using a neural network model. A lateral compensation vector is a displacement correction parameter in a Cartesian coordinate system, such as a two-dimensional vector containing X / Y translations, used to correct for planar position deviations of the end effector. Angle correction matrix refers to rotation correction parameters expressed using Euler angles or quaternions, such as a homogeneous transformation matrix describing the end effector's posture adjustments.

[0083] Specifically, during the material grasping process, industrial cameras and lidar collect material surface data from two dimensions: optical imaging and 3D ranging. The RGB images captured by the industrial cameras undergo edge detection and feature extraction to identify key points of the material's contour. The dense point cloud data generated by the lidar scan is aligned with the visual feature points using a registration algorithm to establish a unified material spatial coordinate system. A six-dimensional torque sensor on the end effector monitors the gripping force distribution in real time, while an inertial measurement unit collects three-axis acceleration and angular velocity data. Hardware-synchronized trigger signals align the visual coordinate system, point cloud data, mechanical parameters, and posture data with timestamps and transform them to a common spatial reference. The fused data, processed by Kalman filtering, is input into a 3D convolutional neural network. Through end-to-end training, the model learns the mapping from multi-source data to spatial offsets and outputs 3D offsets that include position and posture deviations. The motion control module decomposes the offsets into lateral translation compensation and rotation angle correction, generating control commands to drive the joint modules. Based on the inverse kinematics solution, the joint axes coordinate their movements, driving the end effector to adjust its position and posture in real time, ultimately achieving dynamic compensation for grasping offsets.

[0084] Compared with existing technologies, traditional single-vision positioning solutions are easily affected by ambient light interference, resulting in accumulated errors in material edge recognition. This solution integrates vision and lidar data to construct a spatial coordinate system, and effectively improves the accuracy of coordinate system establishment through the complementary advantages of optical imaging and three-dimensional ranging. Existing methods do not consider the impact of the mechanical state of the end effector on the offset. This solution introduces torque and inertia data to more comprehensively reflect the dynamic interaction state during the grasping process. In addition, the existing technology uses a fixed threshold compensation strategy, which is difficult to deal with nonlinear offset problems under complex working conditions. This solution uses a neural network model to achieve adaptive offset prediction, significantly improving the intelligence level of compensation control.

[0085] Through the above technical solutions, this application effectively solves the problem of grasping offset caused by traditional single visual positioning. The multi-sensor data fusion mechanism enhances the ability to adapt to the environment and can still accurately perceive the position of materials in reflective or occluded scenes. Spatiotemporal synchronization processing ensures the consistency and real-time performance of multi-source data, laying the foundation for precise compensation. The neural network model achieves high-precision offset prediction through multi-dimensional feature learning, and combines composite compensation instruction generation with joint collaborative control to ultimately achieve sub-millimeter grasping accuracy. This method can significantly reduce the failure rate of material grasping and improve the stability of the production line in scenarios such as automated assembly and precision parts handling.

[0086] In one possible implementation, reference Figure 2 , the step S100 includes steps S110 to S140, wherein:

[0087] Step S110, performing edge detection and feature point extraction on the feature image data to obtain a material contour key point set;

[0088] Step S120, registering the three-dimensional point cloud data with the feature point set to establish a transformation matrix between the visual coordinate system and the point cloud coordinate system;

[0089] Step S130, performing coordinate transformation on the feature point set in the visual coordinate system based on the transformation matrix to generate a three-dimensional space coordinate sequence that integrates visual and point cloud features;

[0090] In step S140 , the three-dimensional space coordinate sequence is dynamically compensated by timestamp alignment, and the compensated three-dimensional space coordinate sequence is spatially mapped to the robot base coordinate system, and the material center point is used as the origin of the coordinate system to construct the material space coordinate system.

[0091] Among them, edge detection refers to the process of identifying the contour boundary of the material surface through image processing technology. Specifically, it can be implemented by the Canny operator or the Sobel operator to extract the geometric boundary information of the material in the two-dimensional image. Feature point extraction refers to the selection of positioning points with significant features from the material surface image. Specifically, it can be implemented by the SIFT algorithm or the ORB algorithm to establish the corresponding relationship between the material under different viewing angles. The transformation matrix refers to the spatial transformation relationship between different coordinate systems. Specifically, it can be implemented by the rotation and translation matrix in the rigid body transformation to unify the data of the visual sensor and the lidar into a common reference system. Timestamp alignment refers to the process of time synchronization of multi-source sensor data. Specifically, it can be implemented by the interpolation compensation method or the nearest neighbor matching method to eliminate the spatiotemporal errors caused by the sampling delays of different sensors.

[0092] Specifically, during the material grasping process, the visual sensor captures a feature image containing the material's surface texture. An edge detection algorithm extracts a set of pixel coordinates representing the material's contour edges, while a lidar acquires a three-dimensional point cloud of the material's surface. A feature point matching algorithm is used to associate the two-dimensional image feature points with the corresponding spatial points in the three-dimensional point cloud. The rotation and translation parameters between the two sensor coordinate systems are calculated to form a transformation matrix. Based on this matrix, the visual feature points are converted to the point cloud coordinate system, generating a three-dimensional coordinate sequence that integrates the features of the two sensors. Because the movement of the robotic arm may cause the collected data to be out of sync, a timestamp alignment method is used to dynamically interpolate and compensate for the coordinate sequence to eliminate coordinate drift caused by movement. Finally, the compensated three-dimensional coordinates are mapped to the robotic arm's base coordinate system, and a high-precision material spatial coordinate system is established with the material's center of mass as the origin.

[0093] Compared with existing technologies, traditional methods rely solely on a single visual sensor to construct a coordinate system, which is susceptible to lighting changes or reflection interference, leading to unstable feature extraction. This solution fuses visual images with laser point cloud data, supplementing depth information in the spatial dimension and implementing dynamic compensation in the temporal dimension, effectively improving the accuracy and stability of coordinate system construction.

[0094] Through the above technical solution, this application solves the positioning deviation problem of the material space coordinate system caused by sensor errors and motion interference, provides an accurate spatial reference for subsequent offset compensation, and enables the robotic arm to accurately identify the relative offset between the center position of the material and the grasping point, significantly improving the positioning accuracy and operational reliability of material grasping.

[0095] In one possible implementation, reference Figure 3 , step S300 includes steps S310 to S340, wherein:

[0096] Step S310 , using a timestamp alignment algorithm to perform cross-modal synchronization on the material space coordinate system, moment data, and inertia data;

[0097] Step S320, converting the torque data into the material space coordinate system through the space conversion matrix, and performing vector synthesis with the posture data;

[0098] Step S330 , performing sliding window sampling on the multi-dimensional data stream after time-space synchronization to establish a time-space correlation matrix including spatial position, mechanical state and motion trajectory;

[0099] Step S340: Using a Kalman filter to perform noise suppression and state estimation on the spatiotemporal correlation matrix to generate corresponding multi-source fusion data.

[0100] In this embodiment, the timestamp alignment algorithm refers to a method for achieving cross-modal data synchronization by unifying the time base of different sensor data. Specifically, it can be implemented by hardware clock synchronization or software interpolation compensation. It is used in the solution to eliminate time delay differences in visual, torque, and inertial data. The spatial transformation matrix refers to a mathematical transformation tool that maps mechanical data in different coordinate systems to a unified spatial coordinate system. Specifically, it can be implemented by a rigid transformation matrix or an affine transformation matrix. It is used in the solution to achieve the unification of the mechanical state of torque data and the material space coordinate system. Sliding window sampling refers to a data processing method that dynamically intercepts a continuous data stream at a fixed time length. Specifically, it can be implemented by a fixed step size or an adaptive window length. It is used in the solution to extract multidimensional data segments with spatiotemporal correlation. The Kalman filter refers to a recursive algorithm that optimally estimates a dynamic system through the state equation and the observation equation. Specifically, it can be implemented by a linear or extended Kalman filter. It is used in the solution to suppress sensor noise and estimate the true state of the system.

[0101] Specifically, during the robotic arm's material grasping process, the material's spatial coordinate system, constructed by the visual sensor and lidar, and the torque and posture data of the end effector are synchronized at the microsecond level through a timestamp alignment algorithm, eliminating data misalignment caused by differences in sensor sampling frequencies. The torque data is converted to the material's spatial coordinate system using a pre-calibrated spatial transformation matrix and vector-superimposed with the posture data to obtain a composite mechanical state. The synchronized multidimensional data stream is divided into multiple time windows for sampling, each containing spatial position, torque vector, and motion trajectory data, thereby forming a spatiotemporal correlation matrix. The Kalman filter filters the noisy data in the correlation matrix based on the system dynamics model, and combines historical state predictions to predict the current optimal estimate, ultimately generating multi-source fused data containing denoised spatial coordinates, mechanical parameters, and motion trajectories.

[0102] Compared with existing technologies, traditional material handling systems typically use raw sensor data directly without performing spatiotemporal alignment and fusion. This results in time deviations and coordinate inconsistencies between different modal data, which in turn affects the accuracy of offset calculations. This solution achieves spatiotemporal consistency of multidimensional data through cross-sensor time synchronization and spatial coordinate mapping. Combined with the dynamic estimation capabilities of the Kalman filter, it effectively suppresses sensor noise and improves the accuracy and reliability of multi-source data fusion.

[0103] Through the above technical solution, this application effectively solves the problem of inaccurate compensation instructions caused by sensor data asynchrony and noise interference during the robotic arm grasping process, realizes high-precision multi-source data spatiotemporal fusion, and provides an accurate data basis for subsequent offset calculation, thereby significantly improving the position compensation accuracy and operational stability of material grasping.

[0104] In one possible implementation, reference Figure 4 The method further includes steps S410 to S440, wherein:

[0105] Step S410: collecting historical offset data sets of various material specifications and grasping postures, and performing normalization and sample balancing on the data sets;

[0106] Step S420: performing enhancement processing on the data set to construct corresponding training set and validation set;

[0107] Step S430, initializing network parameters using a transfer learning strategy for the training set, and constructing a dual-channel feature extraction network including a three-dimensional convolution kernel and an attention mechanism;

[0108] In step S440, the network weights are optimized through end-to-end training, and a dynamic learning rate decay strategy is used for iterative optimization until the mean absolute error of the model on the validation set is less than a preset threshold, so as to obtain a pre-trained convolutional neural network model.

[0109] In this embodiment, the historical offset data set refers to a data set containing actual offsets under different material forms and robotic arm grasping postures. Specifically, it can be achieved by synchronously collecting actual grasping deviation data under different working conditions through multiple sensors, and is used to cover offset patterns in diverse scenarios. Normalization processing refers to eliminating the dimensional differences in data from different sensors. Specifically, it can be achieved by using maximum and minimum value scaling or Z-score normalization methods to ensure the consistency of the numerical range of the neural network input. Sample balance refers to adjusting the distribution ratio of samples with different offsets in the data set. Specifically, it can be achieved by oversampling minority class samples or undersampling majority class samples to avoid category imbalance problems during model training. Enhancement processing refers to expanding the diversity of training data. Specifically, it can be achieved by using random rotation, translation or adding Gaussian noise to improve the robustness of the model to noise interference. The transfer learning strategy refers to reusing the feature extraction capability of the pre-trained model. Specifically, it can be achieved by using the convolution layer parameters pre-trained on the ImageNet dataset as the initialization weights to accelerate the network convergence speed. A three-dimensional convolution kernel refers to a stereo filter that can simultaneously extract spatiotemporal features. Specifically, it can be achieved by setting a multi-channel convolution layer stacking structure to capture spatial correlations in multi-source data. The attention mechanism is a computational module that dynamically adjusts the weights of feature map channels. This can be achieved through channel attention or spatial attention modules, enhancing the network's ability to focus on key features. A dynamic learning rate decay strategy involves gradually reducing the optimization step size as training progresses. This can be achieved using exponential decay or cosine annealing algorithms, balancing model convergence speed and accuracy.

[0110] Specifically, during the model training phase, historical offset data covering different material sizes, weights, and gripping angles are first collected, the raw data is standardized to eliminate sensor range differences, and the number of various offset samples is balanced through oversampling. Subsequently, random geometric transformations and noise injection operations are applied to the training set to expand the data scale, and a validation set is divided to evaluate the generalization performance of the model. When constructing a dual-channel network, transfer learning is used to load the pre-trained image feature extraction layer parameters, the three-dimensional convolution layer processes the visual and mechanical feature data in parallel, and the attention module automatically selects high-contribution feature channels. During the training process, end-to-end backpropagation is used to update the network weights, and the loss function is gradually optimized in combination with a dynamically adjusted learning rate strategy. The training is terminated when the mean absolute error on the validation set has not decreased for three consecutive training cycles, and finally a neural network model that can accurately predict the three-dimensional offset is obtained.

[0111] Compared with existing technologies, traditional robotic arm offset compensation models typically rely on a single type of sensor data and lack data augmentation, resulting in insufficient generalization capabilities under complex working conditions. Existing methods often employ a fixed learning rate strategy for full-parameter training, which suffers from slow convergence and a tendency to fall into local optimality. Few existing technologies have combined transfer learning with three-dimensional convolution for multimodal data fusion, making it difficult to effectively extract cross-domain feature correlations.

[0112] Through the above technical solution, this application solves the problem of insufficient generalization ability of traditional compensation models due to the single-source training data, effectively improving the neural network's adaptability to different material specifications and grasping postures. By combining transfer learning with a dynamic learning rate decay strategy, the model training cycle is shortened and the parameter optimization efficiency is improved, overcoming the slow convergence speed of traditional full-parameter training methods. The dual-channel structure of three-dimensional convolution and attention mechanism enhances the model's feature extraction ability for multi-source heterogeneous data, making the prediction results more accurate and reliable.

[0113] In one possible implementation, reference Figure 5 , the step S500 includes steps S510 to S530, wherein:

[0114] Step S510, performing kinematic decomposition on the three-dimensional offset to obtain a lateral displacement component and an attitude deflection component;

[0115] Step S520, converting the lateral displacement component into a compensation vector in a Cartesian coordinate system, and converting the posture deflection component into an angle correction matrix based on a homogeneous transformation matrix;

[0116] Step S530 , adjusting the weight ratio of the compensation vector and the angle correction matrix according to the real-time load state of the robot arm, and generating an offset compensation instruction including the compensation vector and the angle correction matrix through a weighted fusion algorithm.

[0117] In this embodiment, kinematic decomposition refers to the process of decomposing the offset in three-dimensional space into linear translation and rotation components. Specifically, this can be achieved using homogeneous coordinate transformation matrix operations to distinguish motion components in different directions. The compensation vector in the Cartesian coordinate system refers to the linear displacement based on the robot arm's base coordinate system. Specifically, the lateral displacement component can be mapped to the robot arm's motion plane through a coordinate transformation formula to achieve accurate compensation of translation errors. The homogeneous transformation matrix refers to a four-dimensional matrix containing rotation and translation information. Specifically, it can be generated through Euler angle or quaternion conversion of the posture deflection component and is used to describe the posture correction requirements of the end effector. Weight proportional adjustment refers to the dynamic allocation of translation and rotation compensation priorities based on the current load torque of the joint module. Specifically, it can be achieved using a dynamic calculation module of the proportional coefficient based on torque sensor data to adapt to the mechanical constraints under different working conditions. The weighted fusion algorithm refers to an operation method that normalizes and superimposes the compensation amounts of different dimensions. Specifically, a comprehensive compensation instruction can be generated through a linear weighting or nonlinear interpolation algorithm to ensure the coordination of multi-dimensional compensation.

[0118] Specifically, during the material grasping process, the three-dimensional offset output by the convolutional neural network is first input into the kinematic decomposition module, where the lateral displacement and attitude deflection components are separated through homogeneous coordinate transformation. The lateral displacement component undergoes coordinate system transformation to generate a compensation vector in Cartesian space, for example, a millimeter-level translation correction in the XY plane. The attitude deflection component is converted into a rotational correction instruction for the end effector using a homogeneous transformation matrix, for example, an angle correction value around the Z axis. Simultaneously, load torque data from each joint of the robotic arm is collected in real time. When the load of a joint approaches a threshold, the weight of the compensation in the corresponding direction is reduced to avoid execution errors due to overload. Finally, the compensation vector and the angle correction matrix are input into the weighted fusion module, which generates a comprehensive compensation instruction using dynamically assigned weight coefficients. For example, translation compensation is given a higher weight under low load conditions, while angle correction is prioritized under high load conditions.

[0119] Compared to existing technologies, traditional approaches typically employ only single-direction translation compensation or fixed-ratio multi-dimensional compensation, failing to dynamically adjust compensation strategies based on real-time load conditions. For example, existing technologies use a preset fixed value for the weight ratio of the compensation vector to the angle correction matrix, which can easily lead to compensation overshoot or joint overload when the robot's load suddenly changes. This solution dynamically adjusts weights through real-time load state feedback. For example, when grasping heavy materials, it reduces the translation compensation amplitude and increases the posture correction priority, thereby maintaining compensation accuracy while avoiding overshooting the robot's dynamics.

[0120] Through the above technical solution, this application can optimize the compensation strategy according to the actual working state of the robot arm and solve the problem of compensation error accumulation caused by fixed weights in traditional methods. Through the dynamic weight adjustment mechanism, posture stability can be prioritized under high-load conditions, and precise displacement compensation can be achieved under low-load conditions, thereby improving the adaptability and robustness of the material grasping process. For example, when handling fragile materials, the system automatically increases the angle correction weight to reduce the surface stress concentration caused by the clamping posture deviation, effectively reducing the risk of material damage.

[0121] In one possible implementation, reference Figure 6 , step S600 includes steps S610 to S640, wherein:

[0122] Step S610, determining a translation compensation component and a rotation compensation component of the end effector based on the offset compensation instruction;

[0123] Step S620, converting the translation compensation component into a displacement increment of each joint axis of the joint module through an inverse kinematics algorithm, and converting the rotation compensation component into an angular correction value of each joint axis of the joint module based on a Jacobian matrix;

[0124] Step S630, dynamically compensating the displacement increment and the angle correction amount according to the dynamic parameters of the joint module to generate a target posture instruction for each joint axis;

[0125] Step S640: Send the target posture instruction to the joint module to drive each joint axis of the joint module to perform a corresponding offset compensation action to compensate for the offset that occurs during the material grasping process.

[0126] In this embodiment, the translation compensation component refers to the linear displacement that the end effector needs to move along the coordinate axis in the Cartesian coordinate system. Specifically, the kinematic inverse solution algorithm can be used to map the Cartesian space displacement to the joint space, for example, the joint angle change can be solved by the Newton-Raphson method or analytical method. The rotation compensation component refers to the posture adjustment amount of the end effector around each coordinate axis. Specifically, the homogeneous transformation matrix can be used to describe the rotation change, for example, the posture correction amount is decomposed into the angular velocity or angular displacement increment of each joint axis through the Jacobian matrix. The dynamic parameters refer to the physical properties of the joint module that affect the motion accuracy, such as the inertia, friction coefficient and load inertia. Specifically, they can be obtained in real time through parameter identification or sensor feedback, for example, by jointly estimating the torque sensor data and the motor current signal. Dynamic compensation refers to the feedforward or feedback correction of the joint motion command based on the dynamic model, for example, compensating for the displacement increment by calculating the inertia torque deviation caused by the joint acceleration.

[0127] Specifically, the translational compensation component and the rotational compensation component are obtained by analyzing the lateral compensation vector and the angle correction matrix in the offset compensation instruction. The Cartesian space displacement is converted into the displacement increment of each joint axis through the kinematic inverse algorithm. For example, when the lateral compensation vector is a 5 mm translation along the X-axis, the change in the rotation angle of each joint can be calculated by the inverse algorithm. The Jacobian matrix is ​​used to establish a mapping relationship between the end effector posture change and the joint angular velocity. For example, when it is necessary to rotate 0.1 radians around the Z axis, the proportional relationship of the angle correction of each joint is obtained by inverting the Jacobian matrix. The dynamic parameters are input into the dynamic compensation module. For example, when the joint load inertia increases, the calculation of the displacement increment needs to add the lag compensation caused by the inertia moment. The final target posture instruction contains the precise angle and movement speed of each joint axis, driving the joint module to move in coordination to eliminate the actual offset of the end effector.

[0128] Compared with existing technologies, traditional methods often directly apply the results of the inverse kinematic solution without considering the dynamic characteristics of the joints, resulting in trajectory deviations due to inertia during high-speed motion. This solution introduces a dynamic compensation mechanism for dynamic parameters, pre-compensating the joint displacement increments for inertia and friction during the inverse calculation phase, effectively suppressing end-positioning errors caused by load changes. For example, when grasping materials with sudden changes in mass, traditional methods may cause overshoot due to failure to adjust the joint acceleration limits in a timely manner. However, this solution adaptively corrects the displacement increments by updating the dynamic parameters in real time.

[0129] Through the above technical solution, this application achieves precise control of the robot arm's joint motion and dynamic error suppression, solves the problem of end-effector compensation lag caused by joint inertia and load changes, and improves the real-time compensation accuracy of material grasping offset. For example, in high-speed grasping scenarios, dynamic compensation can eliminate end-position deviations caused by joint motor response delays, ensuring that the three-dimensional offset between the material center point and the grasping point is stably controlled within the sub-millimeter range after the compensation action is executed.

[0130] In one possible implementation, reference Figure 7 The method further includes steps S710 to S720, wherein:

[0131] Step S710 , monitoring the actual offset of the end effector in real time during the process of the joint module performing the offset compensation action;

[0132] In step S720, if the actual offset exceeds the preset threshold, the process returns to the step of collecting feature image data of the material surface through the visual sensor, and collecting three-dimensional point cloud data of the material surface through the lidar, and constructing a material space coordinate system based on the image data and the three-dimensional point cloud data, so as to re-obtain the offset compensation instruction until the actual offset of the end effector meets the preset accuracy requirement.

[0133] In this embodiment, real-time monitoring of the actual offset refers to continuously measuring the relative position deviation between the end effector and the material through a laser tracker or encoder. Specifically, this can be achieved by using a dynamic error analysis algorithm based on a time series to determine whether the compensation action has achieved the expected accuracy. The preset threshold refers to the dynamic error tolerance value set according to the maximum position deviation allowed for the material grasping task. Specifically, it can be determined through experimental calibration or process parameter configuration, and is used to trigger the iterative update of the compensation instruction. The return execution step refers to reactivating the data acquisition process of vision and lidar when compensation failure is detected. Specifically, it can be achieved through an event-driven mechanism to reconstruct the coordinate system in a dynamic environment and generate more accurate compensation parameters.

[0134] Specifically, during the compensation action, the actual offset of the end effector is collected in real time and compared with a preset threshold. If the deviation exceeds the threshold, the visual sensor and lidar immediately initiate data collection, reconstruct the material's spatial coordinate system, and generate new offset compensation instructions. This process repeats until the actual offset of the end effector is controlled within the preset accuracy range, thus forming a closed-loop compensation control mechanism.

[0135] Compared to existing technologies, which typically only perform a single visual positioning before grasping, this solution cannot correct for accumulated errors in real time during dynamic grasping. This solution, by introducing real-time monitoring and iterative compensation mechanisms, can continuously correct offsets under dynamic conditions such as material vibration, arm joint hysteresis, or external interference, thus preventing grasping failures caused by error accumulation.

[0136] Through the above technical solution, this application can dynamically eliminate positioning error drift during the robot arm's movement, significantly improving gripping stability under complex working conditions. Through a closed-loop feedback mechanism, compensation instructions are continuously optimized to ensure that the material gripping process remains within the preset accuracy range, effectively solving the problem of secondary offset caused by environmental interference or mechanical errors.

[0137] In one possible implementation, reference Figure 8 The method further includes steps S810 to S840, wherein:

[0138] Step S810, after the end effector grabs the material, collecting pressure distribution data of the material contact surface and calculating the pressure gradient change rate;

[0139] Step S820: comparing the pressure gradient change rate with a preset safe grasping threshold, and identifying a slip trend of the contact surface between the material and the end effector when an abnormal pressure distribution is detected;

[0140] Step S830, dynamically correcting the compensation priority of the three-dimensional offset according to the slip trend, and generating a secondary compensation instruction including dynamic compensation parameters;

[0141] Step S840: adjusting the clamping force and compensation timing of the end effector according to the secondary compensation instruction until the pressure gradient change rate returns to a preset safety threshold range.

[0142] In this embodiment, the pressure gradient change rate refers to the rate of change of the pressure distribution on the material contact surface per unit time. Specifically, this can be achieved by using a pressure sensor array to collect pressure values ​​in various areas of the contact surface and differentially calculating the pressure changes in adjacent time windows. Its function is to identify abnormal contact stability between the material and the actuator in advance by quantifying the non-uniform changes in the pressure distribution. The slip trend refers to the relative displacement tendency between the material and the contact surface of the clamping mechanism due to insufficient friction coefficient or uneven distribution of clamping force. Specifically, it can be judged by the relationship between the pressure gradient change rate and a preset threshold. When the pressure distribution shows an asymmetric change with a sudden drop in pressure in the edge area and an increase in pressure in the center area, it can be determined that there is a slip trend. The dynamic compensation parameter refers to a combination of compensation parameters that are dynamically adjusted according to the severity of the slip trend. Specifically, it may include parameters such as the clamping force increment coefficient and the compensation action response priority coefficient, which are used to prioritize the suppression of slip risks during the three-dimensional offset compensation process.

[0143] Specifically, after the end effector completes the initial grasping, the pressure sensor array arranged on the contact surface of the gripper collects the pressure distribution data of the contact area in real time. By calculating the pressure gradient change rate of adjacent sampling cycles, local pressure anomalies caused by uneven material surface or insufficient clamping force can be detected. When it is detected that the pressure gradient change rate in a specific area exceeds the safety threshold, the system determines that there is a slip risk and triggers the secondary compensation mechanism. At this time, the compensation priority of the three-dimensional offset is dynamically adjusted, and the offset component related to the direction of the slip trend is corrected first. The secondary compensation instruction not only includes the spatial position correction parameters, but also synchronously adjusts the force curve of the clamping mechanism and the execution timing of the compensation action, and suppresses the slip trend by increasing the clamping force or changing the pressure distribution pattern of the contact surface until the pressure gradient change rate returns to a safe range.

[0144] Compared to existing technologies, traditional robotic gripping control relies solely on visual positioning data and lacks real-time monitoring of the mechanical state of the contact surface. This solution combines pressure distribution data with slip trends to proactively adjust compensation strategies when physical contact anomalies occur, effectively addressing potential grip failures caused by relying solely on spatial positioning data.

[0145] Through the above technical solution, the present application can monitor the mechanical state of the contact surface in real time during the material grasping process, and actively adjust the clamping parameters and compensation action timing when slippage occurs, so as to avoid grasping failure caused by material slippage and significantly improve the stability and reliability of the grasping process.

[0146] In one possible implementation, reference Figure 9 The method further includes steps S910 to S940, wherein:

[0147] Step S910: constructing an adaptive grasping strategy library based on material types, wherein the strategy library includes grasping parameters and compensation strategies for different material types;

[0148] Step S920, matching corresponding grabbing parameters and compensation strategies from the strategy library according to the type of the material to be grabbed;

[0149] Step S930: During the material grabbing process, monitor the changes in material type in real time and dynamically adjust the grabbing parameters and compensation strategy to adapt to the changes in material type;

[0150] Step S940: When the material type suddenly changes or cannot be matched to the corresponding grasping strategy, the exception handling mechanism is triggered to re-identify the material and re-plan the corresponding grasping strategy.

[0151] In this embodiment, the adaptive grasping strategy library refers to a set of grasping parameters associated with the physical properties of materials. Specifically, it can be implemented by a classified storage structure based on material density, surface friction coefficient and geometric dimensions, and rapid retrieval is achieved by establishing a mapping relationship between material feature codes and grasping parameters. Dynamic adjustment of grasping parameters refers to changes in material types detected in real time. Specifically, it can be achieved by identifying the surface texture or weight distribution characteristics of the material through an online classifier, and updating the clamping force threshold and compensation calculation coefficient in a linked manner. The exception handling mechanism refers to the emergency response process when unregistered materials or parameter mismatches are detected. Specifically, it can be implemented by the collaborative use of multi-sensor cross-validation and manual intervention request modules to ensure the safe operation of the system under unknown working conditions.

[0152] Specifically, before the robotic arm performs a grasping task, a grasping strategy database containing different material types such as metal parts, plastic parts, and glass products is pre-established. When the visual system recognizes that the material on the conveyor belt is a metal gear, the control system automatically calls the corresponding clamping force parameters, compensation response speed, and allowable offset threshold in the database. During the grasping process, if the infrared sensor detects an abnormal increase in the surface temperature of the material, resulting in a change in the friction coefficient, the system immediately adjusts the anti-slip correction coefficient in the compensation strategy. When a new type of composite material workpiece appears, the system automatically triggers the three-dimensional contour scanning and mechanical property analysis process, generates a temporary grasping strategy, and stores it in the strategy library for subsequent calls.

[0153] In some implementations, spectral analysis based on material surface reflectivity can be used to identify material types, combining historical captured data to optimize parameter combinations in the strategy library. When mixed material delivery scenarios are detected, the system can run multiple compensation strategy models in parallel and select the optimal solution based on confidence assessment.

[0154] Compared to existing technologies, traditional robotic gripping systems typically use fixed parameter configurations and are unable to adapt to mixed-product production scenarios. This solution, by establishing a dynamic strategy library and real-time type perception mechanism, addresses the issue of gripping failures caused by sudden changes in material properties and avoids the operational lag associated with frequent manual parameter adjustments.

[0155] Through the above technical solution, the present application can effectively cope with the complex working conditions of dynamic changes in material types in the production line, significantly reduce the grasping failure rate caused by misjudgment of material types, and at the same time reduce the system downtime under sudden abnormal working conditions, thereby improving the grasping robustness in multi-variety flexible production scenarios.

[0156] In an embodiment of the present application, a real-time compensation method for material grasping offset based on multi-sensor data fusion collects characteristic image data of the material surface through a visual sensor, and collects three-dimensional point cloud data of the material surface through a lidar, and constructs a material space coordinate system based on the image data and the three-dimensional point cloud data; and collects the torque data of the end effector through a torque sensor, and collects the posture data of the end effector through an inertial measurement unit; then the material space coordinate system, the torque data and the posture data are processed in time and space synchronization to generate corresponding multi-source fusion data; the multi-source fusion data is input into a pre-trained convolutional neural network model to output the three-dimensional offset of the material center point relative to the grasping point; then, based on the three-dimensional offset, an offset compensation instruction including a lateral compensation vector and an angle correction matrix is ​​generated; finally, based on the offset compensation instruction, the joint module is controlled to perform corresponding offset compensation actions to compensate for the offset occurring during the material grasping process. That is, the real-time compensation method for material grasping offset based on multi-sensor data fusion constructs the material space coordinate system by fusing multi-sensor data, and combines the neural network model to generate three-dimensional offset in real time and control the robotic arm compensation, which solves the problems of large positioning error and insufficient compensation accuracy of a single sensor, can improve the accuracy of material grasping, and thus enhance the stability of the automated production line.

[0157] This application also provides a real-time compensation device for material grabbing deviation based on multi-sensor data fusion, refer to Figure 10The real-time compensation device for material grabbing offset based on multi-sensor data fusion includes: a memory 10, a processor 20, and a real-time compensation program for material grabbing offset based on multi-sensor data fusion stored on the memory 10 and executable on the processor 20. The real-time compensation program for material grabbing offset based on multi-sensor data fusion is configured to implement the steps of the real-time compensation method for material grabbing offset based on multi-sensor data fusion.

[0158] The real-time compensation device for material grasping offset based on multi-sensor data fusion provided in this application adopts the real-time compensation method for material grasping offset based on multi-sensor data fusion in the above-mentioned embodiment, which can improve the accuracy of material grasping. Compared with the existing technology, the beneficial effects of the real-time compensation device for material grasping offset based on multi-sensor data fusion provided in this application are the same as the beneficial effects of the real-time compensation method for material grasping offset based on multi-sensor data fusion provided in the above-mentioned embodiment. The other technical features of the real-time compensation device for material grasping offset based on multi-sensor data fusion are the same as the features disclosed in the above-mentioned embodiment method, and are not further described here.

[0159] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A real-time compensation method for material grabbing deviation based on multi-sensor data fusion, characterized in that: Applied to a robotic arm, the robotic arm includes a joint module and an end effector connected to the joint module, the joint module is used to adjust the position of the end effector, and the end effector is used to grasp materials, the method comprising: Collecting characteristic image data of the material surface through a visual sensor and three-dimensional point cloud data of the material surface through a laser radar, and constructing a material space coordinate system based on the image data and the three-dimensional point cloud data; Collecting torque data of the end effector through a torque sensor, and collecting posture data of the end effector through an inertial measurement unit; Performing spatiotemporal synchronization processing on the material space coordinate system, the torque data, and the posture data to generate corresponding multi-source fusion data; Inputting the multi-source fusion data into a pre-trained convolutional neural network model to output a three-dimensional offset of the material center point relative to the grasping point; generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset; Controlling the joint module to perform corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset that occurs during the material grabbing process; The step of constructing a material space coordinate system based on the image data and the three-dimensional point cloud data includes: Performing edge detection and feature point extraction on the feature image data to obtain a material contour key point set; Registering the three-dimensional point cloud data with the feature point set to establish a transformation matrix between the visual coordinate system and the point cloud coordinate system; Performing coordinate transformation on the feature point set in the visual coordinate system based on the transformation matrix to generate a three-dimensional space coordinate sequence that integrates visual and point cloud features; Dynamically compensate the three-dimensional space coordinate sequence by timestamp alignment, and spatially map the compensated three-dimensional space coordinate sequence to the robot arm base coordinate system, and construct the material space coordinate system with the center point of the material as the origin of the coordinate system; The step of performing spatiotemporal synchronization processing on the material space coordinate system, the torque data, and the posture data to generate corresponding multi-source fusion data includes: Use timestamp alignment algorithm to synchronize material space coordinate system, torque data and posture data across modalities; The torque data is converted to the material space coordinate system through the space transformation matrix and vector synthesized with the posture data; Perform sliding window sampling on the multi-dimensional data stream after spatiotemporal synchronization to establish a spatiotemporal correlation matrix containing spatial position, mechanical state and motion trajectory; A Kalman filter is used to perform noise suppression and state estimation on the spatiotemporal correlation matrix to generate corresponding multi-source fusion data; The method further comprises: Collect historical offset datasets for various material specifications and gripping postures, and perform normalization and sample balancing on the datasets; Performing enhancement processing on the dataset to construct corresponding training sets and validation sets; Initializing network parameters using a transfer learning strategy for the training set, and constructing a dual-channel feature extraction network including a three-dimensional convolution kernel and an attention mechanism; The network weights are optimized through end-to-end training, and a dynamic learning rate decay strategy is used for iterative optimization until the mean absolute error of the model on the validation set is less than the preset threshold to obtain a pre-trained convolutional neural network model. The step of generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset comprises: Performing kinematic decomposition on the three-dimensional offset to obtain a lateral displacement component and an attitude deflection component; Converting the lateral displacement component into a compensation vector in a Cartesian coordinate system, and converting the posture deflection component into an angle correction matrix based on a homogeneous transformation matrix; Adjusting the weight ratio of the compensation vector and the angle correction matrix according to the real-time load state of the robot arm, and generating an offset compensation instruction including the compensation vector and the angle correction matrix through a weighted fusion algorithm; The step of controlling the joint module to perform a corresponding offset compensation action based on the offset compensation instruction to compensate for the offset occurring during the material grabbing process includes: determining a translation compensation component and a rotation compensation component of the end effector based on the offset compensation instruction; The translation compensation component is converted into the displacement increment of each joint axis of the joint module by using an inverse kinematics algorithm, and the rotation compensation component is converted into the angular correction value of each joint axis of the joint module based on the Jacobian matrix; Dynamically compensate the displacement increment and the angle correction amount according to the dynamic parameters of the joint module to generate a target posture instruction for each joint axis; The target posture instruction is sent to the joint module to drive each joint axis of the joint module to perform corresponding offset compensation actions to compensate for the offset that occurs during the material grasping process.

2. The real-time compensation method for material grabbing deviation based on multi-sensor data fusion according to claim 1 is characterized in that: The method further comprises: During the process of the joint module performing the offset compensation action, the actual offset of the end effector is monitored in real time; If the actual offset exceeds the preset threshold, the process returns to executing the steps of collecting feature image data of the material surface through the visual sensor, collecting three-dimensional point cloud data of the material surface through the lidar, and constructing the material space coordinate system based on the image data and the three-dimensional point cloud data, so as to re-acquire the offset compensation instruction until the actual offset of the end effector meets the preset accuracy requirement.

3. The real-time compensation method for material grabbing deviation based on multi-sensor data fusion according to claim 1 is characterized in that: The method further comprises: After the end effector grabs the material, collecting pressure distribution data of the material contact surface and calculating the pressure gradient change rate; Comparing the pressure gradient change rate with a preset safe grasping threshold, and identifying a slip trend of the contact surface between the material and the end effector when an abnormal pressure distribution is detected; Dynamically correcting the compensation priority of the three-dimensional offset according to the slip trend, and generating a secondary compensation instruction including dynamic compensation parameters; The clamping force and compensation timing of the end effector are adjusted according to the secondary compensation instruction until the pressure gradient change rate returns to the preset safety threshold range.

4. The real-time compensation method for material grabbing deviation based on multi-sensor data fusion according to claim 1 is characterized in that: The method further comprises: Build an adaptive grasping strategy library based on material types, which contains grasping parameters and compensation strategies for different material types; According to the type of material to be grasped, the corresponding grasping parameters and compensation strategies are matched from the strategy library; During the material grabbing process, the changes in material types are monitored in real time, and the grabbing parameters and compensation strategies are dynamically adjusted to adapt to the changes in material types; When the material type mutates or cannot be matched to the corresponding grasping strategy, the exception handling mechanism is triggered to re-identify the material and plan the corresponding grasping strategy.

5. A real-time compensation device for material grabbing deviation based on multi-sensor data fusion, characterized in that: The real-time compensation device for material grabbing offset based on multi-sensor data fusion includes: a memory, a processor, and a real-time compensation program for material grabbing offset based on multi-sensor data fusion stored in the memory and executable on the processor. The real-time compensation program for material grabbing offset based on multi-sensor data fusion is configured to implement the steps of the real-time compensation method for material grabbing offset based on multi-sensor data fusion as described in any one of claims 1 to 4.

Images