Material grabbing offset real-time compensation method and device based on multi-sensor data fusion

Abstract

The invention discloses a material grabbing offset real-time compensation method and device based on multi-sensor data fusion, relates to the technical field of automatic production, and discloses a material grabbing offset real-time compensation method and device based on multi-sensor data fusion. The three-dimensional offset is generated in real time in combination with the neural network model, mechanical arm compensation is controlled, the problems that a single sensor is large in positioning error and insufficient in compensation precision are solved, the material grabbing accuracy can be improved, and then the stability of an automatic production line is enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of automated production, and particularly to a real-time compensation method and device for material grasping offset based on multi-sensor data fusion. Background Art

[0002] The robotic arm is the core actuator for material grasping in an automated production line. In the prior art, material grasping is generally performed by the robotic arm through a single vision positioning technique. Due to changes in ambient light and interference from the reflection of the material surface, its positioning error is relatively large, resulting in frequent offsets during the material grasping process, low accuracy of material grasping, and affecting the stability and efficiency of the automated production line.

[0003] The above content is only used to assist in understanding the technical solution of the present application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of the present application is to provide a real-time compensation method and device for material grasping offset based on multi-sensor data fusion, aiming to improve the accuracy of material grasping.

[0005] To achieve the above objective, the present application proposes a real-time compensation method for material grasping offset based on multi-sensor data fusion, which is applied to a robotic arm. The robotic arm includes a joint module and an end effector connected to the joint module. The joint module is used to adjust the position of the end effector, and the end effector is used to grasp materials. The method includes:

[0006] Collecting characteristic image data of the material surface through a vision sensor, and collecting three-dimensional point cloud data of the material surface through a lidar, and constructing a material space coordinate system based on the image data and the three-dimensional point cloud data;

[0007] Collecting torque data of the end effector through a torque sensor, and collecting attitude data of the end effector through an inertial measurement unit;

[0008] Performing spatio-temporal synchronization processing on the material space coordinate system, the torque data, and the attitude data to generate corresponding multi-source fusion data;

[0009] Inputting the multi-source fusion data into a pre-trained convolutional neural network model to output the three-dimensional offset of the material center point relative to the grasping point;

[0010] Generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset;

[0011] Controlling the joint module to perform corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset occurring during the material grasping process.

[0012] In one embodiment, the step of constructing a material space coordinate system based on the image data and the three-dimensional point cloud data includes:

[0013] Perform edge detection and feature point extraction on the feature image data to obtain a set of key points of the material contour;

[0014] Register the three-dimensional point cloud data with the set of feature points to establish a transformation matrix between the visual coordinate system and the point cloud coordinate system;

[0015] Based on the transformation matrix, perform coordinate transformation on the set of feature points in the visual coordinate system to generate a three-dimensional space coordinate sequence that fuses visual and point cloud features;

[0016] Perform dynamic compensation on the three-dimensional space coordinate sequence through timestamp alignment, and perform spatial mapping on the compensated three-dimensional space coordinate sequence with the base coordinate system of the robotic arm. Using the center point of the material as the origin of the coordinate system, construct a material space coordinate system.

[0017] In one embodiment, the step of performing spatio-temporal synchronization processing on the material space coordinate system, the torque data, and the attitude data to generate corresponding multi-source fusion data includes:

[0018] Adopt a timestamp alignment algorithm to perform cross-modal synchronization on the material space coordinate system, torque data, and inertial data;

[0019] Convert the torque data to the material space coordinate system through a spatial transformation matrix and perform vector synthesis with the attitude data;

[0020] Perform sliding window sampling on the spatio-temporally synchronized multi-dimensional data stream to establish a spatio-temporal correlation matrix that includes spatial position, mechanical state, and motion trajectory;

[0021] Adopt a Kalman filter to perform noise suppression and state estimation on the spatio-temporal correlation matrix to generate corresponding multi-source fusion data.

[0022] In one embodiment, the method further includes:

[0023] Collect historical offset data sets under various material specifications and grasping postures, and perform normalization processing and sample balancing on the data sets;

[0024] Perform enhancement processing on the data sets to construct corresponding training sets and validation sets;

[0025] Adopt a transfer learning strategy to initialize network parameters for the training set, and construct a dual-channel feature extraction network that includes three-dimensional convolutional kernels and an attention mechanism;

[0026] Optimize the network weights through an end-to-end training method, and adopt a dynamic learning rate decay strategy for iterative optimization until the mean absolute error of the model on the validation set is less than a preset threshold to obtain a pre-trained convolutional neural network model.

[0027] In one embodiment, the step of generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset amount includes:

[0028] Perform kinematic decomposition on the three-dimensional offset amount to obtain a lateral displacement component and an attitude deflection component;

[0029] Convert the lateral displacement component into a compensation vector in the Cartesian coordinate system, and convert the attitude deflection component into an angle correction matrix based on the homogeneous transformation matrix;

[0030] Adjust the weight ratio of the compensation vector and the angle correction matrix according to the real-time load state of the robotic arm, and generate an offset compensation instruction including the compensation vector and the angle correction matrix through a weighted fusion algorithm.

[0031] In one embodiment, the step of controlling the joint module to perform corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset occurring during the material grasping process includes:

[0032] Determine the translational compensation component and the rotational compensation component of the end effector based on the offset compensation instruction;

[0033] Convert the translational compensation component into the displacement increment of each joint axis of the joint module through an inverse kinematic algorithm, and convert the rotational compensation component into the angular correction amount of each joint axis of the joint module based on the Jacobian matrix;

[0034] Dynamically compensate the displacement increment and the angular correction amount according to the dynamic parameters of the joint module to generate the target pose instruction of each joint axis;

[0035] Send the target pose instruction to the joint module to drive each joint axis of the joint module to perform corresponding offset compensation actions to compensate for the offset occurring during the material grasping process.

[0036] In one embodiment, the method further includes:

[0037] During the process of the joint module performing the offset compensation action, continuously monitor the actual offset amount of the end effector;

[0038] If the actual offset exceeds the preset threshold, return to execute the steps of collecting the feature image data of the material surface through a vision sensor, and collecting the three-dimensional point cloud data of the material surface through a lidar, and constructing a material space coordinate system based on the image data and the three-dimensional point cloud data to re-obtain an offset compensation instruction until the actual offset of the end effector meets the preset accuracy requirements.

[0039] In one embodiment, the method further includes:

[0040] After the end effector grasps the material, collect the pressure distribution data of the material contact surface and calculate the pressure gradient change rate;

[0041] Compare the pressure gradient change rate with a preset safe grasping threshold, and when abnormal pressure distribution is detected, identify the slip trend of the contact surface between the material and the end effector;

[0042] Dynamically correct the compensation priority of the three-dimensional offset according to the slip trend, and generate a secondary compensation instruction including dynamic compensation parameters;

[0043] Adjust the clamping force and compensation timing of the end effector according to the secondary compensation instruction until the pressure gradient change rate returns to the preset safe threshold range.

[0044] In one embodiment, the method further includes:

[0045] Construct an adaptive grasping strategy library based on the material type, and the strategy library includes grasping parameters and compensation strategies for different material types;

[0046] According to the type of the material to be grasped, match the corresponding grasping parameters and compensation strategies from the strategy library;

[0047] During the material grasping process, real-time monitor the change of the material type, and dynamically adjust the grasping parameters and compensation strategies to adapt to the change of the material type;

[0048] When the material type mutates or no corresponding grasping strategy can be matched, trigger an exception handling mechanism to re-identify the material and plan the corresponding grasping strategy.

[0049] In addition, to achieve the above object, the present application also proposes a real-time offset compensation device for material grasping based on multi-sensor data fusion. The real-time offset compensation device for material grasping based on multi-sensor data fusion includes: a memory, a processor, and a real-time offset compensation program for material grasping based on multi-sensor data fusion stored on the memory and executable on the processor. The real-time offset compensation program for material grasping based on multi-sensor data fusion is configured to implement the steps of the real-time offset compensation method for material grasping based on multi-sensor data fusion.

[0050] As can be seen from the above, a real-time compensation method and device for material grasping offset based on multi-sensor data fusion provided by the present application constructs a material space coordinate system by fusing multi-sensor data, generates a three-dimensional offset in real time in combination with a neural network model, and controls the robotic arm to compensate, solving the problems of large positioning error and insufficient compensation accuracy of a single sensor, being able to improve the accuracy of material grasping, and further enhancing the stability of the automated production line. Brief Description of the Drawings

[0051] The drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0052] To more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0053] Figure 1 It is a schematic flowchart provided for an embodiment of the real-time compensation method for material grasping offset based on multi-sensor data fusion of the present application;

[0054] Figure 2 For the present application Figure 1 It is a detailed schematic flowchart of step S100;

[0055] Figure 3 For the present application Figure 1 It is a detailed schematic flowchart of step S300;

[0056] Figure 4 It is a schematic flowchart provided for another embodiment of the real-time compensation method for material grasping offset based on multi-sensor data fusion of the present application;

[0057] Figure 5 For the present application Figure 1 It is a detailed flowchart of step S500;

[0058] Figure 6 For the present application Figure 1 It is a detailed flowchart of step S600;

[0059] Figure 7 It is a schematic flowchart provided for yet another embodiment of the real-time compensation method for material grasping offset based on multi-sensor data fusion of the present application;

[0060] Figure 8 It is a schematic flowchart provided for still another embodiment of the real-time compensation method for material grasping offset based on multi-sensor data fusion of the present application;

[0061] Figure 9 This is a schematic flowchart provided for another embodiment of the real-time compensation method for material grasping offset based on multi-sensor data fusion in this application;

[0062] Figure 10 This is a schematic structural diagram provided for an embodiment of the device for real-time compensation of material grasping offset based on multi-sensor data fusion in this application.

[0063] Explanation of the reference numerals in the accompanying drawings:

[0064] 10. Memory; 20. Processor.

[0065] The realization of the purpose, functional characteristics and advantages of this application will be further described in conjunction with the embodiments with reference to the accompanying drawings. Specific embodiments

[0066] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not used to limit this application.

[0067] For a better understanding of the technical solutions of this application, the following will be described in detail in conjunction with the drawings in the specification and specific embodiments.

[0068] The main solution of the embodiment of this application is: collecting characteristic image data of the material surface through a vision sensor, and collecting three-dimensional point cloud data of the material surface through a lidar, and constructing a material space coordinate system based on the image data and the three-dimensional point cloud data; collecting torque data of the end effector through a torque sensor, and collecting attitude data of the end effector through an inertial measurement unit; then performing spatio-temporal synchronization processing on the material space coordinate system, the torque data and the attitude data to generate corresponding multi-source fusion data; inputting the multi-source fusion data into a pre-trained convolutional neural network model to output the three-dimensional offset of the material center point relative to the grasping point; then generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset; finally, controlling the joint module to perform corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset occurring during the material grasping process.

[0069] In this embodiment, for the convenience of description, the following will be described with the device for real-time compensation of material grasping offset based on multi-sensor data fusion as the execution subject.

[0070] Since the robotic arm in the prior art is the core execution mechanism for material grasping in an automated production line, in the prior art, material grasping is generally performed by the robotic arm through a single vision positioning technology. Due to changes in ambient light and interference from the reflection of the material surface, its positioning error is relatively large, resulting in frequent deviations during the material grasping process, and the accuracy of material grasping is relatively low, affecting the stability and efficiency of the automated production line.

[0071] The solution provided by this application constructs a material space coordinate system by fusing multi-sensor data, combines a neural network model to generate a three-dimensional offset in real time, and controls the robotic arm to compensate, solving the problems of large positioning errors and insufficient compensation accuracy of a single sensor, being able to improve the accuracy of material grasping, and further enhancing the stability of the automated production line.

[0072] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of implementing the above functions, a real-time compensation device for material grasping offset based on multi-sensor data fusion, etc. Hereinafter, the real-time compensation device for material grasping offset based on multi-sensor data fusion will be taken as an example to illustrate this embodiment and the following embodiments.

[0073] In the prior art, material grasping in an automated production line usually relies on a robotic arm to complete, and the positioning accuracy of the robotic arm directly affects the success rate of material grasping. Traditional methods mainly use a single vision sensor for positioning, identify the position of the material through image processing, and control the movement of the robotic arm. However, in a complex industrial environment, factors such as changes in lighting conditions, reflection or occlusion of the material surface are likely to cause visual recognition errors, and then cause grasping deviations. For example, in the scenario of grasping metal parts, strong light reflection may cause the vision sensor to misjudge the edge position of the material, making the actual grasping point of the end effector deviate from the target center, and ultimately resulting in material dropping or assembly misalignment.

[0074] To solve the above problems, it is necessary to consider how to improve the real-time perception and compensation ability of the offset during the robotic arm grasping process. The traditional single-sensor solution has insufficient robustness in a dynamic environment and is difficult to accurately reflect the comprehensive state of the material position and the robotic arm posture. Through analysis, it is found that multi-modal data fusion can effectively make up for the limitations of a single sensor. For example, combining vision and three-dimensional point cloud data to construct a more accurate coordinate system, and at the same time introducing mechanical and kinematic parameters to enhance state perception. Further, real-time data processing and dynamic compensation mechanisms become the key to solving grasping deviations. It is necessary to establish a cross-sensor spatio-temporal synchronization framework to ensure the consistency of multi-source data in the time and space dimensions. Based on this, it is necessary to design a neural network model suitable for multi-dimensional data to achieve rapid prediction of the offset, and generate a composite control instruction including translation and rotation compensation, and finally achieve real-time compensation through the coordinated action of the joint module.

[0075] Based on this, an embodiment of the present application provides a real-time compensation method for material grasping offset based on multi-sensor data fusion, which is applied to a robotic arm. The robotic arm includes a joint module and an end effector connected to the joint module. The joint module is used to adjust the position of the end effector, and the end effector is used to grasp materials. Refer to Figure 1 , the method includes steps S100 to S600, where:

[0076] Step S100: Collect feature image data of the material surface through a vision sensor, and collect three-dimensional point cloud data of the material surface through a lidar, and construct a material space coordinate system based on the image data and the three-dimensional point cloud data;

[0077] Step S200: Collect torque data of the end effector through a torque sensor, and collect attitude data of the end effector through an inertial measurement unit;

[0078] Step S300: Perform spatio-temporal synchronization processing on the material space coordinate system, the torque data, and the attitude data to generate corresponding multi-source fusion data;

[0079] Step S400: Input the multi-source fusion data into a pre-trained convolutional neural network model to output the three-dimensional offset of the material center point relative to the grasping point;

[0080] Step S500: Generate an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset;

[0081] Step S600: Control the joint module to perform corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset occurring during the material grasping process.

[0082] Among them, the vision sensor refers to an image acquisition device used to capture the optical characteristics of the material surface. Specifically, it can be realized by using an industrial camera in combination with a ring light source, and the contour features of the material are extracted by obtaining high-contrast images. The lidar refers to a sensor that obtains the three-dimensional coordinates of an object through laser scanning. For example, a line-scanning lidar using the TOF ranging principle is used to construct a point cloud model of the material surface. The spatio-temporal synchronization processing refers to the operation of aligning the timestamps and unifying the spatial coordinates of multi-source data. Specifically, a hardware trigger synchronization mechanism can be adopted to ensure the time consistency of vision, lidar, and mechanical data, and the spatial reference alignment is achieved through a coordinate transformation matrix. The three-dimensional offset refers to the deviation of the actual position of the material from the expected grasping point in the three XYZ axes, which is obtained by non-linearly mapping multi-dimensional input data through a neural network model. The lateral compensation vector refers to the displacement correction parameter in the Cartesian coordinate system. For example, a two-dimensional vector containing the translation amounts in the X / Y directions is used to correct the planar position deviation of the end effector. The angle correction matrix refers to the rotation correction parameter represented by Euler angles or quaternions. For example, a homogeneous transformation matrix is used to describe the attitude adjustment amount of the end effector.

[0083] Specifically, during the material grasping process, the industrial camera and the lidar respectively collect the surface data of the material from the two dimensions of optical imaging and three-dimensional ranging. The RGB image obtained by the industrial camera undergoes edge detection and feature extraction to identify the key points of the outer contour of the material. The dense point cloud data generated by the lidar scanning is aligned with the visual feature points through a registration algorithm to establish a unified material space coordinate system. The six-axis force sensor on the end effector real-time monitors the clamping force distribution, and the inertial measurement unit collects the three-axis acceleration and angular velocity data of the end. Through the hardware synchronization trigger signal, the timestamps of the vision coordinate system, point cloud data, mechanical parameters, and attitude data are aligned and transformed to the same spatial reference. The fused data processed by the Kalman filter is input into a three-dimensional convolutional neural network. The model learns the mapping relationship from multi-source data to spatial offset through end-to-end training and outputs a three-dimensional offset containing position and attitude deviations. The motion control module decomposes the offset into lateral translation compensation and rotation angle correction to generate control instructions for driving the joint module. Each joint axis moves collaboratively according to the inverse kinematics solution result to drive the end effector to complete real-time position and attitude adjustment, and finally realizes the dynamic compensation of the grasping offset.

[0084] Compared with the prior art, the traditional single-vision positioning scheme is vulnerable to environmental light interference, leading to the accumulation of material edge recognition errors. In contrast, this scheme combines vision and lidar data to construct a spatial coordinate system, effectively improving the accuracy of coordinate system establishment through the complementary advantages of optical imaging and three-dimensional ranging. The existing methods do not consider the influence of the mechanical state of the end effector on the offset. This scheme introduces torque and inertia data to more comprehensively reflect the dynamic interaction state during the grasping process. In addition, the existing technology adopts a fixed-threshold compensation strategy, which is difficult to handle the non-linear offset problem under complex working conditions. This scheme realizes the adaptive offset prediction through a neural network model, significantly enhancing the intelligent level of compensation control.

[0085] Through the above technical solutions, this application effectively solves the grasping offset problem caused by traditional single-vision positioning. The multi-sensor data fusion mechanism enhances the environmental adaptability and can still accurately perceive the position of the material in reflective or occluded scenarios. The spatio-temporal synchronization processing ensures the consistency and real-time nature of multi-source data, laying a foundation for precise compensation. The neural network model realizes high-precision offset prediction through multi-dimensional feature learning, combined with the generation of composite compensation commands and joint collaborative control, ultimately achieving a grasping accuracy of sub-millimeter level. This method can significantly reduce the material grasping failure rate and improve the stability of the production line in scenarios such as automated assembly and precision part handling.

[0086] In a feasible implementation manner, referring to Figure 2 , step S100 includes steps S110 to S140, where:

[0087] Step S110: Perform edge detection and feature point extraction on the feature image data to obtain a set of key points of the material contour;

[0088] Step S120: Register the three-dimensional point cloud data with the feature point set to establish a transformation matrix between the visual coordinate system and the point cloud coordinate system;

[0089] Step S130: Based on the transformation matrix, perform coordinate transformation on the set of feature points in the visual coordinate system to generate a three-dimensional spatial coordinate sequence that combines visual and point cloud features;

[0090] Step S140: Dynamically compensate the three-dimensional spatial coordinate sequence through timestamp alignment, and perform spatial mapping between the compensated three-dimensional spatial coordinate sequence and the base coordinate system of the robotic arm. Using the center point of the material as the origin of the coordinate system, construct a material spatial coordinate system.

[0091] Among them, edge detection refers to the process of identifying the contour boundary of the material surface through image processing technology. Specifically, the Canny operator or Sobel operator can be used to achieve this, which is used to extract the geometric boundary information of the material in the two-dimensional image. Feature point extraction refers to selecting the positioning points with significant features from the material surface image. Specifically, the SIFT algorithm or ORB algorithm can be used to achieve this, which is used to establish the corresponding relationship of the material under different perspectives. The transformation matrix refers to the spatial conversion relationship between different coordinate systems. Specifically, the rotation and translation matrix in rigid body transformation can be used to achieve this, which is used to unify the data of the vision sensor and lidar into a common reference system. Timestamp alignment refers to the process of time synchronization of multi-source sensor data. Specifically, the interpolation compensation method or the nearest neighbor matching method can be used to achieve this, which is used to eliminate the spatio-temporal error caused by the sampling delay of different sensors.

[0092] Specifically, during the material grasping process, the vision sensor collects the feature image containing the material surface texture, extracts the set of pixel coordinates of the material contour edge through the edge detection algorithm, and at the same time the lidar obtains the three-dimensional spatial point cloud of the material surface. The two-dimensional image feature points are associated with the corresponding spatial points in the three-dimensional point cloud through the feature point matching algorithm, and the rotation and translation parameters between the two sensor coordinate systems are calculated to form the transformation matrix. Based on this matrix, the vision feature points are transformed into the point cloud coordinate system, and a three-dimensional coordinate sequence integrating the features of the two sensors is generated. Since the movement of the robotic arm may cause the time asynchronization of the collected data, the timestamp alignment method is used to perform dynamic interpolation compensation on the coordinate sequence to eliminate the coordinate drift caused by the movement. Finally, the compensated three-dimensional coordinates are mapped to the base coordinate system of the robotic arm, and a high-precision material space coordinate system is established with the centroid of the material as the origin.

[0093] Compared with the prior art, the traditional method only relies on a single vision sensor to construct the coordinate system, and is easily affected by the change of illumination or the reflection interference, resulting in unstable feature extraction. This solution effectively improves the accuracy and stability of the coordinate system construction by fusing the vision image and the lidar point cloud data, supplementing the depth information in the spatial dimension and implementing dynamic compensation in the time dimension.

[0094] Through the above technical solution, this application solves the positioning deviation problem of the material space coordinate system caused by sensor errors and motion interference, provides an accurate spatial reference for subsequent offset compensation, enables the robotic arm to accurately identify the relative offset between the material center position and the grasping point, and significantly improves the positioning accuracy and operation reliability of material grasping.

[0095] In a feasible implementation manner, with reference to Figure 3 , step S300 includes steps S310 to S340, where:

[0096] Step S310, perform cross-modal synchronization on the material space coordinate system, torque data, and inertial data using a timestamp alignment algorithm;

[0097] Step S320, convert the torque data to the material space coordinate system through a space transformation matrix and perform vector synthesis with the attitude data;

[0098] Step S330, perform sliding window sampling on the spatio-temporally synchronized multi-dimensional data stream to establish a spatio-temporal correlation matrix including spatial position, mechanical state, and motion trajectory;

[0099] Step S340, use a Kalman filter to suppress noise and estimate the state of the spatio-temporal correlation matrix to generate corresponding multi-source fusion data.

[0100] In this embodiment, the timestamp alignment algorithm refers to a method for cross-modal data synchronization by unifying the time bases of different sensor data, which can be specifically implemented by means of hardware clock synchronization or software interpolation compensation, and is used in the solution to eliminate the time delay differences in visual, torque, and inertial data. The space transformation matrix refers to a mathematical transformation tool for mapping mechanical data in different coordinate systems to a unified space coordinate system, which can be specifically implemented by a rigid transformation matrix or an affine transformation matrix, and is used in the solution to unify the mechanical states of torque data and the material space coordinate system. Sliding window sampling refers to a data processing method for dynamically intercepting a continuous data stream at a fixed time length, which can be specifically implemented by a fixed step size or an adaptive window length, and is used in the solution to extract multi-dimensional data segments with spatio-temporal correlation. The Kalman filter refers to a recursive algorithm for optimal estimation of a dynamic system through state equations and observation equations, which can be specifically implemented by a linear or extended Kalman filter, and is used in the solution to suppress sensor noise and estimate the true state of the system.

[0101] Specifically, during the process of the robotic arm grasping the material, the material space coordinate system constructed by the visual sensor and the lidar, the torque data and the attitude data of the end effector achieve microsecond-level time synchronization through the timestamp alignment algorithm, eliminating data misalignment caused by differences in sensor sampling frequencies. The torque data is converted to the material space coordinate system through a pre-calibrated space transformation matrix and vectorially superimposed with the attitude data to obtain a synthetic mechanical state. The synchronized multi-dimensional data stream is divided into multiple time windows for sampling, and each window contains spatial position, torque vector, and motion trajectory data, thereby forming a spatio-temporal correlation matrix. The Kalman filter filters the noise data in the correlation matrix according to the system dynamics model, and at the same time combines the historical state to predict the current optimal estimated value, and finally generates multi-source fusion data including denoised spatial coordinates, mechanical parameters, and motion trajectories.

[0102] Compared with the prior art, traditional material grasping systems usually directly use the original sensor data without spatio-temporal alignment and fusion, resulting in time deviation and coordinate inconsistency of different modality data, thus affecting the accuracy of offset calculation. This solution realizes the spatio-temporal consistency of multi-dimensional data through cross-sensor time synchronization and spatial coordinate mapping, and effectively suppresses sensor noise by combining the dynamic estimation ability of Kalman filtering, improving the accuracy and reliability of multi-source data fusion.

[0103] Through the above technical solution, this application effectively solves the problem of inaccurate compensation instructions caused by asynchronous sensor data and noise interference during the robotic arm grasping process, realizes high-precision spatio-temporal fusion of multi-source data, provides an accurate data basis for subsequent offset calculation, and thus significantly improves the position compensation accuracy and operation stability of material grasping.

[0104] In a feasible implementation, referring to Figure 4 , the method further includes steps S410 to S440, where:

[0105] Step S410, collect the historical offset data sets under various material specifications and grasping postures, and perform normalization processing and sample balancing on the data sets;

[0106] Step S420, perform augmentation processing on the data sets to construct corresponding training sets and validation sets;

[0107] Step S430, initialize the network parameters of the training set using a transfer learning strategy, and construct a dual-channel feature extraction network including three-dimensional convolutional kernels and attention mechanisms;

[0108] Step S440, optimize the network weights through an end-to-end training method, and perform iterative optimization using a dynamic learning rate decay strategy until the mean absolute error of the model on the validation set is less than a preset threshold to obtain a pre-trained convolutional neural network model.

[0109] In this embodiment, the historical offset dataset refers to a data set containing the actual offsets under different material forms and robotic arm grasping postures. Specifically, it can be achieved by synchronously collecting the actual grasping deviation data under different working conditions through multiple sensors, and is used to cover the offset patterns in diverse scenarios. Normalization processing refers to eliminating the dimensional differences of data from different sensors. Specifically, it can be achieved by using the maximum-minimum scaling or Z-score standardization method to ensure the consistency of the numerical range input to the neural network. Sample balancing refers to adjusting the distribution ratio of samples with different offsets in the data set. Specifically, it can be achieved by oversampling the minority class samples or undersampling the majority class samples to avoid the problem of class imbalance during model training. Augmentation processing refers to expanding the diversity of training data. Specifically, it can be achieved by using methods such as random rotation, translation, or adding Gaussian noise to enhance the robustness of the model to noise interference. The transfer learning strategy refers to reusing the feature extraction ability of the pre-trained model. Specifically, it can be achieved by using the convolutional layer parameters pre-trained on the ImageNet dataset as the initial weights to accelerate the network convergence speed. The three-dimensional convolutional kernel refers to a three-dimensional filter that can simultaneously extract spatio-temporal features. Specifically, it can be achieved by setting a multi-channel convolutional layer stacking structure and is used to capture the spatial correlation in multi-source data. The attention mechanism refers to a calculation module that dynamically adjusts the channel weights of the feature map. Specifically, it can be achieved by a channel attention or spatial attention module to enhance the network's focusing ability on key features. The dynamic learning rate decay strategy refers to gradually reducing the optimization step size according to the training process. Specifically, it can be achieved by using the exponential decay or cosine annealing algorithm to balance the model convergence speed and accuracy.

[0110] Specifically, in the model training stage, first collect historical offset data covering different material sizes, weights, and grasping angles, perform standardization processing on the original data to eliminate the sensor range differences, and balance the number of offset samples of each type through oversampling. Subsequently, apply random geometric transformation and noise injection operations to the training set to expand the data scale, and divide out a validation set to evaluate the model generalization performance. When constructing the dual-channel network, use the transfer learning method to load the pre-trained image feature extraction layer parameters. The three-dimensional convolutional layer processes the visual and mechanical feature data in parallel, and the attention module automatically screens the high-contribution feature channels. During the training process, use end-to-end backpropagation to update the network weights, and gradually optimize the loss function in combination with the dynamically adjusted learning rate strategy. When the mean absolute error on the validation set does not decrease for three consecutive training cycles, terminate the training, and finally obtain a neural network model that can accurately predict the three-dimensional offset.

[0111] Compared with the prior art, the traditional robotic arm offset compensation model usually relies on a single type of sensor data and lacks data augmentation means, resulting in insufficient generalization ability of the model under complex working conditions. Most of the existing methods adopt a fixed learning rate strategy for full-parameter training, which has the problems of slow convergence speed and being easily trapped in local optima. There is little in the prior art that combines transfer learning and three-dimensional convolution and applies it to the multi-modal data fusion scenario, making it difficult to effectively extract cross-domain feature correlations.

[0112] Through the above technical solutions, this application solves the problem of insufficient generalization ability caused by single training data of the traditional compensation model, and effectively improves the adaptability of the neural network to different material specifications and grasping postures. By combining transfer learning and the dynamic learning rate decay strategy, the model training cycle is shortened and the parameter optimization efficiency is improved, overcoming the defect of slow convergence speed of the traditional full-parameter training method. The dual-channel structure of three-dimensional convolution and the attention mechanism enhances the model's feature extraction ability for multi-source heterogeneous data, making the prediction results more accurate and reliable.

[0113] In a feasible implementation manner, referring to Figure 5 , the step S500 includes steps S510 to S530, where:

[0114] Step S510, kinematically decompose the three-dimensional offset to obtain a lateral displacement component and an attitude deflection component;

[0115] Step S520, convert the lateral displacement component into a compensation vector in the Cartesian coordinate system, and convert the attitude deflection component into an angle correction matrix based on the homogeneous transformation matrix;

[0116] Step S530, adjust the weight ratio of the compensation vector and the angle correction matrix according to the real-time load state of the robotic arm, and generate an offset compensation instruction including the compensation vector and the angle correction matrix through a weighted fusion algorithm.

[0117] In this embodiment, kinematic decomposition refers to the process of decomposing the offset in three-dimensional space into linear translation and rotation components. Specifically, it can be implemented by homogeneous coordinate transformation matrix operations, which is used to distinguish motion components in different directions. The compensation vector in the Cartesian coordinate system refers to the linear displacement quantity based on the base coordinate system of the robotic arm. Specifically, the lateral displacement component can be mapped to the motion plane of the robotic arm through the coordinate transformation formula to achieve precise compensation for translational errors. The homogeneous transformation matrix refers to a four-dimensional matrix containing rotation and translation information. Specifically, it can be generated by converting the Euler angles or quaternions of the attitude deflection components, which is used to describe the attitude correction requirements of the end effector. The weight ratio adjustment refers to dynamically allocating the priorities of translational and rotational compensations according to the current load torque of the joint module. Specifically, it can be implemented by a proportional coefficient dynamic calculation module based on torque sensor data to adapt to the mechanical constraints under different working conditions. The weighted fusion algorithm refers to an operation method of normalizing and superimposing compensation quantities in different dimensions. Specifically, a comprehensive compensation instruction can be generated through linear weighting or non-linear interpolation algorithms to ensure the coordination of multi-dimensional compensations.

[0118] Specifically, during the material grasping process, first, the three-dimensional offset output by the convolutional neural network is input into the kinematic decomposition module, and the lateral displacement and attitude deflection components are separated through homogeneous coordinate transformation. The lateral displacement component generates a compensation vector in the Cartesian space after coordinate transformation. For example, a translational correction amount in millimeters is generated in the X-Y plane. The attitude deflection component is converted into a rotation correction instruction for the end effector through the homogeneous transformation matrix. For example, an angle correction value around the Z axis is generated. At the same time, the load torque data of each joint of the robotic arm are collected in real time. When it is detected that the load of a certain joint approaches the threshold, the weight ratio of the corresponding direction compensation amount is reduced to avoid execution errors caused by overload. Finally, the compensation vector and the angle correction matrix are input into the weighted fusion module, and a comprehensive compensation instruction is generated through the dynamically allocated weight coefficients. For example, a higher weight is given to translational compensation in the low-load state, while angle correction is preferentially executed in the high-load state.

[0119] Compared with the prior art, traditional methods usually only adopt translational compensation in a single direction or multi-dimensional compensation with a fixed ratio, and cannot dynamically adjust the compensation strategy according to the real-time load state. For example, in the prior art, the weight ratio of the compensation vector and the angle correction matrix adopts a preset fixed value, which is likely to cause compensation overshoot or joint overload when the load of the robotic arm suddenly changes. This solution dynamically adjusts the weight through real-time load state feedback. For example, when grasping heavy materials, the translational compensation amplitude is reduced and the priority of attitude correction is increased, so as to avoid exceeding the dynamics of the robotic arm while maintaining the compensation accuracy.

[0120] Through the above technical solution, the present application can optimize the compensation strategy according to the actual working state of the robotic arm, and solve the problem of cumulative compensation error caused by fixed weights in traditional methods. Through the dynamic weight adjustment mechanism, the attitude stability can be preferentially guaranteed under high-load working conditions, and accurate displacement compensation can be achieved under low-load working conditions, thereby improving the adaptability and robustness of the material grasping process. For example, when handling fragile materials, the system automatically increases the weight of angle correction to reduce the surface stress concentration caused by the clamping attitude deviation, effectively reducing the risk of material breakage.

[0121] In a feasible implementation, referring to Figure 6 , step S600 includes steps S610 to S640, where:

[0122] Step S610, determining the translational compensation component and rotational compensation component of the end effector based on the offset compensation instruction;

[0123] Step S620, converting the translational compensation component into the displacement increment of each joint axis of the joint module through the inverse kinematics algorithm, and converting the rotational compensation component into the angular correction amount of each joint axis of the joint module based on the Jacobian matrix;

[0124] Step S630, dynamically compensating the displacement increment and angular correction amount according to the dynamic parameters of the joint module to generate the target pose instruction of each joint axis;

[0125] Step S640, sending the target pose instruction to the joint module to drive each joint axis of the joint module to perform the corresponding offset compensation action to compensate for the offset occurring during the material grasping process.

[0126] In this embodiment, the translational compensation component refers to the linear displacement amount that the end effector needs to move along the coordinate axis direction in the Cartesian coordinate system. Specifically, the inverse kinematics algorithm can be used to map the Cartesian space displacement to the joint space, for example, by using the Newton-Raphson method or the analytical method to solve the joint angle change amount. The rotational compensation component refers to the attitude adjustment amount of the end effector around each coordinate axis. Specifically, the homogeneous transformation matrix can be used to describe the rotational change, for example, by using the Jacobian matrix to decompose the attitude correction amount into the angular velocity or angular displacement increment of each joint axis. The dynamic parameters refer to the physical characteristics such as the inertia, friction coefficient, and load inertia of the joint module that affect the motion accuracy. Specifically, they can be obtained in real time through parameter identification or sensor feedback, for example, by using the torque sensor data and the motor current signal for joint estimation. Dynamic compensation refers to the feedforward or feedback correction of the joint motion instruction based on the dynamic model, for example, by compensating the displacement increment by calculating the inertial torque deviation caused by the joint acceleration.

[0127] Specifically, the translation compensation component and the rotation compensation component are obtained by parsing the lateral compensation vector and the angle correction matrix in the offset compensation instruction. The Cartesian space displacement is converted into the displacement increment of each joint axis through the inverse kinematics algorithm. For example, when the lateral compensation vector is a translation of 5 millimeters along the X-axis, the change in the rotation angle of each joint can be calculated through the inverse kinematics algorithm. The Jacobian matrix is used to establish the mapping relationship between the attitude change of the end effector and the joint angular velocity. For example, when it is necessary to rotate 0.1 radians around the Z-axis, the proportional relationship of the correction amount of each joint angle is obtained by inverting the Jacobian matrix. The dynamic parameters are input into the dynamic compensation module. For example, when the joint load inertia increases, the calculation of the displacement increment needs to superimpose the lag compensation amount caused by the inertia torque. The finally generated target pose instruction contains the precise angles and motion speeds of each joint axis, driving the joint modules to act cooperatively to eliminate the actual offset of the end effector.

[0128] Compared with the prior art, traditional methods usually directly apply the results of inverse kinematics without considering the joint dynamic characteristics, resulting in trajectory deviations due to inertia during high-speed motion. This solution effectively suppresses the end positioning error caused by load changes by introducing a dynamic compensation mechanism for dynamic parameters and pre-compensating the inertia torque and friction force for the joint displacement increment during the inverse kinematics calculation stage. For example, when grasping materials with a sudden change in mass, traditional methods may produce overshoot due to the failure to adjust the joint acceleration limit in time, while this solution can adaptively correct the displacement increment by updating the dynamic parameters in real time.

[0129] Through the above technical solution, this application realizes the precise control of the joint motion of the robotic arm and the suppression of dynamic errors, solves the problem of lag in the compensation action of the end effector caused by joint inertia and load changes, and improves the real-time compensation accuracy for the offset of material grasping. For example, in a high-speed grasping scenario, the dynamic compensation can eliminate the end position deviation caused by the response delay of the joint motor, ensuring that the three-dimensional offset between the center point of the material and the grasping point is stably controlled within the sub-millimeter range after the compensation action is executed.

[0130] In a feasible implementation manner, referring to Figure 7 , the method further includes step S710 to step S720, where:

[0131] Step S710, during the process of the joint module executing the offset compensation action, the actual offset of the end effector is monitored in real time;

[0132] Step S720, if the actual offset exceeds the preset threshold, then return to execute the steps of collecting the characteristic image data of the material surface through the vision sensor, and collecting the three-dimensional point cloud data of the material surface through the lidar, and constructing the material space coordinate system based on the image data and the three-dimensional point cloud data, so as to re-obtain the offset compensation instruction until the actual offset of the end effector meets the preset accuracy requirements.

[0133] In this embodiment, real-time monitoring of the actual offset means continuously measuring the relative position deviation between the end effector and the material through a laser tracker or an encoder. Specifically, it can be implemented by using a dynamic error analysis algorithm based on time series, which is used to determine whether the compensation action has reached the expected accuracy. The preset threshold refers to the dynamic error tolerance value set according to the maximum position deviation allowed for the material grasping task. Specifically, it can be determined through experimental calibration or process parameter configuration, and is used to trigger the iterative update of the compensation instruction. The return execution step means reactivating the data acquisition process of the vision and lidar when it is detected that the compensation fails. Specifically, it can be implemented through an event-driven mechanism, which is used to reconstruct the coordinate system in a dynamic environment and generate more accurate compensation parameters.

[0134] Specifically, during the execution of the compensation action, the actual offset of the end effector is collected in real time and compared with the preset threshold. If the deviation exceeds the threshold range, the vision sensor and the lidar immediately start the data acquisition process, reconstruct the material space coordinate system, and generate a new offset compensation instruction. This process is executed in a loop until the actual offset of the end effector is controlled within the preset accuracy range, thereby forming a closed-loop compensation control mechanism.

[0135] Compared with the prior art, the prior art usually only performs single visual positioning before grasping and cannot correct the cumulative error in real time during the dynamic grasping process. However, this solution can continuously correct the offset under dynamic conditions such as material vibration, robot arm joint backlash, or external interference by introducing a real-time monitoring and iterative compensation mechanism, avoiding grasping failures caused by error accumulation.

[0136] Through the above technical solution, the present application can dynamically eliminate the positioning error drift problem during the movement of the robot arm, significantly improving the grasping stability under complex working conditions. By continuously optimizing the compensation instruction through a closed-loop feedback mechanism, it is ensured that the material grasping process is always within the preset accuracy range, effectively solving the problem of secondary offset caused by environmental interference or mechanical errors.

[0137] In a feasible implementation manner, referring to Figure 8 , the method further includes step S810 to step S840, where:

[0138] Step S810, after the end effector grasps the material, collect the pressure distribution data of the material contact surface and calculate the pressure gradient change rate;

[0139] Step S820, compare the pressure gradient change rate with a preset safe grasping threshold, and when abnormal pressure distribution is detected, identify the slip trend of the contact surface between the material and the end effector;

[0140] Step S830: Dynamically correct the compensation priority of the three-dimensional offset according to the slipping trend, and generate a secondary compensation instruction including dynamic compensation parameters;

[0141] Step S840: Adjust the clamping force and compensation timing sequence of the end effector according to the secondary compensation instruction until the pressure gradient change rate returns to the preset safe threshold range.

[0142] In this embodiment, the pressure gradient change rate refers to the change rate of the pressure distribution on the material contact surface per unit time. Specifically, a pressure sensor array can be used to collect the pressure values of each area on the contact surface, and the pressure change amount between adjacent time windows can be calculated through differential calculation to achieve it. Its function is to identify the abnormal contact stability between the material and the actuator in advance by quantifying the non-uniform change of the pressure distribution. The slipping trend refers to the relative displacement tendency between the contact surface of the material and the clamping mechanism due to insufficient friction coefficient or uneven clamping force distribution. Specifically, it can be judged by the relationship between the pressure gradient change rate and the preset threshold. When the pressure distribution shows an asymmetric change with a sudden drop in pressure in the edge area and an increase in pressure in the central area, it can be determined that there is a slipping trend. The dynamic compensation parameter refers to a combination of compensation parameters dynamically adjusted according to the severity of the slipping trend, and specifically may include parameters such as the clamping force increment coefficient and the compensation action response priority coefficient, which are used to preferentially suppress the slipping risk during the three-dimensional offset compensation process.

[0143] Specifically, after the end effector completes the initial grasping, the pressure sensor array arranged on the contact surface of the gripper collects the pressure distribution data of the contact area in real time. By calculating the pressure gradient change rate of adjacent sampling periods, local pressure anomalies caused by uneven material surfaces or insufficient clamping force can be detected. When the pressure gradient change rate in a specific area exceeds the safe threshold, the system determines that there is a slipping risk and triggers the secondary compensation mechanism. At this time, the compensation priority of the three-dimensional offset is dynamically adjusted to preferentially correct the offset component related to the slipping trend direction. The secondary compensation instruction not only includes the spatial position correction parameters, but also synchronously adjusts the force application curve of the clamping mechanism and the compensation action execution timing sequence, and suppresses the slipping trend by increasing the clamping force or changing the contact surface pressure distribution mode until the pressure gradient change rate returns to the safe range.

[0144] Compared with the prior art, the traditional robotic arm grasping control only relies on visual positioning data and lacks real-time monitoring of the mechanical state of the contact surface. This solution actively adjusts the compensation strategy when physical contact anomalies occur through the joint analysis of pressure distribution data and slipping trends, effectively making up for the clamping failure problem that may be caused by simply relying on spatial positioning data.

[0145] Through the above technical solution, the present application can monitor the mechanical state of the contact surface in real time during the material grasping process, and actively adjust the clamping parameters and compensation action timing when slippage occurs, so as to avoid grasping failure caused by material slippage and significantly improve the stability and reliability of the grasping process.

[0146] In one possible implementation, reference Figure 9 The method further includes steps S910 to S940, wherein:

[0147] Step S910, constructing an adaptive grasping strategy library based on material types, wherein the strategy library contains grasping parameters and compensation strategies for different material types;

[0148] Step S920, matching corresponding grabbing parameters and compensation strategies from the strategy library according to the type of the material to be grabbed;

[0149] Step S930, during the material grabbing process, monitor the changes in material types in real time, and dynamically adjust the grabbing parameters and compensation strategies to adapt to the changes in material types;

[0150] Step S940: When the material type suddenly changes or cannot be matched to the corresponding grasping strategy, the exception handling mechanism is triggered to re-identify the material and plan the corresponding grasping strategy.

[0151] In this embodiment, the adaptive grasping strategy library refers to a set of grasping parameters associated with the physical properties of materials. Specifically, it can be implemented by a classified storage structure based on material density, surface friction coefficient and geometric dimensions, and rapid retrieval is achieved by establishing a mapping relationship between material feature codes and grasping parameters. Dynamic adjustment of grasping parameters refers to changes in material types detected in real time. Specifically, the surface texture or weight distribution characteristics of the material can be identified by an online classifier, and the clamping force threshold and compensation calculation coefficient are updated in a linked manner. The exception handling mechanism refers to the emergency response process when unregistered materials or parameter mismatches are detected. Specifically, it can be implemented in collaboration with multi-sensor cross-validation and manual intervention request modules to ensure the safe operation of the system under unknown working conditions.

[0152] Specifically, before the robotic arm performs the grasping task, a grasping strategy database containing different material types such as metal parts, plastic parts, and glass products is pre-established. When the visual system recognizes that the material on the conveyor belt is a metal gear, the control system automatically calls the corresponding clamping force parameters, compensation response speed, and allowable offset threshold in the database. During the grasping process, if the infrared sensor detects that the surface temperature of the material has increased abnormally, causing the friction coefficient to change, the system immediately adjusts the anti-slip correction coefficient in the compensation strategy. When a new type of composite material workpiece appears, the system automatically triggers the three-dimensional contour scanning and mechanical property analysis process, generates a temporary grasping strategy, and stores it in the strategy library for subsequent calls.

[0153] In some specific embodiments, spectral analysis technology based on the surface reflectivity of the material can be used to achieve type recognition, and the parameter combinations in the strategy library can be optimized by combining historical grasping data. When a material mixed transportation scenario is detected, the system can run multiple compensation strategy models in parallel and select the optimal solution for execution through confidence evaluation.

[0154] Compared with the prior art, traditional robotic arm grasping systems usually adopt fixed parameter configurations and cannot adapt to multi-variety mixed-flow production scenarios. This solution solves the problem of grasping failure caused by sudden changes in material properties by establishing a dynamic strategy library and a real-time type perception mechanism, and avoids the operation lag of manually adjusting parameters frequently.

[0155] Through the above technical solution, this application can effectively cope with the complex working conditions of dynamic changes in material types on the production line, significantly reduce the grasping failure rate caused by misjudgment of material types, and at the same time reduce the system downtime under sudden abnormal working conditions, improving the grasping robustness in multi-variety flexible production scenarios.

[0156] In the embodiments of this application, the real-time compensation method for material grasping offset based on multi-sensor data fusion collects characteristic image data of the material surface through a vision sensor, and collects three-dimensional point cloud data of the material surface through a lidar, and constructs a material space coordinate system based on the image data and the three-dimensional point cloud data; and collects torque data of the end effector through a torque sensor, and collects attitude data of the end effector through an inertial measurement unit; then performs spatio-temporal synchronization processing on the material space coordinate system, the torque data, and the attitude data to generate corresponding multi-source fusion data; and inputs the multi-source fusion data into a pre-trained convolutional neural network model to output the three-dimensional offset of the material center point relative to the grasping point; then generates an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset; and finally controls the joint module to execute corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset occurring during the material grasping process. That is, the real-time compensation method for material grasping offset based on multi-sensor data fusion constructs a material space coordinate system by fusing multi-sensor data, generates a three-dimensional offset in real time in combination with a neural network model, and controls the robotic arm to compensate, solving the problems of large positioning error and insufficient compensation accuracy of a single sensor, and being able to improve the accuracy of material grasping, thereby enhancing the stability of the automated production line.

[0157] This application also provides a device for real-time compensation of material grasping offset based on multi-sensor data fusion, refer to Figure 10, the real-time compensation device for material grasping offset based on multi-sensor data fusion includes: a memory 10, a processor 20, and a real-time compensation program for material grasping offset based on multi-sensor data fusion stored on the memory 10 and executable on the processor 20. The real-time compensation program for material grasping offset based on multi-sensor data fusion is configured to implement the steps of the real-time compensation method for material grasping offset based on multi-sensor data fusion as described above.

[0158] The real-time compensation device for material grasping offset based on multi-sensor data fusion provided in this application adopts the real-time compensation method for material grasping offset based on multi-sensor data fusion in the above embodiment, which can improve the accuracy of material grasping. Compared with the prior art, the beneficial effects of the real-time compensation device for material grasping offset based on multi-sensor data fusion provided in this application are the same as those of the real-time compensation method for material grasping offset based on multi-sensor data fusion provided in the above embodiment, and other technical features in the real-time compensation device for material grasping offset based on multi-sensor data fusion are the same as the features disclosed in the method of the above embodiment, which will not be elaborated here.

[0159] The above are only partial embodiments of this application, and thus do not limit the patent scope of this application. Any equivalent structural transformation made under the technical concept of this application by using the content of the specification and drawings of this application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of this application.

Claims

1. A real-time compensation method for material grasping offset based on multi-sensor data fusion, characterized in that, Applied to a robotic arm, the robotic arm includes a joint module and an end effector connected to the joint module. The joint module is used to adjust the position of the end effector, and the end effector is used to grasp materials. The method includes: Collecting characteristic image data of the material surface through a vision sensor, and collecting three-dimensional point cloud data of the material surface through a lidar, and constructing a material space coordinate system based on the image data and the three-dimensional point cloud data; Collecting the torque data of the end effector through a torque sensor, and collecting the attitude data of the end effector through an inertial measurement unit; Performing spatio-temporal synchronization processing on the material space coordinate system, the torque data, and the attitude data to generate corresponding multi-source fusion data; Inputting the multi-source fusion data into a pre-trained convolutional neural network model to output the three-dimensional offset of the material center point relative to the grasping point; Generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset; Controlling the joint module to perform corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset occurring during the material grasping process.

2. The real-time compensation method for material grasping offset based on multi-sensor data fusion according to claim 1, wherein, The step of constructing a material space coordinate system based on the image data and the three-dimensional point cloud data includes: Performing edge detection and feature point extraction on the characteristic image data to obtain a set of key points of the material contour; Registering the three-dimensional point cloud data with the feature point set to establish a transformation matrix between the vision coordinate system and the point cloud coordinate system; Performing coordinate transformation on the set of feature points in the vision coordinate system based on the transformation matrix to generate a three-dimensional space coordinate sequence integrating vision and point cloud features; Performing dynamic compensation on the three-dimensional space coordinate sequence through timestamp alignment, and performing spatial mapping on the compensated three-dimensional space coordinate sequence with the base coordinate system of the robotic arm, and constructing a material space coordinate system with the material center point as the coordinate origin.

3. The real-time compensation method for material grasping offset based on multi-sensor data fusion according to claim 2, wherein The step of performing spatio-temporal synchronization processing on the material space coordinate system, the torque data, and the attitude data to generate corresponding multi-source fusion data includes: Adopting a timestamp alignment algorithm to perform cross-modal synchronization on the material space coordinate system, torque data, and inertial data; Converting the torque data to the material space coordinate system through a spatial transformation matrix and performing vector synthesis with the attitude data; Performing sliding window sampling on the spatio-temporally synchronized multi-dimensional data stream to establish a spatio-temporal correlation matrix including spatial position, mechanical state, and motion trajectory; Adopting a Kalman filter to perform noise suppression and state estimation on the spatio-temporal correlation matrix to generate corresponding multi-source fusion data.

4. The real-time compensation method for material grasping offset based on multi-sensor data fusion according to claim 3, characterized in that The method further includes: Collecting a historical offset data set under various material specifications and grasping postures, and performing normalization processing and sample balancing on the data set; Performing enhancement processing on the data set to construct corresponding training sets and validation sets; Initializing network parameters for the training set using a transfer learning strategy to construct a dual-channel feature extraction network including three-dimensional convolutional kernels and an attention mechanism; Optimize the network weights through end-to-end training, and adopt a dynamic learning rate decay strategy for iterative optimization until the mean absolute error of the model on the validation set is less than a preset threshold to obtain a pre-trained convolutional neural network model.

5. The real-time compensation method for material grasping offset based on multi-sensor data fusion according to claim 4, characterized in that, The step of generating an offset compensation instruction including a lateral compensation vector and an angle correction matrix based on the three-dimensional offset amount includes: Perform kinematic decomposition on the three-dimensional offset amount to obtain a lateral displacement component and an attitude deflection component; Convert the lateral displacement component into a compensation vector in the Cartesian coordinate system, and convert the attitude deflection component into an angle correction matrix based on the homogeneous transformation matrix; Adjust the weight ratio of the compensation vector and the angle correction matrix according to the real-time load state of the robotic arm, and generate an offset compensation instruction including the compensation vector and the angle correction matrix through a weighted fusion algorithm.

6. The real-time compensation method for material grasping offset based on multi-sensor data fusion according to claim 5, wherein, The step of controlling the joint module to perform corresponding offset compensation actions based on the offset compensation instruction to compensate for the offset occurring during the material grasping process includes: Determine the translational compensation component and the rotational compensation component of the end effector based on the offset compensation instruction; Convert the translational compensation component into the displacement increment of each joint axis of the joint module through an inverse kinematics algorithm, and convert the rotational compensation component into the angular correction amount of each joint axis of the joint module based on the Jacobian matrix; Dynamically compensate the displacement increment and the angular correction amount according to the dynamic parameters of the joint module to generate the target pose instruction of each joint axis; Send the target pose instruction to the joint module to drive each joint axis of the joint module to perform corresponding offset compensation actions to compensate for the offset occurring during the material grasping process.

7. The real-time compensation method for material grasping offset based on multi-sensor data fusion according to claim 6, wherein, The method further includes: During the process of the joint module performing the offset compensation action, real-time monitor the actual offset amount of the end effector; If the actual offset amount exceeds the preset threshold, return to execute the steps of collecting the characteristic image data of the material surface through the vision sensor, collecting the three-dimensional point cloud data of the material surface through the lidar, and constructing the material space coordinate system based on the image data and the three-dimensional point cloud data to re-obtain the offset compensation instruction until the actual offset amount of the end effector meets the preset accuracy requirements.

8. The real-time compensation method for material grasping offset based on multi-sensor data fusion according to claim 1, characterized in that The method further includes: After the end effector grasps the material, collect the pressure distribution data of the material contact surface and calculate the pressure gradient change rate; Compare the pressure gradient change rate with a preset safe grasping threshold, and when abnormal pressure distribution is detected, identify the slip trend of the contact surface between the material and the end effector; Dynamically correct the compensation priority of the three-dimensional offset amount according to the slip trend to generate a secondary compensation instruction including dynamic compensation parameters; Adjust the clamping force and compensation timing sequence of the end effector according to the secondary compensation instruction until the pressure gradient change rate returns to the preset safe threshold range.

9. The real-time compensation method for material grasping offset based on multi-sensor data fusion according to claim 1, characterized in that The method further includes: Construct an adaptive grasping strategy library based on the material type, and the strategy library includes grasping parameters and compensation strategies for different material types; According to the type of the material to be grasped, match the corresponding grasping parameters and compensation strategies from the strategy library; During the material grasping process, the change of the material type is monitored in real time, and the grasping parameters and compensation strategies are dynamically adjusted to adapt to the change of the material type; When the material type mutates or the corresponding grasping strategy cannot be matched, an exception handling mechanism is triggered to re-identify the material and plan the corresponding grasping strategy.

10. A real-time compensation device for material grasping offset based on multi-sensor data fusion, characterized in that, The real-time compensation device for material grasping offset based on multi-sensor data fusion includes: a memory, a processor, and a real-time compensation program for material grasping offset based on multi-sensor data fusion stored on the memory and executable on the processor. The real-time compensation program for material grasping offset based on multi-sensor data fusion is configured to implement the steps of the real-time compensation method for material grasping offset based on multi-sensor data fusion according to any one of claims 1 to 9.

Images