Intelligent hand-eye calibration and adaptive correction system and method

Abstract

The invention relates to the field of robot vision positioning, and discloses an intelligent hand-eye calibration and self-adaptive correction system and method. According to the method, a mechanical arm is controlled to drive a camera to collect multi-modal calibration data, a convolutional neural network is utilized to identify a calibration plate mark point, and a three-dimensional coordinate under a camera coordinate system is calculated in combination with depth information; meanwhile, on the basis of an encoder and torque data, flexible deformation of the mechanical arm is compensated through a kinematic model and a self-adaptive rigidity model, and three-dimensional coordinates under a base coordinate system are obtained. And obtaining a hand-eye transformation matrix by solving a transformation relation between the two point sets. And repeatedly calibrating before and after operation, inputting the difference of the two transformation matrixes into the fault diagnosis neural network, outputting fault type probability distribution, and generating a maintenance strategy. According to the invention, high-precision hand-eye calibration, automatic compensation of flexible deformation and intelligent diagnosis of system state change can be realized, so that the long-term precision and reliability of a visual positioning system are improved.

Description

Technical Field

[0001] This invention relates to the field of robot vision positioning technology, and more specifically, to an intelligent hand-eye calibration and adaptive correction system and method. Background Technology

[0002] In industrial automation, the collaborative operation of robotic arms and vision systems is crucial for intelligent manufacturing. Hand-eye calibration, as the core step in establishing the spatial transformation relationship between the camera and the robotic arm's base coordinate system, directly affects the accuracy of visual positioning. Traditional methods are mostly designed for small, rigid robotic arms, assuming no structural deformation. However, in large robotic arm applications, long reach and heavy loads lead to significant flexible deformation, causing the actual end effector pose to deviate from the theoretical model, thus limiting the accuracy of traditional calibration methods.

[0003] The main problems with existing technologies are that they do not consider the flexible deformation of the robotic arm and use theoretical end-effector poses for calibration, which introduces systematic errors; calibration is usually a one-time process and cannot adapt to the system state drift caused by wear, loosening, and temperature changes during long-term operation, resulting in a decrease in accuracy over time; there is a lack of effective fault diagnosis mechanisms, making it difficult to quickly locate the cause when the system is abnormal, maintenance relies on experience, which is inefficient and easily increases downtime losses.

[0004] Furthermore, calibration involves multimodal data such as images, depth, encoders, and torque sensors. Existing methods do not adequately utilize the fusion of multi-source information and fail to fully explore the system state contained within. Simultaneously, traditional image processing methods exhibit poor robustness in recognizing calibration board markers at long distances or under varying lighting conditions, easily leading to missed or false detections and affecting calibration reliability. Therefore, there is an urgent need for an intelligent hand-eye calibration method capable of compensating for flexible deformation and achieving intelligent monitoring and fault diagnosis to ensure the long-term stable operation of the vision system. Summary of the Invention

[0005] This invention provides an intelligent hand-eye calibration and adaptive correction system and method, which solves the technical problems in related technologies such as the influence of flexible deformation on the hand-eye calibration accuracy of large robotic arms, the decrease in calibration accuracy over time, and the lack of intelligent fault diagnosis mechanisms.

[0006] This invention provides an intelligent hand-eye calibration and adaptive correction method, comprising the following steps: S1, control the robotic arm and camera bracket to move together to the preset pose, and simultaneously collect color images, depth images, encoder values ​​of motors at each joint and torque sensor values ​​of each joint to obtain a multimodal calibration dataset; S2, based on the color image and depth image in the multimodal calibration dataset, identify the marker points on the calibration board and extract the pixel coordinates, combine the in-camera participation depth value to calculate the three-dimensional spatial position of the marker points, and obtain the set of three-dimensional coordinates of the marker points in the camera coordinate system; S3. Based on the encoder values ​​and torque values ​​in the multimodal calibration dataset, the compensated end pose is calculated by combining the mechanical arm kinematic model with the flexible deformation compensation module. The coordinate transformation is performed by combining the calibration plate installation pose with the physical coordinates of the marker points to obtain the three-dimensional coordinate set of the marker points in the mechanical arm base coordinate system. S4. Based on the set of three-dimensional coordinates of the marker points in the camera coordinate system and the base coordinate system, solve for the rotation matrix and translation vector of the two point sets to obtain the hand-eye transformation matrix. S5. Based on the hand-eye transformation matrix, repeat the above steps before and after the robotic arm operation to obtain the current transformation matrix. Input the initial and current transformation matrices into the fault diagnosis model, output the fault type probability distribution, and generate a maintenance strategy based on the fault type.

[0007] In a preferred embodiment, S1 includes: Control the robotic arm and camera support to move in coordination to a preset calibration pose sequence, which includes a preset number of different spatial positions and posture combinations; For each target pose, after the robotic arm reaches the target pose and stabilizes, it synchronously acquires color images, depth images, encoder values ​​of motors at each joint of the robotic arm, torque sensor values ​​of torque sensors at each joint of the robotic arm, and encoder values ​​of motors at each joint of the camera bracket through hardware trigger signals. The collected data and collection timestamps are packaged into a set of calibration data; all calibration poses are traversed to obtain a multimodal calibration dataset; The acquired multimodal calibration dataset is preprocessed and quality checked. The color images are denoised and contrast enhanced, and the depth images are filtered to remove depth measurement noise.

[0008] In a preferred embodiment, S2 includes: The color image is input into a pre-trained convolutional neural network to identify marker points and combinations of marker points, and outputs the bounding box coordinate confidence score of each combination of marker points on the calibration board, retaining the detection results whose confidence scores exceed the preset confidence threshold; Calculate the coordinates of each bounding box and the corresponding pixel coordinates of the marker points; spatially sort the identified marker points according to the known arrangement order of the marker points on the calibration board. Read the camera intrinsic parameter matrix, and for the pixel coordinates of each marked point, extract the depth value of the corresponding position from the depth image; use the inverse transformation of the pinhole camera imaging model to convert the pixel coordinates and depth values ​​into three-dimensional coordinates in the camera coordinate system.

[0009] In a preferred embodiment, S3 includes: The theoretical pose of the robotic arm end effector is calculated based on the encoder values ​​through the forward kinematics model of the robotic arm. The forward kinematics model of the robotic arm adopts the DH parameter representation. According to the DH parameters and joint angles, the homogeneous transformation matrix of each joint coordinate system relative to the previous joint coordinate system is calculated in sequence. The transformation matrices of each joint are multiplied in sequence to obtain the theoretical pose transformation matrix of the robotic arm end effector coordinate system relative to the base coordinate system. Flexible deformation compensation is based on an adaptive stiffness model. A multi-factor coupled model of joint stiffness is established. The stiffness model realizes the nonlinear mapping from input parameters to stiffness coefficients through a radial basis function neural network. By combining the installation pose of the calibration plate and the physical coordinates of the marker points, a coordinate system chain transformation is performed to obtain the set of three-dimensional coordinates of the marker points in the coordinate system of the robot arm base.

[0010] In a preferred embodiment, the radial basis function neural network includes an input layer, a hidden layer, and an output layer; The input layer receives parameters such as joint torque, joint angle, joint temperature, and cumulative running time, and normalizes each parameter to eliminate the influence of dimensions. The hidden layer contains several radial basis function neurons, each neuron corresponding to a center vector and a width parameter. The output of the neuron is calculated using Gaussian radial basis functions. The output layer is a single linear neuron. The stiffness coefficient is obtained by performing an inner product operation between the hidden layer output vector and the weight vector and adding a bias. The flexible deformation angle of the joint is calculated based on the stiffness coefficient and the current torque value. The forward kinematics calculation is then performed again based on the compensated actual joint angle to obtain the compensated end-effector pose transformation matrix.

[0011] In a preferred embodiment, S4 includes: Center the two point sets separately, calculate the centroid coordinates of the point sets, and the coordinates of the centered points are equal to the original coordinates minus the centroid coordinates; Construct a covariance matrix based on the two centered point sets, and then perform singular value decomposition on the covariance matrix to obtain an orthogonal matrix. The rotation matrix is ​​calculated based on the results of singular value decomposition, and the determinant and orthogonality of the rotation matrix are verified. The translation vector is calculated based on the rotation matrix and the centroid coordinates of the two point sets; the rotation matrix and translation vector are combined into a homogeneous transformation matrix to obtain the hand-eye transformation matrix; Calculate the calibration residual and evaluate the calibration accuracy. For each pair of corresponding points in the point set, use the eye-hand transformation matrix to transform the point coordinates in the camera coordinate system to obtain the predicted coordinates. Calculate the Euclidean distance between the predicted coordinates and the coordinates of the marked point in the base coordinate system as the calibration residual.

[0012] In a preferred embodiment, S5 includes: The calibration review process is executed under preset trigger conditions, including time-triggered, event-triggered, and periodic trigger conditions. A simplified calibration pose sequence is used to repeatedly perform the data acquisition, marker point identification, coordinate calculation and transformation matrix solution process to obtain the hand-eye transformation matrix at the current moment; Extract the difference features from the initial transformation matrix and the current transformation matrix, including the change in rotation angle, the change in translation, the calibration residual statistics, the pose residual and the deformation parameters. Combine the extracted parameters to form the original feature vector and then normalize it. The normalized feature vector is input into the fault diagnosis neural network model, and the probability distribution of fault types is calculated through forward propagation.

[0013] In a preferred embodiment, the fault diagnosis neural network model adopts a fully connected neural network, which includes an input layer, two hidden layers and an output layer; The input layer receives a normalized feature vector; the first and second hidden layers use the ReLU activation function, and Dropout regularization is applied after the second hidden layer. The output layer corresponds to multiple fault types. The original output vector is normalized by applying the Softmax activation function to obtain the probability vector. The category with the highest probability value in the probability vector is selected as the predicted fault type. If the probability of the normal state is the highest and exceeds the first preset threshold, the system is determined to be in a normal state. If the probability of one of the other fault types is the highest and exceeds the second preset threshold, the system is determined to have a fault of the corresponding type.

[0014] In a preferred embodiment, the predicted fault types include normal state, robotic arm origin offset, camera support structure offset, and robotic arm flexible deformation. Generate corresponding maintenance strategies for different fault types; Establish a maintenance feedback mechanism. After completing maintenance, maintenance personnel will provide feedback on the actual fault type and maintenance results. The feedback information will be associated with the corresponding feature vectors to form new labeled samples. An online learning strategy is used to continuously optimize the fault diagnosis model. After accumulating a preset number of feedback samples, new samples are added to the training dataset to retrain the neural network model.

[0015] In a preferred embodiment, an intelligent hand-eye calibration and adaptive correction system is used to perform the steps of the aforementioned intelligent hand-eye calibration and adaptive correction method, including: The data acquisition module is used to control the robotic arm and camera bracket to move in coordination to a preset pose, and to simultaneously acquire color images, depth images, encoder values ​​of motors at each joint, and torque sensor values ​​of each joint to obtain a multimodal calibration dataset. The marker recognition module is used to identify markers on the calibration board and extract pixel coordinates based on color images and depth images in the multimodal calibration dataset. It also calculates the three-dimensional spatial position of the markers by combining the in-camera depth values, and obtains a set of three-dimensional coordinates of the markers in the camera coordinate system. The coordinate transformation module is used to calculate the compensated end pose based on the encoder values ​​and torque values ​​in the multimodal calibration dataset, through the kinematic model of the robotic arm combined with the flexible deformation compensation module, and to perform coordinate transformation by combining the installation pose of the calibration plate and the physical coordinates of the marker points to obtain the three-dimensional coordinate set of the marker points in the coordinate system of the robotic arm base. The hand-eye calibration module is used to solve for the rotation matrix and translation vector of the two point sets based on the three-dimensional coordinate sets of the marker points in the camera coordinate system and the base coordinate system, so as to obtain the hand-eye transformation matrix. The fault diagnosis module is used to obtain the current transformation matrix by repeatedly executing the above steps before and after the robotic arm operation based on the hand-eye transformation matrix. The initial and current transformation matrices are input into the fault diagnosis model, which outputs the probability distribution of fault types and generates maintenance strategies based on the fault types.

[0016] The beneficial effects of this invention are as follows: An adaptive stiffness model is constructed to compensate for the flexible deformation of a robotic arm in real time. This model employs a radial basis function neural network to map multiple factors such as joint torque, angle, temperature, and runtime into stiffness coefficients. It automatically adjusts parameters based on the actual state of the robotic arm and calculates the compensated end-effector pose, thereby eliminating the impact of deformation on calibration accuracy. This method is particularly suitable for large robotic arms, improving calibration accuracy to the millimeter level. Furthermore, the model supports incremental learning, automatically updating parameters based on continuously collected calibration data to adapt to robotic arm aging and wear, eliminating the need for periodic manual recalibration and effectively reducing maintenance costs. A neural network-based intelligent fault diagnosis mechanism was established. By analyzing the differences between the initial and current hand-eye transformation matrices, multi-dimensional features, including rotation angle changes, translation changes, calibration residual statistics, pose residuals, and deformation parameters, were extracted and input into a fully connected neural network to output the probability distribution of various faults. The system can automatically identify various fault types, such as robotic arm origin offset, camera support structure offset, and flexible deformation, and generate targeted maintenance strategies and inspection suggestions accordingly. This mechanism realizes a shift from passive maintenance to proactive prevention, helping maintenance personnel quickly locate the cause of faults, reduce troubleshooting time and unplanned downtime, and improve system availability and reliability. Attached Figure Description

[0017] Figure 1 This is a flowchart of an intelligent hand-eye calibration and adaptive correction method according to the present invention; Figure 2 This is a flowchart of an intelligent hand-eye calibration and adaptive correction method according to the present invention. Detailed Implementation

[0018] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0019] At least one embodiment of the present invention discloses an intelligent hand-eye calibration and adaptive correction method, such as Figures 1 to 2 As shown, it includes the following steps: S1, control the robotic arm and camera bracket to move together to the preset pose, and simultaneously collect color images, depth images, encoder values ​​of motors at each joint and torque sensor values ​​of each joint to obtain a multimodal calibration dataset; S11 controls the robotic arm and camera bracket to move in coordination to a preset calibration pose sequence, and simultaneously collects multimodal data; In one embodiment of the present invention, in order to obtain multimodal data of the calibration board in different poses, an intelligent calibration program is started to control the robotic arm and the camera support to move in coordination to a preset calibration pose sequence; the calibration pose sequence includes a preset number of different spatial positions and posture combinations, such as 12 to 18 poses, to ensure that the calibration board presents different angles and distances within the camera's field of view. For each target pose, after the robotic arm reaches and stabilizes in the target pose, the following data are synchronously collected via a hardware trigger signal: Color image data is acquired by a color camera, and the image format is RGB three-channel color image, which is used for visual recognition of the marker points on the calibration board. Depth image data is acquired through a depth camera. The resolution of the depth image is consistent with that of the color image. The depth measurement range is set according to the actual application scenario, such as 0.5 meters to 5 meters. The depth value is in millimeters and is used to obtain the spatial depth information of the marker point. The encoder values ​​of each joint motor of the robotic arm are read by the robotic arm controller. These values ​​include the absolute position values ​​of the encoders of the robotic arm joints, which are expressed in angles and are used to calculate the end-effector pose of the robotic arm. The torque sensor values ​​of each joint of the robotic arm are collected by the torque sensor, including the current torque output value of the robotic arm joint, and the value is in Newton-meter, which is used for flexible deformation compensation calculation; The encoder values ​​of each joint motor of the camera bracket are read by the camera bracket controller. The encoder position values ​​of the three joints are used to record the status of the camera bracket.

[0020] The acquired color images, depth images, robotic arm encoder values, torque sensor values, camera mount encoder values, and acquisition timestamps are packaged into a single set of calibration data. The acquisition process is repeated for all calibration poses to obtain a preliminary multimodal calibration dataset.

[0021] S12, preprocessing and quality checking of the collected multimodal data; In one embodiment of the present invention, the acquired multimodal calibration dataset is preprocessed and quality checked; the color image is preprocessed, including denoising, contrast enhancement and other operations, to improve image quality for subsequent marker recognition; the depth image is filtered, using median filtering or bilateral filtering to remove depth measurement noise and smooth the depth data; Perform a quality check on each set of calibration data, checking whether the color and depth images are clear and complete, and whether the calibration board is completely within the camera's field of view; check whether the encoder and torque values ​​are within a reasonable range and whether there are any abnormal jumps; if a set of data has obvious quality problems, mark the set of data as to be reviewed, and re-acquire the data of that pose if necessary. The data, after preprocessing and quality checks, constitute a complete multimodal calibration dataset, which is used for subsequent marker identification and coordinate calculation.

[0022] S2, based on the color image and depth image in the multimodal calibration dataset, identify the marker points on the calibration board and extract the pixel coordinates, combine the in-camera participation depth value to calculate the three-dimensional spatial position of the marker points, and obtain the set of three-dimensional coordinates of the marker points in the camera coordinate system; S21, a convolutional neural network is used to identify the marker points on the calibration board and extract the pixel coordinates of the marker points; Color and depth images of each calibration dataset are extracted from the multimodal calibration dataset. For each dataset, the color image is input into a pre-trained convolutional neural network for marker recognition. This convolutional neural network adopts a multi-task object detection architecture, which not only recognizes individual markers but also recognizes pre-trained marker combination patterns. Image features are extracted through multi-layer convolution operations, and the output includes the bounding box coordinates and confidence scores of individual markers, as well as the bounding box coordinates, combination type labels, and confidence scores of marker combination patterns. The marker combination patterns include a variety of predefined geometric structures, specifically: horizontal three-point combinations, which are horizontal linear combinations of three adjacent markers in the same row; vertical three-point combinations, which are vertical linear combinations of three adjacent markers in the same column; L-shaped four-point combinations, which are L-shaped structures of four adjacent markers in two rows and two columns; square-shaped four-point combinations, which are rectangular structures of four adjacent markers in two rows and two columns; and cross-shaped five-point combinations, which are cross structures of the center point and its four adjacent points above, below, left, and right. These combination patterns are learned during the training phase by labeling a large number of calibration board images containing different combinations, enabling the network to recognize the spatial relationship features of the markers. Set a first confidence threshold for a single marker point and a second confidence threshold for a combination of marker points, for example, 0.85 and 0.80 respectively. Retain detection results with confidence exceeding the thresholds and filter out uncertain detections with low confidence. For each identified single marker point, calculate the geometric center of its bounding box as the marker point's pixel coordinates. For each identified combination of marker points, calculate the pixel coordinates of each marker point within the combination based on the combination type, bounding box coordinates, and the known physical spacing of the marker points. Pixel coordinates are represented in the image coordinate system. The recognition results are fused and quality checked. First, a candidate set of marker points is established, and the marker point coordinates derived from the recognition results of individual marker points and combined recognition results are uniformly added to the candidate set. For duplicate marker points with close positions in the candidate set, a weighted average method is used for fusion. The weights are determined according to the detection confidence level. The higher the confidence level, the greater the weight. The fused pixel coordinates are the weighted average of the coordinates of each candidate point. Through fusion processing, the complementary information of single-point recognition and combined recognition can be comprehensively utilized to improve the accuracy and robustness of marker point localization. To address the issue of insufficient recognition of individual marker points, a combination compensation mechanism is activated. When the number of recognized individual marker points is less than a preset threshold, the recognized marker point combinations are analyzed to deduce the location of missing marker points from the combinations. For example, if a horizontal three-point combination is recognized, but only two marker points are recognized by single-point detection, the pixel coordinates of the third point are calculated based on the bounding box of the combination and the positions of the two already recognized points. The calculation method involves using linear interpolation or extrapolation to calculate the coordinates of the missing point based on the spacing of the recognized points and the geometric constraints of the combination. For L-shaped or tic-tac-toe four-point combinations, if some points are missing, the coordinates of the missing points are also calculated based on the positions of the recognized points and the rectangular structural constraints of the combination. Through the combination compensation mechanism, even if some marker points are not recognized by single-point detection due to occlusion, lighting, or distance, their positions can still be recovered through the combination information, ensuring that the recognition quantity meets the requirements. The number of fused and compensated markers is counted. If the number of identified markers is less than the preset minimum marker count threshold, the marker recognition quality of the data set is deemed insufficient, and the data set is marked as invalid and not included in subsequent coordinate calculations. The proportion of valid data in all calibration data is counted. If the proportion of valid data is lower than the preset valid data proportion threshold, the calibration data acquisition quality is indicated, and it is recommended to repeat step S1 for data acquisition. For valid calibration data, the identified markers are spatially sorted according to the known arrangement order of the markers on the calibration board. The sorting method is as follows: first, the markers are divided into 3 groups according to the pixel x-coordinate values ​​from smallest to largest, corresponding to the 3 columns of the calibration board. Within each group, the markers are then arranged according to the pixel y-coordinate values ​​from smallest to largest, corresponding to the 5 rows of each column. The correspondence between pixel coordinates and physical markers on the calibration board is established in the order from left to right and from top to bottom to ensure the correctness of subsequent coordinate transformations. S22, combined with the camera's internal depth values, converts pixel coordinates into three-dimensional coordinates in the camera coordinate system; Read the camera intrinsic parameter matrix. The camera intrinsic parameters include focal length parameters fx and fy, and principal point coordinates cx and cy. These parameters are obtained through camera factory calibration or field calibration. For the pixel coordinates of each marker point, extract the corresponding depth value from the depth image. The depth value represents the distance from that pixel to the camera optical center, in millimeters. The inverse transformation of the pinhole camera imaging model is used to convert pixel coordinates and depth values ​​into three-dimensional coordinates in the camera coordinate system. The conversion calculation process is as follows: the Z coordinate in the camera coordinate system is equal to the depth value; the X coordinate is calculated by subtracting the principal point's x-coordinate from the pixel's x-coordinate to obtain the pixel offset, multiplying the pixel offset by the depth value, and then dividing by the lateral focal length fx to obtain the X coordinate; the Y coordinate is calculated by subtracting the principal point's y-coordinate from the pixel's y-coordinate to obtain the pixel offset, multiplying the pixel offset by the depth value, and then dividing by the longitudinal focal length fy to obtain the Y coordinate; through the above calculations, the three-dimensional coordinates of the marker point in the camera coordinate system are obtained, in the form of X_cam, Y_cam, and Z_cam, in millimeters; Repeat the above identification, extraction, and transformation process for all valid calibration data in the multimodal calibration dataset to obtain the 3D coordinates of the marker points corresponding to each set of calibration data; summarize the 3D coordinates of the marker points in all valid calibration data to form a set of 3D coordinates of marker points in the camera coordinate system, which contains the 3D coordinates of all valid marker points in multiple calibration poses; record the calibration pose number and marker point number to which each marker point belongs to ensure that it can be correctly matched with the marker points in the base coordinate system in the future; S23, perform quality assessment and anomaly handling on depth data and marker sorting; In one embodiment of the present invention, for cases where depth values ​​are missing in a depth image, a neighborhood interpolation method is used to repair the depth values. When the depth value of the depth map position corresponding to the pixel coordinates of a marker point is zero or invalid, the valid depth values ​​within a 5×5 neighborhood around that position are extracted. The valid depth values ​​are statistically analyzed, and outliers deviating from the median by more than twice the standard deviation are removed. The weighted average of the remaining valid depth values ​​is calculated as the depth estimate of the marker point. The weight is determined based on the distance between the pixel and the center of the marker point; the closer the distance, the greater the weight. If the number of valid depth values ​​in the neighborhood is less than 15, i.e., the proportion of valid data is less than 60%, the depth data of the marker point is determined to be unreliable, and the marker point is marked as invalid and removed from the set of marker points of the calibration data. To address the issue of small marker size in images due to the distance between the calibration board and the camera in large robotic arm scenarios, a multi-scale training strategy is employed during the convolutional neural network training phase. The training dataset includes calibration board images acquired at different distances, ensuring the model's ability to recognize markers and combinations of different sizes. The training dataset not only labels the position of individual markers but also labels the bounding boxes and combination types for various patterns, including horizontal three-point combinations, vertical three-point combinations, L-shaped four-point combinations, grid-shaped four-point combinations, and cross-shaped five-point combinations. Through multi-task learning, the network simultaneously learns both single-point detection and combination detection tasks, sharing the underlying feature extraction layer and outputting single-point and combination results using independent detection heads. During the inference phase, the average distance between the calibration board and the camera is estimated based on the depth image. When the distance exceeds 3 meters, the input image is locally magnified to improve the image resolution of the markers and enhance recognition accuracy. When the distance is too great for individual markers to be easily identified, combinations of markers, with their larger visual features and stronger structural constraints, achieve a higher recognition success rate. To address the issue of incorrect marker sorting, a geometric consistency verification mechanism is established. The three-dimensional spatial distance between adjacent markers is calculated, and the actual spatial distance is compared with the theoretical distance based on the known physical distance between markers on the calibration board. Verification includes the row spacing between adjacent markers in the same column and the column spacing between adjacent markers in the same row. If the deviation between the spatial distance of adjacent markers and the theoretical distance exceeds a preset geometric deviation threshold, it is determined that the marker sorting may be incorrect, triggering a re-sorting process. The re-sorting process employs a geometric constraint-based optimization method. By minimizing the sum of squared deviations between the distances of all adjacent markers and the theoretical distance, while constraining the marker arrangement to a preset matrix structure, the optimal marker arrangement order is searched to obtain the correct sorting result. If the geometric consistency requirement is still not met after re-sorting, i.e., the minimum deviation still exceeds the threshold, the set of calibration data is marked as invalid data. S3. Based on the encoder values ​​and torque values ​​in the multimodal calibration dataset, the compensated end pose is calculated by combining the mechanical arm kinematic model with the flexible deformation compensation module. The coordinate transformation is performed by combining the calibration plate installation pose with the physical coordinates of the marker points to obtain the three-dimensional coordinate set of the marker points in the mechanical arm base coordinate system. S31, calculates the theoretical pose of the robotic arm end effector based on encoder values ​​using a forward kinematics model; Extract the encoder values ​​of each joint motor and the torque sensor values ​​of each joint of the robotic arm from the multimodal calibration dataset; for each set of data, first calculate the theoretical pose of the robotic arm end effector based on the encoder values ​​using the forward kinematics model of the robotic arm; The forward kinematics model of the robotic arm adopts an improved DH parameter representation method to read the kinematic parameters of the robotic arm, including the length of each link, the torsion angle of the link, and the joint offset distance. The encoder values ​​are converted into the angle values ​​of each joint. For example, for a 6-DOF robotic arm, 6 joint angles are obtained. Based on the DH parameters and joint angles, the homogeneous transformation matrix of each joint coordinate system relative to the previous joint coordinate system is calculated in sequence. Each transformation matrix is ​​a 4×4 matrix, which includes rotation and translation parts. The transformation matrices of each joint are multiplied in sequence to obtain the theoretical pose transformation matrix of the robotic arm end-effector coordinate system relative to the base coordinate system. This matrix describes the position and attitude of the end-effector. In one embodiment of the present invention, the forward kinematics calculation process is illustrated using a 6-DOF robotic arm as an example. The DH parameters of the robotic arm include the link length a, link torsion angle α, joint offset distance d, and joint angle θ of each joint. Taking the first joint as an example, its DH parameters include the link length, link torsion angle, joint offset distance, and joint angle read by the encoder. The DH parameters of the remaining joints are defined in the same way. According to the standard transformation criteria of the DH parameter method, the transformation of each joint coordinate system relative to the previous joint coordinate system is composed of four basic transformations: first, rotate by an angle θ around the Z-axis of the previous joint; then translate by a distance d along the Z-axis; then translate by a distance a along the new X-axis; and finally rotate by an angle α around the X-axis. These four transformations are combined to form a homogeneous transformation matrix. The general form of the matrix is ​​a 4×4 matrix, where the rotation part is composed of a combination of trigonometric functions of θ and α, and the translation part is determined by the parameters a and d. The transformation matrix of each joint is calculated according to the above criteria. The first row of each transformation matrix T is cosθ, negative sinθ multiplied by cosα, sinθ multiplied by sinα, and a multiplied by cosθ; the second row is sinθ, cosθ multiplied by cosα, negative cosθ multiplied by sinα, and a multiplied by sinθ; the third row is 0, sinα, cosα, and d; and the fourth row is 0, 0, 0, and 1. Substitute the DH parameters and joint angles corresponding to each joint into the above general matrix form to calculate the transformation matrices of T1 to T6 in sequence. According to the chain rule of coordinate transformation, the transformation of the robotic arm's end-effector coordinate system relative to the base coordinate system is equal to the sequential combination of the transformations of each joint coordinate system. Since each joint transformation matrix T describes the relationship between two adjacent coordinate systems, to obtain the total transformation of the end-effector relative to the base, it is necessary to multiply the transformation matrices of all joints sequentially from the base to the end-effector, because the multiplication of homogeneous transformation matrices represents a continuous transformation of the coordinate systems. The transformation matrix T of the end-effector relative to the base is calculated as T1 multiplied by T2 multiplied by T3 multiplied by T4 multiplied by T5 multiplied by T6. Matrix multiplication is performed sequentially from left to right, with each multiplication transforming the current coordinate system to the next, ultimately resulting in the complete transformation from the base coordinate system to the end-effector coordinate system. The resulting T matrix is ​​a 4×4 matrix, where the upper left 3×3 submatrix is ​​the rotation matrix R, describing the attitude of the end-effector coordinate system, and the upper right 3×1 subvector is the translation vector P, describing the position of the end-effector coordinate system. The three components of the translation vector P represent the spatial coordinates of the end-effector in the base coordinate system. S32, flexible deformation compensation is performed based on an adaptive stiffness model to obtain the compensated end pose; In one embodiment of the present invention, in order to address the problem of flexible deformation of a large robotic arm under its own weight and load, an adaptive flexible deformation compensation module is introduced to correct the theoretical pose; the module adopts a dynamic stiffness model based on historical data self-learning, which can automatically adjust the stiffness parameters according to the actual operating state of the robotic arm. A multi-factor coupled model for joint stiffness is established, which considers multiple influencing factors such as torque, pose, temperature, and running time. The stiffness model is expressed as Ki = f(τ_i, θ_i, T, t_run), where Ki is the equivalent stiffness coefficient of the i-th joint, τ_i is the joint torque, θ_i is the joint angle, T is the joint temperature, and t_run is the cumulative running time. The function f represents the nonlinear mapping relationship from the input parameters to the stiffness coefficient, which is implemented through a radial basis function neural network (RBFNN). Specifically, the RBFNN network consists of three layers: an input layer, a hidden layer, and an output layer. The input layer is responsible for receiving and preprocessing the input parameters. Since the input parameters τ_i, θ_i, T, and t_run have different units (torque τ_i is in Newton-meters, joint angle θ_i is in radians, temperature T is in degrees Celsius, and runtime t_run is in hours), each parameter needs to be normalized to eliminate the influence of the units. The maximum and minimum values ​​of each parameter are determined according to the working range of the robotic arm. For example, the torque range is 0 to 300 Newton-meters, the angle range is -180 degrees to 180 degrees (converted to -π to π radians), the temperature range is 0 to 60 degrees Celsius, and the runtime range is 0 to 10000 hours. The normalized torque, angle, temperature, and runtime values ​​are arranged in order to form the input vector x, which is a column vector containing these normalized values. The hidden layer contains several radial basis function neurons, each implementing local response characteristics. The j-th neuron corresponds to a center vector c_j and a width parameter σ_j. The center vector c_j has the same dimension and structure as the input vector x, representing the center position of the neuron in the input space. The output h_j of the j-th neuron is calculated using Gaussian radial basis functions. The calculation process is as follows: first, calculate the difference vector between the input vector x and the center vector c_j; then, calculate the Euclidean norm of the difference vector to obtain the distance d_j, which is equal to the square root of the sum of the squares of the elements of the difference vector; substitute the distance d_j into the Gaussian radial basis function... The Gaussian function is used to calculate the neuron output h_j, which is equal to exp(negative d_j squared) divided by 2σ_j squared. The Gaussian function's properties mean that when the input vector x is close to the center vector c_j, the neuron output h_j is close to 1; when the input vector x is far from the center vector c_j, the neuron output h_j is close to 0. The width parameter σ_j controls the width of the Gaussian function; a larger σ_j indicates a wider response range for the neuron. The outputs of all neurons in the hidden layer are calculated separately, and the outputs h_j of each neuron are arranged in order to form the hidden layer output vector h. The output layer consists of a single linear neuron, responsible for mapping the output of the hidden layer to stiffness coefficients. Each output layer neuron corresponds to a weight vector w and a bias scalar b. The dimension of the weight vector w is the same as the number of hidden layer neurons, and the j-th element w_j represents the contribution weight of the j-th hidden layer neuron to the output. The stiffness coefficient K_i is calculated by performing an inner product operation between the hidden layer output vector h and the weight vector w, i.e., summing w_j over all j, multiplying by h_j, and then adding the bias b. The result is K_i, which is equal to the sum of w_j multiplied by h_j plus b. This calculation process realizes a weighted linear combination of the hidden layer outputs. In one embodiment of the present invention, the training process of the RBFNN network is as follows: First, training data is collected. The training data is obtained through experimental measurement. Calibration experiments are performed on the robotic arm under different working states. State parameters such as torque, angle, temperature, and runtime of each joint are recorded. At the same time, the actual pose of the robotic arm end is measured by a high-precision external measuring device such as a laser tracker. The actual pose is compared with the theoretical pose to calculate the actual deformation angle of each joint. The actual stiffness coefficient K_target under this state is obtained by back-calculating based on the deformation angle and torque value. The actual stiffness coefficient K_target is equal to the joint torque divided by the deformation angle. Through multiple experiments, measurements are taken under different combinations of torque, angle, temperature, and runtime to obtain multiple sets of input-output sample pairs. Each set of samples contains an input vector x and the corresponding target stiffness coefficient K_target. The number of hidden layer neurons and the position of the center vector c_j are determined. The center vector is obtained by clustering the input data of the training samples. The K-means clustering algorithm is used to divide the input vector of the training samples into several classes, and the center of each class is used as the center vector c_j of a radial basis function neuron. The width parameter σ_j is determined according to the distance between the center vectors, and is usually set to a number of times the average distance between adjacent center vectors to make the response regions of each neuron overlap appropriately. After determining the center vector and width parameters, the weight vector w and bias b of the output layer are trained. The training data contains multiple input-output sample pairs, each containing an input vector x and a corresponding target stiffness coefficient K_target. For each training sample, the input vector x is input into the network, the hidden layer output vector h is calculated, and then the network output K_pred is calculated based on the current weight vector w and bias b. The error between the predicted output K_pred and the target output K_target is calculated, and the error is equal to K_target minus K_pred. The weight vector w and bias b are solved using the least squares method to minimize the sum of squared errors of all training samples. The hidden layer output vectors h of all training samples are arranged in rows to form a matrix H, and the target stiffness coefficients K_target of all training samples are arranged in column vector Y. The solution of the weight vector w and bias b is then transformed into the solution of a system of linear equations, and the optimal weights and biases are obtained through matrix operations. After training, all parameters of the RBFNN network are determined, including the center vector c_j, width parameter σ_j, output layer weight vector w, and bias b of each neuron. For new input parameters, forward propagation is performed in the order of input layer normalization, hidden layer radial basis function calculation, and output layer weighted summation to obtain the corresponding stiffness coefficient prediction values. During each calibration process, the state parameters such as torque, angle, and temperature of each joint are recorded, as well as the actual deformation amount obtained by back-calculation through calibration residuals. The actual deformation amount is compared with the deformation amount predicted by the current stiffness model, and the prediction error is calculated. After accumulating a certain amount of calibration data, the network parameters of RBFNN are updated using an incremental learning algorithm. The incremental learning process still uses recursive least squares (RLS) to update the output layer weights, avoiding retraining with all historical data. For newly collected data samples, the hidden layer output vector h_new is calculated first, and then the current network's predicted output K_pred for that sample is calculated. The prediction error e is calculated as the actual stiffness value minus K_pred. According to the update formula of recursive least squares, the weight vector w is adjusted. The update formula is w_new equal to w_old plus the gain vector multiplied by the prediction error e. The gain vector is calculated based on the hidden layer output h_new and the covariance matrix. At the same time, the covariance matrix is ​​updated to reflect the information of the new samples. A forgetting factor λ is introduced, with a value ranging from 0.95 to 0.99. When updating the covariance matrix, historical information is attenuated, so that the model assigns higher weights to recent data and gradually forgets historical data, thereby adapting to the aging and wear characteristics of the robotic arm. For each joint, the feature vector is input into the RBFNN model based on the current torque, angle, temperature and running time, and the stiffness coefficient K_i of the joint is calculated by forward propagation; the flexible deformation angle of the joint is calculated based on the stiffness coefficient and the current torque value, and the deformation angle Δθ_i is equal to the torque τ_i divided by the stiffness coefficient K_i. Taking the second joint as an example, the calculation process of the adaptive stiffness model is explained as follows: Assume that the current state of the second joint is a torque of 150 N·m, a joint angle of 45 degrees, a temperature of 35 degrees Celsius, and a cumulative running time of 500 hours. After normalizing these four parameters, they are input into the RBFNN model, and the model outputs a stiffness coefficient of 12,500 N·m per radian. The deformation angle of the joint is calculated to be 150 divided by 12,500, which gives 0.012 radians, or approximately 0.69 degrees. The encoder reads an angle of 45 degrees, which is the theoretical angle. The actual joint will have an additional angle offset due to flexible deformation. The compensated actual joint angle is equal to the encoder angle of 45 degrees plus the deformation angle of 0.69 degrees, which gives 45.69 degrees, which is converted to 0.7974 radians. The above calculations were performed on each of the six joints of the robotic arm to obtain the actual joint angles after compensation for each joint; the state parameters and compensation results of this calibration were also recorded as training samples for subsequent model updates. Based on the compensated actual joint angles, the forward kinematics calculation is re-executed to obtain the compensated pose transformation matrix of the robotic arm end effector after considering flexible deformation; this compensated pose transformation matrix more realistically reflects the actual end effector pose of the large robotic arm under the current load state. The adaptive stiffness model can automatically adapt to the aging, wear and environmental changes of the robotic arm, eliminating the need for manual recalibration of stiffness parameters periodically, and significantly improving the accuracy and stability of long-term operation. S33, combining the installation pose of the calibration plate and the physical coordinates of the marker points, performs a coordinate system chain transformation; The installation pose parameters of the calibration board in the end-effector coordinate system are read. The installation pose describes the spatial relationship between the calibration board coordinate system and the end-effector coordinate system, including relative position and relative orientation. The installation pose is obtained as follows: During calibration board installation, a coordinate measuring machine or laser tracker is used to measure the position coordinates of the origin of the calibration board coordinate system in the end-effector coordinate system, and the direction vectors of each axis of the calibration board coordinate system in the end-effector coordinate system are measured. A 4×4 homogeneous transformation matrix is ​​constructed based on the position and orientation, denoted as the transformation matrix T_end_to_board from the end-effector to the calibration board. This transformation matrix remains fixed after the calibration board installation is completed and is stored in the system configuration file for subsequent use.

[0023] Read the design parameters of the calibration board. The physical coordinates of the marker points on the calibration board in the calibration board coordinate system are known. The calibration board coordinate system is defined with the origin located at the geometric center of the calibration board, the X-axis along the column direction of the marker points, the Y-axis along the row direction of the marker points, and the Z-axis perpendicular to the plane of the calibration board and pointing outward. According to the calibration board design drawings, the marker points are arranged in a preset matrix form, and the spacing between adjacent marker points is a preset value. Taking one embodiment as an example, the calibration plate has 15 circular markers arranged in a matrix of 3 columns and 5 rows, with a spacing of 140 mm between adjacent markers. The coordinates of each marker are calculated as follows: The first column of markers is located in the negative X-axis direction, with an X-coordinate of -140 mm. It contains 5 markers, with Y-coordinates of 280, 140, 0, -140, and -280 mm from top to bottom. The corresponding coordinates are -140, 280, 0; -140, 140, 0; -140, 0, 0; -140, -140, 0; and -140, -280, 0. The second column of markers is located at the origin of the X-axis, with an X-coordinate of 0 mm. It contains 5 markers, with Y-coordinates of 280, 140, 0, -140, and -280 mm from top to bottom. The marker with a Y-coordinate of 0 is the origin of the calibration plate coordinate system, with corresponding coordinates of 0, 280, 0, 0, 140, 0, 0, 0, 0, -140, 0, 0, -280, 0. The third column of markers is located in the positive X-axis direction, with an X-coordinate of 140 mm. It contains 5 markers, and their Y-coordinates from top to bottom are 280, 140, 0, -140, and -280 mm, corresponding to coordinates of 140, 280, 0; 140, 140, 0; 140, 0, 0; 140, -140, 0; and 140, -280, 0. The Z-coordinate of all markers is 0, in millimeters; the markers are numbered 1 to 15 from left to right and top to bottom. For each marker point, a chain transformation of the coordinate system is performed, transforming the marker point from the calibration board coordinate system to the end-effector coordinate system and then to the base coordinate system. The three-dimensional coordinates of the marker point are expanded into four-dimensional homogeneous coordinates, with the fourth dimension having an element of 1. The homogeneous coordinates of the marker point in the end-effector coordinate system are calculated, which is equal to the end-effector to calibration board transformation matrix T_end_to_board multiplied by the homogeneous coordinates of the marker point in the calibration board coordinate system. The matrix multiplication yields a 4×1 homogeneous coordinate vector. The homogeneous coordinates of the marker point in the base coordinate system are calculated, which is equal to the end-effector compensation pose transformation matrix calculated by S32 multiplied by the homogeneous coordinates of the marker point in the end-effector coordinate system. The three-dimensional coordinates are extracted from the homogeneous coordinates. The first three elements of the homogeneous coordinate vector are taken as the three-dimensional coordinates of the marker point in the robot arm base coordinate system, in the form of X_base, Y_base, and Z_base, with the unit being millimeters. Repeat the above compensation calculation and coordinate transformation process for all calibration data in the multimodal calibration dataset to obtain the three-dimensional coordinates of the marker points corresponding to each set of calibration data in the base coordinate system; summarize the three-dimensional coordinates of the marker points in all calibration data to form a set of three-dimensional coordinates of the marker points in the robot arm base coordinate system. This set contains the coordinates of all marker points under multiple calibration poses; this set of three-dimensional coordinates corresponds one-to-one with the set of three-dimensional coordinates of the marker points in the camera coordinate system obtained in S2 according to the calibration pose number and the marker point number. The corresponding point pairs are used for subsequent hand-eye calibration calculations.

[0024] S4. Based on the set of three-dimensional coordinates of the marker points in the camera coordinate system and the base coordinate system, solve for the rotation matrix and translation vector of the two point sets to obtain the hand-eye transformation matrix. S41, perform centering on the two point sets; The set of three-dimensional coordinates of the marker points in the camera coordinate system is obtained from S2, denoted as point set P_cam. The set of three-dimensional coordinates of the marker points in the robot arm base coordinate system is obtained from S3, denoted as point set P_base. The points in the two point sets correspond one-to-one with the calibration pose number and the marker point number, ensuring that the i-th point represents the coordinates of the same physical marker point in the same calibration pose in both coordinate systems. The number of points in the point sets is the same, denoted as N points, where N is equal to the number of effective calibration poses multiplied by the number of effectively identified marker points in each pose. The goal of hand-eye calibration is to solve the spatial transformation relationship from the camera coordinate system to the robot arm base coordinate system. This transformation relationship consists of a rotation matrix R and a translation vector t, which satisfies the relationship P_base equals R multiplied by P_cam plus t, where R is a 3×3 orthogonal rotation matrix and t is a 3×1 translation vector. In one embodiment of the present invention, in order to eliminate the influence of translation components on the solution of rotation matrix, the two point sets are first centered; the centroid coordinates of point set P_cam are calculated, the X component of the centroid is equal to the sum of the X coordinates of all points in the point set divided by the number of points N, the Y component and Z component are calculated in the same way, and the centroid coordinates are denoted as C_cam; similarly, the centroid coordinates of point set P_base are calculated and denoted as C_base; Center each point in the point set P_cam. The coordinates of the centered point are equal to the original coordinates minus the centroid coordinates C_cam. The resulting centered point set is denoted as P_cam_cen. The same centering process is performed on the point set P_base. The resulting centered point set is denoted as P_base_cen. The centroids of both centered point sets are located at the origin. Taking a point set containing N points as an example, the centering process is explained as follows: First, calculate the sum of the X, Y, and Z coordinates of all points in the point set P_cam, and divide each sum by the number of points N to obtain the three components of the centroid coordinate C_cam. Similarly, calculate the centroid coordinate C_base of the point set P_base. For each point in the point set, subtract the corresponding centroid coordinates from the X, Y, and Z components of its original coordinates to obtain the centered coordinates. Perform the same centering operation on all points to obtain the centered point sets P_cam_cen and P_base_cen.

[0025] S42, construct the covariance matrix and perform singular value decomposition; Based on the two centered point sets, a 3×3 covariance matrix H is constructed. The covariance matrix H is calculated as follows: for each pair of corresponding points in the point set, the coordinates of the points in P_cam_cen are represented as a 3×1 column vector, and the coordinates of the corresponding points in P_base_cen are represented as a 1×3 row vector. The two are multiplied to obtain a 3×3 matrix. The 3×3 matrices obtained from all N pairs of points are summed to obtain the covariance matrix H. Specifically, the covariance matrix H can be represented as a matrix multiplication form, where H is equal to the transpose of P_cam_cen multiplied by the matrix P_base_cen, where both P_cam_cen and P_base_cen are represented as N×3 matrices, with each row representing the three-dimensional coordinates of a point; or it can be represented as matrix elements, where the i-th row and j-th column element H_ij of the covariance matrix H is calculated by iterating through all N points, multiplying the i-th coordinate component of each point in P_cam_cen by the j-th coordinate component of that point in P_base_cen, and then summing the product results for all points, where i and j take values ​​of 1, 2, and 3, corresponding to the X, Y, and Z coordinate components, respectively; Singular value decomposition is performed on the covariance matrix H. The decomposition is H equal to the transpose of U multiplied by Sigma multiplied by V, where U and V are both 3×3 orthogonal matrices, Sigma is a 3×3 diagonal matrix, and the diagonal elements are singular values. Singular value decomposition is achieved through numerical computation methods, using the Jacobi iterative algorithm or the Golub-Reinsch algorithm to iteratively calculate the U, Sigma, and V matrices. S43, Calculate the rotation matrix and verify its validity; Based on the results of singular value decomposition, the rotation matrix R is calculated; the rotation matrix R is equal to V multiplied by the transpose of U; since singular value decomposition guarantees that U and V are both orthogonal matrices, R is also an orthogonal matrix, satisfying the property of rotation matrices; The validity of the calculated rotation matrix R is verified. First, the determinant of R is verified by calculating the determinant det(R). Under normal circumstances, det(R) should be equal to positive 1, indicating that R is a pure rotation transformation. If det(R) is equal to negative 1, it means that the transformation contains mirror reflection. In this case, R needs to be corrected. The correction method is to invert the elements of the third column of matrix V, that is, multiply the three elements of the third column by negative 1 respectively, and then recalculate R, which is equal to the corrected V multiplied by the transpose of U. The determinant of the corrected R is positive 1. To verify the orthogonality of R, calculate R multiplied by its transpose. The result should be a 3×3 identity matrix, with diagonal elements equal to 1 and off-diagonal elements equal to 0. In actual calculations, due to numerical errors, the absolute value of the off-diagonal elements should be less than the preset orthogonality threshold, and the deviation of the diagonal elements from 1 should be less than this threshold. If the verification fails, it indicates that the point set data contains significant noise or incorrect correspondences, suggesting a calibration failure and recommending a check of data quality. In one embodiment of the present invention, the calculation and verification process of the rotation matrix is ​​described. After obtaining orthogonal matrices U and V through singular value decomposition, the rotation matrix R is calculated to be equal to V multiplied by the transpose of U. The determinant of R is verified by calculating the value of det(R). If the result is positive 1, the verification is passed. If the result is negative 1, the third column of the V matrix is ​​inverted and R is recalculated. The orthogonality of the corrected R is verified by calculating R multiplied by the transpose of R and checking whether the diagonal elements of the resulting matrix are close to 1 and whether the off-diagonal elements are close to 0. If the orthogonality requirement is met, the verification is passed. S44, calculate the translation vector and construct the hand-eye transformation matrix; Based on the obtained rotation matrix R and the centroid coordinates of the two point sets, the translation vector t is calculated. The formula for calculating the translation vector t is that t equals C_base minus R multiplied by C_cam. The physical meaning of this formula is that after transforming the centroid of the camera coordinate system through the rotation matrix R, the translation vector t should coincide with the centroid of the base coordinate system. The rotation matrix R and the translation vector t are combined into a 4×4 homogeneous transformation matrix, denoted as the hand-eye transformation matrix T_hand_eye; the upper left 3×3 submatrix of the matrix is ​​the rotation matrix R, the upper right 3×1 subvector is the translation vector t, and the bottom 1×4 row vector is 0, 0, 0, 1; this hand-eye transformation matrix describes the complete spatial transformation relationship from the camera coordinate system to the robot arm base coordinate system; In one embodiment of the present invention, the calculation process of the translation vector is given; given the centroid of the camera coordinate system C_cam and the centroid of the base coordinate system C_base, and the rotation matrix R, first calculate R multiplied by C_cam to obtain the rotated centroid coordinates, then calculate C_base minus the rotated centroid coordinates to obtain the translation vector t; fill the 9 elements of the rotation matrix R into the upper left 3×3 position of the hand-eye transformation matrix T_hand_eye, fill the 3 elements of the translation vector t into the upper right 3×1 position, and set the bottom row to 0, 0, 0, 1 to complete the construction of the hand-eye transformation matrix; S45, Calculate the calibration residuals and evaluate the calibration accuracy; To evaluate the accuracy of the calibration results, the calibration residual is calculated. For each pair of corresponding points in the point set, the point coordinates in the camera coordinate system are converted into homogeneous coordinates, and the transformation is performed using the hand-eye transformation matrix T_hand_eye to obtain the predicted coordinates of the transformed point in the base coordinate system. The predicted coordinates are compared with the coordinates of the marked point in the base coordinate system obtained by S3, and the Euclidean distance between the two is calculated. This distance is the calibration residual of the point. Calculate the calibration residuals for all N points, and statistically analyze the mean and standard deviation of the residuals. The mean of the residuals reflects the systematic error of the calibration, and the standard deviation of the residuals reflects the random error of the calibration. Set the residual threshold to 5 mm. If the mean of the residuals is less than 2 mm and the standard deviation is less than 3 mm, the calibration accuracy is considered to meet the requirements. If the residuals exceed the threshold, analyze the residual distribution, identify outliers with large residuals, remove the outliers from the point set, and re-execute the calibration calculations from S41 to S44 to iteratively optimize the calibration results. In one embodiment of the present invention, for cases where there are a small number of outliers in the point set, the RANSAC random sampling consensus algorithm is used to improve the calibration robustness. In each iteration, a minimum subset containing 8 to 12 points is randomly selected from the point set. The hand-eye transformation matrix is ​​calculated based on the subset, and this matrix is ​​used to transform all points and calculate the residuals. The number of points with residuals less than a threshold is counted as the number of inliers. This iteration is repeated 100 to 200 times, and the iteration result with the largest number of inliers is selected. The final hand-eye transformation matrix is ​​recalculated based on the set of inliers of this result. This method can effectively eliminate the influence of outliers and improve calibration accuracy and stability. The verified hand-eye transformation matrix T_hand_eye is stored in the system configuration file as the core parameter of the robotic arm vision positioning system. It is used to transform the coordinates of the target object observed by the camera to the coordinate system of the robotic arm base during subsequent operations, so as to achieve accurate visual guidance and positioning. S5. Based on the hand-eye transformation matrix, repeat the above steps before and after the robotic arm operation to obtain the current transformation matrix. Input the initial and current transformation matrices into the fault diagnosis model, output the fault type probability distribution, and generate a maintenance strategy based on the fault type. After the robotic arm completes the initial calibration, the hand-eye transformation matrix T_hand_eye obtained in S4 is recorded as the initial transformation matrix T_initial, which serves as a benchmark reference for the normal state of the system. During the actual operation of the robotic arm, the calibration review process is triggered at specific times. By comparing the difference between the current transformation matrix and the initial transformation matrix, real-time monitoring and intelligent diagnosis of the system state are achieved. S51, under preset trigger conditions, execute the calibration review process to obtain the current hand-eye transformation matrix; Multiple calibration and re-check trigger conditions are set, including three types: time-triggered, event-triggered, and periodic-triggered. The time-triggered condition automatically performs the calibration and re-check before the first operation after each startup of the robotic arm. The event-triggered condition immediately performs the calibration and re-check after detecting a collision, abnormal torque, or emergency stop event of the robotic arm. The periodic-triggered condition automatically performs the calibration and re-check when the continuous running time of the robotic arm reaches a preset cycle. The cycle length is set according to the work intensity, usually from 4 hours to 8 hours. When the triggering conditions are met, the calibration review process is automatically started. To improve review efficiency, the review process adopts a simplified calibration pose sequence, selecting 5 to 8 representative poses from the 12 to 18 poses of the complete calibration. These poses cover the main area of ​​the workspace. The robotic arm and camera support are controlled to move according to the simplified pose sequence, and the data acquisition, marker identification, coordinate calculation and transformation matrix solution processes of S1 to S4 are repeated to obtain the hand-eye transformation matrix T_current at the current moment. During the execution of S4, the calibration residual is also calculated and the accuracy is evaluated to obtain the current calibration residual data. This residual data is used for subsequent fault diagnosis feature extraction. S52, extract the difference features between the initial transformation matrix and the current transformation matrix, and construct the fault diagnosis input feature vector; The system extracts difference features from the initial transformation matrix T_initial and the current transformation matrix T_current for fault diagnosis; For the rotation component, Euler angles are extracted from the initial rotation matrix R_initial and the current rotation matrix R_current. Using the ZYX Euler angle order, the rotation matrix is ​​decomposed into three rotation angles around the Z-axis, Y-axis, and X-axis, denoted as α, β, and γ, respectively. The differences between the initial and current Euler angles are calculated to obtain the changes in the three rotation angles Δα, Δβ, and Δγ. The Euler angle differences directly reflect the degree of camera pose deflection in the three directions. For the translation component, the translation components in the X, Y, and Z directions are extracted from the initial translation vector t_initial and the current translation vector t_current; the translation difference vector t_diff is calculated, which is equal to t_current minus t_initial, to obtain the translation changes Δx, Δy, and Δz in the three directions; the total distance d_trans of the translation difference is calculated, which is equal to the Euclidean norm of the translation difference vector; the translation feature reflects the spatial offset of the camera position. Read the calibration residual data calculated by S45 during the current calibration process. The calibration residual is the Euclidean distance between the predicted coordinates of each marker point after transformation by the hand-eye transformation matrix and the coordinates of the marker point in the base coordinate system. Calculate the average value e_mean of the residuals of all marker points, which reflects the overall accuracy of the calibration. Calculate the standard deviation e_std of the residuals, which reflects the stability of the calibration accuracy. Calculate the maximum value e_max of the residuals, which reflects the degree of error at the point of maximum deviation. The average residuals under different calibration poses were statistically analyzed. Since the simplified calibration uses 5 to 8 poses, the average residuals of all marked points under each pose were calculated to obtain 5 to 8 pose residual values. To unify the feature dimensions, the residual values ​​of the 3 poses with the largest residuals were selected and denoted as e_pose1, e_pose2, and e_pose3, respectively. The pose residual features can reflect the distribution characteristics of faults in the workspace. For example, a significantly larger residual in some poses may indicate that the robotic arm is deformed under a specific pose. Read the flexible deformation compensation data calculated by S32 during the current calibration process; extract the deformation angle compensation amount for each joint of the robot arm; calculate the average value θ_def_mean of the deformation angle of each joint, which reflects the overall degree of flexible deformation of the robot arm; calculate the maximum value θ_def_max of the deformation angle of each joint, which reflects the joint with the most severe deformation. The above 15 parameters are combined and concatenated into the original feature vector F_raw, including the rotation angle changes Δα, Δβ, Δγ, the translation changes Δx, Δy, Δz, the total translation distance d_trans, the residual statistics e_mean, e_std, e_max, the pose residuals e_pose1, e_pose2, e_pose3, and the deformation parameters θ_def_mean, θ_def_max. The original feature vector is then normalized using the maximum-minimum normalization method, scaling each feature to the range of 0 to 1. The normalization formula is: the normalized value equals the original value minus the minimum value of the feature, divided by the difference between the maximum and minimum values ​​of the feature. The maximum and minimum values ​​of the features are obtained statistically from historical calibration data and stored in the system configuration. S53, Construct a fault diagnosis neural network model and calculate the probability distribution of fault types through forward propagation. The system uses a fully connected neural network to build a fault diagnosis model. The model takes the normalized feature vector F_norm extracted by S52 as input and outputs the probability distribution of four fault types. The four fault types are: normal state, robot arm origin offset, camera support structure offset, and robot arm flexible deformation. The neural network consists of an input layer, two hidden layers, and an output layer. The input layer receives a 15-dimensional normalized feature vector F_norm, which includes rotation angle changes Δα, Δβ, Δγ, translation changes Δx, Δy, Δz, total translation distance d_trans, residual statistics e_mean, e_std, e_max, pose residuals e_pose1, e_pose2, e_pose3, and deformation parameters θ_def_mean, θ_def_max. The first hidden layer contains 32 neurons and uses the ReLU activation function. The 15-dimensional feature vector of the input layer is multiplied by the weight matrix of the first hidden layer and a bias is added to obtain a 32-dimensional intermediate vector. The ReLU activation function is applied to each element of the intermediate vector, setting negative values ​​to zero and positive values ​​to remain unchanged, thus obtaining the output of the first hidden layer. The second hidden layer contains 16 neurons and uses the ReLU activation function. The 32-dimensional output of the first hidden layer is multiplied by the weight matrix of the second hidden layer and a bias is added to obtain a 16-dimensional intermediate vector. The ReLU activation function is applied to the intermediate vector to obtain the output of the second hidden layer. Dropout regularization is applied after the second hidden layer with a dropout ratio of 0.3. During the training phase, 30% of the neuron outputs are randomly set to zero to prevent overfitting. The output layer contains 4 neurons, corresponding to 4 fault types. The 16-dimensional output of the second hidden layer is multiplied by the weight matrix of the output layer and a bias is added to obtain a 4-dimensional original output vector logits. The logits are normalized by applying the Softmax activation function. The Softmax function calculates the exponent value of each element and then divides it by the sum of the exponent values ​​of all elements to obtain a 4-dimensional probability vector P. The four components of the probability vector P represent the probability of normal state, the probability of robot arm origin offset, the probability of camera bracket offset, and the probability of robot arm deformation, respectively. The sum of the four probability values ​​is equal to 1. The forward propagation process of the neural network is as follows: the input is a 15-dimensional feature vector F_norm, which is transformed by the first hidden layer and activated by ReLU to obtain a 32-dimensional vector. After the transformation by the second hidden layer and activated by ReLU, a 16-dimensional vector is obtained. After the transformation by the output layer and activated by Softmax, a 4-dimensional probability vector P is obtained. The fault type corresponding to the component with the largest probability value in the probability vector P is the prediction result of the model. S54, a fault diagnosis neural network model is trained based on a labeled dataset, and the network parameters are optimized using the cross-entropy loss function; A fault diagnosis training dataset was constructed, comprising historical calibration data and artificially simulated fault data. Historical calibration data was derived from calibration review results recorded during long-term operation of the robotic arm. Each data point contains a 15-dimensional feature vector extracted using the S52 method and a corresponding fault type label, which was added by maintenance personnel based on actual maintenance records. Artificially simulated fault data was generated by artificially introducing known faults into a normal system, including adjusting the robotic arm's zero-position to simulate origin offset, loosening support bolts to simulate support offset, and increasing end-effector load to simulate flexible deformation. A calibration review process was performed on each simulated fault, extracting feature vectors and labeling the corresponding fault type. The training dataset contains thousands of samples, with a relatively balanced number of samples across the four fault types, each fault type comprising no less than 20% of the total samples. Train the neural network model; during training, each training sample contains an input feature vector F_norm and a true fault type label y. The label y is a one-hot encoded 4-dimensional vector, with the position corresponding to the fault type set to 1 and the other positions set to 0; input the feature vector F_norm into the neural network for forward propagation to obtain the predicted probability vector P; The cross-entropy loss function is used to measure the difference between the predicted probability and the true label. The cross-entropy loss function is calculated by multiplying the corresponding component of the true label by the logarithm of the predicted probability for each of the four categories, then summing the results and taking the negative value. The smaller the cross-entropy loss value, the more accurate the prediction. For multiple samples in a batch, the cross-entropy loss of each sample is calculated, and then the average is taken to obtain the batch loss. The backpropagation algorithm and Adam optimizer are used to update the network parameters. The backpropagation algorithm calculates the gradient of each parameter layer by layer from the output layer to the input layer based on the gradient of the loss function with respect to the network parameters. The Adam optimizer combines momentum and adaptive learning rate strategies to update the weights and bias parameters of the network based on the gradient. The learning rate is set to 0.001, the batch size is set to 32, and the number of training iterations is set to 100 to 200. During training, the dataset is divided into a training set and a validation set. The training set is used for parameter updates, and the validation set is used to evaluate the model's generalization performance. After every few training rounds, the accuracy is calculated on the validation set. If the validation accuracy does not improve for several consecutive rounds, training is stopped early to prevent overfitting. After training, the model achieves a fault classification accuracy of over 90% on the validation set. S55 determines faults based on the probability distribution of fault types and generates targeted maintenance strategies according to the fault types. After obtaining the probability vector P output by the fault diagnosis neural network model, fault determination is performed. The probability vector P contains four components, corresponding to four fault types: normal state, robot arm origin offset, camera support structure offset, and robot arm flexible deformation. The category with the highest probability value in the probability vector P is selected as the predicted fault type. If the probability of the normal state is the highest and exceeds the threshold of 0.8, the system is determined to be in a normal state and no maintenance is required. If the probability of one of the robot arm origin offset, camera support structure offset, or robot arm flexible deformation is the highest and exceeds the threshold of 0.6, the system is determined to have the corresponding type of fault and maintenance is required. If all probability values ​​are lower than the determination threshold, it indicates that the fault type is uncertain, and it is recommended to perform a complete calibration process for re-evaluation. For different fault types, corresponding maintenance strategies are generated. When the fault is determined to be a robot arm origin offset, this fault type is mainly caused by encoder zero drift, zero-position sensor failure, encoder error caused by electrical system failure, etc. The maintenance strategy includes: first, controlling the robot arm to perform an origin return operation, moving each joint to the zero position, and checking whether the zero-position sensor signal of each joint is triggered normally; reading the encoder value of each joint at the current zero position, comparing it with the zero-position reference value stored in the system, and calculating the deviation; if the deviation exceeds the allowable range, for example, more than 0.1 degrees, then updating the zero-position reference value to the current measurement value; after completing the zero-position update, re-executing the complete calibration process, including all steps from S1 to S4, establishing a new hand-eye transformation matrix and replacing the initial transformation matrix T_initial; the maintenance strategy also includes inspection suggestions, prompting maintenance personnel to check whether the encoder battery power is sufficient, whether the zero-position sensor installation status is loose, whether the transmission system fastening bolts are loose, and whether the electrical connection is reliable, etc. When the fault is determined to be a camera bracket structural misalignment, this type of fault is mainly caused by loose bracket base, wear of bracket joints, structural loosening due to vibration, camera installation position misalignment, and vision system malfunctions such as lens impact. Maintenance strategies include: using the camera to perform visual measurements on pre-set fixed reference objects within the workspace; the reference objects are calibration blocks or markers with known precise coordinates; transforming the reference object coordinates measured by the camera to the base coordinate system using the current hand-eye transformation matrix, comparing them with the actual coordinates of the reference object, and calculating the coordinate deviation; analyzing the direction and magnitude of the deviation; if the deviation is mainly manifested as overall translation, it is determined to be an overall misalignment of the bracket base; if the deviation is manifested as rotation or nonlinear change... The issue identified as a loose support joint. The maintenance strategy recommends that maintenance personnel check the tightness of the mounting bolts between the support base and the ground, and retighten them to the specified torque using a torque wrench. Inspect the connecting components and transmission mechanisms of each support joint, checking for wear on the joint bearings, proper gear meshing, and looseness of the transmission belt. Check for loose camera mounting bolts and whether the camera lens has been subjected to external impact. Check for camera intrinsic parameters drift, and recalibrate the camera to update the intrinsic parameter matrix if necessary. Check for lens contamination and clean the lens if necessary. Check the camera power supply and data connection for proper function, and verify image acquisition quality. Replace damaged components if necessary. After maintenance, re-execute the complete calibration process to verify the maintenance effectiveness. When the fault is determined to be a flexible deformation of the robotic arm, this type of failure is mainly caused by joint wear, increased clearance in the transmission system, fatigue damage to the linkage structure, thermal deformation due to temperature changes, and loose installation of the calibration plate. Maintenance strategies include: analyzing the spatial distribution pattern of calibration residuals obtained during the S51 calibration review process, identifying calibration poses with large residuals, and determining the joints primarily related to the deformation based on the joint angle configuration of these poses; conducting detailed inspections of the relevant joints, using dial indicators or laser rangefinders to measure joint position changes under different loads, and quantifying joint clearance and hysteresis; checking the integrity of the linkage structure for cracks, deformation, or fatigue damage; and checking for loose installation of the calibration plate, re-fixing it if necessary. The system's installation posture is measured; system temperature changes are monitored. If temperature changes cause thermal deformation, it is recommended to wait for the system temperature to stabilize before recalibrating, or to record temperature compensation parameters in the system configuration. Maintenance strategies include adjusting or repairing the joints, such as adjusting joint preload, replacing worn bearings or reducers, and repairing or replacing deformed links. If the joint's mechanical properties have recovered but deformation persists, the parameters of the adaptive stiffness model in S32 are updated, and the current calibration data is incorporated into the training using an incremental learning algorithm to update the RBFNN network weights. If necessary, the DH kinematic parameters of the robotic arm are recalibrated, and parameters such as link length or joint offset are corrected. After maintenance, a complete calibration process is performed to verify whether the system accuracy has recovered to the required range. The maintenance strategy is generated following a principle of progressing from simple to complex, prioritizing rapid diagnosis and simple maintenance operations. If the problem persists, further in-depth inspection and complex repairs are then performed. Fault diagnosis results, maintenance strategies, and inspection recommendations are integrated into a maintenance report. This report includes information such as fault type, fault probability, feature vector values, maintenance steps, estimated maintenance time, and relevant fault history. The maintenance report is displayed through the system interface and simultaneously sent to the operations and maintenance management system, supporting timely response and handling by operations and maintenance personnel. The report also includes potential job tasks affected by the fault and suggested maintenance priorities, helping operations and maintenance personnel to rationally plan maintenance schedules. In one embodiment of the present invention, the fault diagnosis neural network model can be expanded according to actual application needs to identify more subdivided fault types. By adjusting the number of neurons in the output layer and increasing the corresponding training samples, the four basic fault types can be further refined. For example, the origin offset of the robotic arm can be subdivided into subtypes such as zero-position loss due to insufficient encoder battery power, mechanical damage to the zero-position sensor, and encoder signal interference; the offset of the camera bracket structure can be subdivided into subtypes such as loose bracket base bolts, wear of bracket joint bearings, loose camera lens, and camera intrinsic parameter drift; and the flexible deformation of the robotic arm can be subdivided into subtypes such as wear of joint reducers, thermal deformation of connecting rods, increased clearance in the transmission system, and loose installation of calibration plates. By refining the fault types, more accurate fault location and more targeted maintenance guidance can be provided to maintenance personnel. When expanding the fault types, the feature extraction method remains unchanged, still extracting 15-dimensional features such as rotation angle change, translation change, calibration residual statistics, pose residual, and deformation parameters from the initial and current transformation matrices. The number of neurons in the neural network output layer is adjusted to match the number of expanded fault types; for example, when expanding to 8 or 12 fault types, the output layer contains 8 or 12 neurons respectively. The training dataset is reconstructed, collecting a sufficient number of training samples for each sub-fault type through historical maintenance records, artificially simulated fault experiments, and expert annotation. The same training method is used, employing the cross-entropy loss function and Adam optimizer to train the expanded neural network model, ensuring the reliability of sub-fault type identification. For the expanded and subdivided fault types, maintenance strategies are refined accordingly. For example, when the fault is identified as insufficient encoder battery power, the maintenance strategy focuses on replacing the encoder battery and verifying zero-position accuracy. When the fault is identified as loose bracket base bolts, the maintenance strategy focuses on using a torque wrench to tighten the bolts to the specified torque and checking the flatness of the base. When the fault is identified as wear of the joint reducer, the maintenance strategy focuses on measuring the reducer hysteresis, assessing the degree of wear, and deciding whether to replace the reducer. Through refined fault diagnosis and maintenance strategies, maintenance efficiency can be significantly improved, unnecessary inspection steps can be reduced, and system downtime can be shortened. The extended fault diagnosis model also supports online learning and continuous optimization. When maintenance personnel report that an actual fault is of a certain sub-type, the system associates the feedback information with the feature vector to form new labeled samples for incremental training of the model. Through continuous learning, the model can continuously improve the accuracy of identifying various sub-type faults and adapt to changes in fault characteristics under different working environments and usage conditions. S56, continuously optimizes the fault diagnosis model based on maintenance feedback data, and adopts an online learning strategy to improve diagnostic accuracy; Establish a maintenance feedback mechanism so that after completing maintenance, maintenance personnel can provide feedback on the actual fault type and maintenance results through the system interface. Feedback information includes whether the system diagnosis is accurate, the actual cause of the fault found, and the effectiveness of the maintenance measures. The feedback information is then associated with the corresponding feature vectors to form new labeled samples. An online learning strategy is adopted to continuously optimize the fault diagnosis model. After accumulating a certain number of feedback samples, new samples are added to the training dataset to retrain the neural network model. The online learning process uses a small learning rate to fine-tune the original model parameters and avoid forgetting historical knowledge. Through continuous learning, the model can adapt to the aging characteristics of the mechanical system and new fault modes, maintaining diagnostic accuracy. In one embodiment of the present invention, to address the interpretability of the decision-making process of the fault diagnosis model, the contribution of each feature to the final judgment is recorded during each diagnosis; by analyzing the importance of features, maintenance personnel can understand the basis for the model's judgment and increase their trust in the diagnosis results; a diagnosis history query function is also provided to support comparative analysis of diagnosis results at different times and identify the evolution trend of the system status. The entire process of calibration review, fault diagnosis, and maintenance execution is recorded in a historical database to form a complete equipment health record. Based on historical data, predictive maintenance analysis can be performed to estimate the time when future failures may occur, supporting maintenance personnel to arrange maintenance plans in advance and reducing the impact of unplanned downtime on production. Through intelligent diagnosis and adaptive correction, a shift from passive maintenance to proactive prevention is achieved, improving the reliability and availability of the large robotic arm vision positioning system.

[0026] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for intelligent hand-eye calibration and adaptive correction, characterized in that, Includes the following steps: S1, control the robotic arm and camera bracket to move together to the preset pose, and simultaneously collect color images, depth images, encoder values ​​of motors at each joint and torque sensor values ​​of each joint to obtain a multimodal calibration dataset; S2, based on the color image and depth image in the multimodal calibration dataset, identify the marker points on the calibration board and extract the pixel coordinates, combine the in-camera participation depth value to calculate the three-dimensional spatial position of the marker points, and obtain the set of three-dimensional coordinates of the marker points in the camera coordinate system; S3. Based on the encoder values ​​and torque values ​​in the multimodal calibration dataset, the compensated end pose is calculated by combining the mechanical arm kinematic model with the flexible deformation compensation module. The coordinate transformation is performed by combining the calibration plate installation pose with the physical coordinates of the marker points to obtain the three-dimensional coordinate set of the marker points in the mechanical arm base coordinate system. S4. Based on the set of three-dimensional coordinates of the marker points in the camera coordinate system and the base coordinate system, solve for the rotation matrix and translation vector of the two point sets to obtain the hand-eye transformation matrix. S5. Based on the hand-eye transformation matrix, repeat the above steps before and after the robotic arm operation to obtain the current transformation matrix. Input the initial and current transformation matrices into the fault diagnosis model, output the fault type probability distribution, and generate a maintenance strategy based on the fault type.

2. The intelligent hand-eye calibration and adaptive correction method according to claim 1, characterized in that, S1 includes: Control the robotic arm and camera support to move in coordination to a preset calibration pose sequence, which includes a preset number of different spatial positions and posture combinations; For each target pose, after the robotic arm reaches the target pose and stabilizes, it synchronously acquires color images, depth images, encoder values ​​of motors at each joint of the robotic arm, torque sensor values ​​of torque sensors at each joint of the robotic arm, and encoder values ​​of motors at each joint of the camera bracket through hardware trigger signals. The collected data and collection timestamps are packaged into a set of calibration data; all calibration poses are traversed to obtain a multimodal calibration dataset; The acquired multimodal calibration dataset is preprocessed and quality checked. The color images are denoised and contrast enhanced, and the depth images are filtered to remove depth measurement noise.

3. The intelligent hand-eye calibration and adaptive correction method according to claim 1, characterized in that, S2 includes: The color image is input into a pre-trained convolutional neural network to identify marker points and combinations of marker points, and outputs the bounding box coordinate confidence score of each combination of marker points on the calibration board, retaining the detection results whose confidence scores exceed the preset confidence threshold; Calculate the coordinates of each bounding box and the corresponding pixel coordinates of the marker points; spatially sort the identified marker points according to the known arrangement order of the marker points on the calibration board. Read the camera intrinsic parameter matrix, and for the pixel coordinates of each marked point, extract the depth value of the corresponding position from the depth image; use the inverse transformation of the pinhole camera imaging model to convert the pixel coordinates and depth values ​​into three-dimensional coordinates in the camera coordinate system.

4. The intelligent hand-eye calibration and adaptive correction method according to claim 1, characterized in that, S3 includes: The theoretical pose of the robotic arm end effector is calculated based on the encoder values ​​through the forward kinematics model of the robotic arm. The forward kinematics model of the robotic arm adopts the DH parameter representation. According to the DH parameters and joint angles, the homogeneous transformation matrix of each joint coordinate system relative to the previous joint coordinate system is calculated in sequence. The transformation matrices of each joint are multiplied in sequence to obtain the theoretical pose transformation matrix of the robotic arm end effector coordinate system relative to the base coordinate system. Flexible deformation compensation is based on an adaptive stiffness model. A multi-factor coupled model of joint stiffness is established. The stiffness model realizes the nonlinear mapping from input parameters to stiffness coefficients through a radial basis function neural network. By combining the installation pose of the calibration plate and the physical coordinates of the marker points, a coordinate system chain transformation is performed to obtain the set of three-dimensional coordinates of the marker points in the coordinate system of the robot arm base.

5. The intelligent hand-eye calibration and adaptive correction method according to claim 4, characterized in that, The radial basis function neural network includes an input layer, a hidden layer, and an output layer; The input layer receives parameters such as joint torque, joint angle, joint temperature, and cumulative running time, and normalizes each parameter to eliminate the influence of dimensions. The hidden layer contains several radial basis function neurons, each neuron corresponding to a center vector and a width parameter. The output of the neuron is calculated using Gaussian radial basis functions. The output layer is a single linear neuron. The stiffness coefficient is obtained by performing an inner product operation between the hidden layer output vector and the weight vector and adding a bias. The flexible deformation angle of the joint is calculated based on the stiffness coefficient and the current torque value. The forward kinematics calculation is then performed again based on the compensated actual joint angle to obtain the compensated end-effector pose transformation matrix.

6. The intelligent hand-eye calibration and adaptive correction method according to claim 1, characterized in that, S4 includes: Center the two point sets separately, calculate the centroid coordinates of the point sets, and the coordinates of the centered points are equal to the original coordinates minus the centroid coordinates; Construct a covariance matrix based on the two centered point sets, and then perform singular value decomposition on the covariance matrix to obtain an orthogonal matrix. The rotation matrix is ​​calculated based on the results of singular value decomposition, and the determinant and orthogonality of the rotation matrix are verified. The translation vector is calculated based on the rotation matrix and the centroid coordinates of the two point sets; the rotation matrix and translation vector are combined into a homogeneous transformation matrix to obtain the hand-eye transformation matrix; Calculate the calibration residual and evaluate the calibration accuracy. For each pair of corresponding points in the point set, use the eye-hand transformation matrix to transform the point coordinates in the camera coordinate system to obtain the predicted coordinates. Calculate the Euclidean distance between the predicted coordinates and the coordinates of the marked point in the base coordinate system as the calibration residual.

7. The intelligent hand-eye calibration and adaptive correction method according to claim 1, characterized in that, S5 includes: The calibration review process is executed under preset trigger conditions, including time-triggered, event-triggered, and periodic trigger conditions. A simplified calibration pose sequence is used to repeatedly perform the data acquisition, marker point identification, coordinate calculation and transformation matrix solution process to obtain the hand-eye transformation matrix at the current moment; Extract the difference features from the initial transformation matrix and the current transformation matrix, including the change in rotation angle, the change in translation, the calibration residual statistics, the pose residual and the deformation parameters. Combine the extracted parameters to form the original feature vector and then normalize it. The normalized feature vector is input into the fault diagnosis neural network model, and the probability distribution of fault types is calculated through forward propagation.

8. The intelligent hand-eye calibration and adaptive correction method according to claim 7, characterized in that, The fault diagnosis neural network model adopts a fully connected neural network, which includes an input layer, two hidden layers and an output layer; The input layer receives a normalized feature vector; the first and second hidden layers use the ReLU activation function, and Dropout regularization is applied after the second hidden layer. The output layer corresponds to various fault types. The original output vector is normalized by applying the Softmax activation function to obtain the probability vector. The category with the highest probability value in the probability vector is selected as the predicted fault type. If the probability of the normal state is the highest and exceeds the first preset threshold, the system is determined to be in a normal state. If the probability of one of the other fault types is the highest and exceeds the second preset threshold, the system is determined to have a fault of the corresponding type.

9. The intelligent hand-eye calibration and adaptive correction method according to claim 8, characterized in that, The predicted fault types include normal state, robotic arm origin offset, camera support structure offset, and robotic arm flexible deformation. Generate corresponding maintenance strategies for different fault types; Establish a maintenance feedback mechanism. After completing maintenance, maintenance personnel will provide feedback on the actual fault type and maintenance results. The feedback information will be associated with the corresponding feature vectors to form new labeled samples. An online learning strategy is used to continuously optimize the fault diagnosis model. After accumulating a preset number of feedback samples, new samples are added to the training dataset to retrain the neural network model.

10. An intelligent hand-eye calibration and adaptive correction system, used to perform the steps of the intelligent hand-eye calibration and adaptive correction method as described in any one of claims 1-9, characterized in that, include: The data acquisition module is used to control the robotic arm and camera bracket to move in coordination to a preset pose, and to simultaneously acquire color images, depth images, encoder values ​​of motors at each joint, and torque sensor values ​​of each joint to obtain a multimodal calibration dataset. The marker recognition module is used to identify markers on the calibration board and extract pixel coordinates based on color images and depth images in the multimodal calibration dataset. It also calculates the three-dimensional spatial position of the markers by combining the in-camera depth values, and obtains a set of three-dimensional coordinates of the markers in the camera coordinate system. The coordinate transformation module is used to calculate the compensated end pose based on the encoder values ​​and torque values ​​in the multimodal calibration dataset, through the kinematic model of the robotic arm combined with the flexible deformation compensation module, and to perform coordinate transformation by combining the installation pose of the calibration plate and the physical coordinates of the marker points to obtain the three-dimensional coordinate set of the marker points in the coordinate system of the robotic arm base. The hand-eye calibration module is used to solve for the rotation matrix and translation vector of the two point sets based on the three-dimensional coordinate sets of the marker points in the camera coordinate system and the base coordinate system, so as to obtain the hand-eye transformation matrix. The fault diagnosis module is used to obtain the current transformation matrix by repeatedly executing the above steps before and after the robotic arm operation based on the hand-eye transformation matrix. The initial and current transformation matrices are input into the fault diagnosis model, which outputs the probability distribution of fault types and generates maintenance strategies based on the fault types.

Images