Human-robot collaboration-oriented robot hand-eye calibration system and method

Abstract

The application provides a human-robot collaboration-oriented robot hand-eye calibration system and method, wherein the human-robot collaboration-oriented robot hand-eye calibration system realizes real-time, accurate and stable calibration of the hand-eye transformation relationship in the human-robot collaboration process through the synergistic effect of dynamic contour tracking, task context perception and adaptive calibration decision, can adaptively adjust the calibration strategy according to the dynamic operation scene and task demand, and enhances the user interaction experience through visual guidance, thereby effectively improving the adaptability, reliability and overall operation efficiency of the calibration system in complex collaboration tasks.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a robot hand-eye calibration system and method for human-robot collaboration. Background Technology

[0002] Robot hand-eye calibration technology is a key technology in the field of robot perception and control. It provides the foundation for robots to achieve vision-based localization, grasping, and manipulation by accurately solving the coordinate transformation relationship between the robot's end effector and vision sensors. With the development of intelligent manufacturing, flexible production lines, and human-robot collaborative operation modes, this technology has gradually evolved from fixed calibration applicable to structured industrial environments to online calibration adaptable to dynamic unstructured environments. Especially in scenarios requiring close collaboration between humans and robots to complete complex tasks, such as precision assembly, medical assistance, and service robots, higher demands are placed on the real-time performance, adaptability, and user-friendliness of the calibration system.

[0003] However, existing technologies mostly focus on improving calibration accuracy under static or preset trajectories, lacking the ability to jointly perceive and adaptively decide on the calibration state and task context during dynamic human-machine collaboration. This makes it difficult for the calibration system to maintain optimal performance and reliability when task requirements, user operations, or environmental conditions change, thus limiting its practical effectiveness in efficient and safe human-machine collaboration applications. Summary of the Invention

[0004] In view of this, the present invention provides a robot hand-eye calibration system for human-robot collaboration. One or more embodiments of this specification also relate to a robot hand-eye calibration method for human-robot collaboration, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the present invention, a robot hand-eye calibration system for human-robot collaboration is provided, comprising: The dynamic contour tracking module is used to capture and track the dynamic contour of the operator's hand in real time and output contour data. The hand-eye calibration calculation module, which is associated with the dynamic contour tracking module, is used to receive contour data and, in combination with the motion data of the robot's end effector, calculate the hand-eye transformation matrix from the camera coordinate system to the robot's end effector coordinate system, and output the calibration results. The visual guidance feedback module, which is associated with the hand-eye calibration calculation module and the dynamic contour tracking module, is used to receive the calibration confidence score and contour data in the calibration results, generate visual guidance information and provide feedback to the user. The task context awareness module is used to parse task instructions and operation sequences and output task context information; The adaptive calibration decision module, which is associated with the hand-eye calibration calculation module, the task context perception module, and the visual guidance feedback module, is used to receive the calibration confidence score, task context information, and visual guidance information from the calibration results, generate calibration parameter adjustment instructions and task switching signals, and send them to the hand-eye calibration calculation module and the visual guidance feedback module.

[0006] In some implementations, the dynamic contour tracking module includes a depth camera and a PointNet++-based 3D segmentation network. The depth camera acquires RGB-D point cloud data at a preset frame rate and a preset resolution, and the 3D segmentation network outputs the 3D key point coordinates of the human hand contour.

[0007] In some implementations, the hand-eye calibration calculation module uses an extended Kalman filter algorithm to fuse contour data, inertial measurement unit data, and robot encoder data, and uses a Gaussian process regression model for error compensation to output calibration results with sub-millimeter accuracy.

[0008] In some implementations, the hand-eye calibration calculation module outputs a calibration confidence score, and the steps for calculating the calibration confidence score include: Obtain the reference baseline sequence, the current estimated sequence, and the uncertainty sequence of the current estimate of the calibration parameters; based on the reference baseline sequence, the current estimated sequence, and the uncertainty sequence, and combined with the pre-calculated dynamic sensitivity adjustment factor and the preset first weighting coefficient, generate the parameter error assessment value; Obtain the error vector sequence of consistency verification error and the tolerance threshold sequence; based on the error vector sequence and the tolerance threshold sequence, and combined with the preset second weighting coefficient, generate the position error evaluation value; Obtain the environmental disturbance intensity sequence and normalize it; based on the normalized environmental disturbance intensity sequence, combined with the preset third weighting coefficient, generate the environmental disturbance assessment value; By combining the combined parameter error assessment value, the position error assessment value, and the environmental interference assessment value, a comprehensive value of calibration uncertainty is generated. The calibration confidence score is calculated based on the comprehensive value of the calibration uncertainty.

[0009] In some implementations, the step of calculating the dynamic sensitivity modulator factor includes: Obtain the task complexity metric and the sequence of attitude angle change rate components; Based on the task complexity metric, the task complexity contribution value is obtained by mapping through a preset scaling factor and a sigmoid function. Based on the attitude angle change rate component sequence, the values ​​are adjusted using a preset normalized reference value and mapped using the hyperbolic tangent function to obtain the operational dynamics contribution value. By combining the preset normal base, the contribution value of task complexity and the contribution value of operation dynamism, as well as the preset adjustment coefficient, a dynamic sensitivity adjustment factor is generated.

[0010] In some implementations, the visual guidance feedback module includes an augmented reality device that dynamically adjusts the transparency and position of the visual guidance information based on a Transformer attention model and renders the feedback content at a refresh rate of 60fps.

[0011] In some implementations, the task context-aware module includes a BERT-based natural language processing model and a long short-term memory network. The natural language processing model parses speech or text instructions, and the long short-term memory network analyzes historical operation sequences and predicts task probability distributions.

[0012] In some implementations, the adaptive calibration decision module uses a deep reinforcement learning algorithm as the core controller. The deep reinforcement learning algorithm generates calibration parameter adjustment instructions and task switching signals based on calibration error, task context information, and user behavior patterns.

[0013] In some implementations, the system also includes a distributed processing architecture that interconnects the modules via a high-speed data bus to ensure that the data stream transmission latency is less than 2 milliseconds.

[0014] According to a second aspect of the present invention, a robot hand-eye calibration method for human-robot collaboration is provided, applied to the aforementioned robot hand-eye calibration system for human-robot collaboration, the method comprising: The dynamic contour tracking module captures and tracks the dynamic contour of the operator's hand in real time and outputs contour data. The hand-eye calibration calculation module receives contour data and combines it with the motion data of the robot's end effector to calculate the hand-eye transformation matrix from the camera coordinate system to the robot's end effector coordinate system, and outputs the calibration results. The visual guidance feedback module receives the calibration confidence score and contour data from the calibration results, generates visual guidance information, and provides feedback to the user. The task context awareness module parses task instructions and operation sequences, and outputs task context information. The adaptive calibration decision module receives the calibration confidence score, task context information, and visual guidance information from the calibration results, generates calibration parameter adjustment instructions and task switching signals, and sends them to the hand-eye calibration calculation module and the visual guidance feedback module.

[0015] At least one embodiment of the present invention achieves real-time, accurate and stable calibration of hand-eye relationship changes during human-machine collaboration through the synergistic effect of dynamic contour tracking, task context awareness and adaptive calibration decision. It can adaptively adjust the calibration strategy according to the dynamic operation scenario and task requirements, and enhance the user interaction experience through visual guidance, thereby effectively improving the adaptability, reliability and overall operation efficiency of the calibration system in complex collaborative tasks. Attached Figure Description

[0016] Figure 1 This is a simplified structural diagram of a robot hand-eye calibration system for human-machine collaboration provided by the present invention; Figure 2 This is a flowchart of a robot hand-eye calibration method for human-machine collaboration provided by the present invention. Detailed Implementation

[0017] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0018] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the one or more embodiments of this specification. The singular forms “a” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The modifications “a” and “a plurality” as used in this disclosure are illustrative and not restrictive, and those skilled in the art will understand that they should be understood as “one or more” unless the context clearly indicates otherwise.

[0019] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0020] See Figure 1 , Figure 1This diagram illustrates a simplified structural schematic of a robot hand-eye calibration system for human-robot collaboration, according to some embodiments of this specification, specifically including: The dynamic contour tracking module is used to capture and track the dynamic contour of the operator's hand in real time and output contour data. The hand-eye calibration calculation module, which is associated with the dynamic contour tracking module, is used to receive contour data and, in combination with the motion data of the robot's end effector, calculate the hand-eye transformation matrix from the camera coordinate system to the robot's end effector coordinate system, and output the calibration results. The visual guidance feedback module, which is associated with the hand-eye calibration calculation module and the dynamic contour tracking module, is used to receive the calibration confidence score and contour data in the calibration results, generate visual guidance information and provide feedback to the user. The task context awareness module is used to parse task instructions and operation sequences and output task context information; The adaptive calibration decision module, which is associated with the hand-eye calibration calculation module, the task context perception module, and the visual guidance feedback module, is used to receive the calibration confidence score, task context information, and visual guidance information from the calibration results, generate calibration parameter adjustment instructions and task switching signals, and send them to the hand-eye calibration calculation module and the visual guidance feedback module.

[0021] A dynamic contour tracking module can refer to a functional unit used to capture and track changes in the shape of a human hand in real time. For example, this module collects point cloud data through a depth sensor and continuously outputs the spatial coordinate information of the human hand contour using computer vision algorithms, providing accurate dynamic input data for subsequent hand-eye calibration calculations. Contour data can refer to a set of information describing the shape and position of the human hand contour. For example, it can be achieved by the operator wearing or holding a calibration tool with a known, fixed geometric relationship to the robot's end effector, or by establishing a known spatial offset relationship between specific anatomical key points of the human hand and the tool center point of the robot's end effector through a one-time initialization process. This information set can include the three-dimensional coordinates of the key points of the human hand or a set of contour boundary points, providing real-time posture information of the human hand in space.

[0022] The hand-eye calibration calculation module can refer to the processing unit responsible for calculating the transformation relationship between the human hand coordinate system and the robot eye coordinate system. For example, this module receives contour data and uses mathematical optimization algorithms to solve the rotation matrix and translation vector, and can output calibration results that represent the spatial relationship between the human hand and the robot vision system.

[0023] The visual guidance feedback module can be a functional component that provides users with visual prompts and guidance information. For example, this module receives the calibration confidence score and contour data from the calibration results and generates guidance graphics or text overlaid on the real scene using display technologies such as augmented reality. This helps users adjust their operations to optimize the calibration process. Visual guidance information refers to guidance content presented to users visually. For example, this information is dynamically generated by the visual guidance feedback module based on the current calibration status and task requirements, and may include virtual arrows, highlighted areas, or text prompts to guide users to perform the correct calibration actions in real time.

[0024] A task context-aware module can refer to a functional unit used to understand and analyze the background and environmental conditions of the current operation task. For example, this module infers the task type, step requirements, and environmental constraints by parsing natural language instructions and analyzing operation sequences, and can output an understanding result containing task semantics. Task context information can refer to a set of data describing the task-related background, requirements, and status. For example, this information is output by the task context-aware module after comprehensively parsing instructions and historical operations, and includes semantic features such as task type, key steps, and accuracy requirements, used to provide task-dimensional input for adaptive decision-making.

[0025] The adaptive calibration decision module can refer to the core control unit that autonomously adjusts the calibration strategy based on the system state. For example, this module receives calibration confidence scores, task context information, and visual guidance information from the calibration results, and uses intelligent decision algorithms to generate parameter adjustment and task switching instructions, enabling closed-loop optimization control of the calibration process. Calibration parameter adjustment instructions can refer to commands used to modify the configuration parameters of the hand-eye calibration calculation process. For example, this instruction is generated by the adaptive calibration decision module based on calibration errors and task requirements, specifying the adjustment direction and magnitude of parameters such as filter coefficients and convergence thresholds, used to dynamically optimize the performance and accuracy of the calibration algorithm. Task switching signals can refer to control signals that trigger the system to switch between different working modes or task stages. For example, this signal is issued by the adaptive calibration decision module when it detects calibration completion or a task change, coordinating the behavior transitions of various modules in the system to adapt to new collaborative task requirements. The calibration confidence score can refer to a numerical indicator that quantitatively evaluates the reliability of the hand-eye calibration results. For example, it is calculated by considering factors such as the stability of comprehensive parameters, pose consistency, and environmental interference. The range is usually between 0 and 1, which is used to dynamically reflect the reliability of the current calibration result for use by subsequent decision-making modules.

[0026] As a concrete example: In a human-machine collaborative scenario of precision assembly of an automotive engine, the operator issues the voice command "Start piston installation." The task context awareness module uses a BERT model to parse this command, identifies the task type as "precision assembly," and calls a pre-stored operation sequence (including steps such as "calibration start," "coarse alignment," and "fine correction"). The dynamic contour tracking module uses a depth camera to acquire RGB-D point clouds of the operator's hand at a rate of 30 frames per second, and extracts the 3D coordinates of key points such as the wrist and fingertips as contour data through PointNet++ network segmentation. The hand-eye calibration calculation module receives these coordinates, combines them with the joint angles fed back by the robot encoder, uses the extended Kalman filter algorithm to solve the hand-eye transformation matrix in real time, and outputs the calibration results including the transformation matrix and covariance. The visual guidance feedback module uses AR glasses to overlay virtual installation path arrows onto the real piston at a refresh rate of 60 frames per second to guide the operator's movement. The adaptive calibration decision module continuously monitors the confidence level of the calibration results and the progress of the task. When it is determined that the coarse alignment is completed and the accuracy meets the requirements of "precision assembly", it generates calibration parameter adjustment instructions (such as tightening the process noise covariance of EKF) and sends a task switching signal to drive the system into the "fine calibration" stage. At the same time, the visual guidance information is also changed to a high-precision alignment indicator circle.

[0027] By integrating dynamic contour tracking, task context awareness, and adaptive decision-making mechanisms, this invention achieves real-time, accurate, and intelligent control of the hand-eye calibration process in a human-machine collaborative dynamic environment, effectively improving the adaptability, reliability, and overall collaborative efficiency of the calibration system when facing complex tasks and changing operations.

[0028] The beneficial effects of one of the embodiments in this specification include at least the following: through the synergistic effect of dynamic contour tracking, task context awareness and adaptive calibration decision, real-time, accurate and stable calibration of the hand-eye relationship is achieved in the process of human-computer collaboration. It can adaptively adjust the calibration strategy according to the dynamic operation scenario and task requirements, and enhance the user interaction experience through visual guidance, thereby effectively improving the adaptability, reliability and overall operation efficiency of the calibration system in complex collaborative tasks.

[0029] In some implementations, the dynamic contour tracking module includes a depth camera and a PointNet++-based 3D segmentation network. The depth camera acquires RGB-D point cloud data at a preset frame rate and a preset resolution, and the 3D segmentation network outputs the 3D key point coordinates of the human hand contour.

[0030] A depth camera can refer to an image sensor capable of simultaneously acquiring color and depth information of a scene. For example, such a device measures pixel distances by actively projecting infrared structured light or using time-of-flight methods, outputting an RGB-D image containing 3D spatial point cloud data, providing high-precision, real-time visual input for dynamic contour tracking. A PointNet++-based 3D segmentation network can refer to a deep learning architecture for processing 3D point cloud data. For example, this network uses hierarchical feature learning and ensemble abstraction layers to progressively extract local and global features of the point cloud, achieving semantic segmentation of different objects in 3D space, and can accurately separate hand point clouds from complex backgrounds. RGB-D point cloud data can refer to a set of 3D spatial points that simultaneously contains color and depth information. For example, this data is acquired by a depth camera, with each point containing 3D coordinates and a corresponding RGB color value, providing dense 3D geometric and texture information of the scene. A 3D segmentation network can refer to a neural network model specifically designed for partitioning or classifying 3D data. For example, this network receives raw point cloud data, learns point features through operations such as multilayer perceptrons and max pooling, and can output a semantic category label for each point. The 3D key point coordinates of the human hand outline can refer to the data describing the positions of the main joints and feature points of the human hand in three-dimensional space. For example, these coordinates are output by a 3D segmentation network after processing the point cloud, including the three-dimensional coordinates of parts such as fingertips and knuckles, which are used to accurately represent the real-time posture and shape of the human hand.

[0031] As a concrete example: in a human-robot collaborative scenario of medical robot-assisted surgery, a depth camera (such as the Intel RealSense D435) is fixed to the robot's end effector, acquiring RGB-D data at a resolution of 1280x720 at a rate of 30 frames per second. This raw data is fed into a 3D segmentation network trained on the PointNet++ architecture. This network first constructs a multi-layered hierarchical structure of the point set through farthest-point sampling and grouping operations, learns local features using a multilayer perceptron, and then performs upsampling through a feature propagation layer, ultimately outputting the probability that each point belongs to either the "hand" or the "background." The segmented hand point cloud is further processed by a keypoint detection head to regress and calculate the 3D coordinates of 21 key points, such as the thumb tip and index finger tip. These precise 3D keypoint coordinates, as contour data, are transmitted in real-time to the hand-eye calibration calculation module, providing a high-precision input source for subsequent coordinate transformation calculations.

[0032] By employing a technical solution that combines a depth camera with an advanced 3D segmentation network, this invention can extract the three-dimensional contour information of a human hand from complex backgrounds in real time and accurately, providing a stable and reliable data foundation for hand-eye calibration and significantly improving the calibration accuracy and robustness in dynamic human-machine collaboration scenarios.

[0033] In some implementations, the hand-eye calibration calculation module uses an extended Kalman filter algorithm to fuse contour data, inertial measurement unit data, and robot encoder data, and uses a Gaussian process regression model for error compensation to output calibration results with sub-millimeter accuracy.

[0034] The Extended Kalman Filter (EKF) algorithm can refer to an optimization algorithm for state estimation of nonlinear systems. For example, this algorithm approximates the nonlinear function using a first-order Taylor expansion, combining the system state equation and observation equations for prediction and update steps, and is used to achieve optimal estimation of the system state in noisy dynamic systems. Inertial Measurement Unit (IMU) data refers to motion sensing information collected by sensors such as accelerometers and gyroscopes. For example, this data includes real-time measurements of three-axis acceleration and three-axis angular velocity, providing information on the vehicle's motion acceleration and rotational angular velocity in space. Robot encoder data refers to sensor readings reflecting the robot's joint positions and motion states. For example, this data comes from photoelectric or magnetic encoders on each joint axis of the robot, providing absolute or relative rotation angle information for accurately calculating the pose of the robot's end effector in the base coordinate system. A Gaussian process regression model can refer to a nonparametric probabilistic model based on Bayesian theory. For example, this model uses Gaussian process priors to model the function distribution, calculates the covariance between samples using a kernel function, and can predict the function values ​​at unknown points and provide uncertainty estimates. Error compensation refers to the technical means of correcting the deviation between the system output and the true value through algorithms. For example, this process uses a Gaussian process regression model to learn the pattern and distribution of calibration errors and performs reverse correction on the original output of the hand-eye calibration calculation module to improve the final accuracy of the calibration results. Sub-millimeter accuracy calibration results refer to outputs that achieve extremely high accuracy after error compensation. For example, this result indicates that the translation error of the hand-eye transformation matrix is ​​less than one millimeter and the rotation error is less than one-tenth of a degree, which can meet the stringent positioning accuracy requirements of applications such as precision assembly and medical surgery.

[0035] As a concrete example: In a minimally invasive surgical robot system, the hand-eye calibration calculation module synchronously receives the coordinates of key hand points (contour data) from the dynamic contour tracking module, the angular velocity and acceleration output from the IMU (Inertial Measurement Unit) integrated on the surgical instruments (IMU data), and the joint angles fed back by the encoders of each robot joint (robot encoder data). The Extended Kalman Filter (EKF) algorithm constructs a nonlinear model based on the system state (hand-eye transformation matrix) and observations (multi-source sensor data) to predict and update the state, initially fusing these heterogeneous data. Subsequently, a Gaussian process regression (GPR) model is activated. Based on the complex nonlinear mapping relationship between the position and posture residuals learned from historical calibration data and the surgical scenario context (such as instrument type and field of view), it compensates for errors in the initial EKF calibration results. Ultimately, the system outputs calibration results with sub-millimeter accuracy, where the translational error is stable below 0.5 mm and the posture error is below 0.1 degrees, ensuring precise operation of the surgical instruments under visual guidance.

[0036] By fusing multi-source sensor data and employing advanced filtering and machine learning error compensation techniques, this invention significantly improves the accuracy and stability of hand-eye calibration in dynamic, unstructured environments, providing reliable technical support for high-precision human-machine collaborative tasks.

[0037] In some implementations, the hand-eye calibration calculation module outputs a calibration confidence score. The steps for calculating the calibration confidence score include: obtaining a reference sequence, a current estimation sequence, and a current estimation uncertainty sequence of calibration parameters; generating a parameter error assessment value based on the reference sequence, the current estimation sequence, and the uncertainty sequence, combined with a pre-calculated dynamic sensitivity adjustment factor and a preset first weighting coefficient; obtaining an error vector sequence and a tolerance threshold sequence of consistency verification errors; generating a position error assessment value based on the error vector sequence and the tolerance threshold sequence, combined with a preset second weighting coefficient; obtaining and normalizing an environmental interference intensity sequence; generating an environmental interference assessment value based on the normalized environmental interference intensity sequence, combined with a preset third weighting coefficient; generating a comprehensive calibration uncertainty value by combining the parameter error assessment value, the position error assessment value, and the environmental interference assessment value; and calculating the calibration confidence score based on the comprehensive calibration uncertainty value.

[0038] The reference baseline sequence is obtained by calculating the arithmetic mean of the calibration parameter estimates output by the extended Kalman filter algorithm during the stable operation phase of the system, within a sliding window of a preset time length. This mean is continuously updated as the window slides. The current estimation sequence is obtained through real-time iterative output from the hand-eye calibration calculation module. The uncertainty sequence is obtained by evaluating the uncertainty of the current estimate using a Gaussian process regression model within the hand-eye calibration calculation module. The error vector sequence is obtained by moving the robot's end effector to a set of preset verification poses using the current hand-eye transformation matrix. For each verification pose, the difference vector between the 3D position of key points on the hand or its held tool actually observed by the dynamic contour tracking module and the position of the same key point predicted based on the robot's kinematics model and known hand-end geometry is calculated. The tolerance threshold sequence is a specific value obtained by mapping the task accuracy requirement information output by the task context awareness module. The environmental disturbance intensity sequence is obtained by obtaining the raw measurements of environmental disturbance factors from the dynamic contour tracking module and normalizing them based on predefined reference values. The dynamic sensitivity adjustment factor can refer to a parameter that adaptively adjusts according to task complexity and operational dynamics. For example, it is calculated using a specific formula from the contribution values ​​of task complexity and operational dynamism, with a coefficient greater than 1. This coefficient is used to amplify the weight of parameter fluctuations in highly dynamic or complex task scenarios when calculating confidence levels. The preset first weighting coefficient can refer to a pre-defined constant used to balance the importance of parameter errors in the comprehensive evaluation. For example, it is a scalar value between 0 and 1, used when calculating the comprehensive value of calibration uncertainty, and can adjust the influence of parameter stability on the final confidence score according to the system design focus. The parameter error assessment value can refer to a sub-score obtained by comparing the current parameter estimate with a reference benchmark, and incorporating adjustments for uncertainty and dynamism. For example, it is generated through weighted calculations based on the difference between the reference benchmark sequence and the current estimated sequence, the uncertainty sequence, and the dynamic sensitivity adjustment factor, and is used to quantitatively measure the deviation and instability of the calibration parameters themselves.

[0039] The error vector sequence of consistency verification error can refer to the hand-eye calibration pose error data sequence obtained through additional verification methods. For example, this sequence is formed by applying the calibration results to known test movements and comparing the difference (translation and rotation errors) between the calculated pose and the actual pose, reflecting the consistency accuracy of the calibration results in actual space. The preset second weighting coefficient can refer to a pre-set constant used to balance the importance of position errors in the comprehensive evaluation. For example, it is another scalar value between 0 and 1, used when calculating the comprehensive value of calibration uncertainty, and can adjust the contribution of pose consistency error according to the absolute accuracy requirements of the application scenario. The position error evaluation value can refer to a sub-score obtained by analyzing whether the consistency verification error exceeds the tolerance threshold. For example, it is calculated based on the comparison of the error vector sequence and the tolerance threshold sequence, combined with the second weighting coefficient, and is used to quantify the pose deviation risk that the calibration results may generate in actual space applications. The preset third weighting coefficient can refer to a pre-set constant used to balance the importance of environmental interference in the comprehensive evaluation. For example, it is the third scalar value between 0 and 1, used in calculating the comprehensive calibration uncertainty value. It allows adjustment of the weights of environmental interference factors based on the system's emphasis on environmental robustness. The environmental interference assessment value can refer to a sub-score obtained by analyzing the normalized environmental interference intensity. For example, it is calculated by combining the normalized interference sequence with the third weighting coefficient, used to transform the impact of environmental factors on the sensing system into a measure of the negative contribution to the overall calibration confidence. The comprehensive calibration uncertainty value can refer to the overall uncertainty measure that integrates the assessment results of parameters, pose, and environment. For example, it is the weighted sum of parameter error assessment values, position error assessment values, and environmental interference assessment values, or the output of other forms of comprehensive functions, forming a single indicator that comprehensively reflects all risk factors in the current calibration process.

[0040] As a concrete example: During system operation, the hand-eye calibration calculation module continuously outputs the current hand-eye transformation matrix estimation sequence and its covariance matrix (as an uncertainty sequence), while maintaining a high-confidence historical parameter database as a reference sequence. The online verification unit drives the robot to perform a set of preset micro-movements, generating a sequence of error vectors for consistency verification, and comparing it with a preset tolerance threshold sequence based on task accuracy requirements. The environment perception unit synchronously outputs the noise ratio of the depth image as an environmental interference intensity sequence. The dynamic sensitivity adjustment factor is jointly calculated by the task complexity (e.g., "precision assembly") provided by the task context perception module and the rate of change of hand posture angle calculated by the dynamic contour tracking module. Subsequently, using preset first, second, and third weighting coefficients (e.g., 0.4, 0.4, 0.2), the parameter error evaluation value, the position error evaluation value (the sum of each component of the error vector compared with the threshold), and the environmental interference evaluation value (weighted after normalizing the interference intensity) are calculated. The sum of the three yields the comprehensive calibration uncertainty value U. Finally, the calibration confidence score S is calculated using the formula S=1 / (1+U) or a similar mapping function, and is fed back to the adaptive calibration decision module in real time.

[0041] By quantifying the reliability of calibration results into a comprehensive confidence score, this scheme can dynamically and multidimensionally evaluate the calibration status, providing accurate and reliable basis for subsequent adaptive adjustments and decisions of the system, thereby improving the overall robustness and safety of human-machine collaborative tasks.

[0042] In some implementations, the step of calculating the dynamic sensitivity adjustment factor includes: obtaining a task complexity metric and a sequence of attitude angle change rate components; mapping the task complexity metric using a preset scaling factor and a sigmoid function to obtain a task complexity contribution value; adjusting the attitude angle change rate component sequence using a preset normalized baseline value and mapping it using a hyperbolic tangent function to obtain an operational dynamics contribution value; and combining a preset positive constant base, the task complexity contribution value, the operational dynamics contribution value, and a preset adjustment factor to generate the dynamic sensitivity adjustment factor.

[0043] The attitude angle change rate component sequence refers to a series of values ​​describing how quickly the operator's hand orientation changes. For example, this sequence is calculated from the 3D keypoint coordinates of the hand output by the dynamic contour tracking module, and is usually decomposed into angular velocities or difference sequences around the X, Y, and Z axes, reflecting the agility and smoothness of the user's operation. The attitude angle change rate component sequence can be generated through the following steps: Using the Euler angle representation of the hand posture output by the dynamic contour tracking module, for each Euler angle component, calculate the original difference between two consecutive frames, perform a modulo-2π periodic correction to eliminate jumps caused by rotation around a full circle, obtaining a smooth angle change amount, and then adjust this smooth angle change amount based on the frame time interval to obtain the angular velocity component.

[0044] A task complexity metric can refer to a numerical value that quantifies the difficulty or precision requirements of a task to be performed. For example, after parsing the current task instruction through a task context awareness module, a preset complexity level can be mapped based on the parsed task type, number of target objects, and number of operation steps. Alternatively, it can be obtained by calculating the entropy value of the task probability distribution output by a Long Short-Term Memory (LSTM) network. A task complexity contribution value can refer to an intermediate quantity that characterizes the influence of task factors on the adjustment factor after nonlinear mapping. For example, it is calculated by adjusting the task complexity metric with a preset scaling factor and then using a sigmoid function to make the adjustment factor's response to task complexity smoother and avoid abrupt changes. A preset normalization baseline value can refer to a reference constant value used to adjust the attitude angle change rate component sequence to a typical range. For example, it might be an empirical value representing the average angle change rate under normal operation. Dividing the actual change rate component sequence by this baseline value makes dynamic data from different operators comparable. An operation dynamics contribution value can refer to an intermediate quantity that quantifies the influence of the intensity of the user's hand movements on the adjustment factor. For example, it transforms continuous operational dynamics into a standardized contribution metric by taking the absolute value and averaging the sequence of attitude angle change rate components adjusted to a normalized baseline, and then mapping it using the hyperbolic tangent function. The preset positive constant base can refer to a fundamental multiplicative constant in the adjustment factor calculation formula. For example, it could be a value slightly greater than 1 (e.g., 1.0), serving as the starting point for the adjustment factor calculation, ensuring that even at low complexity and low dynamics, the adjustment factor has a minimum influence base. The preset adjustment coefficient can refer to a constant that controls the combined influence of task complexity and operational dynamics. For example, it could be a positive real number, multiplied by the sum of the task complexity contribution and the operational dynamics contribution, ultimately determining the extent to which these two factors increase the dynamic sensitivity adjustment factor.

[0045] By adaptively calculating adjustment factors based on task complexity and real-time operational dynamics, this scheme enables the assessment of calibration confidence to intelligently adapt to different collaborative scenarios and operator habits. It automatically raises the assessment standard when high-precision complex tasks are required or when operations are unstable, thereby ensuring the contextual rationality of calibration quality judgment.

[0046] In some implementations, the visual guidance feedback module includes an augmented reality device that dynamically adjusts the transparency and position of the visual guidance information based on a Transformer attention model and renders the feedback content at a refresh rate of 60fps.

[0047] Augmented reality devices can refer to display devices that overlay computer-generated virtual information onto a real-world view. For example, these devices use optical or video perspective technology to fuse virtual images with real-world scenes through head-mounted displays or smart glasses, providing users with an intuitive and interactive visual experience. The Transformer attention model can refer to a deep learning architecture based on a self-attention mechanism. For instance, this model assigns different attention weights by calculating the correlation between all positions in the input sequence, effectively capturing long-distance dependencies and handling sequence-to-sequence mapping tasks. The transparency of visual guidance information refers to the visibility parameter of virtual information on the display screen. For example, this parameter is dynamically calculated by the Transformer attention model based on the current calibration task stage and the user's focus, and its value is continuously adjustable between completely transparent and completely opaque, used to balance the intensity of information cues and the degree of occlusion of the real field of vision. The position of visual guidance information refers to the spatial coordinates of virtual information on the display screen. For example, these coordinates are calculated in real-time by the Transformer attention model based on the relationship between the human hand contour, robot state, and task target in three-dimensional space, ensuring that the guidance information always appears in the most appropriate place in the user's field of vision. Feedback content can refer to all prompts and status information presented to the user through the visual channel. For example, this content includes graphic or text elements such as calibration status indicators, next operation guidance, and error warning icons, which can help users understand the system status and perform the correct collaborative actions.

[0048] As a concrete example, in a human-machine collaboration scenario for industrial assembly training, operators wear augmented reality devices such as the Microsoft HoloLens 2. The Transformer attention model built into the visual guidance feedback module continuously receives calibration results from the hand-eye calibration calculation module and real-time hand position data from the dynamic contour tracking module. Through its encoder-decoder structure and self-attention mechanism, the model analyzes the current task context (e.g., "tightening the screw") and the user's gaze point (obtained through eye tracking), dynamically calculating the optimal attributes of the visual guidance information. For example, when the user's hand approaches the target screw, the transparency (transparency of the visual guidance information) of a virtual arrow (feedback content) is adjusted to semi-transparent to avoid excessive occlusion; its three-dimensional position (position of the visual guidance information) is precisely anchored above the screw. All these virtual elements are rendered and displayed by the AR device at a refresh rate of 60 frames per second, ensuring the real-time nature, accuracy, and immersive experience of the visual guidance.

[0049] By applying augmented reality technology and an intelligent attention model, this invention can generate dynamic visual guidance that is highly adaptive to the environment and tasks and fits the user's intentions, significantly improving the naturalness and efficiency of human-computer interaction and reducing the cognitive load on the operator.

[0050] In some implementations, the task context-aware module includes a BERT-based natural language processing model and a long short-term memory network. The natural language processing model parses speech or text instructions, and the long short-term memory network analyzes historical operation sequences and predicts task probability distributions.

[0051] BERT-based natural language processing models can refer to pre-trained language understanding models based on the Transformer architecture. For example, these models are pre-trained on large-scale corpora using masked language models and next-sentence prediction tasks, capturing the deep semantics of words within context. They are used for intent recognition and semantic parsing of user-inputted speech or text commands. Natural language processing models can refer to a collective term for computer algorithms capable of understanding and processing human language. For instance, these models use techniques such as word embedding and sequence modeling to convert unstructured text or speech into structured, machine-readable information, serving as a bridge for natural interaction between machines and humans. Speech or text commands refer to operational commands issued by users to the system in the form of sound or written text. For example, in an industrial setting, a user saying "Start calibration" or inputting the text "Execute precision verification mode" contains the user's explicit intention for the system to perform a task, driving the system into the corresponding working state. Long Short-Term Memory (LSTM) networks can refer to a special type of recurrent neural network structure. For example, by introducing input, forget, and output gate mechanisms, this network can effectively learn and remember long-term dependencies in sequential data, used for modeling and analyzing the temporal patterns of historical operational data. Historical operation sequences refer to a list of actions performed by the user and robot in chronological order over a past period. For example, this sequence might include ordered steps such as "hand approach," "tool pickup," and "fine alignment," along with their timestamps, reflecting the task execution flow and user habits. Task probability distributions refer to predictions of the likelihood of the system being in or about to transition to different task states. For instance, this distribution, output by a Long Short-Term Memory network after analyzing historical operation sequences, is represented by probability values ​​such as "precision assembly: 0.8," "tool change: 0.15," and "pause: 0.05," providing a forward-looking task context for the system's adaptive decisions.

[0052] As a concrete example: In a human-machine collaboration scenario for maintaining complex equipment, an engineer speaks the command via microphone: "Check and calibrate the end effector of the third robotic arm." The task context awareness module first uses a BERT-based model (Bidirectional Encoder Representations from Transformers) to recognize and semantically parse the voice command, extracting the core intent "calibrate," the target "third robotic arm," and the "end effector." Simultaneously, the module's Long Short-Term Memory (LSTM) network reads recently stored historical operation sequences, recording previous steps such as "system self-check," "arm extension initialization," and "safety confirmation." The LSTM analyzes this sequence using its gating mechanism, predicting the most likely next task step and its probability, such as outputting the probability distribution of "perform hand-eye calibration: 0.75," "perform accuracy test: 0.20," and "enter standby: 0.05." These parsed command semantics and predicted task probabilities together constitute the task context information, which is then passed to the adaptive calibration decision module.

[0053] By integrating advanced natural language processing and sequence prediction technologies, this invention enables the system to accurately understand user intent and predict task progress, providing human-computer collaboration with more intelligent and task-oriented context-aware capabilities, and enhancing the smoothness and predictability of collaboration.

[0054] In some implementations, the adaptive calibration decision module uses a deep reinforcement learning algorithm as the core controller. The deep reinforcement learning algorithm generates calibration parameter adjustment instructions and task switching signals based on calibration error, task context information, and user behavior patterns.

[0055] Deep reinforcement learning algorithms can refer to a machine learning method that combines deep neural networks with reinforcement learning principles. For example, this algorithm trains a policy network using the state, action, and reward sequences generated by the interaction between an agent and its environment, enabling it to solve complex decision-making problems in high-dimensional state spaces. A core controller refers to the core processing unit in a system responsible for integrating various information and making the highest-level decisions. For instance, this unit, based on deep reinforcement learning algorithms, receives multiple input signals and performs joint inference, outputting control commands that guide the behavior of the entire system. Calibration error refers to the deviation between the transformation matrix output by the hand-eye calibration calculation module and the actual spatial transformation relationship. For example, this error includes position and posture error components, typically quantified by Euclidean distance and rotation angle difference, used to assess the accuracy of the current calibration result. User behavior patterns refer to the user's personalized or habitual operating methods during operation. For example, these patterns include temporal characteristics such as the user's operation speed, preferred paths, and pause habits, which can be extracted from historical interaction data to make system decisions more adaptable to individual user differences. Calibration parameter adjustment commands are commands issued by the decision module to modify configuration parameters in the hand-eye calibration calculation process. For example, this command might specify the direction and magnitude of adjustments to parameters such as the process noise covariance of the extended Kalman filter or the kernel function bandwidth of Gaussian process regression, enabling real-time optimization of the calibration algorithm's performance. Task switching signals are control commands that indicate the system's transition between different operating modes or task stages. For instance, this signal is generated when the deep reinforcement learning algorithm determines that the current task is complete or another task is more urgent, coordinating the synchronous switching of operating states among system modules to adapt to new task requirements.

[0056] As a concrete example, in the human-machine collaboration unit of a flexible production line, the core controller of the adaptive calibration decision module employs a deep reinforcement learning algorithm called proximal policy optimization. The controller's state space includes calibration errors (such as the difference between the hand-eye transformation matrix of the current frame and the historical average), task probability distributions from the task context awareness module (such as "assembly: 0.6, detection: 0.4"), and user behavior patterns learned from long-term operational data (such as the operator typically performing rapid coarse calibration followed by fine-tuning). The agent outputs an action based on the current state, which is decoded into specific calibration parameter adjustment instructions (such as "increase the innovative covariance of the filter") and task switching signals (such as "immediately switch from 'general calibration' mode to 'precision assembly' mode"). These instructions and signals are sent in real-time to the hand-eye calibration calculation module and the visual guidance feedback module, achieving closed-loop intelligent control of the entire calibration process and task execution.

[0057] By employing deep reinforcement learning as the core decision-making mechanism, this invention enables the system to comprehensively consider real-time performance, task objectives, and user habits, and autonomously generate the optimal control strategy, significantly improving the intelligence level and overall operational efficiency of the human-machine collaborative system in dynamically changing environments.

[0058] In some implementations, the system also includes a distributed processing architecture that interconnects the modules via a high-speed data bus to ensure that the data stream transmission latency is less than 2 milliseconds.

[0059] Distributed processing architecture can refer to a computer architecture that decomposes system functions into multiple independent computing nodes and interconnects them through a communication network. For example, this architecture deploys different functional modules such as dynamic contour tracking and hand-eye calibration calculation on dedicated processors to improve the data processing capabilities and reliability of the entire system.

[0060] As a concrete example, a human-machine collaborative calibration system for an automotive assembly line employs a distributed processing architecture based on heterogeneous computing. The dynamic contour tracking module is deployed on a dedicated vision processing unit, the hand-eye calibration calculation module runs on a real-time FPGA, and complex logic such as task context awareness and adaptive decision-making resides on a multi-core CPU. These modules are interconnected via a high-speed PCIe data bus. Contour data (such as hand key point clouds) generated from the vision processing unit is directly transmitted to the FPGA via the PCIe bus using DMA, with the transmission latency of the entire data stream strictly controlled within 2 milliseconds. Similarly, calibration results and task context information also flow efficiently between modules via this bus, ensuring low latency throughout the entire link from image acquisition to decision output, meeting the real-time requirements of high-cycle assembly operations.

[0061] By adopting a distributed processing architecture and high-speed data bus interconnection technology, this invention ensures efficient and timely data exchange within the system, providing a solid hardware foundation for the parallel collaborative work of complex algorithm modules, thereby guaranteeing the real-time performance and responsiveness of the entire human-machine collaborative calibration system.

[0062] Corresponding to the above system embodiments, this specification also provides embodiments of robot hand-eye calibration methods for human-machine collaboration. Figure 2 A flowchart illustrating a robot hand-eye calibration method for human-robot collaboration, as provided in some embodiments of this specification, is shown. Figure 2 As shown, the specific steps include: The dynamic contour tracking module captures and tracks the dynamic contour of the operator's hand in real time and outputs contour data. The hand-eye calibration calculation module receives contour data and combines it with the motion data of the robot's end effector to calculate the hand-eye transformation matrix from the camera coordinate system to the robot's end effector coordinate system, and outputs the calibration results. The visual guidance feedback module receives the calibration confidence score and contour data from the calibration results, generates visual guidance information, and provides feedback to the user. The task context awareness module parses task instructions and operation sequences, and outputs task context information. The adaptive calibration decision module receives the calibration confidence score, task context information, and visual guidance information from the calibration results, generates calibration parameter adjustment instructions and task switching signals, and sends them to the hand-eye calibration calculation module and the visual guidance feedback module.

[0063] The above is an illustrative scheme of a robot hand-eye calibration method for human-machine collaboration according to this embodiment. It should be noted that the technical solution of this robot hand-eye calibration method for human-machine collaboration belongs to the same concept as the technical solution of the robot hand-eye calibration system for human-machine collaboration described above. For details not described in detail in the technical solution of the robot hand-eye calibration method for human-machine collaboration, please refer to the description of the technical solution of the robot hand-eye calibration system for human-machine collaboration described above.

[0064] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0065] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this invention. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A robot hand-eye calibration system for human-robot collaboration, characterized in that, include: The dynamic contour tracking module is used to capture and track the dynamic contour of the operator's hand in real time and output contour data. The hand-eye calibration calculation module, associated with the dynamic contour tracking module, is used to receive the contour data and, in conjunction with the motion data of the robot end effector, calculate the hand-eye transformation matrix from the camera coordinate system to the robot end effector coordinate system, and output the calibration result. The visual guidance feedback module, which is associated with the hand-eye calibration calculation module and the dynamic contour tracking module, is used to receive the calibration results and the contour data, generate visual guidance information and provide feedback to the user. The task context awareness module is used to parse task instructions and operation sequences and output task context information; An adaptive calibration decision module, associated with the hand-eye calibration calculation module and the task context awareness module, is used to receive the calibration confidence score, the task context information, and feature information reflecting the dynamics of user operations from the calibration results, generate calibration parameter adjustment instructions and task switching signals, and send them to the hand-eye calibration calculation module and the visual guidance feedback module.

2. The system according to claim 1, characterized in that, The dynamic contour tracking module includes a depth camera and a PointNet++-based 3D segmentation network. The depth camera acquires RGB-D point cloud data at a preset frame rate and a preset resolution. The 3D segmentation network outputs the 3D key point coordinates of the operator's hand contour.

3. The system according to claim 2, characterized in that, The hand-eye calibration calculation module uses an extended Kalman filter algorithm to fuse contour data, inertial measurement unit data, and robot encoder data, and uses a Gaussian process regression model for error compensation to output calibration results with sub-millimeter accuracy.

4. The system according to claim 3, characterized in that, The hand-eye calibration calculation module outputs a calibration confidence score, and the steps for calculating the calibration confidence score include: Obtain the reference baseline sequence, the current estimation sequence, and the uncertainty sequence of the current estimation of the calibration parameters; based on the reference baseline sequence, the current estimation sequence, and the uncertainty sequence, and combined with the pre-calculated dynamic sensitivity adjustment factor and the preset first weighting coefficient, generate the parameter error evaluation value; Obtain the error vector sequence of the consistency verification error and the tolerance threshold sequence; based on the error vector sequence and the tolerance threshold sequence, and combined with the preset second weighting coefficient, generate the position error evaluation value; Obtain the environmental interference intensity sequence and normalize it; based on the normalized environmental interference intensity sequence, and combined with the preset third weighting coefficient, generate an environmental interference assessment value; By combining the parameter error assessment value, the position error assessment value, and the environmental interference assessment value, a comprehensive calibration uncertainty value is generated. Based on the comprehensive value of the calibration uncertainty, the calibration confidence score is calculated.

5. The system according to claim 4, characterized in that, The steps for calculating the dynamic sensitivity adjustment factor include: Obtain the task complexity metric and the sequence of attitude angle change rate components; Based on the task complexity metric, the task complexity contribution value is obtained by mapping through a preset scaling factor and a sigmoid function. Based on the attitude angle change rate component sequence, the values ​​are adjusted using a preset normalized reference value and mapped using the hyperbolic tangent function to obtain the operational dynamics contribution value. The dynamic sensitivity adjustment factor is generated by combining the preset normal base, the contribution value of task complexity and the contribution value of operation dynamics, and the preset adjustment coefficient.

6. The system according to claim 1, characterized in that, The visual guidance feedback module includes an augmented reality device, which dynamically adjusts the transparency and position of the visual guidance information based on the Transformer attention model and renders the feedback content at a preset refresh rate.

7. The system according to claim 1, characterized in that, The task context awareness module includes a BERT-based natural language processing model and a long short-term memory network. The natural language processing model parses speech or text instructions, and the long short-term memory network analyzes historical operation sequences and predicts task probability distribution.

8. The system according to claim 1, characterized in that, The adaptive calibration decision module uses a deep reinforcement learning algorithm as the core controller. The deep reinforcement learning algorithm generates calibration parameter adjustment instructions and task switching signals based on calibration error, task context information and user behavior patterns.

9. The system according to claim 1, characterized in that, The system also includes a distributed processing architecture, which interconnects the modules through a high-speed data bus to ensure that the data stream transmission latency is less than 2 milliseconds.

10. A dynamic hand-eye calibration processing method based on contour tracking, characterized in that, The method is applied to the robot hand-eye calibration system for human-robot collaboration as described in any one of claims 1 to 9, and the method includes: The dynamic contour tracking module captures and tracks the dynamic contour of the operator's hand in real time and outputs contour data. The contour data is received by the hand-eye calibration calculation module, and combined with the motion data of the robot end effector, the hand-eye transformation matrix from the camera coordinate system to the robot end effector coordinate system is calculated, and the calibration result is output. The visual guidance feedback module receives the calibration confidence score and the contour data from the calibration results, generates visual guidance information, and feeds it back to the user. The task context awareness module parses task instructions and operation sequences, and outputs task context information. The adaptive calibration decision module receives the calibration confidence score, the task context information, and the visual guidance information from the calibration results, generates calibration parameter adjustment instructions and task switching signals, and sends them to the hand-eye calibration calculation module and the visual guidance feedback module.

Images