Robot forward and inverse kinematics optimization method based on depth camera hand-eye calibration

Abstract

The invention discloses a robot forward and inverse kinematics optimization method and system based on depth camera hand-eye calibration, and the method comprises the steps: installing a depth camera on an end effector of a robot, and synchronously collecting an environment RGB image and laser radar depth information; identifying feature points of the calibration plate through target detection, obtaining three-dimensional coordinates of the feature points in combination with depth information, and simultaneously solving a hand-eye calibration conversion matrix to unify coordinates of multiple sensors; on the basis of DH parameters of the robot, the coordinate transformation matrixes of all the joints are superposed, and the pose of the end effector relative to the base coordinate system is obtained; a homotopy continuation method is combined with DNN to solve inverse kinematics, the DNN predicts an easy-to-reach point or an initial joint angle, the homotopy continuation method constructs a convex group contract syntactic function, a prediction-correction mechanism tracks a path until a parameter is 1, a target solution is obtained, and kinematics constraint is introduced to guarantee a solution method; and finally, a control signal is generated according to the solved joint angle, and the end effector is driven to move according to an expected track. And the operation precision and the real-time performance of the robot are improved.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, and more specifically, relates to a method and system for optimizing robot forward and inverse kinematics based on depth camera hand-eye calibration. Background Technology

[0002] With the deep application of robotics technology in high-precision machining, automated assembly, and other fields, hand-eye calibration and inverse kinematics solutions have gradually become key factors restricting the accuracy of robot control. Among them, hand-eye calibration plays a core role in establishing the spatial relationship between the robot's end effector and the vision sensor, directly determining whether the robot can achieve precise positioning and motion control based on visual information; while inverse kinematics serves as a bridge connecting the robot's joint angles and the end effector's pose. Especially in seven-DOF redundant robot systems, the results of solving these two methods directly affect the robot's operational accuracy and response efficiency.

[0003] However, existing technical solutions still face significant bottlenecks in practical applications. In the handwritten calibration stage, traditional methods rely excessively on complex mathematical models and a large number of calibration samples, which not only makes the calibration process cumbersome and inefficient, but also makes it difficult to adapt to dynamic application scenarios. When there are interference factors such as changes in lighting or object occlusion in the environment, the calibration accuracy is easily affected, resulting in a deviation between the pose data output by the vision sensor and the actual movement requirements of the robot, which in turn causes positioning errors in the end effector and fails to meet the requirements of high-precision operations.

[0004] In solving forward and inverse kinematics, the limitations of traditional methods are even more pronounced. For multi-degree-of-freedom robot systems, forward kinematics calculations often suffer from insufficient accuracy due to the complexity of joint association models, making it difficult to accurately reflect the correspondence between joint angles and end-effector poses. Inverse kinematics solutions face even more severe challenges, not only exhibiting high computational complexity and poor real-time performance, but also being prone to multiple solutions or no solutions at all. Especially when the robot is in a singular configuration, the failure rate increases significantly, directly affecting the robot's real-time response speed and control accuracy, making it unsuitable for the stringent motion control requirements of high-precision machining, automated assembly, and other scenarios.

[0005] Currently, the shortcomings of robot hand-eye calibration and forward and inverse kinematics solutions are superimposed, further amplifying the operational errors of robots in dynamic environments. Traditional methods can no longer meet the dual requirements of accuracy and real-time performance in practical applications. There is an urgent need for a technical solution that can achieve joint optimization of the two to break through existing bottlenecks and improve the robot's performance in complex scenarios. Summary of the Invention

[0006] This invention aims to solve the problems of large hand-eye calibration errors and complex forward and inverse kinematics solutions in existing robots. By optimizing the joint solution of hand-eye calibration and forward and inverse kinematics, the influence of illumination and occlusion on calibration in dynamic environments is reduced, the kinematics solution process of multi-degree-of-freedom robots (especially seven-degree-of-freedom redundant systems) is simplified, and the robot's operational accuracy and real-time performance are improved, meeting the needs of high-precision machining, automated assembly and other scenarios.

[0007] To address the aforementioned deficiencies or improvement needs of existing technologies, as a first aspect of this invention, a method for optimizing robot forward and inverse kinematics based on depth camera hand-eye calibration is also provided, comprising:

[0008] S1. Install a depth camera on the robot's end effector to simultaneously acquire RGB image visual information and LiDAR depth information of the environment;

[0009] S2. Based on the target detection method, detect the feature points of the calibration board, combine the depth information to obtain its three-dimensional coordinates, and then solve the hand-eye calibration transformation matrix to achieve the unification of multi-sensor coordinates;

[0010] S3. Based on the robot's DH parameters, the position and orientation of the end effector relative to the robot's base coordinate system are obtained by multiplying and superimposing the coordinate transformation matrices of each joint;

[0011] S4. Homotopy extension combined with a deep neural network (DNN) is used to solve the inverse kinematics. The DNN does not directly output the target joint angles, but predicts the initial values ​​of the joint angles on easily reachable points close to the desired end pose or on the homotopy path, providing a high-quality starting point for path tracing in the homotopy extension method. The homotopy extension method constructs a convex combined homotopy function, starts from the known solution of the auxiliary system, and uses a prediction-correction mechanism to track the homotopy path as the parameters change from 0 to 1. Finally, when the parameters reach 1, the solution to the target inverse kinematics problem is obtained, thus stably bypassing singularities and completing the global convergence solution. At the same time, kinematic constraints are introduced and the legality of the solution is guaranteed by a soft constraint method.

[0012] S5. Based on the joint angles obtained from the inverse kinematics optimization solution, generate control signals for each joint of the robot, drive the robot's end effector to move along the desired trajectory, and complete the corresponding task.

[0013] Furthermore, in S1, the RGB image is used to extract visual features, and the lidar data is used to measure the distance between the object and the sensor.

[0014] Furthermore, the method for calculating the hand-eye calibration transformation matrix in S2 is as follows:

[0015] First, using a point P0 in the calibration board coordinate system as a reference, and based on the known camera extrinsic parameters, i.e., the transformation matrix T1, we can transform to point P1 in the camera coordinate system. Then, using the hand-eye calibration matrix to be determined, i.e., the transformation matrix X, we can transform to point P2 in the robot's end effector coordinate system. Finally, using the known robot's own parameters, i.e., the transformation matrix T3, we can transform to point P3 in the robot's base coordinate system. Therefore, we can obtain the following relationship:

[0016] T3XT1P0=P3

[0017] Changing the pose of the robotic arm's end effector keeps the coordinates of P0 and P3 unchanged, while the coordinates of P1 and P2 change. Therefore, the above relationship becomes:

[0018] T3′XT1′P0=P3

[0019] Combining the above two equations, we can obtain the following relationship:

[0020] T3XT1=T3′XT1′

[0021] Multiply the above expression by T3′ on the left. -1 Right multiply by T1 -1 ,available:

[0022] T3′ -1 T3X=XT1′T1 -1

[0023] The above equation can be understood as an equation in the form of AX = XB, and the hand-eye transformation matrix X can be solved.

[0024] Furthermore, the coordinate transformation matrix of each joint in S3 is constructed based on the DH parameter method and is used to describe the position and attitude transformation relationship between adjacent joint coordinate systems of the robot. This matrix is ​​a 4×4 homogeneous transformation matrix that integrates the rotation and translation information of the joint, and its specific form is as follows:

[0025]

[0026] Where, θ i Let α be the rotation angle of the i-th joint; i Let a be the torsion angle of the i-th link; i d is the length of the i-th link; i Let be the offset of the i-th joint.

[0027] Furthermore, the homotopy function in S4 is:

[0028] H(θ,t)=(1-t)*G(θ)+t*F(θ)=0

[0029] Where H(θ,t) is the constructed homotopy function; θ is the robot's joint angle vector; t is the homotopy parameter; G(θ) is the constructed simple auxiliary system, with the target point set as an end point X0 corresponding to a known configuration, i.e., G(θ) = f(θ) - X0 = 0, and its solution θ 0 Given that X0 is the known end-effector pose; f(θ) is the robot's positive kinematics function.

[0030] Furthermore, the specific process in S4 of tracking the homotopy path during the parameter change from 0 to 1 using a prediction-correction mechanism is as follows:

[0031] Let the current tracking point be (θ) k ,t k ), where θ is the robot joint angle vector; t is the homotopy parameter t∈[0,1]; H(θ,t) is the constructed convex combinatorial homotopy function, connecting the known easily solvable problem G(θ)=0 and the target inverse kinematics problem F(θ=0;

[0032] During the prediction phase, a path tangent vector field is defined based on the local geometric characteristics of the current point to determine the direction of parameter evolution:

[0033]

[0034] in, The rate of change of the homotopy parameter t; θ is the rate of change of the joint angle vector θ; k t is the joint angle vector at the k-th iteration; k J is the homotopy parameter at the k-th iteration; H Let H be the Jacobian matrix of θ, reflecting the effect of joint angle changes on the homotopy function; The partial derivative of H with respect to t represents the trend of parameter variation;

[0035] Based on the initial joint angle θ0 predicted by the deep neural network, the next prediction point is generated using the Euler method:

[0036]

[0037] t k+1 =t k +Δt

[0038] in, The joint angle vector predicted in step k; t k+1 t is the homotopy parameter at the (k+1)th iteration; k Δt is the homotopy parameter at the k-th iteration; Δt is a fixed parameter step size to ensure a smooth transition from t=0 to t=1.

[0039] During the correction phase, starting from the predicted point, the deviation is corrected using constraint equations to ensure that the evolution always follows the homotopy path; constraint equations are constructed to force the homotopy function to be satisfied.

[0040] H(θ,t k+1 ) = 0

[0041] The correction point is solved using the Gauss-Newton method by utilizing the continuity of the robot's forward kinematics model f(θ):

[0042]

[0043] Repeat the above prediction-correction loop until t k+1 =1, at which point θ is obtained k+1 That is, to satisfy the desired end pose X d Solution of joint angles.

[0044] Furthermore, the kinematic constraints in S4 are specifically as follows:

[0045] A soft constraint method was adopted, which introduced a penalty term for the joint angle range into the objective function. In this way, during the iteration process, the robot control system not only optimizes the error, but also takes into account the physical limitations of each joint, ensuring that the robot can operate safely and stably in practical applications.

[0046] E total =E+λ·C constrains

[0047] Among them, E total This represents the total optimization error, E is the basic error term, and C... constrains λ is the constraint penalty term, and λ is a parameter that controls the weight of the penalty term.

[0048] Furthermore, the objective function is specifically:

[0049] In each iteration, the error between the current target position and the actual position of the robot's end effector is calculated; the optimization objective is to minimize this error, thereby obtaining the optimal joint angle.

[0050] E = ||P target -P current ||

[0051] Among them, P target It is the position of the target end effector, P current The position of the end effector is calculated using the current joint angle, and E is the error value.

[0052] As a second aspect of the present invention, a robot forward and inverse kinematics optimization system based on depth camera hand-eye calibration is also provided, comprising:

[0053] The multi-source data acquisition unit is used to install a depth camera on the robot's end effector to simultaneously acquire RGB image visual information and LiDAR depth information of the environment;

[0054] The hand-eye calibration fusion unit is used to detect feature points of the calibration board based on the target detection method, combine depth information to obtain its three-dimensional coordinates, and then solve the hand-eye calibration transformation matrix to achieve the unification of multi-sensor coordinates.

[0055] The positive kinematics calculation unit is used to obtain the position and orientation of the end effector relative to the robot's base coordinate system by multiplying and superimposing the coordinate transformation matrices of each joint based on the robot's DH parameters.

[0056] The inverse kinematics solving unit is used to solve inverse kinematics using the homotopy extension method combined with a deep neural network (DNN). The DNN does not directly output the target joint angles, but instead predicts the initial joint angle values ​​at easily reachable points close to the desired end pose or on the homotopy path, providing a high-quality starting point for path tracing using the homotopy extension method. The homotopy extension method constructs a convex combined homotopy function, starting from the known solution of the auxiliary system, and uses a prediction-correction mechanism to track the homotopy path as the parameters change from 0 to 1. Finally, when the parameters reach 1, it obtains the solution to the target inverse kinematics problem, thus stably bypassing singularities and achieving global convergence. Simultaneously, kinematic constraints are introduced, and the validity of the solution is guaranteed through soft constraint methods.

[0057] The robot control output unit is used to generate control signals for each joint of the robot based on the joint angles obtained from inverse kinematics optimization, and drive the robot end effector to move along the desired trajectory to complete the corresponding task.

[0058] As a third aspect of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, which is executed by a processor, any step of the robot forward and inverse kinematics optimization method based on depth camera hand-eye calibration.

[0059] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0060] 1. The robot forward and inverse kinematics optimization method based on hand-eye calibration using a depth camera, as described in this invention, involves mounting a depth camera on the robot's end effector to simultaneously acquire environmental RGB image visual information and LiDAR depth information. This is combined with target detection methods to detect feature points on a calibration board and obtain their 3D coordinates. The hand-eye calibration transformation matrix is ​​then solved simultaneously to achieve unified coordinates across multiple sensors. This technical feature ensures precise alignment between the visual sensor and the robot's base coordinate system, effectively eliminating pose calculation deviations caused by inconsistencies in coordinates between multiple data sources. It provides a consistent and reliable spatial coordinate reference for subsequent forward and inverse kinematics solutions, avoiding end effector pose positioning errors caused by coordinate misalignment and ensuring the fundamental accuracy of kinematic calculations.

[0061] 2. The robot forward and inverse kinematics optimization method based on depth camera hand-eye calibration of the present invention calculates the position and orientation of the end effector relative to the robot's base coordinate system by multiplying and superimposing the coordinate transformation matrices of each joint sequentially based on the robot's DH parameters. This technique leverages the standardized modeling advantages of the DH parameter method to accurately describe the position and orientation transformation relationships between adjacent joint coordinate systems. This ensures that the forward kinematics calculation process has clear mathematical logic and traceability, accurately reflecting the impact of joint angle changes on the end effector's pose. It avoids the pose calculation errors caused by ambiguity in joint relationships in traditional modeling methods, providing accurate forward kinematics model support for inverse kinematics solutions.

[0062] 3. The robot forward and inverse kinematics optimization method based on depth camera hand-eye calibration of the present invention solves the inverse kinematics by using the homotopy extension method combined with a deep neural network (DNN). The DNN predicts easily reachable points or initial joint angles of the homotopy path close to the desired end pose. The homotopy extension method constructs a convex combination homotopy function and uses a prediction-correction algorithm to track the path to obtain the target solution. At the same time, kinematic constraints are introduced and the legality of the solution is guaranteed by a soft constraint method. This technical feature solves the problems of traditional inverse kinematics solving methods being sensitive to initial values ​​and lacking global convergence guarantees. The high-quality initial points provided by the DNN accelerate the convergence process, the homotopy extension method ensures the global convergence of the solution and can bypass singular points, and the soft constraint method guarantees the kinematic legality of the solution. Finally, high-precision and high-reliability inverse kinematics solutions are achieved, providing key technical support for precise trajectory control of robots. Attached Figure Description

[0063] Figure 1 This is a flowchart of a robot forward and inverse kinematics optimization method based on depth camera hand-eye calibration, according to an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the Eye-in-Hand system according to an embodiment of the present invention;

[0065] Figure 3 This is a flowchart illustrating the transformation matrix calculation according to an embodiment of the present invention;

[0066] Figure 4 This is a schematic diagram of the robotic arm configuration according to an embodiment of the present invention;

[0067] Figure 5 This is a schematic diagram of the forward propagation of a three-layer DNN according to an embodiment of the present invention;

[0068] Figure 6 This is a flowchart illustrating the DNN and prediction process in an embodiment of the present invention.

[0069] Figure 7 This is a schematic diagram of the system units in an embodiment of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0071] Example 1

[0072] Please refer to Figure 1 This embodiment 1 provides a method for optimizing the forward and inverse kinematics of a robot based on hand-eye calibration using a depth camera, including:

[0073] S1. Install a depth camera on the robot's end effector to simultaneously acquire RGB image visual information and LiDAR depth information of the environment;

[0074] S2. Based on the target detection method, detect the feature points of the calibration board, combine the depth information to obtain its three-dimensional coordinates, and then solve the hand-eye calibration transformation matrix to achieve the unification of multi-sensor coordinates;

[0075] S3. Based on the robot's DH parameters, the position and orientation of the end effector relative to the robot's base coordinate system are obtained by multiplying and superimposing the coordinate transformation matrices of each joint;

[0076] S4. Homotopy extension combined with a deep neural network (DNN) is used to solve the inverse kinematics. The DNN does not directly output the target joint angles, but predicts the initial values ​​of the joint angles on easily reachable points close to the desired end pose or on the homotopy path, providing a high-quality starting point for path tracing in the homotopy extension method. The homotopy extension method constructs a convex combined homotopy function, starts from the known solution of the auxiliary system, and uses a prediction-correction mechanism to track the homotopy path as the parameters change from 0 to 1. Finally, when the parameters reach 1, the solution to the target inverse kinematics problem is obtained, thus stably bypassing singularities and completing the global convergence solution. At the same time, kinematic constraints are introduced and the legality of the solution is guaranteed by a soft constraint method.

[0077] S5. Based on the joint angles obtained from the inverse kinematics optimization solution, generate control signals for each joint of the robot, drive the robot's end effector to move along the desired trajectory, and complete the corresponding task.

[0078] This embodiment 1 further elaborates on the above steps.

[0079] (1) Multi-source data acquisition

[0080] The method proposed in Embodiment 1 achieves adaptive calibration in dynamic environments by combining camera visual data and laser scanning information; please refer to... Figure 2 The L515 camera was used for "eye-to-hand" calibration.

[0081] In the process of robot hand-eye calibration, data acquisition is a crucial step, determining the accuracy and reliability of subsequent calibration and kinematic calculations. The hand-eye calibration scheme in Embodiment 1 utilizes multi-sensor data fusion, leveraging the collaboration of depth cameras and LiDAR to provide high-precision visual and spatial positioning information.

[0082] 1.1 Camera and LiDAR Integration: To ensure high-precision calibration in complex environments, we mounted a depth camera on the robot's end effector and used the camera's built-in LiDAR to acquire depth information from the environment. The RGB images captured by the camera provide visual information, while the LiDAR obtains the precise distance between objects in the environment and the sensor by emitting lasers and receiving reflected signals. This information will be used to accurately measure the positions of the calibration board and the robot's end effector.

[0083] 1.2 Calibration Plate Placement and Image Acquisition; In actual operation, the calibration plate is fixed on the worktable, and the robot's end effector adjusts its position through movement. The camera captures images of the calibration plate and, combined with the known geometric patterns on the calibration plate, extracts key feature points. These image data and extracted feature points serve as the basis for subsequent calibration calculations. Through the pattern features of the calibration plate, we can clearly calculate the transformation relationship between the camera coordinate system and the robot coordinate system.

[0084] 1.3 Multimodal Data Synchronization; During data acquisition, it is essential to ensure the time synchronization of camera images and LiDAR data. To this end, this embodiment 1 employs high-precision timestamp technology to ensure accurate data correspondence between the two, and uses a synchronization algorithm to eliminate errors caused by data acquisition delays. This method effectively eliminates time differences between different sensors, ensuring the accuracy of subsequent calculations.

[0085] 1.4 Depth Information Processing and Fusion: The depth information acquired by the LiDAR will be processed into point cloud data, which reflects the three-dimensional spatial structure of the environment. Combined with RGB image data, this provides the robot with more comprehensive environmental perception, helping to accurately determine the precise position of the robot's end effector relative to the worktable and the environment. Simultaneously, the fusion of point cloud data and image data enhances the accuracy and robustness of visual calibration, especially under complex lighting conditions or object occlusion.

[0086] By using precise data acquisition and multi-sensor data fusion, the method in this embodiment 1 can effectively improve the accuracy of the hand-eye calibration process and provide more reliable spatial data for subsequent robot control and path planning.

[0087] (2) Hand-eye alignment fusion

[0088] There are many methods for testing calibration boards. The current mainstream method is to use the detection function in OpenCV to detect grid points. This method has high requirements for the experimental environment. Furthermore, since the pose data is obtained entirely by detecting grid points and comparing them with the actual size of the calibration board, the robustness and reliability of this detection in complex environments are poor.

[0089] The main idea of ​​this embodiment 1 is to use an L515 depth camera to obtain real-time depth information of the target point, and then perform fitting calculations based on the depth information. This embodiment 1 uses YOLOv8 as the target detection method. YOLOv8, as a deep learning one-stage detector, relies on data-driven feature learning for its calibration board detection, making it suitable for complex scenes (such as industrial environments and uncontrolled lighting) and custom requirements. After detecting the target point using YOLOv8, the coordinates of the target point in the camera coordinate system can be obtained by combining the camera's depth information. The plane can then be fitted using the coordinates of multiple detected points. For example, if three points A(x1,y1,z1), B(x2,y2,z2), and C(x3,y3,z3) are detected, with point A as the origin, then:

[0090] AB = BA (1)

[0091] AC = CA (2)

[0092] z = AB × AC (3)

[0093] x=AB (4)

[0094] y = z × x (5)

[0095] From the above calculations, the transformation matrix from the calibration plate plane to the camera coordinate system can be obtained as follows:

[0096] R = (x, y, z) (6)

[0097] The translation matrix can be directly obtained from the camera's depth information.

[0098] Please refer to Figure 3 After data acquisition, the next step is to calculate the eye-to-hand transformation matrix. First, using a point P0 in the calibration board coordinate system as a reference, and based on the known camera extrinsic parameters (transformation matrix T1), the system can be transformed to point P1 in the camera coordinate system. Then, using the desired hand-eye calibration matrix (transformation matrix X), the system can be transformed to point P2 in the robot's end effector coordinate system. Finally, based on the known robot's own parameters (transformation matrix T3), the system can be transformed to point P3 in the robot's base coordinate system. Therefore, the following relationship can be obtained:

[0099] T3XT1P0=P3 (7)

[0100] Changing the pose of the robotic arm's end effector keeps the coordinates of P0 and P3 unchanged, while the coordinates of P1 and P2 change. Therefore, the above relationship becomes:

[0101] T3′XT1′P0=P3 (8)

[0102] Combining the above two equations, we can obtain the following relationship:

[0103] T3XT1=T3′XT1′ (9)

[0104] Multiply the above expression by T3′ on the left. -1 Right multiply by T1 -1 ,available:

[0105] T3′ -1 T3X=XT1′T1 -1 (10)

[0106] The above equation can be understood as an equation in the form of AX = XB, and the hand-eye transformation matrix X can be solved.

[0107] In actual robot operation, environmental and task conditions are often dynamically changing, which poses a challenge to the hand-eye calibration process. To ensure high-precision calibration in these dynamic environments, especially when the robot is performing complex operations, a preferred embodiment also proposes an adaptive adjustment mechanism as follows:

[0108] The calibration parameters are dynamically updated. As the robot's end effector moves, especially when the robot switches between different task environments, the calibration parameters may change. For example, environmental factors such as lighting, object positions, and the robot's own load can all affect the accuracy of hand-eye calibration. Therefore, the calibration parameters must be adjusted in real time to cope with these changes. In this embodiment 1, by combining sensor data and visual information from the robot's end effector, a dynamic optimization algorithm is used to continuously adjust the transformation matrix of hand-eye calibration to ensure calibration accuracy.

[0109] Optimization is based on real-time feedback; through real-time data feedback from LiDAR and cameras, the robot can automatically adjust to changes in the current environment. Specifically, when the robot performs a specific task, the system makes small adjustments to the calibration matrix based on real-time perceived environmental information (such as depth data and image features). This adaptive adjustment ensures that the robot can maintain accurate hand-eye calibration in different operating environments, even with slight disturbances or positional shifts.

[0110] Real-time control of the detection: Real-time detection is required during the camera's inspection of the calibration board. This means that the calibration board is inspected on the screen in real time to ensure that each detected grid point coincides with a grid point on the calibration board; otherwise, data needs to be re-acquired. Furthermore, it is crucial that the detected calibration board pose corresponds one-to-one with the pose matrix of the current robotic arm end effector. The specific control method is to perform detection in a stationary state, meaning that the checkerboard detection is only performed after the robotic arm has moved to the corresponding position.

[0111] To ensure reliable detection results, we adhere to two principles during pose acquisition: first, the number of data sets acquired should be at least five to increase the accuracy of the solution; second, the pose differences of the robot should be as large as possible, meaning that all six joints of the robotic arm should change. This is to avoid singular values ​​that could lead to solution failure. This paper suggests that the acquired poses be distributed as follows: upper left, upper, upper right; right, lower right, lower, lower left, left, and top center of the calibration board.

[0112] Error Compensation and Control: To further improve the robustness of the system, the method in Embodiment 1 combines an error compensation algorithm to correct the calibration error in real time. By calculating the error magnitude under the current environment, the system can automatically adjust the calibration matrix and perform error compensation through optimized control strategies (e.g., initiating recalibration when the feature point offset of the calibration board is >0.5mm), ensuring the positioning accuracy and motion control stability of the robot in dynamic environments.

[0113] The introduction of adaptive adjustment technology ensures that the robot system can perform high-precision hand-eye calibration in constantly changing environments, improving the robustness and accuracy of the calibration process. Whether in complex dynamic environments or with changes in the robot's own operations, the system can maintain excellent performance under various conditions.

[0114] (3) Forward kinematics calculation

[0115] This embodiment 1 proposes a joint solution method based on numerical optimization and deep learning, combining forward and inverse kinematics optimization problems.

[0116] Forward kinematics refers to the calculation of the position and orientation of a robot's end effector using a kinematic model, given the robot's joint angles. Forward kinematics is one of the most fundamental computational tasks in robot motion control, and its purpose is to determine the precise position and orientation of the robot's end effector relative to the robot's base coordinate system.

[0117] Please refer to Figure 4 In this embodiment 1, we take the KUKA LBR iiwa 14R820 seven-DOF robot as an example and use the robot's Denavit-Hartenberg (DH) parameters to establish a forward kinematics model. DH parameters are a standard method for describing the relative positions of the robot's joint coordinate systems in a normalized manner. In this method, each joint is defined by four parameters: joint angle θ, joint offset d, link length a, and link torsion angle α. These parameters are used to describe the transformation from one joint coordinate system to the next.

[0118] The specific steps for solving the forward kinematics are as follows:

[0119] 3.1 Establish the robot model; using the DH parameter table, the relationships between the robot's joints and links are described as a series of transformation matrices. The transformation matrix of each joint is multiplied by the transformation matrix of the previous joint to obtain the complete transformation from the base coordinate system to the end effector coordinate system. The DH parameter table is shown in Table 1:

[0120] Table 1 DH Parameter Table

[0121] i <![CDATA[θ i ]]> <![CDATA[α i (rad)]]> <![CDATA[d i ]]> <![CDATA[a i ]]> 1 <![CDATA[θ1]]> -90° <![CDATA[L1]]> 0 2 <![CDATA[θ2]]> 90° 0 0 3 <![CDATA[θ3]]> -90° <![CDATA[L2]]> 0 4 <![CDATA[θ4]]> 90° 0 0 5 <![CDATA[θ5]]> -90° <![CDATA[L3]]> 0 6 <![CDATA[θ6]]> 90° 0 0 7 <![CDATA[θ7]]> 0 <![CDATA[L4]]> 0

[0122] 3.2 Coordinate Transformation; For a seven-DOF robot, the coordinate transformation of each joint consists of a rotation matrix and a displacement vector. By multiplying and superimposing the matrix equation (11) for each joint, the position and orientation of the end effector relative to the robot's base coordinate system can be calculated, as shown in equation (12) below:

[0123]

[0124] Where, θ i Let α be the rotation angle of the i-th joint; i Let a be the torsion angle of the i-th link; i d is the length of the i-th link; i Let be the offset of the i-th joint.

[0125] 3.3 Calculate the position and orientation of the end effector; using the coordinate system transformation described above, the spatial position and orientation (rotation matrix or quaternion) of the robot's end effector are obtained through the forward kinematic function f(θ). This information is crucial for subsequent task execution (such as path planning, control algorithms, etc.).

[0126] The core advantage of forward kinematics is its simple calculation process and explicit mathematical expression. It can directly output the spatial position and orientation of the robot's end effector given the angles of each joint (control input). For most industrial robot applications, forward kinematics is the most commonly used fundamental calculation.

[0127] (4) Solving inverse kinematics

[0128] Inverse kinematics is the process that is the opposite of forward kinematics. Given the desired position and orientation of the robot's end effector, the task of inverse kinematics is to solve for the required angles (control inputs) of each joint. Inverse kinematics plays a crucial role in robot path planning, control, and task execution.

[0129] Traditional inverse kinematics solutions face two main challenges:

[0130] 1. Multiple-solution problem: In some cases, the target position and orientation of a robot end effector may correspond to multiple joint angle solutions. For example, when there are many degrees of freedom in multiple joints, the robot end effector can achieve the same target position and orientation through different joint configurations, but effective strategy selection or searching for suitable solutions is required.

[0131] 2. Cases where solutions cannot be found: For certain task objectives, the target position of the robot's end effector may exceed the robot's workspace, making it impossible to find a valid joint angle. A mechanism is needed to identify and handle this situation.

[0132] To effectively address these issues, this embodiment 1 compares the advantages and disadvantages of two methods—the Newton-Raphson method and the homotopy extension method—and ultimately selects the homotopy extension method for complex inverse kinematics calculations, then combines it with a deep neural network to solve the inverse kinematics problem. The specific steps are as follows:

[0133] 4.1 Newton-Raphson Method; The Newton-Raphson method, as shown in Equation (13), is an iterative optimization method widely used to solve nonlinear equations. In inverse kinematics, we take the error between the desired position and the actual calculated position of the robot's end effector as the objective function and minimize the error by iteratively solving it. Each iteration updates the joint angle values ​​until the error converges to a sufficiently small range.

[0134]

[0135] Although the Newton-Raphson method is a classic iterative method for solving nonlinear equations, it has significant limitations when applied to robot inverse kinematics problems with high degrees of freedom, singular points, or near the workspace boundary:

[0136] a. Initial value sensitivity: The convergence and convergence rate of Newton's method are highly dependent on the quality of the initial joint angle guess x0. A poor initial guess may cause the iteration to diverge, converge to an incorrect solution (especially in the multi-solution region), or even fail to converge at all.

[0137] b. Local convergence: Newton's method typically exhibits quadratic convergence only in the vicinity of the solution. If the initial guess is far from the true solution, Newton's method may fail to find it even if the solution exists.

[0138] c. Ill-conditioned Jacobian matrix: When the robot approaches a singular configuration, the kinematic Jacobian matrix tends to be ill-conditioned (large condition number). At this time, the calculation of the iteration step size of Newton's method is unstable, which can easily lead to numerical oscillation, slow convergence or even failure.

[0139] d. Cannot guarantee global convergence: Newton's method lacks a mechanism to ensure that starting from any initial point, it eventually converges to a solution (if one exists).

[0140] These limitations make Newton's method challenging in real-world robot control scenarios that require high reliability and precision, especially when dealing with complex configuration spaces or dynamic environments, where Newton's method is not a good choice.

[0141] 4.2 Homotopy Continuation Method; In order to overcome the inherent defects of the Newton-Raphson method, especially its serious initial value dependence and lack of global convergence guarantee, this embodiment 1 innovatively proposes to use the Homotopy Continuation Method as the core numerical optimization framework for solving inverse kinematics, and deeply integrates it with the subsequent deep neural network.

[0142] The core idea of ​​homotopy continuation is to construct a continuous parameterized path (homopy path).

[0143] H(x,t)=0), connect a known easily solvable problem H(x,0)=G(x)=0 to the target difficult problem H(x,1)=F(x)=0 (i.e. our inverse kinematic equation F(θ)=X). d -f(θ) = 0, where X is the desired end pose and f(θ) is the positive kinematic function. The parameter t ∈ [0,1] is called the homotopy parameter. The solution steps are as follows:

[0144] A. Constructing the homotopy function: Choosing convex combinatorial homotopy as the homotopy function:

[0145] H(θ,t)=(1-t)*G(θ)+t*F(θ)=0 (14)

[0146] Where H(θ,t) is the constructed homotopy function; θ is the robot's joint angle vector; t is the homotopy parameter; G(θ) is the constructed simple auxiliary system, with the target point set as an end point X0 corresponding to a known configuration, i.e., G(θ) = f(θ) - X0 = 0, and its solution θ 0 Given that X0 is the known end-effector pose; f(θ) is the robot's positive kinematics function.

[0147] B. Tracing the homotopy path: From the known solution (θ) 0 Starting from a point F(θ,t) = 0, the path tracing algorithm uses a prediction-correction mechanism to track the trajectory of the path H(θ,t) = 0 as t changes from 0 to 1. The prediction step (e.g., tangent vector prediction) provides an approximate direction of the path, while the correction step (using a quasi-Newton method for local correction) pulls the point back onto the path. Thus, even if the objective problem F(θ) = 0 is inherently difficult to solve, a well-designed path tracing algorithm can usually guarantee tracing from a simple starting point to the objective solution (θ). * ,t=1).

[0148] C. Obtaining the target solution: When t successfully tracks to 1, the corresponding θ is the solution θ of the target inverse kinematics problem F(θ) = 0. * .

[0149] The above method not only fundamentally solves the core problems of Newton's method being extremely sensitive to initial conditions and lacking global convergence guarantees, but also bypasses singular points and is more stable than Newton's method in directly solving F(θ) = 0, thus providing a basis for subsequent selection of the optimal solution.

[0150] Preferably, in a preferred embodiment, the specific process of using a prediction-correction mechanism to track the homotopy path during the parameter change from 0 to 1 is as follows:

[0151] Let the current tracking point be (θ) k ,t k), where θ is the robot joint angle vector; t is the homotopy parameter t∈[0,1]; H(θ,t) is the constructed convex combinatorial homotopy function, connecting the known easily solvable problem G(θ)=0 and the target inverse kinematics problem F(θ=0;

[0152] During the prediction phase, a path tangent vector field is defined based on the local geometric characteristics of the current point to determine the direction of parameter evolution:

[0153]

[0154] in, The rate of change of the homotopy parameter t; θ is the rate of change of the joint angle vector θ; k t is the joint angle vector at the k-th iteration; k J is the homotopy parameter at the k-th iteration; H Let H be the Jacobian matrix of θ, reflecting the effect of joint angle changes on the homotopy function; The partial derivative of H with respect to t represents the trend of parameter variation;

[0155] Based on the initial joint angle θ0 predicted by the deep neural network, the next prediction point is generated using the Euler method:

[0156]

[0157] t k+1 =t k +Δt

[0158] in, The joint angle vector predicted in step k; t k+1 t is the homotopy parameter at the (k+1)th iteration; k Δt is the homotopy parameter at the k-th iteration; Δt is a fixed parameter step size to ensure a smooth transition from t=0 to t=1.

[0159] During the correction phase, starting from the predicted point, the deviation is corrected using constraint equations to ensure that the evolution always follows the homotopy path; constraint equations are constructed to force the homotopy function to be satisfied.

[0160] H(θ,t k+1 ) = 0

[0161] The correction point is solved using the Gauss-Newton method by utilizing the continuity of the robot's forward kinematics model f(θ):

[0162]

[0163] Repeat the above prediction-correction loop until t k+1 =1, at which point θ is obtained k+1 That is, to satisfy the desired end pose Xd Solution of joint angles.

[0164] 4.3 Efficient integration with Deep Neural Networks (DNNs); please refer to... Figure 5 To further improve the efficiency and accuracy of inverse kinematics solving, we combine deep neural networks (DNNs) to accelerate the solution process. By training a deep learning model and using the output of the previous layer to calculate the output of the next layer, the network can predict the corresponding output given the target position of the end effector, thereby shortening the computation time of traditional algorithms.

[0165] The input vector X of the above algorithm is the robot's current position xyz and the attitude quaternion xyzw. The output layer is the Lth layer, and the loss function is the computation error J(W,b,X,Y), where b is the bias vector.

[0166] By using DNNs to predict high-quality "homotopy origin" and "path initial value", the goal of training DNNs is no longer to directly predict joint angle θ, but rather to predict:

[0167] (a) A construction suggestion for one or more simple auxiliary systems G(θ) = 0 (e.g., predicting a point X close to the target point). d (The known easily reachable point X0).

[0168] (b) For a selected homotopy path, predict the θ value on the path at t=0 or some intermediate t time as a high-quality initial guess for the path tracing algorithm.

[0169] The accurate predictions provided by DNNs significantly reduce the number of prediction-correction steps required for path tracing, especially avoiding a large number of invalid iterations in regions far from the solution. This addresses the major drawback of homotopy methods, which typically have a higher computational cost than Newton's method, thus significantly accelerating path tracing. Simultaneously, DNNs can learn the robot's configuration space structure, helping to predict possible path bifurcation points or directly suggesting multiple potential homotopy starting points, assisting the algorithm in finding all or the optimal solution.

[0170] 4.4 Iterative Solution and Convergence; please refer to... Figure 6 In practical applications, the initial solution provided by deep neural networks The homotopy extension method will be further optimized to ensure the accuracy and convergence of the solution. By combining deep learning with numerical optimization methods, we can effectively avoid the multiple solutions problem and low computational efficiency of traditional inverse kinematics methods.

[0171] 4.5 Constraint Handling; To address the unsolvable situation of inverse kinematics, the method in Embodiment 1 incorporates kinematic constraints during the solution process to ensure that the target position of the robot's end effector remains within the robot's reachable workspace. The validity of the solution is guaranteed by dynamically adjusting the target position and attitude.

[0172] This embodiment 1 creatively solves the inherent problems of initial value sensitivity and lack of global convergence in the Newton-Raphson method for inverse kinematics solving by introducing the homotopy extension method. Combined with the intelligent prediction of the homotopy path starting point by a DNN, it retains the theoretical advantages of global convergence and strong robustness of the homotopy method while overcoming its computational bottleneck, achieving high-precision, high-robustness, and high-efficiency inverse kinematics solving. It is particularly suitable for robot applications with complex configurations, singularities, or high reliability requirements.

[0173] 4.6 Optimization and Iteration; Solving inverse kinematics is typically a complex nonlinear problem, especially in multi-degree-of-freedom robot systems. When faced with solutions involving multiple joint angles and complex target positions, direct solution methods may suffer from long computation times and low accuracy. Therefore, optimization and iterative methods are crucial for improving the efficiency and accuracy of kinematics solutions. Example 1 proposes an iterative optimization process based on an adaptive learning optimization algorithm, combining deep learning and numerical optimization techniques to ensure that the robot system can efficiently solve forward and inverse kinematics in complex environments.

[0174] 4.6.1 Adaptive Learning Optimization Algorithm; The core idea of ​​optimization iteration is to gradually adjust the angles of each joint of the robot according to the given objective, and to achieve the optimization by minimizing the objective function. In order to solve the problems of slow convergence speed and accuracy of solution in traditional methods, this embodiment 1 adopts an adaptive learning optimization algorithm.

[0175] Objective function: In each iteration, we calculate the error between the current target position and the actual position of the robot's end effector (i.e., the objective function). The optimization objective is to minimize this error, thereby obtaining the optimal joint angle.

[0176] E = ||P target -P current ‖ (15)

[0177] Among them, P target It is the position of the target end effector, P current The position of the end effector is calculated using the current joint angle, and E is the error value.

[0178] Adaptive learning rate: To ensure the optimization process converges within a reasonable time and avoids getting trapped in local optima, we introduce an adaptive learning rate σ. This learning rate is dynamically adjusted according to the error magnitude as iterations progress. When the error is large, the learning rate is large, thus accelerating convergence; when the error tends to converge, the learning rate is small, thus improving accuracy and avoiding over-adjustment.

[0179] The incremental optimization strategy involves breaking down the objective function into multiple smaller steps during optimization. Each iteration solves only a portion of the problem, gradually approaching the final solution through incremental optimization. This strategy not only improves computational efficiency but also avoids algorithm instability caused by excessive computational complexity.

[0180] 4.6.2 Combining Deep Learning with Optimization; The method in Example 1 not only relies on traditional numerical optimization techniques but also incorporates a deep learning model to enhance the stability and convergence speed of the optimization process. Specifically, the deep learning model can learn optimization patterns under different robot postures and environmental conditions based on previous training data, providing suitable initial solutions for each solution in practical applications, thereby accelerating the convergence process.

[0181] Network training: By training on a large amount of robot operation data, the deep neural network learns the kinematic characteristics of the robot's end effector and the relationships between the joint angles. During actual kinematic solving, the network outputs a set of initial joint angle solutions based on the current target position. This initial solution serves as the starting point for optimization iterations.

[0182] The optimization process incorporates a deep learning model; the initial joint angle solution provided by the deep learning model is further adjusted using a numerical optimization algorithm. Through iterative optimization, we can start from the initial solution of the deep learning network and further optimize the joint angles until the error converges. This process significantly reduces the multiple-solution problem in traditional inverse kinematics algorithms, ensuring that the optimization process converges to the optimal solution within a reasonable time.

[0183] 4.6.3 Introduction of Constraints; During the optimization iteration process, the robot joint angles are subject to certain physical and kinematic constraints, such as the maximum rotation angle of the joint, range of motion, and load limits. To ensure that the solution not only meets the requirements of the target position and orientation, but also conforms to the robot's working range and actual operation requirements, we introduce constraints into the optimization algorithm.

[0184] Constraint Handling Method: We adopted a soft constraint method, introducing a penalty term for the joint angle range into the objective function. In this way, during the iteration process, the robot control system not only optimizes the error but also considers the physical limitations of each joint, ensuring the robot can operate safely and stably in practical applications.

[0185] E total =E+λ·C constrains (16)

[0186] Among them, E total This represents the total optimization error, E is the basic error term, and C... constrains λ is the constraint penalty term, and λ is a parameter that controls the weight of the penalty term.

[0187] Kinematic feasibility check: After each iteration, the system checks whether the current joint angles meet kinematic feasibility requirements. If a joint angle is found to be outside the robot's maneuverability range, the optimization process will automatically adjust that joint angle to ensure the physical rationality of the entire kinematic solution.

[0188] 4.6.4 Convergence and Stability Guarantees; Convergence and stability are two key factors during the optimization iteration process. To ensure that the algorithm can converge stably in complex environments, we adopted the following strategies:

[0189] Stepwise convergence detection: After each optimization, we monitor the change in error. If the error change is very small (i.e., reaching the convergence threshold, such as a positional error of 0.05 mm), the iteration stops. This mechanism avoids over-iteration, saves computational resources, and improves optimization efficiency.

[0190] Enhanced robustness: With the assistance of deep learning models and the introduction of constraints, the optimization algorithm can maintain robustness in complex dynamic environments, and can still effectively converge to the correct solution even when faced with external disturbances or complex scenarios.

[0191] (5) Robot control output

[0192] Based on the joint angle data obtained from the inverse kinematics optimization solution, the data is first verified in real time to confirm that each joint angle conforms to the robot's kinematic constraints, avoiding exceeding joint limit angles or the risk of motion interference. After verification, the joint angle data is converted into digital signals that the robot control system can recognize. Combined with the motion characteristics of each joint (such as maximum speed and acceleration limits), control commands adapted to different joints are generated.

[0193] These control commands are transmitted to the robot's joint actuators via a communication module. The actuators adjust the motor output torque and speed according to the commands, driving each joint to rotate precisely at a set angle. During the movement, the robot's position sensors collect the actual angle information of each joint in real time and feed it back to the control system for comparison with the target angle. If there is a deviation, the control signals are adjusted in time to ensure the accuracy of joint movement.

[0194] Meanwhile, the pose information of the end effector is monitored in real time by a vision sensor or position encoder, continuously compared with the pose data of the desired trajectory, and the joint control commands are dynamically corrected to ensure that the end effector always moves along the desired trajectory. When the end effector completes all target pose actions on the trajectory, the control system sends a task completion signal, the joint actuators stop outputting, and the robot returns to the initial state or standby state, completing the corresponding task.

[0195] Example 2

[0196] Please refer to Figure 7 This embodiment 2 provides a robot forward and inverse kinematics optimization system based on depth camera hand-eye calibration, including:

[0197] The multi-source data acquisition unit is used to install a depth camera on the robot's end effector to simultaneously acquire RGB image visual information and LiDAR depth information of the environment;

[0198] The hand-eye calibration fusion unit is used to detect feature points of the calibration board based on the target detection method, combine depth information to obtain its three-dimensional coordinates, and then solve the hand-eye calibration transformation matrix to achieve the unification of multi-sensor coordinates.

[0199] The positive kinematics calculation unit is used to obtain the position and orientation of the end effector relative to the robot's base coordinate system by multiplying and superimposing the coordinate transformation matrices of each joint based on the robot's DH parameters.

[0200] The inverse kinematics solving unit is used to solve inverse kinematics using the homotopy extension method combined with a deep neural network (DNN). The DNN does not directly output the target joint angles, but instead predicts the initial joint angle values ​​at easily reachable points close to the desired end pose or on the homotopy path, providing a high-quality starting point for path tracing using the homotopy extension method. The homotopy extension method constructs a convex combined homotopy function, starting from the known solution of the auxiliary system, and uses a prediction-correction mechanism to track the homotopy path as the parameters change from 0 to 1. Finally, when the parameters reach 1, it obtains the solution to the target inverse kinematics problem, thus stably bypassing singularities and achieving global convergence. Simultaneously, kinematic constraints are introduced, and the validity of the solution is guaranteed through soft constraint methods.

[0201] The robot control output unit is used to generate control signals for each joint of the robot based on the joint angles obtained from inverse kinematics optimization, and drive the robot end effector to move along the desired trajectory to complete the corresponding task.

[0202] Example 3

[0203] This embodiment 3 also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement any step of a robot forward and inverse kinematics optimization method based on depth camera hand-eye calibration.

[0204] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0205] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0206] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robot forward and inverse kinematics optimization method based on depth camera hand-eye calibration, characterized in that, Comprise: S1. Install a depth camera on the robot end effector, synchronously collect the RGB image visual information and the laser radar depth information of the environment; S2. Detect the feature points of the calibration board based on a target detection method, obtain their three-dimensional coordinates in combination with the depth information, and then solve the hand-eye calibration conversion matrix to realize the unification of multiple sensor coordinates; S3. Based on the DH parameters of the robot, the coordinate transformation matrix of each joint is multiplied to obtain the position and attitude of the end effector relative to the robot base coordinate system; S4. Use homotopy continuation method combined with deep neural network DNN to solve inverse kinematics; DNN does not directly output the target joint angle, but predicts the easy-to-reach point or the initial value of the joint angle on the homotopy path close to the expected end pose, providing a high-quality starting point for path tracking of the homotopy continuation method; The homotopy continuation method constructs a convex homotopy function, starting from the known solution of the auxiliary system, and tracks the homotopy path in the process of parameter change from 0 to 1 through a prediction-correction mechanism, and finally obtains the solution of the target inverse kinematics problem when the parameter reaches 1, thereby stably bypassing the singular point to complete global convergence solution; At the same time, introduce kinematic constraints and ensure the legality of the solution through soft constraint method; S5. According to the joint angle obtained by inverse kinematics optimization, generate control signals for each joint of the robot to drive the robot end effector to move according to the expected trajectory and complete the corresponding task.

2. The robot forward and inverse kinematics optimization method based on depth camera hand-eye calibration according to claim 1, characterized in that, The RGB image in S1 is used to extract visual features, and the laser radar data is used to measure the distance between the object and the sensor.

3. The robot forward and inverse kinematics optimization method based on hand-eye calibration of depth camera according to claim 1, characterized in that, The calculation method of the hand-eye calibration conversion matrix in S2 is as follows: First, take a point P0 in the calibration board coordinate system as a reference, according to the known camera external parameter, i.e. conversion matrix T1, it can be converted to point P1 in the camera coordinate system; then according to the hand-eye calibration matrix to be solved, i.e. conversion matrix X, it can be converted to point P2 in the robot end coordinate system; then according to the known robot itself parameter, i.e. conversion matrix T3, it can be converted to point P3 in the robot base coordinate system; therefore the relationship is as follows: T3XT1P0=P3 Change the position of the robot arm, the coordinates of P0 and P3 remain unchanged, only the coordinates of P1 and P2 change, so the above relationship becomes: T3'XT1'P0=P3 Solving the hand-eye conversion matrix X can be obtained as follows: The above formula can be understood as the equation form of AX=XB, and the hand-eye conversion matrix X can be solved. T3' * T1 * T2 * T3' = T1 * T2 -1 T3' * T1 * T2 * T3' = T1 * T2 -1 , which gives T3' -1 T3X = XT1'T1 -1 The coordinate transformation matrix of each joint in S3 is constructed based on DH parameter method, which is used to describe the position and attitude transformation relationship between adjacent joint coordinate systems of the robot; The matrix is a 4x4 homogeneous transformation matrix, which integrates the rotation and translation information of the joint, and the specific form is as follows:

4. The robot forward and inverse kinematics optimization method based on hand-eye calibration of depth camera according to claim 1, characterized in that, The homotopy function in S4 is: where θ i is the rotation angle of the i-th joint; α i is the twist angle of the i-th link; a i is the length of the i-th link; d i is the offset of the i-th joint.

5. The robot forward and inverse kinematics optimization method based on hand-eye calibration of depth camera according to claim 1, characterized in that, H(theta, t) = (1-t)*G(theta) + t*F(theta) = 0 The specific process of tracking the homotopy path of the parameter from 0 to 1 in the prediction-correction mechanism in S4 is as follows: where H(θ, t) is the constructed homotopy function; θ is the joint angle vector of the robot; t is the homotopy parameter; G(θ) is the constructed simple auxiliary system, the target point is set as the end point X0 corresponding to a known configuration, i.e. G(θ) = f(θ) - X0 = 0, and the solution θ 0 It is known that X0 is a known end pose; and f(θ) is the forward kinematics function of the robot.

6. The robot forward and inverse kinematics optimization method based on depth camera hand-eye calibration according to claim 1, characterized in that, In the prediction stage, the path tangent vector field is defined based on the local geometric characteristics of the current point to determine the parameter evolution direction: Let the current tracking point be (θ k ,t k ), where θ is the robot joint angle vector; t is the homotopy parameter t ∈ [0, 1]; H(θ, t) is the constructed convex homotopy function connecting the known easy problem G(θ) = 0 and the target inverse kinematics problem F(θ) = 0; ​ wherein, is the rate of change of homotopy parameter t; is the rate of change of joint angle vector θ; θ k is the joint angle vector at the kth iteration; t k is the homotopy parameter at the kth iteration; J H is the Jacobian matrix of H with respect to θ, reflecting the influence of joint angle change on the homotopy function; is the partial derivative of H with respect to t, representing the parameter change trend; The initial joint angle θ0 predicted by the deep neural network is combined to generate the next predicted point by Euler method: t k+1 = t k + Δt wherein, is the joint angle vector predicted for the kth step; t k+1 is the homotopy parameter at the k+1th iteration; t k is the homotopy parameter at the kth iteration; Δt is a fixed parameter step size that ensures a smooth transition from t = 0 to t = 1; In the correction stage, the predicted point is taken as the starting point, and the deviation is corrected by the constraint equation to ensure that the homotopy path is always followed; H(θ, t k+1 ) = 0 The correction point is solved by Gauss-Newton method using the continuity of the robot forward kinematics model f(θ): The above prediction-correction loop is repeated until t k+1 = 1, the resulting θ k+1 is the joint angle solution that satisfies the desired end pose X d .

7. The robot forward and inverse kinematics optimization method based on hand-eye calibration of depth camera according to claim 1, characterized in that, The kinematic constraint in S4 is specifically: A soft constraint method is adopted, and a penalty term for the range of joint angles is introduced into the objective function; in this way, in the iteration process, the robot control system not only optimizes the error, but also considers the physical limitations of each joint, ensuring that the robot can run safely and stably in actual application; E total = E + λ · C constrains where E total is the total optimization error, E is the basic error term, C constrains is the constraint condition penalty term, and λ is a parameter controlling the weight of the penalty term.

8. The robot forward and inverse kinematics optimization method based on hand-eye calibration of depth camera according to claim 7, characterized in that, The objective function is specifically: In each iteration, the error between the current target position and the actual position of the robot end effector is calculated; the optimization goal is to minimize this error to obtain the optimal joint angle; E = ||P target - P current ‖ where P target is the position of the target end effector, P current is the position of the end effector calculated by the current joint angles, and E is an error value.

9. A robot forward / inverse kinematics optimization system based on depth camera hand-eye calibration, characterized in that, It comprises: A multi-source data acquisition unit is used to install a depth camera on the robot end effector and synchronously acquire the RGB image visual information and laser radar depth information of the environment; A hand-eye calibration fusion unit is used to detect the feature points of the calibration board based on a target detection method, obtain their three-dimensional coordinates combined with depth information, and then solve the hand-eye calibration transformation matrix to realize the unification of multi-sensor coordinates; A forward kinematics calculation unit is used to obtain the position and attitude of the end effector relative to the robot base coordinate system by multiplying the coordinate transformation matrices of each joint based on the DH parameters of the robot; An inverse kinematics solving unit is used to solve the inverse kinematics by homotopy continuation method combined with deep neural network DNN; DNN does not directly output the target joint angle, but predicts the easy-to-reach point close to the expected end pose or the initial value of the joint angle on the homotopy path, providing a high-quality starting point for path tracking of the homotopy continuation method; the homotopy continuation method constructs a convex homotopy function from the known solution of the auxiliary system, and tracks the homotopy path in the process of parameter change from 0 to 1 by the prediction-correction mechanism, and finally obtains the solution of the target inverse kinematics problem when the parameter reaches 1, thereby stably bypassing the singular point to complete global convergence solution; Kinematic constraints are also introduced and the legality of the solution is ensured by a soft constraint method; A robot control output unit is used to generate control signals for each joint of the robot according to the joint angles obtained by inverse kinematics optimization, drive the robot end effector to move according to the expected trajectory, and complete the corresponding task.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to perform the robot forward and inverse kinematics optimization method based on the depth camera hand-eye calibration of any one of claims 1-8.

Images