A robot kinematics parameter correction method and system based on reinforcement learning

Abstract

The application relates to a robot kinematic parameter correction method based on reinforcement learning, which comprises the following steps: establishing a Q-learning model; randomly performing any action of modifying kinematic parameters with the probability of epsilon, performing the action of maximizing the Q table in the current state with the probability of 1-epsilon, and modifying the corresponding kinematic parameters; controlling the robot to perform RCM motion; photographing the end of the executing element of the robot by using two photographing devices with two photographing directions perpendicular to each other, and identifying the coordinates; calculating a new state S' and a reward r obtained by taking the action; updating the Q table in the new state S'; judging whether the training times meet the requirements, if not, performing training again; if yes, performing the next step; if the average positioning accuracy of the end of the executing element meets the threshold requirement, obtaining the optimal parameters and saving the output Q table. The method does not need to use other measuring equipment to measure error values, reduces the equipment cost, and does not need to transform and calibrate the robot base to the end executing device, so that the calculation process is simplified.

Description

Technical Field

[0001] This invention relates to the field of robot control, and more specifically, to a method and system for correcting robot kinematic parameters based on reinforcement learning. Background Technology

[0002] Ophthalmic surgery typically requires highly precise manipulation to address visual impairments and other eye conditions such as cataracts, myopia, and hyperopia. Traditional ophthalmic surgery often relies on the surgeon's manual manipulation, which can be affected by factors such as minute hand tremors.

[0003] In recent years, ophthalmic surgical robotics has become an important area for addressing these challenges. Ophthalmic surgical robotic systems offer high-precision manipulation, reducing surgical risks and improving success rates. However, ophthalmic surgical robots are often affected by manufacturing and assembly errors during the production process, causing the actual parameters of the robotic arm to deviate from their nominal values. These parameter errors can lead to positioning and manipulation errors during surgery, thereby reducing the accuracy and success rate of the procedure.

[0004] Existing technologies employ different parameter calibration schemes for different surgical robots. Some surgical robots have specially designed hardware devices for parameter calibration; others utilize self-developed algorithms to correct errors using low-cost measurement methods. However, for ophthalmic surgical robots, current calibration primarily relies on continuous, minor adjustments to parameters based on human experience to meet accuracy requirements.

[0005] Regarding the methods mentioned above, manual adjustment based on experience does not rely on sensor equipment, but it is time-consuming and requires professional calibration personnel; parameter calibration systems using laser trackers to measure position are expensive and require regular calibration; and camera device calibration introduces new errors, which are superimposed on forward and inverse kinematics calculations, increasing the complexity of the error model and reducing the accuracy of kinematic parameter calibration; training neural network models to fit the robot's kinematic parameters requires data collection, cleaning, model training, and testing, which is a large workload. Summary of the Invention

[0006] To overcome the problems of expensive measurement equipment, the need for repeatable calibration, and the complexity of calculation in the existing kinematic parameter calibration, this invention provides a reinforcement learning-based method for correcting robot kinematic parameters. This method not only reduces costs and simplifies calculations, but also corrects robot kinematic parameters.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for correcting robot kinematic parameters based on reinforcement learning, comprising the following steps: Step 1: Select the kinematic parameters to be modified, their initial values, training iterations T, initial state S0, and reward function, and establish a Q-learning model; Step 2: Randomly execute any action that modifies the kinematic parameters with probability ε, and execute the action that makes the Q table take the maximum value in the current state with probability 1-ε, modifying the current value of the corresponding kinematic parameters; the current value is the value of the kinematic parameters at the current moment when Step 2 is executed. If it is the first time Step 2 is executed, the current value is the initial value.

[0008] Step 3: Control the robot to perform RCM movements; Remote Center of Motion (RCM): This is a key point used in robotic surgery to maintain the stability of surgical instruments. During surgery, to prevent the patient's incision from expanding further, the robot is only allowed to rotate or translate around this point.

[0009] Step 4: Use two cameras with perpendicular shooting directions to take pictures of the robot's actuator end effector and identify and obtain the coordinates of the actuator end effector; calculate the new state S' based on the robot's coordinates before and after performing the RCM motion and calculate the reward r obtained by taking this action based on the reward function; Step 5: In the new state S', assume that action a' is performed to maximize the Q-table, and update the Q-table using the Markov property; Step Six: Determine if the number of training iterations has reached T. If not, repeat steps Two through Five; if yes, proceed to step Seven. Step 7: Compare the current values ​​of the kinematic parameters with the manually corrected parameters. If the average positioning accuracy of the actuator end meets the threshold requirement, obtain the optimal parameters and save the output Q table.

[0010] In the above technical solution, the end effector of the robot is used as a fixed RCM point, and the images captured by two cameras are used as the basis for judging the RCM point offset. Then reinforcement learning is performed. At the same time, according to the greedy algorithm of reinforcement learning, the robot is put in different states and performs corresponding kinematic parameter modification actions according to ε until the judgment result of the RCM point offset meets the threshold requirement, that is, the end effector error is less than the preset threshold. At this time, the optimal kinematic parameters of the robot are obtained, and can be adjusted according to the Q table in the future.

[0011] Preferably, the Q-learning model is as follows: State Space: Before the robot performs RCM motion, the image coordinates read by the two cameras are (x01, y01) and (x02, y02). After the robot performs RCM motion, the images are re-captured and the end effector of the actuator is identified. Under the images captured by the two cameras, the end effector coordinates of the actuator are (xnew1, ynew1) and (xnew2, ynew2) respectively. The difference between the image coordinates x and y before and after RCM motion is calculated. The result can be increased, decreased, or unchanged. The difference of the four parameters can produce 3*3*3*3=81 states. Therefore, State=0~80, a total of eighty-one states. Action Space: Sets the correction precision value and modification method for kinematic parameters. The modification method is to increase or decrease the correction precision value based on the current value of the kinematic parameter. Each kinematic parameter is set as an action in the action space, whether it is increased, decreased, or not taken. Reward function: The smallest RCM offset during training is denoted as dis_best. A reward is given when the error RCM_err during the next training iteration is less than dis_best. Where k is adjusted according to RCM_err in the training environment, and k takes a fixed value; when RCM_err is greater than dis_best but smaller than the error of the previous training, a reward is given. A reward is given if the error is larger than the previous training result. ; Preferably, the RCM offset is as follows: In each step of training, the number of RCM movements is set, and Euclidean distance is used as the criterion for RCM offset. The average offset of all RCM movements is taken as the offset for this training session. The RCM offset is defined as: .

[0012] Preferably, the specific value of ε is:

[0013] In the formula, i = 0, 1, 2...T, and T is the number of training iterations.

[0014] Preferably, in step five, the Q table is updated according to the following formula:

[0015] In the formula, Preferably, in step one, the specific process for selecting the kinematic parameters to be modified is as follows: Construct an expression in the base coordinate system of the robot's end effector; the kinematic parameters appearing in the expression are the kinematic parameters to be changed; manually tune the kinematic parameters to be changed and check the tuning results; select the kinematic parameters to be modified based on the tuning results. This process determines the kinematic parameters that most need correction and that, after correction, will best improve the robot's RCM point offset, thereby reducing the number of kinematic parameters that need correction and lowering the computational load.

[0016] Preferably, in step two, if the error between the value of the kinematic parameter after the action modification and the initial value of the corresponding kinematic parameter is greater than 5%, the action modification is reversed with a certain probability to prevent overfitting.

[0017] The function corresponding to a certain probability:

[0018] Let q be a positive number, and let its value satisfy the probability f( The range of is [0, 1]. In the above formula, These are the modified values ​​of the kinematic parameters; These are the initial values ​​for the kinematic parameters.

[0019] Preferably, in step four, Harris corner detection is used to obtain the coordinates of the end of the actuator.

[0020] Preferably, in step five, the end effector of the robot is initialized to the center of the images from the two cameras to avoid the end effector shifting out of the image area due to repeated training.

[0021] A system for correcting robot kinematic parameters, for performing the above-described reinforcement learning-based robot kinematic parameter correction method; comprising a robot, two mutually perpendicular cameras for capturing images of the robot's end effector, and controllers electrically connected to the robot and the cameras respectively.

[0022] A computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described reinforcement learning-based method for correcting robot kinematic parameters.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows: This method uses a camera device to acquire the end image of the actuator and judges the RCM point offset through the image, eliminating the need to use other measuring equipment to measure error values ​​and reducing equipment costs; the end of the actuator is used as a fixed RCM point to constrain it, and reinforcement learning is combined to realize the elimination of coordinate system transformation from robot base to end effector, calibration of camera intrinsic and extrinsic parameters, hand-eye calibration and other processes, simplifying the calculation process; by reducing parameter calibration time and labor costs and reducing the repetition of parameter calibration, the operational threshold of parameter calibration is lowered. Attached Figure Description

[0024] Figure 1 This is a flowchart of a robot kinematics parameter correction method based on reinforcement learning according to the present invention; Figure 2 This is a schematic diagram of a robot kinematic parameter correction system based on reinforcement learning according to the present invention. Detailed Implementation

[0025] The accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. To better illustrate this embodiment, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings. The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0026] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "long," and "short" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0027] The technical solution of the present invention will be further described in detail below through specific embodiments and in conjunction with the accompanying drawings: Example 1 like Figure 1 The following is an embodiment of a robot kinematics parameter correction method based on reinforcement learning, which includes the following steps: Step 1: Set the initial kinematic parameters and their initial values, the number of training iterations T, the initial state S0, and the reward function to establish the Q-learning model; The kinematic parameters that have the greatest impact on the RCM point offset are selected as the kinematic parameters that need to be modified. The specific process for selecting the kinematic parameters to be modified is as follows: Construct an expression in the base coordinate system of the robot's end effector; the kinematic parameters appearing in the expression are the kinematic parameters that need to be changed; manually tune the kinematic parameters that need to be changed and check the tuning results; select the kinematic parameters to be modified based on the tuning results. Determine the kinematic parameters that most need to be corrected and that will best improve the robot's RCM point offset after correction, thereby reducing the number of kinematic parameters that need to be corrected and reducing the amount of computation.

[0028] In this embodiment, taking the robot employing a Parallel Coupled Joint Mechanism (PCJM) as an example, the PCJM transforms the differential displacements of two translational motions into one translational and one rotational motion. The parameters are as follows: point A is the origin; L1 and L2 are the linear displacements of the end-effector joints of the upper and lower motors; the vertical distance between the two parallel motors is dm; the length of the end effector is h; the length from the midpoint of the actuator to the end of the actuator element is Ltool; L and θ are the linear displacement and rotation angles of the entire PCJM, satisfying the following: (1) Based on forward kinematics analysis, the variation relationship of the PCJM from the base point A to the end point D can be obtained as follows: (2) (3) Write X D Y D The partial derivatives of θ with respect to L1, L2, and L5, i.e., the Jacobian matrix J1, are: (4) The serial-parallel hybrid robot connects two PCJMs directly in series after rotating them end-to-end via mechanical components, and adds a tool holder at the end. Let PCJM1 be closer to the base and PCJM2 be closer to the end. The translational displacement and rotation angle of the first PCJM are La, respectively. a, where La corresponds to the displacements of the two parallel motors, L1 and L2; the translational displacement and rotation angle of the second PCJM are Lb and L2, respectively. b, Lb corresponds to the displacements L3 and L4 of the two parallel motors. A vertically moving motor with a displacement of L5 is added to the tool holder. In the entire serial-parallel robot configuration, Ltool and L5 of the first PCJM are 0; the second PCJM is rotated 90° around the y-axis and then mounted at the end of the first PCJM. Based on the forward kinematics derivation, the expression for the robot end effector P(PX,PY,PZ) in the base coordinate system can be obtained as follows: (5) As can be seen from equation (5), when the end point is taken as the fixed point of the RCM, the errors of each kinematic parameter will cause it to shift, and the RCM point cannot be guaranteed to be fixed. Therefore, the kinematic parameters need to be corrected. The five kinematic parameters that need to be changed are Ltool, h1, h2, dm, and dm2. Their physical meanings are as follows: Ltool: The distance from the end of the actuator to the midline between the axes of the two parallel moving joints L3 and L4 of PCJM2. In this embodiment, the actuator is a scalpel, needle, or forceps, and the end of the actuator is the tip of a scalpel, needle, or forceps.

[0029] h1: The distance from the axis of the PCJM2 end effector to the midpoint of the ends of the two parallel traverse joints L3 and L4.

[0030] h2: The distance from the axis of the PCJM1 end effector to the midpoint of the ends of the two parallel traverse joints L1 and L2.

[0031] dm: The distance between the axes of the two parallel translating joints L3 and L4.

[0032] dm2: The distance between the axes of the two parallel translation joints L1 and L2.

[0033] Based on the manual adjustment of the kinematic parameters that need to be changed and the results of the adjustment, the kinematic parameters that need to be modified are Ltool, h1, and h2.

[0034] The Q-learning model is as follows: State Space: Before the robot performs RCM motion, the image coordinates read by the two cameras are (x01, y01) and (x02, y02). After the robot performs RCM motion, the images are re-captured and the end effector of the actuator is identified. Under the images captured by the two cameras, the end effector coordinates of the actuator are (xnew1, ynew1) and (xnew2, ynew2) respectively. The difference between the image coordinates x and y before and after RCM motion is calculated. The result can be increased, decreased, or unchanged. The difference of the four parameters can produce 3*3*3*3=81 states. Therefore, State=0~80, a total of eighty-one states. Action Space: This space sets the correction precision and modification method for kinematic parameters. The modification method involves increasing or decreasing the correction precision. Increasing, decreasing, or not taking any action for each kinematic parameter is considered an action in the action space. The correction precision can have two different values. In this embodiment, the correction precision of Ltool is 0.5 mm and 1 mm, and the precisions of h1 and h2 are 0.1 mm and 0.3 mm, respectively. Each parameter can be increased or decreased. Modification methods include increasing or decreasing the current value of Ltool by 0.5 / 1 mm; increasing or decreasing the current values ​​of h1 and h2 by 0.1 / 0.3 mm; and not taking any action. Therefore, the action space has a total of 3*2*2+1=13 actions.

[0035] Reward function: The smallest RCM offset during training is denoted as dis_best. A reward is given when the error RCM_err during the next training iteration is less than dis_best. Where k is adjusted according to RCM_err in the training environment, and k takes a fixed value; when RCM_err is greater than dis_best but smaller than the error of the previous training, a reward is given. A reward is given if the error is larger than the previous training result. ; RCM offset: In each step of training, the number of RCM movements is set. In this example, it is 10 times. Euclidean distance is used as the criterion for RCM offset. The average offset of the 10 RCM movements is taken as the offset for this training. The RCM offset is defined as: .

[0036] Step 2: Randomly execute any action that modifies the kinematic parameters with probability ε, and execute the action that maximizes the Q-table in the current state with probability 1-ε, modifying the current value of the corresponding kinematic parameter; if the error between the value of the kinematic parameter after the action modification and the initial value of the corresponding kinematic parameter is greater than 5%, execute the action modification in reverse with a certain probability to prevent overfitting.

[0037] The function corresponding to a certain probability:

[0038] q is positive, and its value makes the probability f( The range of is [0, 1]. In the above formula, These are the modified values ​​of the kinematic parameters; These are the initial values ​​for the kinematic parameters.

[0039] The specific value of ε is:

[0040] In the formula, i = 0, 1, 2...T, and T is the number of training iterations.

[0041] Step 3: Control the robot to perform RCM motion; Step 4: Take photos of the robot's actuator end effector using two cameras with perpendicular shooting directions, and obtain the coordinates of the actuator end effector using Harris corner detection; calculate the new state S' based on the robot's coordinates before and after performing RCM motion, and calculate the reward r obtained by taking this action based on the reward function; Step 5: In the new state S', assume that the action a' that makes the Q table take the maximum value is executed, and the Q table is updated using the Markov property. The end effector of the robot is initialized to the center of the images of the two camera devices to avoid the end effector shifting out of the image area due to multiple trainings. In this embodiment, the Q table is updated according to the following formula:

[0042] In the formula, s is the robot's current state, a is the action taken, α and γ are decay factors, r is the reward given by the environment after the robot takes the action, s' is the state at the next moment, and a' is the action assumed to be taken under s'.

[0043] Step Six: Determine if the number of training iterations has reached T. If not, repeat steps Two through Five; if yes, proceed to step Seven. Step 7: Compare the current values ​​of the kinematic parameters with the manually corrected parameters. If the average positioning accuracy of the actuator end meets the threshold requirement, obtain the optimal parameters and save the output Q table.

[0044] In this embodiment, the end effector of the robot's actuator is used as a fixed RCM point. Images captured by two cameras are used as the basis for judging the RCM point offset. Reinforcement learning is then performed. At the same time, according to the greedy algorithm of reinforcement learning, the robot is placed in different states and performs corresponding kinematic parameter modification actions according to ε until the final judgment result of the RCM point offset meets the threshold requirement, that is, the end effector error is less than the preset threshold. At this time, the optimal parameters of the robot's kinematic parameters are obtained, and they can be adjusted according to the Q table in the future.

[0045] The beneficial effects of this embodiment are as follows: This method uses a camera device to acquire the end image of the actuator and determines the RCM point offset through the image, eliminating the need for other measuring devices to measure error values ​​and reducing equipment costs; by using the end of the actuator as a fixed RCM point to constrain it, and by combining reinforcement learning, it eliminates the need for coordinate system transformation from robot base to end effector, calibration of camera intrinsic and extrinsic parameters, hand-eye calibration, and other processes, simplifying the calculation process; by reducing parameter calibration time and labor costs and reducing the repetitiveness of parameter calibration, the operational threshold for parameter calibration is lowered.

[0046] Example 2 An embodiment of a system for correcting robot kinematic parameters, used to execute the reinforcement learning-based robot kinematic parameter correction method of Embodiment 1, such as... Figure 2 As shown, the system includes a robot 1, a camera device for capturing images of the robot's end effector and the cameras being perpendicular to each other, namely a first industrial camera 2 and a second industrial camera 3, and a controller that is electrically connected to the robot 1, the first industrial camera 2, and the second industrial camera 3.

[0047] Example 3 A computer-readable storage medium for storing a computer program that, when executed by a processor, implements the reinforcement learning-based robot kinematic parameter correction method of Embodiment 1.

[0048] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for correcting robot kinematic parameters based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Select the kinematic parameters to be modified, their initial values, training iterations T, initial state S0, and reward function, and establish a Q-learning model; Step 2: Randomly execute any action that modifies the kinematic parameters with probability ε, and execute the action that makes the Q table take the maximum value in the current state with probability 1-ε, modifying the current value of the corresponding kinematic parameters; Step 3: Control the robot to perform RCM motion; Step 4: Use two cameras with perpendicular shooting directions to take pictures of the robot's actuator end effector and identify and obtain the coordinates of the actuator end effector; calculate the new state S' based on the robot's coordinates before and after performing RCM motion and calculate the reward r obtained by taking this action based on the reward function; Step 5: In the new state S', assume that action a' is performed to maximize the Q-table, and update the Q-table using the Markov property; Step Six: Determine if the number of training iterations has reached T. If not, repeat steps Two through Five; if yes, proceed to step Seven. Step 7: Compare the current values ​​of the kinematic parameters with the manually corrected parameters. If the average positioning accuracy of the actuator end meets the threshold requirement, obtain the optimal parameters and save the output Q table.

2. The method for correcting robot kinematic parameters based on reinforcement learning according to claim 1, characterized in that, The Q-learning model is as follows: State Space: Before the robot performs RCM motion, the image coordinates read by the two cameras are (x01, y01) and (x02, y02). After the robot performs RCM motion, the images are re-captured and the end effector of the actuator is identified. Under the images captured by the two cameras, the end effector coordinates of the actuator are (xnew1, ynew1) and (xnew2, ynew2) respectively. The difference between the image coordinates x and y before and after RCM motion is calculated. The result can be increased, decreased, or unchanged. The difference of the four parameters can produce 3*3*3*3=81 states. Therefore, State=0~80, a total of eighty-one states. Action Space: Sets the correction precision value and modification method for kinematic parameters. The modification method is to increase or decrease the correction precision value based on the current value of the kinematic parameter. Each motion parameter, whether increased, decreased, or unchanged, is set as an action in the action space. Reward function: The smallest RCM offset during training is denoted as dis_best. A reward is given when the error RCM_err during the next training iteration is less than dis_best. Where k is adjusted according to RCM_err in the training environment, and k takes a fixed value; when RCM_err is greater than dis_best but smaller than the error of the previous training, a reward is given. A reward is given if the error is larger than the previous training result. .

3. The method for correcting robot kinematic parameters based on reinforcement learning according to claim 2, characterized in that, RCM offset: In each step of training, the number of RCM movements is set. Euclidean distance is used as the criterion for RCM offset. The average offset of all RCM movements is taken as the offset for this training session. The RCM offset is defined as: 。 4. The method for correcting robot kinematic parameters based on reinforcement learning according to claim 3, characterized in that, The specific value of ε is: In the formula, i = 0, 1, 2...T, and T is the number of training iterations.

5. The method for correcting robot kinematic parameters based on reinforcement learning according to claim 3, characterized in that, In step five, the Q table is updated according to the following formula: In the formula, s is the robot's current state, a is the action taken, α and γ are decay factors, r is the reward given by the environment after the robot takes the action, s' is the state at the next moment, and a' is the action assumed to be taken under s'.

6. The method for correcting robot kinematic parameters based on reinforcement learning according to claim 1, characterized in that, In step one, the kinematic parameters to be modified and their initial values ​​are selected. The specific process is as follows: construct an expression in the base coordinate system of the robot's end effector, and the kinematic parameters appearing in the expression are the kinematic parameters to be changed; manually adjust the kinematic parameters to be changed and check the adjustment results, and select the kinematic parameters to be modified based on the adjustment results.

7. The method for correcting robot kinematic parameters based on reinforcement learning according to claim 1, characterized in that, In step two, if the error between the modified kinematic parameter value and the initial kinematic parameter value is greater than 5%, the modified kinematic parameter is executed in reverse with a certain probability.

8. The method for correcting robot kinematic parameters based on reinforcement learning according to claim 1, characterized in that, In step four, Harris corner detection is used to obtain the coordinates of the end of the actuator.

9. The method for correcting robot kinematic parameters based on reinforcement learning according to claim 1, characterized in that, In step five, the robot's actuator end is initialized to the center of the images from the two cameras.

10. A system for correcting the kinematic parameters of a robot, characterized in that, Used to perform the robot kinematic parameter correction method based on reinforcement learning as described in any one of claims 1-9; It includes a robot, two mutually perpendicular camera devices for filming the robot's end effector, and a controller electrically connected to the robot and the camera devices respectively.

Images