A reinforcement learning system and method for bone grinding based on proximal policy optimization
By using a bone resection reinforcement learning system optimized by proximal strategy, and by designing reward and penalty functions using the PPO algorithm and PID control, the system solves the problem of high-precision and high-efficiency control of robotic bone resection in unstructured environments, and achieves rapid convergence and safe operation of robotic bone resection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING JISHUITAN HOSPITAL
- Filing Date
- 2025-02-12
- Publication Date
- 2026-07-17
Smart Images
Figure CN119871420B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot control technology, and in particular relates to a bone grinding reinforcement learning system and method based on proximal strategy optimization. Background Technology
[0002] Robotic bone resection is a delicate maneuver performed in an unstructured environment with abundant contact. Supervised learning-based intelligent robot control faces data limitations. Robots interact frequently with the environment and have diverse contact characteristics. To ensure safe and smooth interaction with the environment, more precise and smooth control of the robot is required, addressing the issue of unstable contact between the robot's end effector and unstructured bone tissue.
[0003] In the field of robot control, the development and application of motion control strategies is an important and popular issue. Robotic laminectomy, which utilizes image-guided navigation in conjunction with the robot, faces challenges due to image registration errors, robot registration errors, and robot system errors. Under fixed-speed position control, it is difficult to meet the high precision and efficiency requirements of cutting. Furthermore, this solution treats the cutting environment as a static environment or the cutting surface as a relatively flat surface. The unstructured environment faced by laminectomy robots includes irregular movements during breathing, bone surface cavities, and deformation caused by forces on the robot's end effector. The robot needs to perform flexible motion control in real time based on the working environment. However, traditional robot control has the following drawbacks: current laminectomy robots use high-power swash-cutting, a strategy that requires sensitive environmental parameters and lacks real-time flexibility. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention proposes a reinforcement learning system and method for bone grinding based on proximal policy optimization. Building upon the Markov property of the robot's bone resection control process, the PPO algorithm is used to train the grinding robot's controller. Experience playback and dense reward functions guide the robot to obtain positive rewards. Parallel computation and disabling graphical displays accelerate the training process. Operator demonstration data and PID control data enable the robot to quickly converge to the expected results. By coordinating the robot's own reinforcement learning strategy and combining it with prior human knowledge or PID control algorithms for guidance, ineffective exploratory behavior is avoided, achieving an efficient learning process.
[0005] To achieve the above objectives, this invention provides a bone grinding reinforcement learning system based on proximal strategy optimization, comprising: a bone grinding force control module and a factor analysis module;
[0006] The bone grinding force control module is used to accelerate robot training by collecting human control experience data or admittance control data, and to train the robot by changing individual factors separately.
[0007] The factor analysis module is used to perform factor variable analysis on the factors transformed by the bone grinding force control module. Specifically, it is used to analyze the factors affecting the bone cutting effect of the robot, obtain the analysis results, and statistically determine the optimal parameter values based on the analysis results.
[0008] Optionally, the bone grinding force control module includes a contact force control submodule, a bone grinding action function submodule, and an action decision submodule;
[0009] The contact force control submodule is used to solve high-dimensional or continuous motion spaces;
[0010] The bone grinding action function submodule is used to design the reward function and the penalty function;
[0011] The action decision submodule is used to adjust the learning rate, optimize the algorithm transformation, introduce an experience replay mechanism, adjust the network structure, pre-train the policy function and value function, carefully design the reward function, and implement parallel computing.
[0012] Optionally, the factor analysis module includes a force sensor, a bone model, an RL model, and a robot;
[0013] The bone model is connected to the force sensor, the sensor is connected to the RL model, the RL model is connected to the robot, and the robot is connected to the bone model.
[0014] The bone model is subjected to pressure contact between the grinding tool and the bone tissue, and the grinding force generated by the extrusion is used to remove the extruded bone tissue.
[0015] The force sensor calibrates the robot, and the RL model establishes the contact force control relationship between the robot's end effector position and bone tissue.
[0016] On the other hand, to achieve the above objectives, the present invention also provides a bone grinding reinforcement learning method based on proximal strategy optimization, comprising:
[0017] Learn from the human process of trial and error to learn the relationship between the end-cutting force state and robot actions, and solve the robot force control Markov decision process problem;
[0018] Optimize contact force control through a proximal strategy;
[0019] Design the reward and penalty functions for the bone grinding action;
[0020] Accelerate the training of robot action decision-making strategies;
[0021] The factors affecting the bone cutting effect of the robot are analyzed.
[0022] Optionally, problems involving robot force-controlled Markov decision processes include:
[0023] Find a policy π(at|st) that enables the robot agent to take an action in the current state and obtain the maximum reward. Discretize the actions and states, and use MDP to transform it into MRP to solve the policy from time 0 to time t.
[0024] τ={s0,a0,s1,a1,s2,a2,...s t-1 .a t-1 ,s t ,a t};
[0025] Once the policy parameter θ is determined, the probability of trajectory τ occurring is:
[0026]
[0027] The robot receives a reward and expected value during one round of interaction with its environment:
[0028]
[0029] Reinforcement learning algorithms require adjusting the actor parameters θ to maximize the reward value R(τ). The policy gradient is calculated by solving for the gradient with respect to the expected value. Perform gradient calculation and update the parameter θ using gradient ascent to maximize the reward value, as shown below:
[0030]
[0031] in, R(τ) represents the difference between the expected values and the reward values. Let θ be the expected value, θ be the actor parameter to be adjusted, and μ be the learning rate.
[0032] Optionally, optimizing contact force control through proximal strategies includes:
[0033] Maximizing the reward R(τ) through gradient ascent can be achieved by transforming it into calculating the policy gradient of the expected reward:
[0034]
[0035] in, R(τ) represents the difference between the expected values and the reward values. Let θ be the expected value, θ be the actor parameter, and τ be the trajectory.
[0036] Optionally, the reward and penalty functions for designing the bone grinding action include:
[0037] Rewards for moving forward r f The reward within time step t is calculated as follows:
[0038] r f =k f (p r -p c );
[0039] Where, p r To control the real-time position of the robot's end effector using contact force; p c k represents the initial position of the robot's end effector. f r is the reward coefficient for the robot's forward movement. f This is used to measure the forward motion of a lateral robot.
[0040] Optionally, accelerating the training of the robot's action decision-making strategy includes:
[0041] The accuracy of the data is further improved by using the PID algorithm. The PID controller with tuned parameters is added to the network, and the collected state is sent to the PID controller to obtain the output. The obtained output is compared with the previous data distribution to obtain the similarity parameter, thus completing the accelerated training of the robot action decision strategy.
[0042] Technical effects of this invention: This invention discloses a reinforcement learning system and method for bone grinding based on proximal policy optimization. Building upon the Markov property of the robot's bone resection control process, the PPO algorithm is used to train the grinding robot's controller. The trained PPO algorithm directly calculates the normal compensation displacement based on the robot's real-time force state during grinding, thus simplifying the robot's grinding force control process. Various methods are used to improve the training process and accelerate the robot's convergence speed, including experience playback, designing dense reward functions, parallel computing, and pre-training using the operator's prior experience and PID control algorithms. Experience playback and dense reward functions guide the robot to obtain positive rewards; parallel computing and disabling graphical displays help accelerate training; utilizing the operator's demonstration data and PID control data enables the robot to quickly converge to the expected effect; by coordinating the robot's own reinforcement learning strategy and combining it with guidance from human prior knowledge or PID control algorithms, ineffective exploration behavior is avoided, achieving an efficient learning process. Attached Figure Description
[0043] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0044] Figure 1 This is a schematic diagram of the structure of a bone grinding reinforcement learning system based on proximal strategy optimization according to an embodiment of the present invention;
[0045] Figure 2 This is a schematic diagram of the architecture of the bone grinding force control module according to an embodiment of the present invention;
[0046] Figure 3 This is a schematic diagram of the architecture of the factor analysis module in an embodiment of the present invention;
[0047] Figure 4 This is a flowchart illustrating a bone grinding reinforcement learning method based on proximal strategy optimization according to an embodiment of the present invention.
[0048] Figure 5 This is a schematic diagram of the hand operation state transition in an embodiment of the present invention;
[0049] Figure 6 This is a schematic diagram of a Markov chain for manual grinding operations according to an embodiment of the present invention;
[0050] Figure 7 This is a schematic diagram of the construction of the Gym simulation environment for the force control of the robotic arm in an embodiment of the present invention, wherein (a) is the state of the environment at the initial stage of training, and (b) is the state of the environment during the training process;
[0051] Figure 8 This is a schematic diagram of the reward curve during the Gym training process according to an embodiment of the present invention, wherein (a) is the reward value curve for each round of training, and (b) is the reward value curve under parallel accelerated training.
[0052] Figure 9 This is a schematic diagram illustrating the use of PID data for contact force control teaching in an embodiment of the present invention;
[0053] Figure 10 This is a schematic diagram of adaptive variable impedance grinding force control according to an embodiment of the present invention;
[0054] Figure 11 This is a schematic diagram illustrating reinforcement learning used for model grinding force control in an embodiment of the present invention. Detailed Implementation
[0055] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0056] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0057] like Figures 1-11 As shown, this embodiment provides a bone grinding reinforcement learning system based on proximal strategy optimization, including: a bone grinding force control module and a factor analysis module;
[0058] The bone grinding force control module is used to accelerate robot training by collecting human control experience data or admittance control data, and to train the robot by changing individual factors separately.
[0059] The factor analysis module is used to perform factor variable analysis on the factors transformed by the bone grinding force control module. Specifically, it is used to analyze the factors affecting the bone cutting effect of the robot, obtain the analysis results, and statistically determine the optimal parameter values based on the analysis results.
[0060] Furthermore, the bone grinding force control module includes a contact force control submodule, a bone grinding action function submodule, and an action decision submodule;
[0061] The contact force control submodule is used to solve high-dimensional or continuous motion spaces;
[0062] The bone grinding action function submodule is used to design the reward function and the penalty function;
[0063] The action decision submodule is used to adjust the learning rate, optimize the algorithm transformation, introduce an experience replay mechanism, adjust the network structure, pre-train the policy function and value function, carefully design the reward function, and implement parallel computing.
[0064] Furthermore, the factor analysis module includes a force sensor, a bone model, an RL model, and a robot;
[0065] The bone model is connected to the force sensor, the sensor is connected to the RL model, the RL model is connected to the robot, and the robot is connected to the bone model.
[0066] The bone model is subjected to pressure contact between the grinding tool and the bone tissue, and the grinding force generated by the extrusion is used to remove the extruded bone tissue.
[0067] The force sensor calibrates the robot, and the RL model establishes the contact force control relationship between the robot's end effector position and bone tissue.
[0068] like Figure 2 As shown, this embodiment also provides a bone grinding reinforcement learning method based on proximal strategy optimization, including: learning the relationship between the end-cutting force state and robot actions by learning the human trial-and-error growth method, and solving the robot force control Markov decision process problem;
[0069] Optimize contact force control through a proximal strategy;
[0070] Design the reward and penalty functions for the bone grinding action;
[0071] Accelerate the training of robot action decision-making strategies;
[0072] The factors affecting the bone cutting effect of the robot are analyzed.
[0073] Furthermore, the data samples for the robot force control Markov decision process problem are represented by a quintuple (S, A, P, γ, R), where S represents the system state of the robot agent in the environment, which, for simplicity, mainly refers to the force state of the robotic arm's end effector in this invention; A represents the action 'a' output by the agent based on the current state and policy, such as moving the end effector upwards away from the bone or downwards towards the bone; P represents the state transition matrix; γ represents the single-step reward discount factor; and R represents the reward value of the reward action. The Markov decision process seeks a policy π(at|st) that enables the robot agent to take an action in the current state and obtain the maximum reward value. This policy is a mapping from the robot state S to the action A. The actions and states are discretized and solved using the MDP to MRP method. The policy for a series of state-action pairs, from time 0 to time t, is as follows:
[0074] τ={s0,a0,s1,a1,s2,a2,...s t-1 .a t-1 ,s t ,a t};
[0075] Once the strategy parameter θ is determined, the probability of this trajectory τ occurring is:
[0076]
[0077] The robot receives a reward and expected value during one round of interaction with its environment:
[0078]
[0079] Reinforcement learning algorithms require adjusting the actor parameters θ to maximize the reward value R(τ). The policy gradient is calculated by solving for the gradient with respect to the expected value. Perform gradient calculations and update the parameter θ using gradient ascent to maximize the reward value.
[0080]
[0081] in, R(τ) represents the difference between the expected values and the reward values. θ is the expected value, μ is the actor parameter to be adjusted, and μ is the learning rate, which is adjusted using the RMSProp or Adam method.
[0082] Furthermore, the goal of reinforcement learning algorithms is to maximize the reward value R(τ) as much as possible through gradient ascent, which is transformed into calculating the policy gradient of the expected reward value.
[0083]
[0084] in, R(τ) represents the difference between the expected values and the reward values. Let θ be the expected value, θ be the actor parameter, and τ be the trajectory.
[0085] Policy gradient is a method for applying the same policy, and resampling is required after each round of sampling and parameter updates. This operation causes sampling to consume most of the time. The PPO algorithm evolved from the policy gradient algorithm, which transforms the same policy into a different policy through importance sampling. The premise that importance sampling is effective is that the two sets of data have similar distributions, i.e., the p-value is limited. θ (s t |a t ) / p θ' (s t |a t The ratio of the near-end policy to the near-end policy is also the key to the PPO algorithm. The objective function for near-end policy optimization is:
[0086]
[0087] In the above formula, the first term is the PPO optimization objective J. θ During training, when optimizing the objective function J, '(θ) is used. θ Based on θ, a KL divergence constraint term is added to limit the similarity of the output actions of the corresponding parameters θ and θ'. This constraint is similar to regularization, where the KL term is called the penalty term, i.e., the penalty for proximal policy optimization. Another improvement method to PPO is proximal policy optimization pruning, which does not include KL divergence in its optimization terms, but instead restricts p through pruning. θ (s t |a t ) / p θ' (s t |a t The ratio of )
[0088]
[0089] Values less than 1-c in the pruning terms are replaced with 1-c, and values greater than 1-c are replaced with 1+c. This will reduce the gap between the demonstration model and the actual learning model after optimization.
[0090] Furthermore, the reward and penalty functions for the bone-grinding action are designed as follows: the reward term rf for forward movement within time step t is composed of the following formula:
[0091] rf =k f (p r -p c );
[0092] Where, p r To control the real-time position of the robot's end effector using contact force; p c k represents the initial position of the robot's end effector. f r is the reward coefficient for the robot's forward movement. f To measure the forward motion of the robot in the lateral direction, the amount of movement during a round of forward motion is proportional to the reward value. Simultaneously, the longitudinal contact force needs to be controlled. The reward function *rn* for longitudinal force control should be set as a function related to the normal contact force error to determine the quality of the action. A smaller normal force error results in a larger reward value, and vice versa.
[0093] r n =k n (f n -f e );
[0094] In the formula, f n f is the actual contact force in the normal direction at the robot's end effector. e k represents the desired contact force in the normal direction at the robot's end effector. n The force deviation penalty coefficient is used to prevent collisions between the robot body and external objects during robot contact control operations, except for the robot's end effector. A penalty term, pcoll, is added to address collisions with external objects. By adding this penalty, the robot learns to avoid collisions while achieving its goal. pcoll is defined as follows:
[0095]
[0096] During laminectomy, excessive contact force between the robot's end effector and the bone should be avoided, as this can easily lead to bone penetration. This is addressed through a penalty point (poF) during robot contact force training.
[0097]
[0098] The optimal strategy π is expressed as follows:
[0099] π(τ)=argmaxE(sum(γ1*r f +γ2*r n +γ3*p coll +γ4*p oF ));
[0100] In the formula, γ is the proportionality coefficient, ranging from (0, 1); to maximize the expected value when the robot executes policy π, it needs to iterate along the direction of gradient change until the expected value reaches its maximum. The states of the robot in reinforcement learning are continuous. The main state perception information of the robotic arm includes kinematic information such as the angular position of each joint of the robotic arm, the angular velocity of the joint, the position signal of the robotic arm end effector, the end effector velocity signal, and dynamic information such as the end effector contact force signal and torque information, thus forming a 12-dimensional state vector as shown in the formula.
[0101] τ = {θ, θ, F, T, P, V};
[0102] In the formula, θ represents the joint angle and joint angular velocity; F and T represent the force and torque sensed by the six-dimensional force sensor at the robot's end effector, respectively; and P and V represent the position and velocity information of the robot's end effector, respectively.
[0103] Furthermore, the training and acceleration method for robot action decision-making strategies specifically involves using traditional, robust control algorithms such as the PID algorithm to further improve data accuracy. A PID controller with tuned parameters is added to the network, the collected state is sent to the PID controller to obtain the output, and the similarity is measured with the previous data distribution to obtain similarity parameters.
[0104] In this invention, in conjunction with the appendix to the specification... Figure 5 The process begins by segmenting and discretizing the operator's manual grinding process, evolving it into a continuous Markov process for the robot. By analyzing a set of data and drawing a partial Markov chain diagram, the operator's operating habits are analyzed. By statistically analyzing a partial Markov decision process data, a state transition diagram and a Markov chain are obtained.
[0105] The operator has the highest probability of performing a VADD action and transitioning to the VLDS state while in the VSDL state (38%); the probability of also performing a VADD action and transitioning to the VSDM state is 12%. Table 1 shows the proportions of the nine states obtained from the data analysis of one operator. To reduce complexity, this invention draws state transition diagrams for the two most prevalent states, including VSDL and VLDS states, and performs state transition analysis by plotting multiple state transition diagrams onto a Markov chain, as shown below. Figure 6 As shown, by artificially adding a reward function to the action, the state value function and the value of each action during the operator's operation are calculated, as shown in Table 1 and... Figure 6Statistical results show that the operator's speed and depth are relatively low (VSDS) and high speed and depth (VLDL) states, while the speed and depth are relatively high (VSDL) and high speed and low depth (VLDS) states are relatively high. When the operator is in one of the states in the lower left or upper right corner of the state transition diagram, the next action has a 33% probability of transitioning to the state in the diagonal direction. It is speculated that the operator's purpose is to adjust the speed and depth to a moderate VMDM state.
[0106] Table 1
[0107] state VSDS VSDM VSDL VMDS VMDM VMDL VLDS VLDM VLDL ％ 5.9 5.9 23.5 0 2.9 8.8 32.4 20.6 0
[0108] Frequent transitions between the lower left and upper right corners indicate that reduced lateral feed speed due to hand movements tends to increase longitudinal depth, while increased lateral feed speed tends to decrease longitudinal depth. The absence of VMDM (Virtual Depth Difference) when the hand maintains a constant speed and depth is likely due to improper threshold settings during data segmentation. This is because the grinding state transitions between the lower left and upper right corners mostly involve passing through the VMDM state. Human grinding is a dynamic closed-loop system process, constantly transitioning from one state to another. Bone penetration and grinding slippage during grinding are clinically considered relatively dangerous actions. When grinding is in the VLDS (Virtual Depth Difference) state, VDDI (Virtual Depth Difference) actions are more frequent. Intuitively, this means that increasing speed and decreasing depth can easily cause tool slippage. Figure 6 As shown, since this is not advisable in clinical practice, the speed needs to be reduced to decrease the possibility of the grinding head slipping. When grinding is in the VSDL state, there are more VADD actions. Intuitively, it can be understood that when the grinding speed decreases and the depth increases, it is easy to cause bone penetration. At this time, the hand action reduces the grinding depth to avoid causing penetration. The above two states and actions are opposite and there are more state transitions. It is speculated that the operator is maintaining balance and keeping the grinding force or output power within a certain range.
[0109] In this invention, the simulation experiment uses a simplified robot model with four motion joints, such as... Figure 7As shown in (a)-(b), the bone model is abstractly represented by a 5x10 grid array. The training objective is to make the robot end effector regularly contact the bone model and effectively remove bone. Its parameters are shown in Table 2. The robot's action space is defined as a four-dimensional torque space, namely Box(-1.0,1.0,(4,)), which represents a 4-dimensional continuous space where the value of each dimension is between -1.0 and 1.0. This is a common method for defining action space or observation space, especially in reinforcement learning environments, where torque is applied to each joint. The observation space integrates the robot's joint angles and velocities, the position and velocity of the robot end effector, the collision force between the end effector and the target, and the amount of target removal or the number of collisions. Specifically, the observation dimension of the collision amount is 2, the observation dimension of the joint angles and velocities is 8, the collision dimension is 1, and the observation dimension of the number of non-collisions is 1. Through the above design, the robot learns how to perform precise bone grinding operations in the simulation environment.
[0110] Table 2
[0111]
[0112] The reward function is designed based on a comprehensive consideration of multiple factors, including the distance the robot moves forward, the amount of material removed, and the positive or negative reward value of a single step. Specifically, the longer the robot's end effector moves forward, the higher the reward value. In the current round, the more blocks are removed through collisions, the greater the reward value. Excessive resection of the bone model at specific longitudinal locations will result in a penalty, as over-resection in a localized area can affect the robot's resection stability and even lead to penetration, which is extremely dangerous and unacceptable for the patient. To incentivize the robot to complete each round's task as quickly as possible, a penalty is added to each step; that is, the more steps taken, the greater the accumulated penalty. The larger the reward value, the more it is set to zero at the beginning of each training round, with a certain degree of random noise added. The action value consists of four random variables, also with random noise introduced. Each training round ends when any of the above-mentioned conditions are met: failure if the robot's end effector fails to reach the predetermined right position within 300 steps, failure if the robot's end effector exceeds the minimum allowable left position, failure if a point on the bone model is punctured, and success if the bone model is successfully and smoothly removed. Through this comprehensive reward mechanism, the robot is guided to learn and complete the resection task while ensuring safety and efficiency. During the robot's training process, the dynamic changes in the reward value are as follows: Figure 8As shown, the robot exhibits a slow upward trend. After a period of training, its movement patterns gradually change from an initial disordered state to an ordered one. Simultaneously, the collision intensity with the environment remains within a preset safety range. After 40,000 rounds of training, totaling 16 hours, the robot's reward function value per round eventually stabilizes at around 250. Notably, after 5,000 rounds of training, or 2 hours, the reward function value already approaches this stable range. Figure 8 As shown in (a); in the subsequent 5000 to 40000 training rounds, the reward function value did not increase significantly; when the graphical user interface (GUI) was turned off during robot training, the robot completed 40000 training rounds in 4 hours, while with the parallel training strategy, the robot's reward value stabilized within 1 hour; although parallel training failed to achieve an acceleration effect proportional to the number of threads, compared to the convergence of single-threaded training after 5000 rounds, 16-thread parallel training converged after 10000 rounds as shown in (a). Figure 8 As shown in (b), the results show that by turning off the robot's graphical display interface and adopting a multi-threaded parallel training strategy, the efficiency of the training process is significantly improved, and the convergence time is reduced to more than a quarter of the original.
[0113] In this invention, in conjunction with the appendix to the specification... Figure 9In the control process implemented using Python, a six-dimensional force / torque sensor is used to acquire force and torque information in real time during the contact process. After the PPO algorithm training converges, the model is applied to a new control task, and the effectiveness of the algorithm is verified by analyzing the error between the actual contact force and the expected contact force at the end of the robotic arm. This experiment is designed to simulate real-world robot operation tasks, aiming to obtain results in the simulation environment that more closely resemble the performance of the physical robot in actual operation. During the simulation experiment, the communication cycle between the robot and the host computer is set to 25ms. Considering the movement of bone tissue during respiration, the difference between the highest and lowest positions of the curved surface is set to 3cm. The expected contact force in the experiment is set to 10N, and this value is adjusted according to specific task requirements. To simplify the experimental process, the position or velocity increment of the end of the robotic arm in the z-direction is controlled, while the position or velocity increment in the x-direction is controlled. To introduce instability in the longitudinal contact force by setting a random variable upwards, the effectiveness of the algorithm was verified. Due to the characteristics of the sensors and the continuous change in the contact position between the robot's end effector and the environment, even small changes in the relative position between the robot's end effector and the contact surface led to drastic changes in the contact force. After 10 hours of training and 10,000 iterations, the reward function value still did not show a convergence trend, indicating that the PPO algorithm in reinforcement learning is difficult to achieve convergence under completely random network initialization and without policy guidance, which is also a common problem faced by reinforcement learning algorithms. In the previous section, the UR robot had force sensors installed at its end effector for orientation control. In the virtual environment, the iiwa robot also had virtual force sensors installed at its end effector, verifying the joint torque control algorithm and the end effector orientation control algorithm. The UR orientation control algorithm in the previous chapter was deployed to the iiwa robot after the development of the basic control program was completed.
[0114] Refer to the instruction manual. Figure 10 The adaptive variable impedance method was also implemented on the VREP simulation platform. Before the robot's end effector contacts the target surface, the robot maintains its posture and moves at a constant speed in the vertical direction until it contacts the surface. In the vertical direction, the robot uses the variable impedance method to calculate and adjust the target speed in the longitudinal direction, while the lateral feed maintains a constant speed mode, the lateral coordinate remains unchanged, and the posture of the robot arm's end effector remains stable. When the robot reaches the predetermined target position in the feed direction, its posture remains unchanged, it returns to its original position, and the control process ends. Figure 10As shown, the curves of force tracking control using the adaptive variable impedance method on the same surface are presented. The tracking error of the PPO algorithm remains within 0.118, while the original force signal deviation of the adaptive variable impedance algorithm is 0.231, with a variance of 157.946 N^2 and a standard deviation of 12.567 N. The filtered signal deviation is 0.106 N, with a variance of 6.459 N^2 and a standard deviation of 2.451 N. Experimental results show that the two algorithms are comparable in force tracking performance with no significant difference. Although the adaptive impedance control method is slightly better than the PPO algorithm in terms of control deviation, the PPO algorithm performs better in terms of control variance, demonstrating its superiority in handling complex unstructured environments.
[0115] Refer to the instruction manual. Figure 11 By acquiring real-time contact force information between the robotic arm's end effector and the environment, motion decision control of the robotic arm is performed. To simplify the experiment and verify the effectiveness of the algorithm, this section sets the robot's motion modes in the x, y, and z directions: random motion, no motion, and incremental control of the PPO algorithm, respectively. After the robot moves vertically downward to the surface of the bone model, the PPO control strategy is used to perform grinding operations with a stable contact force. Compared with the simulation results, the actual contact force error is larger, but overall it still approaches the set contact force. When the expected contact force is set to 10N, the average deviation of the contact force of the original signal is 0.084N, the variance is 50.27N^2, and the standard deviation is 7.090N; the average deviation of the filtered contact force is 0.018N, the variance is 1.980N^2, and the standard deviation is 1.407N. The experimental results show that after 40,000 iterations, the PPO algorithm obtains a relatively stable contact force control strategy.
[0116] This invention discloses a reinforcement learning system and method for bone grinding based on proximal policy optimization. Building upon the Markov property of the robot's bone resection control process, the PPO algorithm is used to train the grinding robot's controller. The trained PPO algorithm directly calculates the normal compensation displacement based on the robot's real-time force state during grinding, thus simplifying the robot's grinding force control flow. Various methods are employed to improve the training process and accelerate the robot's convergence speed, including experience playback, designing dense reward functions, parallel computing, and pre-training using the operator's prior experience and PID control algorithms. Experience playback and dense reward functions guide the robot to obtain positive rewards; parallel computing and disabling graphical displays help accelerate training; utilizing operator demonstration data and PID control data enables the robot to quickly converge to the expected results; and by coordinating the robot's own reinforcement learning strategy and combining it with guidance from human prior knowledge or PID control algorithms, ineffective exploratory behavior is avoided, achieving an efficient learning process.
[0117] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A bone grinding reinforcement learning system based on proximal policy optimization, characterized in that, include: Bone grinding force control module and factor analysis module; The bone grinding force control module is used to accelerate robot training by collecting human control experience data or admittance control data, and to train the robot by changing individual factors separately. The factor analysis module is used to perform factor variable analysis on the factors transformed by the bone grinding force control module. Specifically, it is used to analyze the factors affecting the bone cutting effect of the robot, obtain the analysis results, and statistically determine the optimal parameter values based on the analysis results. The bone grinding force control module includes a contact force control submodule, a bone grinding action function submodule, and an action decision submodule; The contact force control submodule is used to solve the contact force control problem in high-dimensional or continuous motion space; The bone grinding action function submodule is used to design the reward function and the penalty function; The action decision submodule is used to adjust the learning rate, optimize the algorithm transformation, introduce an experience replay mechanism, adjust the network structure, pre-train the policy function and value function, carefully design the reward function, and implement parallel computing. The factor analysis module includes a force sensor, a bone model, an RL model, and a robot; The bone model is connected to the force sensor, the sensor is connected to the RL model, the RL model is connected to the robot, and the robot is connected to the bone model. The bone model is subjected to pressure contact between the grinding tool and the bone tissue, and the grinding force generated by the extrusion is used to remove the extruded bone tissue. The force sensor calibrates the robot, and the RL model establishes the contact force control relationship between the robot's end position and bone tissue. The system includes: By learning from the human process of trial and error, we can study the relationship between the end-effector cutting force state and the robot's actions, and model the robot's force control as a Markov decision process for solution.
2. The bone grinding reinforcement learning system based on proximal strategy optimization as described in claim 1, characterized in that, Modeling robot force control as a Markov decision process includes: Find a strategy π(a) t |s t To enable the robot agent to take actions and obtain the maximum reward value in the current state, the actions and states are discretized, and the MDP is used to transform it into the MRP method for solving the trajectory from time 0 to time t: ； Once the actor parameter θ is determined, the trajectory The probability of this happening is: ； The robot receives a reward and expected value during one round of interaction with its environment: ； Reinforcement learning algorithms require adjusting the actor parameters θ to maximize the reward value R(τ). The policy gradient is calculated by solving for the gradient with respect to the expected value. Perform gradient calculation and update the parameter θ using gradient ascent to maximize the reward value, as shown below: ； in, The difference between the expected values, As a reward value, Let θ be the expected value, θ be the actor parameter to be adjusted, and μ be the learning rate.
3. The bone grinding reinforcement learning system based on proximal strategy optimization as described in claim 1, characterized in that, Optimizing contact force control through proximal strategies includes: The reward value is increased through gradient ascent. Maximizing this is achieved by transforming it into calculating the policy gradient of the expected reward: ； in, As a reward value, Let θ be the expected value, and θ be the actor parameters. For trajectory.
4. The bone grinding reinforcement learning system based on proximal strategy optimization as described in claim 1, characterized in that, The reward and penalty functions for the bone grinding action are designed as follows: Rewards for moving forward r f The reward within time step t is calculated as follows: ； Where, p r To control the real-time position of the robot's end effector using contact force; p c k represents the initial position of the robot's end effector. f r is the reward coefficient for the robot's forward movement. f This is a reward used to measure the amount of forward movement of the robot in the lateral direction.
5. The bone grinding reinforcement learning system based on proximal strategy optimization as described in claim 1, characterized in that, Accelerating the training of robot action decision-making strategies includes: The accuracy of the data is further improved by using the PID algorithm. The PID controller with tuned parameters is added to the network, and the collected state is sent to the PID controller to obtain the output. The obtained output is compared with the previous data distribution to obtain the similarity parameter, thus completing the accelerated training of the robot action decision strategy.