Bone grinding reinforcement learning system and method based on near-end strategy optimization

Abstract

The invention discloses a bone grinding reinforcement learning system and method based on near-end strategy optimization, and the method comprises the steps: learning a relation between a tail end cutting force state and a robot motion in a human trial and error growth mode, and controlling a Markov decision process problem through a robot force; contact force control is optimized through a near-end strategy; designing a reward function and a penalty function of the bone grinding action; performing accelerated training on a robot action decision strategy; and factors influencing the bone cutting effect of the robot are analyzed. On the basis that the robot bone resection control process has the Markov property, the PPO algorithm is adopted to train a controller of the grinding robot, and the robot is guided to obtain a positive value reward; parallelization calculation and closing of graphic display are helpful for increasing the training speed; by coordinating a reinforcement learning strategy of the robot and combining and utilizing priori knowledge of human or a PID control algorithm for guidance, invalid exploration behaviors are avoided, and an efficient learning process is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot control, and in particular relates to a bone grinding reinforcement learning system and method based on proximal strategy optimization. Background Art

[0002] Robotic bone resection is a delicate operation task with rich contacts in unstructured scenarios. The robot's intelligent control method based on supervised learning faces data limitation problems. The robot interacts with the environment frequently and the contact characteristics are diverse. In order to ensure safe and smooth interaction with the environment, the robot needs to be controlled more accurately and smoothly, and the unstable contact between the robot's end tool and unstructured bone tissue is addressed.

[0003] In the field of robot control, the development and application of motion control strategies is an important and hot issue. Robotic laminectomy performed by image navigation and robot cooperation is difficult to meet the high precision and high efficiency requirements of cutting in a fixed speed position control mode due to image registration errors, robot registration errors and robot system errors. In addition, this solution considers the robot's resection environment as a static environment or treats the resection surface as a relatively flat surface. The unstructured environment faced by laminectomy surgical robots includes irregular movements under breathing, bone surface cavities, and deformation of the robot's end due to force. The robot needs to perform real-time and flexible motion control according to the environment in the work scene. However, traditional robot control has the following disadvantages: Currently, laminectomy robots use high-power leveling cutting to cut, and this strategy requires sensitive environmental parameters and does not have real-time flexibility. Summary of the invention

[0004] To solve the above technical problems, the present invention proposes a bone grinding reinforcement learning system and method based on proximal strategy optimization. On the basis that the robot bone resection control process has Markov properties, the PPO algorithm is used to train the controller of the grinding robot. Through experience replay and intensive reward function, the robot is guided to obtain positive rewards; parallel calculation and closing graphic display help to speed up the training speed; the operator's demonstration data and PID control data are used to make the robot quickly converge to the expected effect; by coordinating the robot's own reinforcement learning strategy and combining it with the use of human prior knowledge or PID control algorithm for guidance, invalid exploration behavior is avoided to achieve an efficient learning process.

[0005] On the one hand, to achieve the above-mentioned object, the present invention provides a bone grinding reinforcement learning system based on proximal strategy optimization, comprising: a bone grinding force control module and a factor analysis module;

[0006] The bone grinding force control module is used to accelerate the training of the robot by collecting human control experience data or admittance control data, and to change individual factors to perform robot training;

[0007] The factor analysis module is used to perform factor variable analysis on the factors transformed by the bone grinding force control module, and is specifically used to analyze the factors affecting the robot bone cutting effect, obtain analysis results, and calculate the optimal parameter values ​​for the analysis results.

[0008] Optionally, the bone grinding force control module includes a contact force control submodule, a bone grinding action function submodule and an action decision submodule;

[0009] The contact force control submodule is used to solve high-dimensional or continuous action spaces;

[0010] The bone grinding action function submodule is used to design a reward function and a penalty function;

[0011] The action decision submodule is used to adjust the learning rate, optimize the transformation of the algorithm, introduce the experience replay mechanism, adjust the network structure, pre-train the strategy function and value function, carefully design the reward function and implement parallel computing.

[0012] Optionally, the factor analysis module includes a force sensor, a bone model, an RL model and a robot;

[0013] The bone model is connected to the force sensor, the sensor is connected to the RL model, the RL model is connected to the robot, and the robot is connected to the bone model;

[0014] The bone model is used to simulate the grinding tool and bone tissue in pressure contact, and the grinding force of the tool is generated by extrusion to remove the extruded bone tissue;

[0015] The force sensor calibrates the robot, and the RL model establishes a contact force control relationship between the end position of the robot and the bone tissue.

[0016] On the other hand, to achieve the above purpose, the present invention also provides a bone grinding reinforcement learning method based on proximal strategy optimization, comprising:

[0017] Learn the relationship between the end cutting force state and the robot action by learning the human trial and error growth method, through the robot force control Markov decision process problem;

[0018] Optimizing contact force control through proximal strategies;

[0019] Design reward and penalty functions for the bone grinding action;

[0020] Accelerate the training of robot action decision strategies;

[0021] The factors affecting the effect of robotic bone cutting were analyzed.

[0022] Optional Markov decision process problems controlled by robots include:

[0023] Find a strategy π(at|st) that enables the robot agent to take actions in the current state and obtain the maximum reward value, discretize the actions and states, and use the MDP to MRP method to solve the strategy from time 0 to time t:

[0024] τ={s0,a0,s1,a1,s2,a2,...s t-1 .a t-1 ,s t ,a t};

[0025] When the policy parameter θ is determined, the probability of trajectory τ occurring is:

[0026]

[0027] Get the reward value and expected value in a round of interaction between the robot and the environment:

[0028]

[0029] The reinforcement learning algorithm requires adjusting the actor parameter θ to maximize the reward value R(τ). The policy gradient is calculated by solving the gradient method to calculate the expected value. Do the gradient calculation and update the parameter θ through gradient ascent to make the reward value as large as possible, as shown below:

[0030]

[0031] in, is the difference of expected value, R(τ) is the reward value, is the expected value, θ is the adjusted actor parameter, and μ is the learning rate.

[0032] Optionally, optimized contact force control via proximal strategies includes:

[0033] The reward value R(τ) is maximized through gradient ascent, and converted into a policy gradient for the expected value of the reward:

[0034]

[0035] in, is the difference of expected value, R(τ) is the reward value, is the expected value, θ is the actor parameter, and τ is the trajectory.

[0036] Optionally, the reward function and penalty function for designing the bone grinding action include:

[0037] The forward motion reward r f The reward at time step t is calculated as follows:

[0038] r f =k f (p r -p c );

[0039] Among them, p r The real-time position of the robot end is controlled by the contact force; c is the initial position of the robot end; k f is the robot's forward motion reward coefficient, r f It is used to measure the lateral forward movement of the robot.

[0040] Optional, accelerated training of robot action decision strategies includes:

[0041] The PID algorithm is used to further improve the accuracy of the data. The PID controller with modulated parameters is added to the network. The collected state is sent to the PID controller to obtain the output. The obtained output is measured with the previous data distribution to obtain the similarity parameter, and the robot action decision strategy accelerated training is completed.

[0042] Technical effect of the invention: The present invention discloses a bone grinding reinforcement learning system and method based on proximal strategy optimization. On the basis that the robot bone resection control process has Markov properties, a PPO algorithm is used to train the controller of the grinding robot. The trained PPO algorithm directly calculates the normal compensation displacement according to the real-time force state of the robot grinding, thereby simplifying the control process of the robot grinding force. The training process is improved by a variety of methods to accelerate the convergence speed of the robot, including experience replay, design of intensive reward functions, parallel calculation, and pre-training using the operator's prior experience and PID control algorithm; through experience replay and intensive reward functions, the robot is guided to obtain positive rewards; parallel calculation and closing graphic display help to speed up the training speed; using the operator's demonstration data and PID control data to make the robot quickly converge to the expected effect; by coordinating the robot's own reinforcement learning strategy, and combining the use of human prior knowledge or PID control algorithm for guidance, invalid exploration behavior is avoided to achieve an efficient learning process. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 This is a structural schematic diagram of a bone grinding reinforcement learning system based on proximal strategy optimization according to an embodiment of the present invention;

[0045] Figure 2 Schematic diagram of the structure of the bone grinding force control module according to an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the architecture of the factor analysis module according to an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of a flow chart of a bone grinding reinforcement learning method based on proximal strategy optimization according to an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the manual operation state transition of an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of a Markov chain for manual grinding operation according to an embodiment of the present invention;

[0050] Figure 7 Schematic diagram of the construction of the Gym simulation environment for manipulator force control according to an embodiment of the present invention, wherein (a) is the state of the environment at the beginning of training, and (b) is the state of the environment during training;

[0051] Figure 8 Schematic diagram of the reward curve of the Gym training process in an embodiment of the present invention, wherein (a) is a reward value curve diagram for each round of training, and (b) is a reward value curve diagram under parallel accelerated training;

[0052] Fig. 9 This is a schematic diagram of teaching contact force control using PID data according to an embodiment of the present invention;

[0053] Fig.10 This is a schematic diagram of adaptive variable impedance grinding force control according to an embodiment of the present invention;

[0054] Fig.11 This is a schematic diagram of using reinforcement learning for model grinding force control in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0056] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0057] like Figure 1-Figure 11 As shown, this embodiment provides a bone grinding reinforcement learning system based on proximal strategy optimization, including: a bone grinding force control module and a factor analysis module;

[0058] The bone grinding force control module is used to accelerate the training of the robot by collecting human control experience data or admittance control data, and to change individual factors for robot training;

[0059] The factor analysis module is used to perform factor variable analysis on the factors transformed by the bone grinding force control module, specifically to analyze the factors affecting the robot bone cutting effect, obtain analysis results, and calculate the optimal parameter value for the analysis results.

[0060] Further, the bone grinding force control module includes a contact force control submodule, a bone grinding action function submodule and an action decision submodule;

[0061] The contact force control submodule is used to solve high-dimensional or continuous action spaces;

[0062] The bone grinding action function submodule is used to design a reward function and a penalty function;

[0063] The action decision submodule is used to adjust the learning rate, optimize the transformation of the algorithm, introduce the experience replay mechanism, adjust the network structure, pre-train the strategy function and value function, carefully design the reward function and implement parallel computing.

[0064] Further, the factor analysis module includes a force sensor, a bone model, an RL model and a robot;

[0065] The bone model is connected to the force sensor, the sensor is connected to the RL model, the RL model is connected to the robot, and the robot is connected to the bone model;

[0066] The bone model is used to simulate the grinding tool and bone tissue in pressure contact, and the grinding force of the tool is generated by extrusion to remove the extruded bone tissue;

[0067] The force sensor calibrates the robot, and the RL model establishes a contact force control relationship between the end position of the robot and the bone tissue.

[0068] like Figure 2 As shown, this embodiment also provides a bone grinding reinforcement learning method based on proximal strategy optimization, including: learning the relationship between the end cutting force state and the robot action by learning the human trial and error growth method, and solving the robot force control Markov decision process problem;

[0069] Optimizing contact force control through proximal strategies;

[0070] Design reward and penalty functions for the bone grinding action;

[0071] Accelerate the training of robot action decision strategies;

[0072] The factors affecting the effect of robotic bone cutting were analyzed.

[0073] Furthermore, the data sample of the robot force control Markov decision process problem is represented by a five-tuple (S, A, P, γ, R), where S represents the system state of the robot agent in the environment. To simplify the calculation, it is mainly the force state of the end of the robot arm in the present invention; A represents the action a output by the agent according to the current state and strategy, such as the end moving upward away from the bone and the end moving downward close to the bone; P represents the state transfer matrix; γ represents the single-step reward discount factor; R is the reward value of the reward action Reward; the Markov decision process is to find a strategy π(at|st) that enables the robot agent to take action in the current state and obtain the maximum reward value. The strategy is a mapping from the robot state S to the action A, discretizing the action and state, and using the MDP to MRP method to solve. The strategy of a series of state-action pairs, from time 0 to time t, is as follows:

[0074] τ={s0,a0,s1,a1,s2,a2,...s t-1 .a t-1 ,s t ,a t};

[0075] When the policy parameter θ is determined, the probability of the trajectory τ occurring is:

[0076]

[0077] Get the reward value and expected value in a round of interaction between the robot and the environment:

[0078]

[0079] The reinforcement learning algorithm requires adjusting the actor parameter θ to maximize the reward value R(τ). The policy gradient is calculated by solving the gradient method to calculate the expected value. Do the gradient calculation and update the parameter θ through gradient ascent to make the reward value as large as possible,

[0080]

[0081] in, is the difference of expected value, R(τ) is the reward value, is the expected value, θ is the adjustment actor parameter, and μ is the learning rate, which is adjusted by RMSProp or Adam method.

[0082] Furthermore, the goal of the reinforcement learning algorithm is to maximize the reward value R(τ) as much as possible through gradient ascent, which is converted into a policy gradient for seeking the expected value of the reward.

[0083]

[0084] in, is the difference of expected value, R(τ) is the reward value, is the expected value, θ is the actor parameter, and τ is the trajectory.

[0085] Policy gradient is a same strategy, and after each round of sampling, the parameters must be resampled. This operation causes the sampling operation to take up most of the time. The PPO algorithm is evolved from the policy gradient algorithm. It transforms the same strategy into a different strategy through importance sampling. The premise of importance sampling is that the distribution of the two sets of data is similar, that is, the restriction p θ (s t |a t ) / p θ' (s t |a t ), which is also the key to the PPO algorithm. The proximal strategy optimization objective function is:

[0086]

[0087] The first term in the above formula is the PPO optimization target J θ '(θ), during training, when optimizing the objective function J θ '(θ) adds a KL divergence constraint term to limit the similarity of the corresponding parameters θ and θ' output actions. This constraint is similar to regularization, in which the KL term is called the penalty term, that is, the proximal strategy optimization penalty. Another way to improve PPO is proximal strategy optimization clipping, which does not include KL divergence in the optimization term, but limits p by clipping. θ (s t |a t ) / p θ' (s t |a t ),

[0088]

[0089] In the clipping items, values ​​less than 1-c are replaced with 1-c, and values ​​greater than 1-c are replaced with 1+c, which will reduce the gap between the demonstration model and the actual learning model after optimization.

[0090] Furthermore, the reward function and penalty function of the bone grinding action are designed specifically as follows: the reward for the forward motion reward term rf in time step t is composed of the following formula:

[0091] rf =k f (p r -p c );

[0092] Among them, p r The real-time position of the robot end is controlled by the contact force; c is the initial position of the robot end; k f is the robot's forward motion reward coefficient, r f It is used to measure the amount of lateral robot forward movement. During a round of forward movement of the robot, its movement amount is proportional to the reward value. The robot needs to control the longitudinal contact force while moving forward. The reward function rn of the longitudinal force control should be set to a function related to the normal contact force error to determine the quality of the action. When the normal force error is small, the reward value is large, and vice versa.

[0093] r n =k n (f n -f e );

[0094] In the formula, f n is the actual normal contact force at the end of the robot; f e is the expected normal contact force at the end of the robot; k n is the force deviation penalty coefficient. During the robot contact control operation, except for the end of the robot, other parts of the robot body should avoid contact and collision with external objects. A penalty term pcoll is added when the robot collides with external objects. By adding a penalty method, the robot learns to avoid collision with external objects while achieving the goal. The definition of pcoll is as follows:

[0095]

[0096] During laminectomy, the contact force between the robot tip and the bone should be avoided to be too large. This action can easily cause the risk of bone penetration, which is reflected in the penalty item poF during robot contact force training.

[0097]

[0098] The optimal strategy π is expressed as,

[0099] π(τ)=argmaxE(sum(γ1*r f +γ2*r n +γ3*p coll +γ4*p oF ));

[0100] Where γ is the proportional coefficient, ranging from (0, 1); in order to maximize the expected value when the robot executes the strategy π, it is necessary to iterate along the direction of the gradient change until the expectation reaches the maximum value. The state of the robot reinforcement learning is continuous. The main robot state perception information includes kinematic information such as the angular position of each joint of the robot, the angular velocity of the joint, the position signal of the end position of the robot, the end velocity signal and the dynamic information such as the end contact force signal and the torque information, thereby forming a 12-dimensional state vector as shown in the formula,

[0101] τ = {θ, θ, F, T, P, V};

[0102] Where θ is the joint angle and joint angular velocity; F and T are the force and torque sensed by the six-dimensional force sensor at the end of the robot; P and V are the position information and velocity information of the tool at the end of the robot.

[0103] Furthermore, the training and acceleration method of the robot action decision strategy specifically uses traditional control algorithms with high robustness, such as the PID algorithm, to further improve the accuracy of the data, add the PID controller with modulated parameters to the network, send the collected state to the PID controller to obtain the output, and measure the similarity with the previous data distribution to obtain the similarity parameter.

[0104] In the present invention, in conjunction with the specification Figure 5 , the operator's manual grinding process will first be segmented and discretized to evolve into a robot continuous Markov process. The operator's operating habits will be analyzed by analyzing a set of data and drawing a partial Markov chain diagram. The state transition diagram and Markov chain will be obtained by statistically analyzing some Markov decision process data;

[0105] The probability that the operator performs VADD action and transfers to VLDS state in VSDL state is the highest, which is 38%; the probability that the operator also performs VADD action and transfers to VSDM state is 12%. The proportions of 9 states obtained in the data analysis of an operator are shown in Table 1. To reduce the complexity, the present invention draws two state transition diagrams with the largest proportion, including state VSDL and VLDS for state transition analysis, and draws multiple state transition diagrams into Markov chain, such as Figure 6 As shown in Table 1, adding a reward function to the action artificially calculates the state value function and the value of each action during the operator's operation. Figure 6The statistical results show that the operator's speed is too small and the depth is too small (VSDS) and the speed is too large and the depth is too large (VLDL) account for a small proportion, while the speed is too small and the depth is too large (VSDL) and the speed is too large and the depth is too small (VLDS) account for a large proportion. When the operator is in one of the states in the lower left corner or the upper right corner of the state transition diagram, the next moment the action is transferred to the state in the diagonal direction with a probability of 33%. It is speculated that the operator's purpose is to adjust the speed and depth to a moderate VMDM state.

[0106] Table 1

[0107] state VSDS VSDM VSDL VMDS VMDM VMDL VLDS VLDM VLDL ％ 5.9 5.9 23.5 0 2.9 8.8 32.4 20.6 0

[0108] The frequent state transitions between the lower left and upper right corners indicate that the hand action of reducing the lateral feed speed is likely to cause an increase in the longitudinal depth, and the hand action of increasing the lateral feed speed is likely to cause a decrease in the longitudinal depth. The human hand maintains a constant speed and constant depth state VMDM less often, which is caused by improper threshold setting in the data segmentation process. Because the grinding state will mostly pass through the VMDM state during the transition between the lower left corner and the upper right corner, the human body's grinding operation is a dynamic closed-loop system process, which can be converted from one state to another at any time. Bone penetration and grinding slippage during the grinding process are both clinically dangerous actions. When the grinding is in the VLDS state, there are more VDDI actions. Intuitively, it is easy to cause the tool end to slip when the speed increases and the depth decreases, such as Figure 6 As shown in the figure, since it is not advisable clinically, the speed needs to be reduced to reduce the possibility of the grinding head slipping. When the grinding is in state VSDL, the action VADD is more. Intuitively, when the grinding speed is reduced and the depth is increased, it is easy to cause bone penetration. At this time, the hand action will reduce the grinding depth to avoid penetration. The above two states and actions are opposite and there are more state transitions. It is speculated that the operator is maintaining balance and keeping the grinding force or output power within a certain range.

[0109] In the present invention, the simulation experiment adopts a simplified robot model with four motion joints, such as Figure 7As shown in (a)-(b), the bone model is abstractly represented by a 5x10 grid array. The training goal is to make the robot end regularly contact the bone model and effectively implement bone removal. Its parameters are shown in Table 2. The robot's action space is defined as a four-dimensional torque space, namely Box(-1.0,1.0,(4,)), which represents a 4-dimensional continuous space, where the value of each dimension is between -1.0 and 1.0. This is a common method to define action space or observation space, especially in reinforcement learning environments, where each joint applies a torque, and the observation space integrates the robot's joint angles and speeds, the position and speed of the robot's end, the collision force between the end and the target, and the target removal amount or number of collisions. Specifically, the observation dimension of the collision amount is 2, the observation dimension of the joint angle and speed is 8, the collision dimension is 1, and the observation dimension of the number of non-collisions is 1. Through the above design, the robot learns how to perform precise bone grinding operations in a simulation environment.

[0110] Table 2

[0111]

[0112] The design of the reward function is the result of comprehensive consideration of multiple factors. It is composed of the distance the robot moves forward, the amount of material removed, and the positive or negative reward value brought by a single-step action. Specifically, the longer the distance the robot's end moves forward, the higher the reward value obtained; in the current round, the more blocks are removed by collision, the greater the reward value. If the bone model is resected too deeply at a specific longitudinal position, it will be punished, because excessive resection of local areas will affect the robot's resection stability and even cause penetration. This situation is extremely dangerous and unacceptable to patients. In order to motivate the robot to complete each round of tasks as quickly as possible, a certain penalty is added to each step of the action, that is, the more steps there are, the greater the cumulative penalty. The larger the value, at the beginning of each round of training, all observation values ​​are set to zero, and a certain degree of random noise is added. The action value is composed of four random variables, and random noise is also introduced. When any of the above conditions are met, it is regarded as the end of each round of training. For example, if the end position of the robot fails to reach the predetermined right position within 300 steps, it ends in failure; if the end of the robot exceeds the minimum allowed value on the left, it ends in failure; if a certain position of the bone model is penetrated, it ends in failure; and if the bone model is successfully and smoothly removed, it ends in success. Through the above comprehensive reward mechanism, the robot is guided to learn to complete the removal task under the premise of ensuring safety and efficiency. During the training process of the robot, the dynamic changes of the reward value are as follows Figure 8As shown in the figure, it shows a slowly rising trend. After a certain period of training, the robot's action mode gradually changes from the initial disordered state to an orderly state. At the same time, the collision force with the environment is also maintained within the preset safety range. After 40,000 rounds and a total of 16 hours of training, the robot's reward function value per round finally stabilizes at around 250. It is worth noting that after 5,000 rounds and 2 hours of training, the robot's reward function value is close to this stable range, as shown in the figure. Figure 8 (a) shows that in the subsequent 5000 to 40000 rounds of training, the reward function value did not increase significantly. When the graphical user interface GUI was turned off for robot training, after 40000 rounds of training, the robot completed it within 4 hours, while when the parallel training strategy was adopted, the robot's reward value tended to stabilize within 1 hour. Although parallel training failed to achieve an acceleration effect proportional to the number of threads, compared with the convergence of single-threaded training after 5000 rounds, 16-thread parallel training completed convergence after 10000 rounds. Figure 8 As shown in (b), the results show that by closing the robot's graphical display interface and adopting a multi-threaded parallel training strategy, the efficiency of the training process is significantly improved, and the convergence time is shortened to more than a quarter of the original time.

[0113] In the present invention, in conjunction with the specification Fig. 9In the control process implemented by Python, the six-dimensional force / torque sensor is used to obtain the force and torque information in the contact process in real time. After the PPO algorithm training converges, the model is applied to new control tasks, and the effectiveness of the algorithm is verified by analyzing the error between the actual contact force and the expected contact force at the end of the robot arm. This experimental setting is designed to simulate robot operation tasks in the real world, so that the results obtained in the simulation environment are closer to the performance of the physical robot in actual operation. When conducting simulation experiments, the communication cycle between the robot and the host computer is set to 25ms. At the same time, considering the movement of bone tissue during breathing, the highest and lowest position difference of the surface is set to 3cm. The expected contact force in the experiment is set to 10N, and this value is adjusted according to the specific task requirements. In order to simplify the experimental process, the position or velocity increment of the end of the robot arm in the z direction is controlled, while in the x direction Random variables are set upward to introduce instability in the longitudinal contact force, thereby verifying the effectiveness of the algorithm. Due to the characteristics of the sensor and the continuous change in the contact position between the robot end and the environment, a slight change in the relative position between the robot end and the contact surface leads to a drastic change in the contact force. After 10 hours of training and 10,000 iterations, the reward function value still does not show a convergence trend, indicating that under completely random network initialization and without strategy guidance, the PPO algorithm in reinforcement learning is difficult to converge, which is also a common problem faced by reinforcement learning algorithms. In the previous article, the UR robots are all equipped with force sensors at the end for orientation control. In the virtual environment, virtual force sensors are also installed at the end of the iiwa robot, verifying the joint torque control algorithm and the end orientation control algorithm. In the previous chapter, the UR orientation control algorithm is deployed in the iiwa robot after the basic control program of the iiwa robot is developed.

[0114] In conjunction with the instruction manual Fig.10 ,The adaptive variable impedance method is also implemented on the VREP simulation platform. ,Before the robot end contacts the target surface, the robot maintains the posture and moves at a constant speed in the vertical direction until it contacts the surface. ,In the vertical direction, the robot uses the variable impedance method to calculate and adjust the longitudinal target speed, while the lateral feed maintains a constant speed mode, the lateral coordinates remain unchanged, and the posture of the robot end remains stable. ,When the robot reaches the predetermined target position in the feeding direction, the posture remains unchanged, and the control process ends. Fig.10As shown in the figure, the curve diagram of the force tracking control of the adaptive variable impedance method on the same surface, among which the tracking deviation of the PPO algorithm is kept within 0.118, while the original force signal deviation of the adaptive variable impedance algorithm is 0.231, the variance is 157.946N^2, and the standard deviation is 12.567N; the signal deviation after filtering is 0.106N, the variance is 6.459N^2, and the standard deviation is 2.451N. The experimental results show that the two algorithms are equivalent in force tracking effect and there is no significant difference. Although the adaptive impedance control method is slightly better than the PPO algorithm in control deviation, the PPO algorithm performs better in control variance, indicating its superiority in dealing with complex unstructured environments.

[0115] In conjunction with the instruction manual Fig.11 , by obtaining the contact force information between the end of the robot and the environment in real time, the robot makes action decisions and controls. In order to simplify the experiment and verify the effectiveness of the algorithm, this section sets the robot's action modes in the x, y and z directions: random motion, no motion and incremental control of the PPO algorithm. After the robot moves vertically downward to the surface of the bone model, the PPO control strategy is used to perform grinding operations with stable contact force. Compared with the simulation results, the actual contact force error is larger, but it is still close to the set contact force as a whole. When the expected contact force is set to 10N, the average deviation of the contact force of the original signal is 0.084N, the variance is 50.27N^2, and the standard deviation is 7.090N; the average deviation of the contact force after filtering is 0.018N, the variance is 1.980N^2, and the standard deviation is 1.407N. The experimental results show that after 40,000 iterations, the PPO algorithm obtains a relatively stable contact force control strategy.

[0116] The present invention discloses a bone grinding reinforcement learning system and method based on proximal strategy optimization. On the basis that the robot bone resection control process has Markov properties, a PPO algorithm is used to train the controller of the grinding robot. The trained PPO algorithm directly calculates the normal compensation displacement according to the real-time force state of the robot grinding, thereby simplifying the control process of the robot grinding force. The training process is improved by multiple methods to accelerate the convergence speed of the robot, including experience replay, design of dense reward function, parallel calculation, and pre-training using the operator's prior experience and PID control algorithm; through experience replay and dense reward function, the robot is guided to obtain positive rewards; parallel calculation and closing graphic display are helpful to accelerate the training speed; the operator's demonstration data and PID control data are used to make the robot converge quickly to the expected effect; by coordinating the robot's own reinforcement learning strategy, combined with the use of human prior knowledge or PID control algorithm for guidance, invalid exploration behavior is avoided to achieve an efficient learning process.

[0117] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A bone grinding reinforcement learning system based on proximal strategy optimization, characterized in that: include: Bone grinding force control module and factor analysis module; The bone grinding force control module is used to accelerate the training of the robot by collecting human control experience data or admittance control data, and to change individual factors to perform robot training; The factor analysis module is used to perform factor variable analysis on the factors transformed by the bone grinding force control module, and is specifically used to analyze the factors affecting the robot bone cutting effect, obtain analysis results, and calculate the optimal parameter values ​​for the analysis results.

2. The bone grinding reinforcement learning system based on proximal strategy optimization according to claim 1, characterized in that: The bone grinding force control module includes a contact force control submodule, a bone grinding action function submodule and an action decision submodule; The contact force control submodule is used to solve high-dimensional or continuous action spaces; The bone grinding action function submodule is used to design a reward function and a penalty function; The action decision submodule is used to adjust the learning rate, optimize the transformation of the algorithm, introduce the experience replay mechanism, adjust the network structure, pre-train the strategy function and value function, carefully design the reward function and implement parallel computing.

3. The bone grinding reinforcement learning system based on proximal strategy optimization according to claim 1, characterized in that: The factor analysis module includes a force sensor, a bone model, an RL model and a robot; The bone model is connected to the force sensor, the sensor is connected to the RL model, the RL model is connected to the robot, and the robot is connected to the bone model; The bone model is used to simulate the grinding tool and bone tissue in pressure contact, and the grinding force of the tool is generated by extrusion to remove the extruded bone tissue; The force sensor calibrates the robot, and the RL model establishes a contact force control relationship between the end position of the robot and the bone tissue.

4. A method for a bone grinding reinforcement learning system based on proximal strategy optimization according to any one of claims 1 to 3, characterized in that: include: Learn the relationship between the end cutting force state and the robot action by learning the human trial and error growth method, through the robot force control Markov decision process problem; Optimizing contact force control through proximal strategies; Design reward and penalty functions for the bone grinding action; Accelerate the training of robot action decision strategies; The factors affecting the effect of robotic bone cutting were analyzed.

5. The bone grinding reinforcement learning method based on proximal strategy optimization according to claim 4, characterized in that: Problems of controlling Markov decision processes through robotic force include: Find a strategy π(at|st) that enables the robot agent to take actions in the current state and obtain the maximum reward value, discretize the actions and states, and use the MDP to MRP method to solve the strategy from time 0 to time t: τ={s0,a0,s1,a1,s2,a2,......s t-1 .a t-1 ,s t ,a t }; When the policy parameter θ is determined, the probability of trajectory τ occurring is: Get the reward value and expected value in a round of interaction between the robot and the environment: The reinforcement learning algorithm requires adjusting the actor parameter θ to maximize the reward value R(τ). The policy gradient is calculated by solving the gradient method to calculate the expected value. Do the gradient calculation and update the parameter θ through gradient ascent to make the reward value as large as possible, as shown below: in, is the difference of expected value, R(τ) is the reward value, is the expected value, θ is the adjusted actor parameter, and μ is the learning rate.

6. The bone grinding reinforcement learning method based on proximal strategy optimization according to claim 4, characterized in that: Optimizing contact force control through proximal strategies includes: The reward value R(τ) is maximized through gradient ascent, and converted into a policy gradient for the expected value of the reward: in, is the difference of expected value, R(τ) is the reward value, is the expected value, θ is the actor parameter, and τ is the trajectory.

7. The bone grinding reinforcement learning method based on proximal strategy optimization according to claim 4, characterized in that: The reward and penalty functions for designing the bone grinding action include: The forward motion reward r f The reward at time step t is calculated as follows: r f =k f (p r -p c ); Among them, p r The real-time position of the robot end is controlled by the contact force; c is the initial position of the robot end; k f is the robot's forward motion reward coefficient, r f It is used to measure the lateral forward movement of the robot.

8. The bone grinding reinforcement learning method based on proximal strategy optimization according to claim 4, characterized in that: Accelerated training of robot action decision strategies includes: The PID algorithm is used to further improve the accuracy of the data. The PID controller with modulated parameters is added to the network. The collected state is sent to the PID controller to obtain the output. The obtained output is measured with the previous data distribution to obtain the similarity parameter, and the robot action decision strategy accelerated training is completed.

Images