Robot force control method based on RBF neural network rigidity prediction and reinforcement learning

Abstract

The invention provides a robot force control method based on RBF (Radial Basis Function) neural network rigidity prediction and reinforcement learning, and the method comprises the following specific steps: firstly, collecting the empirical data of robot force tracking through a plurality of groups of PD (Proportional-Digital) control; then, fitting an environmental dynamics model of reinforcement learning by using the empirical data; in order to accelerate the convergence speed of reinforcement learning, fitting the rigidity of the contact environment by adopting an RBF neural network and predicting the environment rigidity, and combining the predicted environment rigidity with a Gaussian model to obtain an environment rigidity selection probability; and finally, according to the environment rigidity selection probability, the selection probability of a P parameter in PD control parameters is corrected in combination with a rigidity model, so that the action selection probability in reinforcement learning is improved, and the optimal parameter of robot force control is quickly found.

Description

Technical Field

[0001] The invention belongs to the field of robot force control and artificial intelligence, and in particular relates to a robot force control method based on RBF neural network stiffness prediction and reinforcement learning. Background Art

[0002] In robot application scenarios in constrained spaces, such as precision machining, medical surgery, and rehabilitation assistance, robots are required to have the ability to accurately perceive the external environment and interact with it efficiently while ensuring the safety of force application. However, due to the insufficient stiffness of the robot (Pan et al., 2023, Optimization of static performance for robot polishing system based on work stiffness evaluation [J]. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture, 2023, 237 (4): 519-531.), when the contact environment changes or there is external interference, it often leads to unstable force during the contact process, which in turn causes fluctuations in contact force and affects the work effect.

[0003] In order to ensure the accuracy of the robot in the process of applying force, a robot force controller is usually used to adjust the stability of the robot when it contacts the external environment (Yang et al., 2018, Force Modeling, Identification, and Feedback Control of Robot-Assisted Needle Insertion: A Survey of the Literature [J]. Sensors, 2018, 18 (2): 561.). Traditional force control methods include impedance control, admittance control, and force / position hybrid control, which can adjust the behavior of the robot during contact. However, when adjusting the parameters of these controllers, multiple factors need to be considered comprehensively. If the adjustment is improper, the controller performance may deteriorate or become unstable. Therefore, many scholars have introduced adaptive or intelligent algorithms based on traditional force controllers to improve their performance. For example, Shi et al. (Shi et al., 2023, Research on Surface Tracking and Constant Force Control of a Grinding Robot [J]. Sensors, 2023, 23(10): 4702.) used a force / position hybrid control strategy based on fuzzy PID, which greatly improved the response speed of the static control strategy; Cao et al. (Cao et al., 2020, Smooth adaptive hybrid impedance control for robotic contact force tracking in dynamic environments [J]. Industrial Robot: the international journal of robotics research and application, 2020, 47(2): 231-242.) combined adaptive algorithms with impedance control to solve the problem of dynamic force tracking in uncertain environments; Wang et al. (Wang et al., 2021, Hybrid force / position control in workspace of robotic manipulator in uncertain environments based on adaptive fuzzy control [J]. Robotics and Autonomous Systems, 2021, 145: 103870.) Combining fuzzy logic and traditional proportional-integral method, a fuzzy proportional-integral control method was proposed to improve the control performance of the normal interaction force; Peng et al. (Peng et al., 2022, Neural Networks Enhanced Optimal Admittance Control of Robot-Environment Interaction Using Reinforcement Learning [J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33 (9): 4551-4561.) proposed a neural network-based adaptive controller with a dynamic learning framework to ensure trajectory tracking performance; Dachang et al. (Dachang et al., 2020, Constant Force PID Control for Robotic Manipulator Based on Fuzzy Neural Network Algorithm [J]. Complexity, 2020, 2020: e3491845.) proposed a fuzzy neural network algorithm that can quickly and effectively adjust the PID controller parameters. However, although these improvements have enhanced the performance of controllers, they usually rely on pre-set rules or models, which means that when the external environment changes beyond expectations, the adaptability of the controller will be limited, making it difficult to autonomously explore and respond to completely new environments.

[0004] Reinforcement learning algorithms have the ability of autonomous learning and can continuously adapt to environmental changes through continuous interaction with the environment, showing good adaptability (Suomalainen et al., 2022, A survey of robot manipulation in contact [J]. Robotics and Autonomous Systems, 2022, 156: 104-224.). Many researchers use reinforcement learning algorithms to explore optimal strategies, such as; Ding et al. (Ding et al., 2023, Impedance control and parameter optimization of surface polishing robot based on reinforcement learning [J]. Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture, 2023, 237 (1-2): 216-228.) A robot impedance control parameter learning algorithm based on reinforcement learning obtained the optimal impedance parameters; Roveda et al. (Roveda et al., 2022, Q-Leaming-based model predictive variable impedance control for physical human-robot collaboration [J]. Artificial Intelligence, 2022, 312: 103771.) proposed a Q-learning algorithm to predict impedance model parameters to assist operators in physical human-machine collaboration tasks; Chen et al. (Chen et al., 2023, Active compliance control of robot peg-in-hole assembly based on combined reinforcement learning [J]. Applied Intelligence, 2023, 53 (24): 30677-30690.) used DQN combined with impedance controller to construct a variable impedance controller, which can make corresponding choices for different working conditions; Bogdanovic et al. (Bogdanovic et al., 2020, Learning Variable Impedance Control for Contact Sensitive Tasks [J].IEEE Robotics and Automation Letters, 2020, 5(4): 6129-6136.) Deep Deterministic Policy Gradient is used to learn the joint positions required for the robot's output impedance. However, when these algorithms are applied to robot force tracking, they usually face problems such as complex design and slow algorithm convergence, which makes it difficult to fully meet actual needs. How to improve the real-time and stability of the system while ensuring learning efficiency remains a challenge that needs to be solved. . Summary of the invention

[0005] The purpose of the present invention is to provide a robot force control method based on RBF (Radial Basis Function) neural network and reinforcement learning, aiming to optimize the deep Q network algorithm and realize the rapid robot force control process. The present invention obtains the initial strategy of robot force control through the PD control algorithm; on the data of the traditional force controller, the RBF neural network is used to fit and predict the stiffness of the contact environment, and reinforcement learning selects the optimal control parameters. In order to accelerate the convergence speed of reinforcement learning, the RBF neural network is used to fit the stiffness of the contact environment and predict the environmental stiffness. The predicted environmental stiffness is combined with the Gaussian model to obtain the environmental stiffness selection probability; finally, according to the environmental stiffness selection probability and combined with the stiffness model, the selection probability of the P parameter in the PD control parameter is corrected, thereby improving the action selection probability in reinforcement learning and quickly finding the optimal parameters of the robot force control.

[0006] In order to achieve the above object, the present invention is implemented as follows: it is a robot force control method based on RBF neural network stiffness prediction and reinforcement learning, including four steps: experience data collection, building a reinforcement learning environment dynamics model based on BP neural network, predicting the stiffness of the contact environment with the RBF neural network model, and optimizing the action selection of reinforcement learning based on the Gaussian model:

[0007] Step 1: Empirical data collection: Use PD control to build an initial strategy and collect robot force signals and robot displacement data;

[0008] Step 2: Construct a reinforcement learning environment dynamics model based on BP neural network: Based on the experimental results of multiple groups of PD control parameters, the relationship between the robot contact state and the robot displacement is mined based on the BP neural network, and an environmental dynamics model is constructed for reinforcement learning offline training. The BP neural network input is the contact state and robot displacement of the previous stage, and the output is the contact state at the next moment;

[0009] Step 3: The RBF neural network model predicts the stiffness of the contact environment: Based on the linear stiffness model, the stiffness of the robot's contact environment is calculated according to the collected contact force and robot displacement data. The environmental stiffness parameters in the previous period are input into the RBF neural network, and the output is the environmental stiffness parameters at the next moment. The RBF neural network model can predict the environmental stiffness.

[0010] Step 4: Optimize the action selection of reinforcement learning based on the Gaussian model: The predicted environmental stiffness is obtained based on the RBF neural network prediction model, and the P parameter in the PD parameter is Gaussianized with the Gaussian model. Since the weight of the P parameter is usually greater than the D parameter, the D parameter is set to a fixed value, and the probability of the Gaussianized P parameter is expanded to the probability of the reinforcement learning action, which is used for the early exploration of reinforcement learning. After a certain number of iterations, the probability of using the ε-greedy strategy is used to smoothly control the trade-off between exploration and utilization.

[0011] In the technical solution, in step 1, preprocessing is performed when collecting robot force signal data, and average value filtering is used.

[0012] In the technical solution, in step 2, the contact force error e of the robot force signal is extracted. t , contact force error change rate Δe t The offset displacement Δx from the robot t , put the collected data into the BP neural network for fitting, and build an environmental dynamics model based on the BP neural network.

[0013] In the technical solution, in step 3, the RBF neural network model predicts the stiffness of the contact environment, and the stiffness of the environment is calculated using the stiffness formula, which is as follows:

[0014] f e =Δx s k s (1)

[0015] Where: f e is the force acting on the object, k s is the stiffness coefficient of the environment, Δx s is the displacement of the object under force. And the stiffness system k of the robot's environment at the previous moment is obtained s (t-3) , k s (t-2) , k s (t-1) , input RBF neural network, output is k s (t) .

[0016] In the technical solution, in step 4, the reinforcement learning is a deep Q network, in which the contact state s is set as the force error and the rate of change of the force error, and the immediate reward can be set as the distance between the actual force and the reference force:

[0017] f=-k f *|f e -f r |. (2)

[0018] In the formula, k r is the scale factor, f e is the actual contact force; f r The reference force is obtained by offline training of reinforcement learning under BP neural network, and the output strategy is the PD parameters of robot force control.

[0019] In the technical solution, in step 4, the reinforcement learning action a is a PD parameter, and the range of the PD parameter is set to k p ∈[k p 1 , k p 2 , ..., k p m ],k d ∈[k d 1 , k d 2 , ..., k d m ], action a randomly selects different parameters from these parameters, that is, a=[u1,u2,...,u T ], where u=(k p , k d ).

[0020] In the technical solution, in step 4, the selection probability of the P parameter in the PD parameter can be realized by a Gaussian model.

[0021]

[0022] In the formula, k p is the parameter in the reinforcement learning action a, k s is the predicted stiffness value, σ is the standard deviation, and exp is the exponential function. After the reinforcement learning action is offline, the probability of the value of different offset actions can be obtained according to the Gaussian model.

[0023] In the technical solution, in step 4, the feature is that the probability of the P parameter selection is expanded to the probability of action a, and the discretized action a needs to be encoded to obtain the action a encoding [(k p 1 , k d1 ), (k p 1 , k d 2 ), (k p 1 , k d 3 )...,(k p m , k d m-2 ), (k p m , k d m-1 ), (k p m , k d m )], corresponding to Considering that the influence of P parameter on the result is dominant in PD control, the D parameter is fixed each time for initial selection, and the average value of the corresponding Gaussian model is set to k s The corresponding k p sequence Then the probability of action a is:

[0024]

[0025] In the technical solution, in step 4, the action selection based on the Gaussian model is used in the early exploration of the reinforcement learning exploration, and the probability of using the ε-greedy strategy is increased after a certain number of iterations to smoothly control the trade-off between exploration and utilization.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] Compared with the traditional force control algorithm, the method of the present invention has significant advantages. By collecting multiple groups of PD control data, an environmental dynamics model of reinforcement learning is constructed, and the convergence speed of the model is accelerated by using RBF neural network. The environmental dynamics model can accurately simulate the contact process between the robot and the workpiece, so that the algorithm can perform a large number of simulations in the offline stage and learn the interaction rules between the robot and the workpiece, thereby effectively reducing the number of interactions with the real environment. This pre-training process not only reduces the cost of adjustment and optimization, but also improves the algorithm's ability to adapt to noise and uncertainty in the actual environment.

[0028] The algorithm of the present invention excels in reliability and safety. The algorithm can flexibly respond to changes and interferences in different processing environments. In particular, the reinforcement learning algorithm can adaptively adjust the control strategy in a constantly changing environment, thereby improving the robustness and stability of the system. At the same time, the RBF neural network provides accurate support for the prediction of environmental stiffness, improves the selection probability of the P parameter in the PD controller, and further optimizes the accuracy of action selection. In this way, the present invention not only improves the accuracy of force control, but also reduces the workload of manually adjusting parameters, making the algorithm more efficient and practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The present invention is a flow chart of the robot force control method based on RBF neural network stiffness prediction and reinforcement learning. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In the present invention, RBF neural network refers to a radial basis function neural network; PD control refers to proportional-derivative control; and P parameter refers to a proportional parameter of PD control.

[0032] Embodiment 1

[0033] like Figure 1 , which is a robot force control method based on RBF neural network and reinforcement learning, including the following steps:

[0034] Including experience data collection, building a reinforcement learning environment dynamics model based on BP neural network, predicting the stiffness of the contact environment with RBF neural network model, and optimizing the action selection of reinforcement learning based on Gaussian model;

[0035] Experience data collection

[0036] Step 1: Collect robot force signals and displacement data. Preprocess the robot force signal data and use average filtering. The robot squeezes the workpiece vertically along the Z direction at a speed of 2 mm / s. When the robot reaches the reference force f along the Z direction, r When Q a When the point is reached, the robot stops moving in the Z direction, turns on the force control mode, and moves horizontally in the X direction at a speed of 1 mm / s for 5 seconds to Qb When the robot applies force on the workpiece surface, the reference force of the robot is set to 20N, that is, f r =20N, the initial strategy is constructed using PD control. The empirical parameters of PD control can ensure that the robot is in contact with the workpiece. After executing different PD control strategies, the robot does not leave the workpiece and collects the difference between the end force of the robot and the reference force e t , the rate of change of error Δe t The offset displacement Δx from the robot t ;

[0037] Construction of reinforcement learning dynamics model based on BP neural network

[0038] Step 2: According to the experimental results under 20 groups of PD control parameters, the robot force signal data and robot displacement are collected; the relationship between the robot contact state and the robot displacement is used to build an environmental dynamics model based on the BP neural network, where the number of intermediate nodes of the neural network is set to 30, and the number of layers of the neural network is designed to be 2;

[0039] RBF neural network model predicts the stiffness of contact environment

[0040] Step 3: Based on the linear stiffness model, calculate the stiffness of the robot system environment according to the collected contact force and robot displacement data. The stiffness formula calculates the stiffness of the environment. The formula is as follows:

[0041] f e =Δx s k s (1)

[0042] Where: f e is the force acting on the object, k s is the stiffness coefficient of the environment, Δx s is the displacement of the object under force. And the stiffness system k of the robot's environment at the previous moment is obtained s (t-3) , k s (t-2) , k s (t-1) , trained 300 times, and obtained the prediction model of RBF neural network, and the output is k s (t) .

[0043] Optimizing action selection in reinforcement learning based on Gaussian model

[0044] Step 4: The predicted environment stiffness is obtained by using the prediction model based on the RBF neural network. The P parameter in the PD parameter is Gaussianized by combining the Gaussian model. Since the weight of the P parameter is usually greater than the D parameter, the selected D parameter is fixed, and the probability of the Gaussianized P parameter is expanded to the probability of the reinforcement learning action, which is used for the early exploration of reinforcement learning. After a certain number of iterations, the probability of using the ε-greedy strategy is used to smoothly control the trade-off between exploration and utilization. The core parameters are as follows:

[0045] Reinforcement learning is a deep Q network, where the contact state s is set as the force error and the rate of change of the force error, and the immediate reward can be set as the distance between the actual force and the reference force:

[0046] r=-k r *|f e -f r |. (2)

[0047] In the formula, k r is the scale factor, f e is the actual contact force; f r The reference force is the robot strategy output under the BP neural network as the PD parameter. The input data s and a are used in the deep Q network, and the output is the Q value, that is, 1-dimensional data. Due to the small dimension of the parameters and the small amount of data, the deep neural network is much smaller than the image dimension, so the number of layers of the neural network can be designed as 2 layers, with 30 nodes in each layer. The step size of reinforcement learning is set to T = 200, and the total number of iterations N is 200.

[0048] Reinforcement learning action a is the PD parameter, and the range of PD parameters is set to k p ∈[k p 1 , k p 2 , ..., k p m ],k d ∈[k d 1 , k d 2 , ..., k d n ], action a randomly selects different parameters from these parameters, that is, a=[u1,u2,...,u T ], where u=(k p , k d ). The PD control parameter is manually adjusted based on experience to k p The range is set to [0.00010.001], and the value is discretized every 0.0001. dThe range of is set to [0.0010.01], and the value is discretized every 0.001. The empirical parameters can ensure that the robot contacts the curved workpiece and does not leave the workpiece. During the experiment, the step length T is divided into 4 stages, each stage starts from k p , k d The state takes the mean and mean square error of each signal as input.

[0049] The selection probability of the P parameter in the PD parameter can be realized through the Gaussian model.

[0050]

[0051] In the formula, k p is the parameter in the reinforcement learning action a, k s is the predicted stiffness value, a is the standard deviation, and exp is the exponential function. The action is taken offline, and the values ​​of different offset actions are obtained based on the values ​​of the Gaussian model:

[0052] The probability of parameter selection of P is extended to the probability of action a. The discretized action a needs to be encoded, and the action a encoding [(k p 1 , k d 1 ), (k p 1 , k d 2 ), (k p 1 , k d 3 )...,(k p m , k d m-2 ), (k p m , k d m-1 ), (k p m , k d n )], corresponding to Considering that the influence of P parameter on the result is dominant in PD control, the D parameter is fixed each time for initial selection, and the average value of the corresponding Gaussian model is set to k s The corresponding k p sequence Then the probability of action a is:

[0053]

[0054] In the early exploration of reinforcement learning, the probability of using the ε-greedy strategy after a certain number of iterations smoothly controls the trade-off between exploration and exploitation.

[0055] Example 2

[0056] In this embodiment, a robot force control method based on RBF neural network and reinforcement learning is described. Compared with Embodiment 1, the environment dynamics model can select RBF neural network; the relationship between the contact state s and the robot offset displacement is constructed; the RBF neural network structure is:

[0057]

[0058] Where n is the node; w is the weight; Γ is the weight matrix; μ is the center position of the Gaussian basis function; are the strategy parameters, including {w i , Γ, μ i}. The output π is the robot position PD parameter.

[0059] Embodiment 2

[0060] In the robot force control method based on RBF neural network and reinforcement learning in this embodiment, compared with embodiment 1, the reinforcement learning algorithm can also be changed to a policy gradient method.

[0061] The policy gradient method mainly consists of two steps: sampling and updating. In the sampling phase, the agent interacts with the environment according to the current policy and generates a series of trajectories. In the updating phase, these sampled trajectories are used to calculate the gradient of the policy and update the parameters. The calculation of the gradient usually uses the Monte Carlo method to estimate the expected value of the return, iteratively update the policy parameters, and gradually approach the optimal value, where the iteration term is the gradient of the objective function, that is,

[0062]

[0063] In the formula, k is the number of iterations; c is the step size parameter; is the partial derivative of θ; is the gradient of the objective function with respect to the parameter vector θ, referred to as the policy gradient, which is decomposed into

[0064]

[0065] In the formula, ln is a logarithm. The part of P(τ, θ) that does not contain θ is eliminated after the derivation process, so in the above formula Can be converted into

[0066]

[0067] It is difficult to directly obtain the value of the policy gradient. You can choose a sampling method that is close to this expression for calculation. The policy gradient is approximated according to the average value of m trajectories to facilitate calculation.

[0068]

[0069] Where m is the order of the trajectory.

[0070] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A robot force control method based on RBF neural network stiffness prediction and reinforcement learning, characterized in that The following steps are involved: Step 1: Empirical data collection: Use PD control to build an initial strategy and collect robot force signals and robot displacement data; Step 2: Construct a reinforcement learning environment dynamics model based on BP neural network: Based on the experimental results of multiple groups of PD control parameters, the relationship between the robot contact state and the robot displacement is mined based on the BP neural network, and an environmental dynamics model is constructed for reinforcement learning offline training. The BP neural network input is the contact state and robot displacement of the previous stage, and the output is the contact state at the next moment; Step 3, RBF neural network model predicts the stiffness of the contact environment: Based on the linear stiffness model, according to the collected contact force and robot displacement data, the stiffness of the robot contact environment is calculated, the environmental stiffness parameters in the previous period are input into the RBF neural network, and the environmental stiffness parameters at the next moment are output, so that the RBF neural network model can predict the environmental stiffness; Step 4: Optimize the action selection of reinforcement learning based on the Gaussian model: The predicted environmental stiffness is obtained based on the RBF neural network prediction model, and the P parameter in the PD parameter is Gaussianized with the Gaussian model. Since the weight of the P parameter is greater than that of the D parameter, the D parameter is set to a fixed value, and the probability of the Gaussianized P parameter is expanded to the probability of the reinforcement learning action, which is used for the early exploration of reinforcement learning. After a certain number of iterations, the probability of using the ε-greedy strategy is used to smoothly control the trade-off between exploration and utilization.

2. The robot force control method based on RBF neural network stiffness prediction and reinforcement learning according to claim 1, characterized in that: In step 1, preprocessing is performed when collecting robot force signal data, and average value filtering is used.

3. The robot force control method based on RBF neural network stiffness prediction and reinforcement learning according to claim 1, characterized in that: In step 2, the contact force error e of the robot force signal is extracted t , contact force error change rate Δe t The offset displacement Δx from the robot t , put the collected data into the BP neural network for fitting, and build an environmental dynamics model based on the BP neural network.

4. The robot force control method based on RBF neural network stiffness prediction and reinforcement learning according to claim 1, characterized in that: The RBF neural network model predicts the stiffness of the contact environment and uses the stiffness formula to calculate the stiffness of the environment. The formula is as follows f e =Δx s k s 。 (1) In the formula, f e is the force acting on the object, k s is the stiffness coefficient of the environment, Δx s is the displacement of the object under force. And the stiffness system k of the robot's environment at the previous moment is obtained s (t-3) , k s (t-2) , k s (t-1) , input RBF neural network, output is k s (t) .

5. The robot force control method based on RBF neural network stiffness prediction and reinforcement learning according to claim 5 is characterized in that: The action selection of reinforcement learning is optimized based on the Gaussian model. The reinforcement learning is a deep Q network, where the contact state s is set to the force error and the rate of change of the force error, and the immediate reward is set to the distance between the actual force and the reference force: r=-k r *|f e -f r |。 (2) In the formula, k r is the scale factor, f e is the actual contact force; f r The reference force is obtained by offline training of reinforcement learning under BP neural network, and the output strategy is the PD parameters of robot force control.

6. The robot force control method based on RBF neural network stiffness prediction and reinforcement learning according to claim 5, characterized in that: In the action selection step of Gaussian model optimization reinforcement learning, the reinforcement learning action a is the PD parameter, and the range of the PD parameter is set to k p ∈[k p 1 , k p 2 , ..., k p m ],k d ∈[k d 1 , k d 2 , ..., k d m ], action a randomly selects different parameters from these parameters, that is, a=[u1,u2,...,u T ], where u=(k p , k d ).

7. The robot force control method based on RBF neural network stiffness prediction and reinforcement learning according to claim 1, characterized in that: In the action selection step of Gaussian model-based optimization reinforcement learning, the selection probability of the P parameter in the PD parameter can be realized through the Gaussian model. In the formula, k p is the parameter in the reinforcement learning action a, k s is the predicted stiffness value, σ is the standard deviation, and exp is the exponential function. After the reinforcement learning action is offline, the probability of the value of different offset actions can be obtained according to the Gaussian model.

8. The robot force control method based on RBF neural network stiffness prediction and reinforcement learning according to claim 7, characterized in that: In the action selection step of Gaussian model optimization reinforcement learning, the probability of P parameter selection is expanded to the probability of action a. The discretized action a needs to be encoded, and the action a encoding [(k p 1 , k d 1 ), (k p 1 , k d 2 ), (k p 1 , k d 3 )...,(k d m , k d n-2 ), (k p m , k d m-1 ), (k p m , k d m )], corresponding to The initial selection parameter D is fixed each time, and the average value of the corresponding Gaussian model is set to k s The corresponding k p sequence Then the probability of action a is:

9. The robot force control method based on RBF neural network stiffness prediction and reinforcement learning according to claim 1, characterized in that: In the action selection step of reinforcement learning optimized based on the Gaussian model, the probability of using the ε-greedy strategy after a certain number of iterations in the early exploration of reinforcement learning is used to smoothly control the trade-off between exploration and exploitation.

Images