Intelligent impedance control method and system for robot arm based on deep reinforcement learning

Abstract

The application discloses a kind of mechanical arm intelligent impedance control method and system based on deep reinforcement learning, it is related to mechanical arm control technical field, method includes: based on the current state data set of the impedance model of rigid mechanical arm in uncertain environment, and then determine the current action of rigid mechanical arm, corresponding reward value, next state data set;The current state data set of rigid mechanical arm, current action, the reward value corresponding to current action and next state data set constitute experience four tuples;Select multiple experience four tuples as training sample from mechanical arm action experience pool to train Actor-Critic neural network in DDPG model, to obtain optimal Actor-Critic neural network, to determine the corresponding optimal action according to the state data set of rigid mechanical arm in uncertain environment.The application improves the force tracking control performance of mechanical arm in complex uncertain environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm control, in particular to a mechanical arm intelligent impedance control method and system based on deep reinforcement learning. BACKGROUND

[0002] At present, mechanical arms are widely used, and many production lines and life scenes can be seen as their epitome. The situation of mechanical arms performing contact tasks is becoming more and more common. For example, surgery, assembly, polishing, etc. Effective force control is required to perform such tasks, so that the mechanical arm is in a compliant state. However, the uncertainty factors of the environment and complex tasks often lead to difficulty in accurately implementing force tracking control, so improving the adaptability of the mechanical arm is an important measure to improve the force control performance of the mechanical arm.

[0003] Impedance control technology is a key method to realize mechanical arm force control and is widely used in some scenarios that have certain requirements for contact force control. However, due to the relatively simple structure of impedance control, it does not have adaptability to dynamically changing environments, and the impedance parameters need to be determined in advance, which is difficult to compensate for the influence of environmental uncertainty factors on the force tracking control performance of the mechanical arm. SUMMARY

[0004] The purpose of the present application is to provide a mechanical arm impedance control method and system based on deep reinforcement learning, which improves the force tracking control performance of the mechanical arm in complex uncertain environments.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A mechanical arm intelligent impedance control method based on deep reinforcement learning, comprising:

[0007] constructing an impedance model of a rigid mechanical arm in an uncertain environment;

[0008] Based on the impedance model of the rigid mechanical arm in the uncertain environment, the current state data set of the rigid mechanical arm is obtained; the current state data set is used to represent the dynamic mapping relationship between the change of the end position of the rigid mechanical arm and the contact force when the rigid mechanical arm contacts with the uncertain environment;

[0009] Based on the reinforcement learning algorithm and the preset mechanical arm action strategy, the current action of the rigid mechanical arm and the corresponding reward value are determined according to the current state data set of the rigid mechanical arm;

[0010] updating a current state data set of the rigid manipulator based on a current action of the rigid manipulator to obtain a next state data set; the current state data set of the rigid manipulator, the current action, a reward value corresponding to the current action, and the next state data set constitute an experience quadruple; a plurality of experience quadruples constitute a manipulator action experience pool;

[0011] selecting a plurality of experience quadruples from the manipulator action experience pool as training samples, training an Actor-Critic neural network in a DDPG model to obtain an optimal Actor-Critic neural network; the optimal Actor-Critic neural network is used to determine a corresponding optimal action according to a state data set of the rigid manipulator in an uncertain environment.

[0012] Optionally, the current state data set of the rigid manipulator is:

[0013] wherein S represents the current state data set of the rigid manipulator, f e represents the contact force of the rigid manipulator with the environment in the force constraint direction; represents the force tracking deviation amount of the rigid manipulator in the force constraint direction, the derivative of the force tracking deviation amount, and the integral of the force tracking deviation amount; represents the actual position and actual velocity of the rigid manipulator in the force constraint direction; represents the position deviation amount of the rigid manipulator in the force constraint direction and the derivative of the position deviation amount.

[0014] Optionally, selecting a plurality of experience quadruples from the manipulator action experience pool as training samples, training an Actor-Critic neural network in a DDPG model to obtain an optimal Actor-Critic neural network, specifically comprising:

[0015] constructing an Actor neural network and a Critic neural network;

[0016] inputting the current state data set of the rigid manipulator in any training sample into the Actor neural network to output a current predicted action; the current predicted action is used to control the rigid manipulator to interact with the uncertain environment to determine a corresponding next state data and a reward value;

[0017] inputting the next state data set corresponding to the current state data set of the rigid manipulator in the training sample and the current predicted action into the Critic neural network for learning, updating the Critic neural network parameters based on minimizing a loss function, and outputting a predicted action value function value;

[0018] updating the actor neural network parameters according to the predicted action value function value based on a deterministic gradient policy;

[0019] updating the actor neural network and the critic neural network respectively based on a plurality of training samples in a soft updating manner to obtain an optimal actor-critic neural network.

[0020] A mechanical arm intelligent impedance control system based on deep reinforcement learning, comprising:

[0021] An impedance model construction module is configured to construct an impedance model of a rigid mechanical arm in an uncertain environment.

[0022] A mechanical arm state acquisition module is configured to acquire a current state data set of the rigid mechanical arm based on the impedance model of the rigid mechanical arm in the uncertain environment, wherein the current state data set is used to represent a dynamic mapping relationship between a change in a position of an end of the rigid mechanical arm and a contact force when the rigid mechanical arm contacts the uncertain environment.

[0023] A mechanical arm action determination module is configured to determine a current action of the rigid mechanical arm and a corresponding reward value based on a reinforcement learning algorithm and a preset mechanical arm action strategy according to the current state data set of the rigid mechanical arm.

[0024] A mechanical arm action experience pool construction module is configured to update the current state data set of the rigid mechanical arm based on the current action of the rigid mechanical arm to obtain a next state data set, wherein the current state data set of the rigid mechanical arm, the current action, the reward value corresponding to the current action, and the next state data set constitute an experience four-tuple, and a plurality of experience four-tuples constitute a mechanical arm action experience pool.

[0025] An optimal action model construction module is configured to select a plurality of experience four-tuples as training samples from the mechanical arm action experience pool, train an actor-critic neural network in a DDPG model, and obtain an optimal actor-critic neural network, wherein the optimal actor-critic neural network is used to determine a corresponding optimal action according to a state data set of the rigid mechanical arm in the uncertain environment.

[0026] According to the embodiments of the present application, the following technical effects are achieved:

[0027] The application discloses a kind of mechanical arm intelligent impedance control method and system based on deep reinforcement learning, based on the impedance model of rigid mechanical arm in uncertain environment, the current state data set of rigid mechanical arm is determined, and then the current action, corresponding reward value and next state data are determined, to obtain experience four-tuple;Multiple experience four-tuples constitute mechanical arm action experience pool. Select multiple experience four-tuples as training samples from the mechanical arm action experience pool, train the Actor-Critic neural network in DDPG model to obtain the optimal Actor-Critic neural network, and the optimal Actor-Critic neural network determines the corresponding optimal action according to the state data set of rigid mechanical arm in uncertain environment. The application improves the mechanical arm impedance control strategy based on deep reinforcement learning, so that the mechanical arm intelligent impedance control can complete the force tracking process of mechanical arm with smaller overshoot, faster response speed and higher tracking accuracy, significantly improve the force tracking control performance of mechanical arm. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0029] Figure 1 The flowchart of the present application based on deep reinforcement learning of mechanical arm intelligent impedance control method is shown.

[0030] Figure 2 The block diagram of the present application intelligent impedance control is shown.

[0031] Figure 3 The Matlab simulation model of the present application IIC is shown.

[0032] Figure 4 The setting of the network structure of the present application IIC is shown.

[0033] Figure 5 The learning process of the present application using DDPG to solve intelligent impedance control strategy is shown.

[0034] Figure 6 The flowchart of the safety learning mechanism of the present application is shown.

[0035] Figure 7 The reward change curve of the present application IIC method in training process is shown.

[0036] Figure 8 The force tracking performance comparison of the present application when environmental stiffness suddenly changes is shown.

[0037] Figure 9 For the force tracking performance comparison of the mechanical arm in the slope surface environment of the application;

[0038] Figure 10 For the force tracking performance comparison of the mechanical arm in the curved surface environment of the application;

[0039] Figure 11 For the process schematic diagram of the intelligent impedance control system of the mechanical arm based on deep reinforcement learning of the application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0041] In a complex uncertain environment, reasonable impedance parameters need to be adjusted in real time to ensure good force control performance. To meet this demand, the intelligent impedance control method and system of the mechanical arm based on deep reinforcement learning of the application realize compliant force control of the mechanical arm in a complex uncertain environment and improve the dynamic and steady-state performance of force tracking control.

[0042] To make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the application will be further described in detail below with reference to the drawings and specific embodiments.

[0043] Embodiment one

[0044] As shown in Figure 1 The application provides an intelligent impedance control method of a mechanical arm based on deep reinforcement learning, which comprises the following steps:

[0045] Step 100, constructing an impedance model of a rigid mechanical arm in an uncertain environment.

[0046] Step 100 specifically comprises the following steps:

[0047] (1) In the Cartesian space, the dynamics model of the mechanical arm is as follows:

[0048]

[0049] Wherein, q, are the joint angle, joint angular velocity and joint angular acceleration of the mechanical arm respectively; D(q)∈R n×n is the inertia matrix of the mechanical arm, is the Coriolis force and centrifugal force matrix of the mechanical arm; G(q)∈R n×1 is the gravity vector of the mechanical arm, is the driving torque of the mechanical arm, and F eLet J(q) be the external force acting on the end effector of the robotic arm, and J(q) be the Jacobian matrix of the robotic arm.

[0050] (2) When the robotic arm comes into contact with an uncertain environment, under the condition of slow contact deformation motion without considering damping factors, the environmental dynamics model is established as follows:

[0051] F e =K e (X e -X); where K e It represents the environmental equivalent stiffness; X represents the actual position of the robotic arm; X e Indicates the location of the environment.

[0052] (3) The desired impedance control model is expressed as:

[0053] Among them, M d B d ,K d These are the desired inertia, damping, and stiffness matrices, respectively; E = X d -X and E f =F d -F e These are positional deviation and force deviation, respectively; X d F represents the desired position where the robotic arm contacts the environment. d This represents the desired contact force at the end of the robotic arm.

[0054] Considering the uncertainties in the actual contact environment, we assume an error in the environmental location estimation. Where, δX e To account for the estimation error caused by environmental location uncertainty, an impedance model considering the influence of uncertain environment is established as follows:

[0055] By decoupling the force from the direction of motion and considering only the contact force constraint in a single dimension, with the relevant variables in this single dimension represented by lowercase letters, the impedance model considering the influence of uncertain environment in the direction of the force constraint is established as follows:

[0056]

[0057] in, δx represents the position estimation deviation in the force constraint direction. e δf represents the estimation error caused by the uncertainty of environmental position and contact force. It can be seen that the parameters in an uncertain environment may be time-varying or discontinuous, which will ultimately affect the force control performance of the robotic arm.

[0058] (4) Optimize the impedance model to make the formula Represented as:

[0059]

[0060] where m0, b0are initial inertia parameter and initial damping parameter, respectively, and μ b and μ k are damping and stiffness adjustment parameters, respectively, thus it can be seen that when μ b and μ k are zero, it degenerates to traditional impedance control.

[0061] Further rearranging the formula , the impedance model of rigid manipulator in uncertain environment can be obtained as follows:

[0062]

[0063]

[0064]

[0065] where and are uncontrollable and controllable terms, respectively; e f represents the force tracking error of rigid manipulator in force constraint direction, δfrepresents the estimation error due to contact force uncertainty, represents the derivative of estimation error due to environment position uncertainty, represents the second derivative of estimation error due to environment position uncertainty, μ b and μ k represent damping adjustment parameter and stiffness adjustment parameter, respectively; represents the position estimation error of rigid manipulator in force constraint direction, e represents the position error of rigid manipulator, represents the second derivative of position error of rigid manipulator, represents the derivative of position estimation error of rigid manipulator in force constraint direction, δx e represents the estimation error due to environment position uncertainty. According to the impedance model of rigid manipulator in uncertain environment, the force tracking control in uncertain environment can be adapted to the change of dynamic unknown environment by adjusting parameters μ b and μ k , so that f2→f1, thus eliminating the influence of complex uncertain environment.

[0066] In step 200, a current state data set of the rigid manipulator is obtained based on an impedance model of the rigid manipulator in an uncertain environment; the current state data set is used to represent a dynamic mapping relationship between a change in a position of an end of the rigid manipulator and a contact force when the rigid manipulator contacts the uncertain environment.

[0067] To optimize the impedance parameter set, a force tracking control process of the manipulator is taken as a reinforcement learning framework based on Markov decision process (MDP) suitable for different environments, appropriate state and action values and a reward function are designed for an intelligent impedance control (IIC) method, and a safe learning mechanism is added to the IIC method. As shown in FIG. 2, an IIC block diagram is shown, P = diag [1 1 0] is a selection matrix of the IIC, and is used to realize decoupling of motion control and force control of the manipulator in different dimensional task spaces. Considering the dynamic mapping relationship between a change in a position of an end of the manipulator and a contact force when the manipulator contacts an unknown environment, the current state data set of the rigid manipulator is determined as follows: Figure 2

[0068] wherein S represents the current state data set of the rigid manipulator, f e represents a contact force of the rigid manipulator in a force constraint direction and the environment; represents a force tracking deviation amount of the rigid manipulator in the force constraint direction, a derivative of the force tracking deviation amount, and an integral of the force tracking deviation amount; represents an actual position and an actual velocity of the rigid manipulator in the force constraint direction; represents a position deviation amount and a derivative of the position deviation amount of the rigid manipulator in the force constraint direction.

[0069] In step 300, a current action of the rigid manipulator and a corresponding reward value are determined based on a reinforcement learning algorithm and a preset manipulator action strategy according to the current state data set of the rigid manipulator.

[0070] In a force control process of the manipulator, it is expected to adjust reasonable impedance parameters according to a real-time contact state to change a compliance of an end of the manipulator, so as to complete tracking of a desired contact force, thereby improving robustness and adaptability of the overall system. Based on this, the preset manipulator action strategy is determined as follows:

[0071] A = [μ b , μ k ]; wherein A represents the action of the rigid manipulator.

[0072] ​Since in the stable parameter range, if there is no constraint on the range of A, it cannot be guaranteed that the tracking effect in the desired force range will be significantly improved. Therefore, in order to ensure that the robot arm can effectively explore on A, thereby reducing the training time and control energy consumption, it is generally necessary to limit the exploration range of the intelligent agent.

[0073] In order to enable the intelligent agent to more efficiently explore the optimal impedance control strategy, a reward function is constructed and a safe learning mechanism is added. Specifically, the reward function is split into several sub-functions, i.e., the reward function is refined using the reward shaping method, and the calculation formula of the reward value corresponding to the current action of the rigid robot arm is:

[0074] reward = λ1r d + λ2r s + λ3r v . Wherein, reward represents the reward value corresponding to the current action of the rigid robot arm, r d is a dynamic reward; r s is a steady-state reward; r v is a task completion reward; λ j,j＝1,2,3 is an allocation weight, which is adjusted according to the balance of dynamic performance, steady-state performance and safety stability in the force tracking control process.

[0075]

[0076] Wherein, i = 1, 2, …, m, m is the maximum interval number set to subdivide the different rewards obtained in different intervals during the dynamic process, and the reward r d is mainly determined according to the dynamic performance of the force tracking control process, in order to reduce the force overshoot and oscillation generated during the dynamic contact process of the robot arm and the environment. According to the actual training effect, the force tracking deviation |e f | is taken as the benchmark for interval processing, i.e., |e f | represents the value of the force tracking deviation of the rigid robot arm in the force constraint direction. w i is the scaling coefficient of |e f | in each interval, which is obtained by dividing the value of the force tracking deviation |e f | as the benchmark. d i is a fixed reward value for reaching each interval, and w i is the scaling coefficient of |e f | in each interval.

[0077] r s = -ln(|e f | + η).

[0078] Wherein, |ef |The logarithm is mainly considered for the steady-state performance of the force control process. In the process of tracking the desired force, the smaller the force deviation is, the better the force tracking effect of the robot is, that is, the order of magnitude of the force tracking accuracy is improved, and the reward value is rapidly increased, so as to guide the agent to control the force tracking steady-state error to a smaller range. At the same time, η represents a preset constant value, and η is set to prevent the logarithmic function from tending to infinity when the logarithmic function tends to zero. The setting of η is related to the desired force tracking steady-state error.

[0079]

[0080] wherein, v1 and v2 represent a fixed reward value obtained by completing a task and a fixed reward value obtained by failing a task respectively, and the task represents that the rigid robot performs the current action in the uncertain environment.

[0081] It should be noted that r v The reward function can encourage the agent to complete the force control task and produce safe and stable behavior in the learning process. The safety learning mechanism is mainly used to verify whether the expected behavior of the robot at each time step is within a set of defined constraint ranges, so as to ensure the safety and stability of the learning process of the robot. As shown in Figure 6 The judgment of task completion or task failure in the task completion reward specifically includes:

[0082] (1) initialize the environment, and determine a preset joint angle range (i.e. not more than q max ), a preset contact force range (i.e. not more than f max ) and a maximum number of steps T for each task process. The preset contact force range is set to prevent the robot and the environment from generating a large impact force, causing potential damage to the robot and the environment.

[0083] (2) for each time step in each task process, obtain the position command x c corresponding to the current action.

[0084] (3) when the position command x c has an inverse solution, it is judged whether the joint angle q c of the rigid robot corresponding to the inverse solution of the position command is within the preset joint angle range.

[0085] (4) if the joint angle q c of the rigid robot is not within the preset joint angle range, it indicates that the task fails; if the joint angle q c of the rigid robot is within the preset joint angle range, it is judged whether the joint angle of the rigid robot corresponding to the inverse solution of the position command and the contact force of the uncertain environment are within the preset contact force range.

[0086] (5) If the joint angle of the rigid manipulator and the contact force of the uncertain environment are in the preset contact force range, it indicates that the task is completed; if the joint angle of the rigid manipulator and the contact force of the uncertain environment are not in the preset contact force range, it indicates that the task fails.

[0087] Step 400, updating the current state data set of the rigid manipulator based on the current action of the rigid manipulator to obtain a next state data set; the current state data set of the rigid manipulator, the current action, the reward value corresponding to the current action and the next state data set constitute an experience four-tuple; a plurality of experience four-tuples constitute a manipulator action experience pool.

[0088] Step 500, selecting a plurality of experience four-tuples from the manipulator action experience pool as training samples, training the Actor-Critic neural network in the DDPG model to obtain an optimal Actor-Critic neural network; the optimal Actor-Critic neural network is used to determine the corresponding optimal action according to the state data set of the rigid manipulator in the uncertain environment.

[0089] Specifically, the DRL (Deep Reinforcement Learning) is used to solve the complex calculation in the high-dimensional dynamic environment, and the DDPG (Deep Deterministic Policy Gradient) algorithm is used to solve the impedance control strategy of the IIC method. The DDPG belongs to a kind of DRL algorithm, which is based on the Actor-Critic architecture, and has the characteristics of value function update and policy update, has certain advantages in solving the problem of continuous action space, and is suitable for the force tracking control problem of the manipulator. When the DDPG algorithm is used to solve the impedance control strategy of the IIC method, the Actor neural network corresponds to the current policy network μ(s t |θ μ ) and the target policy network μ'(s t |θ μ′ ), the Critic neural network corresponds to the current value network Q(s t ,a t |θ Q ) and the target value network Q'(s t ,a t |θ Q′ ), the Actor neural network and the Critic neural network respectively contain two neural networks with the same structure, θ Q , θ Q′ , θ μ , θ μ′ are respectively the parameters of each network.

[0090] DDPG in the application utilizes feature extraction of neural network to optimize policy and value function, and through continuous updating and iteration, finally learns a set of weight parameters to define policy and value network. Figure 4 As shown in the figure, the Actor neural network (corresponding to Actor Net in Figure 4 ) and the Critic neural network (corresponding to Critic Net in Figure 4 ) both adopt fully connected neural network, and each contains 2 and 4 hidden layers respectively, each of which contains 64 neurons. The input layer unit of the Actor neural network is determined by the current state s t , and the output layer adopts tanh activation function, so that the action output of the agent can be within a given range by adding bias; the input layer of the Critic neural network is determined by the state s t and the action a t , receives the current action a t in the third hidden layer, and sums it with the extracted s t features, and the output layer adopts ReLU activation function.

[0091] Step 500, specifically comprising:

[0092] (1) constructing the Actor neural network and the Critic neural network; specifically, initializing the parameters of μ(s t |θ μ ) and Q(s t ,a t |θ Q ) contained in the current network and the experience pool M, and copying the parameters of the current network to μ'(s t |θ μ′ ) and Q'(s t ,a t |θ Q′ ) in the target network. At the same time, the termination condition is also set according to the safety learning mechanism added by the IIC method.

[0093] (2) inputting the current state data set of the rigid manipulator in any training sample into the Actor neural network to output the current predicted action; the current predicted action is used to control the rigid manipulator to interact with the uncertain environment to determine the corresponding next state data and reward value.

[0094] (3) inputting the next state data set corresponding to the current state data set of the rigid manipulator in the training sample and the current predicted action into the Critic neural network for learning, updating the Critic neural network parameters based on minimizing the loss function, and outputting the predicted action value function value.

[0095] Specifically, a plurality of state transition data (s i , a i , r i , s i+1 ) are randomly selected from the experience pool M, s i , a i and r i are respectively the action, state and reward value at the i-th moment, s i and s i+1 are taken as inputs of the Actor, and the outputs are μ(s i |θ μ ) and μ'(s i+1 |θ μ′ ) which are used to calculate the action value function of the Critic. The next state s i+1 and μ'(s i+1 |θ μ′ ) are taken as inputs of Q'(s t , a t |θ Q′ ) to calculate the temporal difference (Temporal-Difference) target y i :

[0096] y i = r i + γQ'(s i+1 , μ'(s i+1 |θ μ′ ) | θ Q′ ).

[0097] The parameters θ Q of the current value network Q(s t , a t | θ Q ) are updated according to the minimization of the loss function L(θ Q ), that is, the loss function is:

[0098]

[0099] wherein L(θ Q ) represents the value of the minimized loss function, N represents the number of training samples, y i represents the temporal difference target, Q(s i , a i | θ Q ) represents the Critic neural network, θ Q represents the parameters of the Critic neural network, s i represents the i-th state input to the Critic neural network, and a i represents the predicted action corresponding to the i-th state input to the Critic neural network; ri represents the reward value of the action corresponding to the i-th state, and γ represents a discount factor, s i+1 represents the next state corresponding to the current predicted action, μ'(s i+1 | θ μ′ ) represents the action output of the Actor neural network updated according to the next state corresponding to the current predicted action, Q'(s i+1 , μ'(s i+1 | θ μ′ )| θ Q′ ) represents the value output of the Critic neural network corresponding to the next state corresponding to the current predicted action.

[0100] (4) Based on the deterministic gradient policy, the Actor neural network parameters are updated according to the predicted action value function value, specifically according to the formula:

[0101] The Actor neural network parameters are updated; wherein θ μ represents the parameters of the Actor neural network μ(s t | θ μ ), N represents the number of training samples, represents the current policy gradient direction, represents the change of Q value caused by the current action under the current state, to calculate the policy gradient to determine the update direction of θ μ . From the above formula, it can be seen that the deterministic policy gradient is mainly related to the policy gradient and the value function.

[0102] (5) Based on multiple training samples, the Actor neural network and the Critic neural network are updated in a soft update manner to obtain the optimal Actor-Critic neural network.

[0103] Wherein, the function formula for updating the parameters θ Q′ and θ μ′ of the target network in a soft update manner is:

[0104]

[0105] Wherein, τ is a smoothing factor, generally τ << 1, usually 0.001.

[0106] In one specific embodiment, in order to ensure that the system can output the optimal impedance control strategy, thereby improving the force control performance of the robot arm, the related parameters involved in the IIC method can be adjusted: 1) Set the adaptive change range of the impedance parameter set, thereby reducing the training time and control energy consumption, and set λ in the reward function according to the training effecti and w i 1) Configure relevant parameters; 2) Configure the Actor neural network and Critic neural network, and optimize the key DDPG parameters, including neural network parameters, noise parameters, maximum number of steps per round, buffer pool size, discount factor and sampling time, so as to adjust the intelligent impedance controller and thus ensure the good force tracking control performance of the robotic arm.

[0107] To verify the effectiveness of this invention in improving the force tracking control performance of robotic arms, such as Figure 3 As shown, this invention uses the MATLAB / Simulink platform to build a simulation model of the intelligent impedance controller; the structures of the designed Actor network and Critic network are as follows. Figure 4 As shown.

[0108] Using the EFORT robotic arm as the simulation object, the learning process of solving the intelligent impedance control strategy using DDPG is as follows: Figure 5 As shown in Table 1. In the experiment, the desired contact force was set to 10 N. Other relevant experimental parameters were set as shown in Table 1.

[0109] Table 1 Experimental Parameter Settings

[0110]

[0111] During training, the maximum number of training rounds was set to 300, and the maximum number of steps per round was T. s The ` / t` parameter uses the `scaling` function from the Matlab library to suggest limits on the action space range, reducing unnecessary exploration time. To ensure the safe and stable behavior of the intelligent impedance controller during the learning process, a safety learning mechanism flowchart is provided as follows: Figure 6 As shown. Ultimately, the reward change curve of the IIC method during training can be obtained as follows: Figure 7 As shown, it can be seen that as the number of training iterations increases, the reward value of IIC gradually increases. In the early stage of learning, IIC is in the policy exploration stage, and it seeks a better impedance control policy through continuous trial and error. Therefore, the cumulative reward value changes significantly in the early stage. After 210 rounds, the cumulative reward curve shows a convergence trend, the change in reward value becomes smaller and tends to be flat, indicating that the training model has reached convergence and IIC has found the optimal impedance control policy.

[0112] By placing the trained robotic arm in three complex and uncertain environments—abrupt changes in environmental stiffness, inclined plane environment, and curved surface environment—the force tracking performance of the IIC method can be compared with that of traditional impedance control (CIC) and adaptive variable impedance control (VIC). Figure 8 , Figure 9 and Figure 10As shown in the three graphs, the horizontal coordinate is time t, the unit is s, and the vertical coordinate is force fe, the unit is N. It can be seen that the force tracking steady-state performance of CIC and the force tracking dynamic performance of VIC are not satisfactory, compared with CIC and VIC, IIC can complete the force tracking process of the robot arm with smaller overshoot, faster response speed and higher tracking accuracy, and the research results show that the IIC method significantly improves the force control performance of the robot arm.

[0113] In summary, the IIC method significantly improves the force control performance of the robot arm.

[0114] Embodiment two

[0115] As Figure 11 shown, the application also provides a robot arm intelligent impedance control system based on deep reinforcement learning, comprising:

[0116] The impedance model construction module 101 is used to construct the impedance model of the rigid robot arm in the uncertain environment.

[0117] The robot arm state acquisition module 201 is used to acquire the current state data set of the rigid robot arm based on the impedance model of the rigid robot arm in the uncertain environment; the current state data set is used to represent the dynamic mapping relationship between the change of the end position of the rigid robot arm and the contact force when the rigid robot arm contacts with the uncertain environment.

[0118] The robot arm action determination module 301 is used to determine the current action of the rigid robot arm and the corresponding reward value according to the current state data set of the rigid robot arm based on the reinforcement learning algorithm and the preset robot arm action strategy.

[0119] The mechanical arm action experience pool construction module 401 is configured to update a current state data set of the rigid mechanical arm based on a current action of the rigid mechanical arm to obtain a next state data set; the current state data set of the rigid mechanical arm, the current action, a reward value corresponding to the current action, and the next state data set constitute an experience four-tuple; and a plurality of experience four-tuples constitute a mechanical arm action experience pool.

[0120] The optimal action model construction module 501 is configured to select a plurality of experience four-tuples from the mechanical arm action experience pool as training samples, train an Actor-Critic neural network in a DDPG model to obtain an optimal Actor-Critic neural network, and use the optimal Actor-Critic neural network to determine an optimal action corresponding to a state data set of the rigid mechanical arm in an uncertain environment.

[0121] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, the description is relatively simple because it corresponds to the method disclosed in the embodiments. The relevant parts can be referred to the description of the method.

[0122] The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. For those skilled in the art, the specific implementation manners and application scope can be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for intelligent impedance control of a robotic arm based on deep reinforcement learning, characterized in that, The methods include: Construct an impedance model for a rigid robotic arm under uncertain conditions; Based on the impedance model of the rigid manipulator in an uncertain environment, the current state dataset of the rigid manipulator is obtained; the current state dataset is used to characterize the dynamic mapping relationship between the change of the end position of the rigid manipulator and the contact force when the rigid manipulator comes into contact with the uncertain environment. The impedance model of the rigid robotic arm under uncertain conditions is as follows: ； ； ； in, These represent the initial inertial parameters and the initial damping parameters, respectively. e f This represents the force tracking deviation of the rigid robotic arm in the direction of the force constraint. This indicates the estimation error caused by the uncertainty of the contact force. The derivative of the estimation error caused by environmental location uncertainty. This represents the second derivative of the estimation error caused by environmental location uncertainty. and These represent the damping adjustment parameters and the stiffness adjustment parameters, respectively. The value 'e' represents the position estimation deviation of the rigid robotic arm in the direction of the force constraint, where 'e' represents the amount of position deviation of the rigid robotic arm. The second derivative of the positional deviation of a rigid robotic arm. The derivative representing the position estimation deviation of the rigid robotic arm in the direction of the force constraint. This indicates the estimation error caused by uncertainty in the environmental location; The preset robotic arm motion strategy is as follows: ;in, A This indicates the movement of a rigid robotic arm; Based on reinforcement learning algorithms and preset robotic arm action strategies, the current action and corresponding reward value of the rigid robotic arm are determined according to the current state dataset of the rigid robotic arm. The formula for calculating the reward value corresponding to the current action of the rigid robotic arm is as follows: ； ； ； ； in, reward This represents the reward value corresponding to the current action of the rigid robotic arm. Dynamic rewards; For steady-state reward; Rewards for completing the task; To assign weights; , The maximum number of intervals is set. A fixed reward value for reaching each interval, This represents the value of the force tracking deviation of the rigid robotic arm in the direction of the force constraint. For each interval pair The scaling factor, the range of which is the value of the force tracking deviation. Obtained by dividing based on the baseline; This represents a preset constant value. and These represent the fixed reward value obtained upon task completion and the fixed reward value obtained upon task failure, respectively, where the task refers to the rigid robotic arm performing the current action under the uncertain environment; The current state dataset of the rigid robotic arm is updated based on the current action of the rigid robotic arm to obtain the next state dataset; the current state dataset of the rigid robotic arm, the current action, the reward value corresponding to the current action, and the next state dataset constitute an experience quadruple; multiple experience quadruples constitute the robotic arm action experience pool. Multiple experience quadruples are selected from the robotic arm action experience pool as training samples to train the Actor-Critic neural network in the DDPG model to obtain the optimal Actor-Critic neural network; the optimal Actor-Critic neural network is used to determine the corresponding optimal action based on the state dataset of the rigid robotic arm in an uncertain environment.

2. The intelligent impedance control method for robotic arms based on deep reinforcement learning according to claim 1, characterized in that, The current state dataset of the rigid robotic arm is as follows: ; in, S This represents the current state dataset of a rigid robotic arm. This represents the contact force between the rigid robotic arm and the environment in the direction of the force constraint. This represents the force tracking deviation of the rigid robotic arm in the force constraint direction, the derivative of the force tracking deviation, and the integral of the force tracking deviation. This indicates the actual position and actual velocity of the rigid robotic arm in the direction of the force constraint. This represents the positional deviation of the rigid robotic arm in the direction of the force constraint and the derivative of the positional deviation.

3. The intelligent impedance control method for robotic arms based on deep reinforcement learning according to claim 1, characterized in that, The determination of whether a task is completed or failed in the task completion reward specifically includes: For each time step in the task process, obtain the position command corresponding to the current action; When the position command has an inverse solution, determine whether the joint angle of the rigid robotic arm corresponding to the inverse solution of the position command is within the preset joint angle range; If the joint angle of the rigid robotic arm is not within the preset joint angle range, the task will fail. If the joint angle of the rigid robotic arm is within the preset joint angle range, then determine whether the joint angle of the rigid robotic arm corresponding to the inverse solution of the position command and the contact force with the uncertain environment are within the preset contact force range. If the joint angle of the rigid robotic arm and the contact force with the uncertain environment are within the preset contact force range, the task is considered complete. If the joint angle of the rigid robotic arm and the contact force with the uncertain environment are not within the preset contact force range, the task will fail.

4. The intelligent impedance control method for robotic arms based on deep reinforcement learning according to claim 1, characterized in that, Multiple experience quadruples are selected from the robotic arm's motion experience pool as training samples to train the Actor-Critic neural network in the DDPG model, in order to obtain the optimal Actor-Critic neural network. Specifically, this includes: Construct Actor neural networks and Critic neural networks; The current state dataset of the rigid robotic arm in any training sample is input into the Actor neural network to output the current predicted action; the current predicted action is used to control the rigid robotic arm to interact with the uncertain environment in order to determine the corresponding next state data and reward value. The next state dataset corresponding to the current state dataset of the rigid robotic arm in the training samples and the current predicted action are input into the Critic neural network for learning. The parameters of the Critic neural network are updated based on minimizing the loss function, and the predicted action value function value is output. Based on a deterministic gradient strategy, the parameters of the Actor neural network are updated according to the predicted action value function value; Based on multiple training samples, a soft update method is used to update the parameters of the Actor neural network and the Critic neural network respectively to obtain the optimal Actor-Critic neural network.

5. The intelligent impedance control method for robotic arms based on deep reinforcement learning according to claim 4, characterized in that, The minimized loss function is: ； ； in, This represents minimizing the value of the loss function, where N represents the number of training samples. y i Represents the temporal difference objective. This represents a Critic neural network. This represents the parameters of the Critic neural network. This represents the i-th state input to the Critic neural network. This represents the predicted action corresponding to the i-th state input into the Critic neural network; r i This represents the reward value for the action corresponding to state i. Indicates the discount factor. This indicates the next state corresponding to the current predicted action. This represents the action output of the Actor neural network after updating the next state corresponding to the current predicted action. This represents the value output of the Critic neural network corresponding to the next state of the current predicted action.

6. The intelligent impedance control method for robotic arms based on deep reinforcement learning according to claim 4, characterized in that, Based on a deterministic gradient strategy, the parameters of the Actor neural network are updated according to the predicted action value function, specifically including: According to the formula ； Update the parameters of the Actor neural network; in, Represents an Actor neural network The parameter N represents the number of training samples. Indicates the current policy gradient direction. This represents the change in Q-value caused by the current action in the current state. Used to calculate the policy gradient to determine The direction of updates.

7. A deep reinforcement learning-based intelligent impedance control system for a robotic arm, used to implement the deep reinforcement learning-based intelligent impedance control method for a robotic arm as described in any one of claims 1-6, characterized in that, The system includes: Impedance model building module, used to build impedance models of rigid robotic arms under uncertain environments; The robotic arm state acquisition module is used to acquire the current state dataset of the rigid robotic arm based on the impedance model of the rigid robotic arm in an uncertain environment; the current state dataset is used to characterize the dynamic mapping relationship between the change of the end position of the rigid robotic arm and the contact force when the rigid robotic arm is in contact with the uncertain environment. The robotic arm motion determination module is used to determine the current motion of the rigid robotic arm and the corresponding reward value based on the current state dataset of the rigid robotic arm, using a reinforcement learning algorithm and a preset robotic arm motion strategy. The robotic arm motion experience pool construction module is used to update the current state dataset of the rigid robotic arm based on the current motion of the rigid robotic arm to obtain the next state dataset; the current state dataset of the rigid robotic arm, the current motion, the reward value corresponding to the current motion, and the next state dataset constitute an experience quadruple; multiple experience quadruples constitute the robotic arm motion experience pool. The optimal action model construction module is used to select multiple experience quadruples from the robotic arm action experience pool as training samples to train the Actor-Critic neural network in the DDPG model to obtain the optimal Actor-Critic neural network; the optimal Actor-Critic neural network is used to determine the corresponding optimal action based on the state dataset of the rigid robotic arm in an uncertain environment.

Images