A reward function and a vibration suppression reinforcement learning algorithm using the same

Abstract

The present invention discloses a reward function of a reinforcement learning method, including a position-related reward function r(Δθ1), a penalty r(σ) for overshoot σ, a reward r(Δθ m ) and the penalty r(M in ), and also disclosed a vibration suppression reinforcement learning algorithm, which converts the control performance indicators of servo drive control into a reward function in reinforcement learning, and realizes the application of reinforcement learning in the field of servo drive control. Compared with the traditional PID control algorithm combined with the reference trajectory input shaping method for vibration suppression, it greatly reduces the accuracy requirements for the product's dynamic model and reduces the calibration cost of product parameter calibration, thereby creating a new idea for a control method.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning, and in particular to a reward function and a vibration suppression reinforcement learning algorithm using the function. Background Art

[0002] Currently, conventional servo controllers use traditional PID control algorithms combined with reference trajectory input shaping for vibration suppression, which requires a relatively accurate known dynamic model of the product. First, a precise dynamic model of the controlled object must be constructed, a step supported by years of experimental data. Secondly, high product consistency is required during the manufacturing, processing, and assembly processes, placing high demands on the control accuracy of the machine tools used in manufacturing. In real-world applications, individual model identification is also required for each specific product. This step requires calibration personnel to perform product parameter calibration, resulting in high calibration costs.

[0003] Modern artificial intelligence (AI) technology is developing rapidly, already having a profound impact on the application of technology and ushering in a new era of technological revolution. Among these, reinforcement learning, which offers the greatest potential for universal solutions, is a leader. It has been applied in fields such as Go, video games, autonomous driving, biomedical design, health prediction, and robotic control.

[0004] At present, reinforcement learning is rarely used in the field of servo drive control, and its application in vibration suppression is even more blank. If reinforcement learning is applied to the field of servo drive control, a new idea of ​​control method can be created. After successful application, it can bring a series of advantages of reinforcement learning, such as solving the problems of various traditional control methods mentioned in the first paragraph in simulation. However, there is currently no reference material on how to convert the control performance indicators of servo drive control into reward functions in reinforcement learning. This is also a stumbling block for reinforcement learning in basic industrial applications. The present invention focuses on the problem of reward function design, that is, how to convert the control application requirements of servo drive vibration suppression in real scenes into learning goals in reinforcement learning, thereby building an effective way for the application of reinforcement learning in this field. Summary of the Invention

[0005] The purpose of the present invention is to provide a reward function and a servo driver vibration suppression reinforcement learning algorithm using this method, so as to solve the problem raised in the above background technology of how to convert the control performance index of servo drive into a reward function in reinforcement learning, and apply reinforcement learning to the field of servo drive, thereby creating a new idea for a control method.

[0006] To achieve the above object, the present invention provides the following technical solution: a reward function of a reinforcement learning method, including four sub-functions, defining the overshoot as σ = θ l_max -θ l_cmdθ l_max is the maximum value reached by the load during the movement, θ l_cmd is the expected load command value; the total reward function is the accumulation of four sub-functions, namely:

[0007] r(s,a)=w1r(Δθ l )+w2r(σ)+w3r(Δθ m )+w4r(M in )

[0008] Where w i represents the weight of the i-th reward function, where

[0009] r(Δθ l ) is a position-related reward function, which uses a combination of exponential decay and linear function, that is, the linear function is followed by exponential decay. Its function form is as follows:

[0010]

[0011] r(σ) is the penalty for overshoot σ, and its function form is as follows:

[0012]

[0013] r(Δθ m ) is the motor position reward, which is in the same form as the load end position reward;

[0014] r(M in ) is the penalty of input torque, and its function form is as follows:

[0015] r(M in )=-|M in |*r(Δθ).

[0016] A vibration suppression reinforcement learning algorithm, using the above-mentioned reward function, is characterized by further comprising the following steps:

[0017] Step 1: Mathematical Modeling

[0018] Build a linear discretized mathematical model of the controlled object. According to Newton's second law and torsional Hooke's law, the robot's single-joint motor load model is abstracted into a mechanical model connected by two rigid bodies, two revolute pairs, and a torsion spring. The dynamic response equation and deformation equation are established respectively. The dynamic response equation and deformation equation are:

[0019]

[0020] Among them, the physical quantity M represents torque, J represents moment of inertia, α represents angular acceleration, B represents rotational damping, ω represents angular velocity, θ represents angle, and the subscripts of the physical quantities represent the corresponding objects, where the subscript m represents the motor, ml represents the coupling, and l represents the load. In particular, M in Represents the input torque, K ml Indicates the equivalent torsional stiffness of the coupling;

[0021] Step 2: Build an MDP-based reinforcement learning model

[0022] Define state S t =(θ m ,θ · m ),θ m is the position of the motor high-speed end, θ · m is the speed of the motor high speed end, action a t is the input torque M in , the return r t is the reward obtained at the current step; the trajectory of a single sampling is defined as τ=(s1,a1,r1,…s t ,a t ,r t ,…,s T ,a T ,r T ), the probability of its occurrence is:

[0023]

[0024] The return of this trajectory is defined as:

[0025]

[0026] Where γ is the discount factor, γ∈(0,1],

[0027] The goal of reinforcement learning is to maximize long-term benefits, that is, to maximize expected returns, which are defined as That is, to find the parameter θ of the action strategy network, and to sample the trajectory obtained by the action strategy under the parameter, the expected return value is maximized, so the optimization problem becomes

[0028]

[0029] The most important function in the above optimization problem is r(s t ,a t ), which is the reward function of the current state action, which determines the learning goal of reinforcement learning and determines whether the training process can converge;

[0030] Step 3: Design of reward function

[0031] Define overshoot as σ = θ l_max -θ l_cmd ,θ l_max is the maximum value reached by the load during the movement, θ l_cmd is the expected load command value; the total reward function is the accumulation of four components, namely:

[0032] r(s,a)=w1r(Δθ l )+w2r(σ)+w3r(Δθ m )+w4r(M in )

[0033] Where w i represents the weight of the i-th reward function, where

[0034] r(Δθ l ) Position-dependent reward function, which uses a combination of exponential decay and linear function, that is, the linear function is followed by exponential decay, and its function form is as follows:

[0035]

[0036] r(σ) is the penalty for overshoot σ, and its function form is as follows:

[0037]

[0038] r(Δθ m ) is the motor position reward, which is in the same form as the load end position reward;

[0039] r(M in ) is the penalty of input torque, and its function form is as follows:

[0040] r(M in )=-|M in |*r(Δθ);

[0041] Step 4: Reinforcement learning model training;

[0042] Use the SAC algorithm or PPO algorithm in reinforcement learning to solve the optimization problem;

[0043] Step 5: Adjust the reward function design based on the SAC or PPO algorithm training results;

[0044] Step 4: After training the action strategy network, apply it to the motor model to check whether the trajectory characteristics meet the control performance requirements of the servo drive. If so, the entire process ends and the obtained action strategy network is the control strategy that meets the control requirements. If not, the reward function design needs to be adjusted and the SAC algorithm is used to train the process again until the requirements are met.

[0045] Step 6: Use the trained policy network to control the motor model.

[0046] Preferably, in step 4, the SAC algorithm includes the following steps:

[0047] The SAC algorithm consists of the following steps:

[0048] 1) Initialize the strategy π network parameters θ and the Q function network parameters φ 1，2 , initialize the experience pool Assign the Q parameter to the target Q network parameter φ targ，1 ←φ1,φ targ，2 ←φ2;

[0049] 2) Observe the state s and select actions a~π from the strategy θ (·|s), execute the action in the environment, observe the state s', benefit r, and end mark d at the next moment, and store the state transition pair (s, a, r, s', d) in the experience pool In , if s' is the end state, reset the environment;

[0050] 3) After collecting K state pairs, perform the following update steps 4) to 8) L times:

[0051] 4) From the experience pool Randomly collect a set of state transition pairs to form B = {(s, a, r, s′, d)},

[0052] 5) Calculate the target value of the Q function:

[0053]

[0054] 6) Use single-step gradient descent to update the weight parameters of the Q function, and its gradient value is:

[0055]

[0056] 7) Use single-step gradient ascent to update the weight parameters of the policy network, and its gradient value is

[0057]

[0058] in, From π θ(·|s) is the sampled value, which is made differentiable with respect to θ through the reparameterization technique.

[0059] 8) Update target network parameters

[0060] φ tar，i ←ρφ tar，i +(1-ρ)φ i , for i=1,2

[0061] 10) Repeat steps 2) to 8) until convergence.

[0062] Preferably, in step 4, the PPO algorithm includes the following steps:

[0063] 1) Define the Actor action strategy network π θ (a t |s t ), representing the state s t a t Probability distribution of Critic state value evaluation network Represents state s t The state value under the condition, Actor outputs At, Critic outputs V(St); The ultimate optimization goal of reinforcement learning is The strategy goal of using SGD to optimize the PPO algorithm is:

[0064]

[0065]

[0066] Among them, ε is a parameter coefficient, which represents the difference between the new action strategy and the old action strategy. is the advantage function under state (s,a);

[0067] 2) Initialize the action strategy network π θ (a t |s t ), State Value Assessment Network The network parameters in ;

[0068] 3) Using multi-threaded parallel technology, each thread uses domain randomization to set J l 、B ml , K ml ,θ tar , determine the simulation environment used by the current thread, and set N environments in total;

[0069] 4) In each thread, starting from the initial state s0, the action policy network π θ (a t |s t) continuously according to the current state s t Select Select action a t (Use normal mapping to maintain the exploratory nature of the action), act on the environment to obtain the state s at the next moment t+1 , and the process is repeated until the trajectory in the thread ends; forming a single sampling trajectory τ=(s0,a0,r0,s1,a1,r1,…s t ,a t ,r t ,…,s T ,a T ,r T ), a total of N trajectories of length T are collected;

[0070] 5) Combine the trajectories generated in each thread into (s t ,a t ,r t ) experience pool; for each state s in the experience pool t , calculate the state value of the current state

[0071] 6) Based on the data in the experience pool and the current state value function Estimate each state-action pair (s t ,a t )

[0072] 7) Update the strategy by maximizing the PPO-Clip target. The gradient direction of the strategy optimization is

[0073]

[0074] Where i represents the i-th trajectory, a total of N trajectories are sampled, and t represents the current time

[0075] Update the action policy network π using policy gradient ascent θ (a t |s t ) to optimize the network parameters

[0076]

[0077] Where α is the update step size, which is obtained using the Adam algorithm;

[0078] 8) According to the principle of minimum mean square error, the state value function is regressed and fitted, and its goal is defined as

[0079]

[0080] V(s t) to obtain the new true state value and use the gradient descent method to update the state value network The network parameters in

[0081] 9) Repeat the process from 3) to 8) until the training times meet certain conditions and the training is terminated.

[0082] Compared with the prior art, the present invention has the following beneficial effects:

[0083] A reward function for reinforcement learning is designed, including the position-dependent reward function r(Δθ l ) is the penalty r(σ) for overshoot σ, the reward r(Δθ) for motor position m ) and the penalty r(M in ), and at the same time, the control performance indicators of the servo drive are converted into the reward function in reinforcement learning, realizing the application of reinforcement learning in the field of servo drive. Compared with the traditional PID control algorithm + reference trajectory input shaping method for vibration suppression, it greatly reduces the accuracy requirements of the product's dynamic model and reduces the calibration cost of product parameter calibration, thereby creating a new idea for control method. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 This is a flow chart of a vibration suppression reinforcement learning algorithm of the present invention;

[0085] Figure 2 It is the interactive process of MDP-based reinforcement learning;

[0086] Figure 3 It is a schematic diagram of the MDP process;

[0087] Figure 4 There are three forms of reward functions related to target values;

[0088] Figure 5 are the forms of the reward functions for the three penalties for overshoot σ;

[0089] Figure 6 The action strategy network structure of the model training algorithm;

[0090] Figure 7 The state-value network structure of the model training algorithm;

[0091] Figure 8 The overall flow chart for adjusting the reward function based on SAC training results;

[0092] Figure 9 Flowchart of the overall solution of the SAC algorithm;

[0093] Figure 10Flowchart of the overall solution of the PPO algorithm;

[0094] Figure 11 Figure 2 shows the vibration suppression control effect obtained by applying the RL generalization method to the motor model under nominal conditions.

[0095] Figure 12 Applied to system parameter J l 、B ml , K ml The vibration suppression control effect diagram obtained when taking a smaller value.

[0096] Figure 13 Applied to system parameter J l 、B ml , K ml The vibration suppression control effect diagram obtained when taking a larger value. DETAILED DESCRIPTION

[0097] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0098] See also Figure 1 and Figure 8 The present invention provides a technical solution: This embodiment designs a reward function for vibration suppression control of a servo driver based on reinforcement learning. This step is a key step in using reinforcement learning to solve actual application problems of industrial servo products. It is a link between building real application scenario problems and reinforcement learning problems. By iteratively trying, selecting and improving during the learning process, a reward function that meets the needs of real applications is finally built in reinforcement learning. Specific implementation method:

[0100] Step 1: Build a linear discretization mathematical model of the controlled object

[0101] Specific steps:

[0102] According to Newton's second law and torsional Hooke's law, the robot's single-joint motor load model is abstracted into a mechanical model connected by two rigid bodies, two revolute pairs, and a torsion spring. The dynamic response equation and deformation equation are established respectively. The dynamic response equation and deformation equation are:

[0103]

[0104] Among them, the physical quantity M represents torque, J represents moment of inertia, α represents angular acceleration, B represents rotational damping, ω represents angular velocity, θ represents angle, and the subscripts of the physical quantities represent the corresponding objects, where subscript m represents motor, ml represents coupling, and l represents load. In particular, M in Represents the input torque, K ml Indicates the equivalent torsional stiffness of the coupling.

[0105] Step 2: Build MDP-based reinforcement learning

[0106] The idealized mathematical form of reinforcement learning is the finite Markov decision process (MDP), which is a theoretical framework for achieving goals through interactive learning. The interactive process is as follows: Figure 2 As shown, the essence of reinforcement learning is to learn action strategy π θ (a t |s t ), which is based on the current state s t Take action t action t Acting on the motor model, the model transfers p(s t+1 |s t ,a t ) Update the state s at the next moment t+1 In this embodiment, the state S t =(θ m ,θ · m ),θ m is the position of the motor high-speed end, θ · m is the speed of the motor high speed end. t is the input torque M in . Revenue r t This is the reward obtained in the current step. The specific design is shown in the third step.

[0107] The MDP process is as follows Figure 3 As shown. Each small triangle represents Figure 1 The interactive process in the play continues until the end of the play.

[0108] The trajectory of a single sampling is defined as τ=(s1,a1,r1,…s t ,a t ,r t ,…,s T ,a T ,r T ), the probability of its occurrence is

[0109]

[0110] The return of this trajectory is defined as:

[0111] Where γ is the discount factor, γ∈(0,1].

[0112] The goal of reinforcement learning is to maximize long-term benefits, that is, to maximize expected returns, which are defined as That is, find the parameter θ of the action strategy network, and sample the trajectory obtained according to the action strategy under the parameter, so the expected return value is maximized. So the optimization problem becomes

[0113]

[0114] The most important function in the above optimization problem is r(s t ,a t ), also known as the reward function for the current state-action, determines the learning goal of reinforcement learning and whether the training process can converge. The next step will focus on the design of the reward function.

[0115] Step 3: Design of reward function

[0116] The goal of the servo driver in the motor vibration suppression control problem is to quickly and smoothly reach the target position from the current position, with the overshoot less than a certain value δ, and to be stable and oscillate-free after reaching the target position, requiring very high control accuracy.

[0117] Here the overshoot is defined as σ = θ l_max -θ l_cmd θ l_max is the maximum value reached by the load during the movement, θ l_cmd is the desired load command value. l The value of cannot be obtained, which is an under-driven system, so conventional control is ineffective. Generally, a large overshoot will occur when reaching the target position and the system will oscillate back and forth.

[0118] First, the reward function should reflect the most direct goal, which is to control the motor to reach the target value θ l_cmd , then the main composition of the designed reward function is related to θ l_cmd Related reward functions, such as Figure 4 As shown, this embodiment proposes the following three forms of reward functions related to the target value:

[0119] Each type of function can have multiple hyperparameter designs. In order to meet the requirements of ultra-high control accuracy in vibration suppression, l -θ l_cmd|It is more reasonable to give a higher reward when the value is lower; the position-related reward function finally used in this embodiment is a combination of exponential decay + linear function, that is, a linear function followed by exponential decay. Its function form is as follows:

[0120]

[0121] The second main component of the reward function is the penalty for overshoot σ. For overshoot penalty, Figure 5 As shown, this embodiment also uses three basic forms of reward functions:

[0122] However, because the overshoot reward is a penalty term, the curve in the above figure needs to be shifted below the horizontal axis. When the overshoot is small, a linear function is used, and when the overshoot is large, a logarithmic decay is used, that is, the overshoot is forcibly suppressed. The function form is as follows:

[0123]

[0124] The third component of the reward function is the reward for the motor position, which has the same form as the position reward for the load end.

[0125] The fourth component of the reward function is the penalty for input torque. This penalty term can reduce energy loss and, after reaching the target value, suppress current oscillations and reduce damage to the mechanical structure. Its function form is as follows:

[0126] r(M in )=-|M in |*r(Δθ)

[0127] In summary, the total reward function is the accumulation of the above four components

[0128] r(s,a)=w1r(Δθ l )+w2r(σ)+w3r(Δθ m )+w4r(M in )

[0129] Where w i Represents the weight of the i-th reward function

[0130] Step 4: Reinforcement Learning Training Process

[0131] After designing the reward function, we use the SAC algorithm in reinforcement learning to solve the optimization problem. The algorithm uses entropy regularization to maximize the trade-off between expected return and entropy (maintaining the exploratory nature of the strategy).

[0132] SAC contains the Actor action policy network π θ (a|s), the goal of action strategy optimization is:

[0133]

[0134] Among them, θ is the network parameter of the action strategy, s is the current state, is the experience pool, ξ is the normal distribution The variables that take values, Q is the state value network, φ is the value network parameter, α is the weight, and π is the action strategy.

[0135]

[0136] in, is the sampling action, μ is the mean, and σ is the standard deviation.

[0137] SAC contains two state-action-value networks, and its optimized loss function is:

[0138]

[0139] Among them, φ i is the parameter of the i-th action value network, d is whether the state is a terminal state, and y is the target value of the current state-action pair. The calculation formula is:

[0140]

[0141] Among them, r is the current income, γ is the discount factor, is the action sampled from the action policy

[0142] The network structures of action strategy and value network are composed of multi-layer fully connected layer + multi-layer LSTM + multi-layer fully connected layer. The network structures are as follows: Figure 6 and Figure 7 As shown in the figure, the difference in structure is that the action strategy network input is state s and output is action a; the action value network input is state-action pair (s, a) and output is action value Q(s, a).

[0143] The SAC algorithm consists of the following steps:

[0144] 1) Initialize the strategy π network parameters θ and the Q function network parameters φ 1，2 , initialize the experience pool Assign the Q parameter to the target Q network parameter φ targ，1 ←φ1,φ targ，2 ←φ2;

[0145] 2) Observe the state s and select actions a~π from the strategy θ (·|s), execute the action in the environment, observe the state s', benefit r, and end mark d at the next moment, and store the state transition pair (s, a, r, s', d) in the experience pool In , if s' is the end state, reset the environment;

[0146] 3) After collecting K state pairs, perform the following update steps 4) to 8) L times:

[0147] 4) From the experience pool Randomly collect a set of state transition pairs to form B = {(s, a, r, s′, d)},

[0148] 5) Calculate the target value of the Q function:

[0149]

[0150] 6) Use single-step gradient descent to update the weight parameters of the Q function, and its gradient value is:

[0151]

[0152] 7) Use single-step gradient ascent to update the weight parameters of the policy network, and its gradient value is

[0153]

[0154] in, From π θ (·|s) is the sampled value, which is made differentiable with respect to θ through the reparameterization technique.

[0155] 8) Update target network parameters

[0156] φ tar，i ←ρφ tar，i +(1-ρ)φ i ,fori=1,2

[0157] 11) Repeat steps 2) to 8) until convergence.

[0158] The PPO algorithm can also be used to solve the reinforcement learning problem, see Figure 10 ,

[0159] After the basic concepts of reinforcement learning are clearly defined, combined with the way the environment changes in the third step, the PPO algorithm in reinforcement learning is used to solve the problem.

[0160] PPO is an online policy update algorithm that requires defining an Actor action policy network π θ (a t |s t ), representing the state s t a t Probability distribution of Critic state value evaluation network Represents state s tThe network architecture of both neural networks consists of multiple fully connected layers + multiple LSTM layers + multiple fully connected layers. The neural network architecture is shown in the figure below. The Actor outputs At and the Critic outputs V(St).

[0161] The ultimate optimization goal of reinforcement learning is The strategy goal of using SGD to optimize the PPO algorithm is:

[0162]

[0163]

[0164] Among them, ε is a parameter coefficient, which represents the difference between the new action strategy and the old action strategy. is the advantage function in state (s,a).

[0165] The overall process of the PPO algorithm using domain generalization technology is as follows:

[0166] 1) Define the Actor action strategy network, which represents the probability distribution of taking actions under a state; the Critic state value evaluation network, which represents the state value under a state. The Actor outputs At and the Critic outputs V(St). The ultimate optimization goal of reinforcement learning is to use SGD to optimize the policy goal in the PPO algorithm as follows:

[0167]

[0168]

[0169] Among them, is a parameter coefficient, which represents the difference between the new action strategy and the old action strategy as the advantage function under the state;

[0170] 2) Initialize the network parameters in the action strategy network and the state value evaluation network;

[0171] 3) Using multi-threaded parallel technology, each thread uses domain randomization settings to determine the simulation environment used by the current thread, and a total of N environments are set;

[0172] 4) In each thread, starting from the initial state s0, the action policy network π θ (a t |s t ) continuously according to the current state s t Select Select action a t (Use normal mapping to maintain the exploratory nature of the action), act on the environment to obtain the state s at the next moment t+1, and the process is repeated until the trajectory in the thread ends; forming a single sampling trajectory τ=(s0,a0,r0,s1,a1,r1,…s t ,a t ,r t ,…,s T ,a T ,r T ). A total of N trajectories of length T are collected.

[0173] 5) Combine the trajectories generated in each thread into (s t ,a t ,r t ) experience pool; for each state s in the experience pool t , calculate the state value of the current state

[0174] 6) Estimate the advantage value of each state-action pair based on the data in the experience pool and the current state-value function;

[0175] 7) Update the strategy by maximizing the PPO-Clip target. The gradient direction of the strategy optimization is

[0176]

[0177] Where i represents the i-th trajectory, a total of N trajectories are sampled, and t represents the current moment using policy gradient ascent to update the network parameters in the action policy network to optimize the parameters, specifically using the Adam algorithm;

[0178] 8) According to the principle of minimum mean square error, the state value function is regressed and fitted, and its goal is defined as

[0179]

[0180] Obtain the new true state value and use the gradient descent method to update the network parameters in the state value network; 9) Repeat the process from 3) to 8) until the training times meet certain conditions and terminate the training.

[0181] The SAC algorithm flow chart is shown in Figure 8 .

[0182] Step 5: Adjust the reward function design based on SAC training results

[0183] The fourth step is to train the action strategy network π θ (a t |s t), and then use it on the motor model to check whether the trajectory characteristics meet the control performance requirements of the servo drive. If so, the entire process ends, and the obtained action strategy network is the control strategy that meets the control requirements; if it does not meet the control requirements, it is necessary to adjust the design of the reward function and retrain the process with the SAC algorithm until the requirements are met. The overall process can be expressed as Figure 9 .

[0184] Finally, step 6: Use the trained policy network to control the motor model

[0185] When using the action policy network for real control, the standard deviation of the normal mapping network needs to be set to 0, similar to setting ε to 0 in the ε-greedy strategy; use the action policy network π with the highest return obtained in the fifth step θ (a t |s t ).

[0186] Taking the nominal state as an example, the RL generalization method is applied to the motor model, and the vibration suppression control effect achieved is as follows Figure 11 As shown, it is applied to the system parameter J l 、B ml , K ml When taking a smaller value, Figure 12 , applied to system parameter J l 、B ml , K ml When taking a larger value, Figure 13 .

[0187] From the three figures above, we can see that under different system parameters, traditional control methods will cause varying degrees of oscillation at the end. Traditional vibration suppression methods can eliminate these oscillations through input shaping, but the final state will still fluctuate within a small range. Reinforcement learning control, on the other hand, reaches the target position in a shorter time while reducing the vibration amplitude, demonstrating optimal performance and being insensitive to changes in system parameters.

[0188] This embodiment takes the real control needs and index requirements of servo drive control into consideration, converts them into equivalent reward functions in reinforcement learning, and builds a bridge between application problems in real industrial scenarios and reinforcement learning.

[0189] This embodiment builds a function library for reward function design, including a composite reward function design library with linear functions, exponential functions, logarithmic functions, and other functions as basis functions. It proposes a combination of different reward function forms and designs different reward functions for different control performance requirements.

[0190] Training based on the reward function continuously modifies the reward function design based on the training results until the control indicators are met, forming a closed loop of reward function design. This opens up a new method and approach for the application of reinforcement learning in the field of servo drive control.

[0191] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A reward function for a reinforcement learning method, characterized in that It includes four sub-functions, and defines the overshoot as σ=θ l_max -θ l_cmd θ l_max is the maximum value reached by the load during the movement, θ l_cmd is the expected load command value; the total reward function is the accumulation of four sub-functions, namely: r(s,a)=w1r(Δθ l )+w2r(σ)+w3r(Δθ m )+w4r(M in ); Where w i represents the weight of the i-th reward function, where r(Δθ l ) is a position-related reward function, which uses a combination of exponential decay and linear function, that is, the linear function is followed by exponential decay. Its function form is as follows: Among them, θ1 is an artificially set initial deflection angle value of the servo motor, r(σ) is the penalty for overshoot σ, and its function form is as follows: Among them, σ1 is an artificially set initial deflection angle value of the servo motor, r(Δθ m ) is the motor position reward, which is in the same form as the load end position reward; r(M in ) is the penalty of input torque, and its function form is as follows: r(M in )=-|M in |*r(Δθ), Among them, |M in |Input torque to the motor.

2. A vibration suppression reinforcement learning algorithm, using the reward function as described in claim 1, characterized in that: The following steps are also included: Step 1: Mathematical Modeling Build a linear discretized mathematical model of the controlled object. According to Newton's second law and torsional Hooke's law, the robot's single-joint motor load model is abstracted into a mechanical model connected by two rigid bodies, two revolute pairs, and a torsion spring. The dynamic response equation and deformation equation are established respectively. The dynamic response equation and deformation equation are: Among them, the physical quantity M represents torque, J represents moment of inertia, α represents angular acceleration, B represents rotational damping, ω represents angular velocity, θ represents angle, and the subscripts of the physical quantities represent the corresponding objects, where the subscript m represents the motor, ml represents the coupling, and l represents the load. In particular, M in Represents the input torque, K ml Indicates the equivalent torsional stiffness of the coupling; Step 2: Build a reinforcement learning model based on MDP to convert the servo drive position control problem into a reinforcement learning problem Define state S t =(θ m ,θ′ m ),θ m is the position of the motor high-speed end, θ′ m is the speed of the motor high speed end, action a t is the input torque M in , the return r t is the reward obtained at the current step; the trajectory of a single sampling is defined as τ=(s1,a1,r1,…s t ,a t ,r t ,…,s T ,a T ,r T ), the probability of its occurrence is: The return of this trajectory is defined as: Where γ is the discount factor, γ∈(0,1], The goal of reinforcement learning is to maximize long-term benefits, that is, to maximize expected returns, which are defined as That is, we find the parameter θ of the action strategy network, and the trajectory obtained by sampling according to the action strategy under this parameter has the maximum expected return value, so the optimization problem becomes: The most important function in the above optimization problem is r(s t ,a t ), which is the reward function of the current state action, which determines the learning goal of reinforcement learning and determines whether the training process can converge; Step 3: Design of reward function Define overshoot as σ = θ l_max -θ l_cmd ,θ l_max is the maximum value reached by the load during the movement, θ l_cmd is the expected load command value; the total reward function is the accumulation of four components, namely: r(s,a)=w1r(Δθ l )+w2r(σ)+w3r(Δθ m )+w4r(M in ); Where w i represents the weight of the i-th reward function, where r(Δθ l ) Position-dependent reward function, which uses a combination of exponential decay and linear function, that is, the linear function is followed by exponential decay, and its function form is as follows: Among them, θ1 is an artificially set initial deflection angle value of the servo motor, r(σ) is the penalty for overshoot σ, and its function form is as follows: Among them, σ1 is an artificially set initial deflection angle value of the servo motor, r(Δθ m ) is the motor position reward, which is in the same form as the load end position reward; r(M in ) is the penalty of input torque, and its function form is as follows: r(M in )=-|M in |*r(△θ); Step 4: Reinforcement learning model training; Use the SAC algorithm or PPO algorithm in reinforcement learning to solve the optimization problem; Step 5: Adjust the reward function design based on the reinforcement learning model training results Step 4: After training the action strategy network, apply it to the motor model to check whether the trajectory characteristics meet the control performance requirements of the servo drive. If so, the entire process ends and the obtained action strategy network is the control strategy that meets the control requirements. If not, the reward function design needs to be adjusted and the process is retrained using the SAC algorithm or PPO algorithm until the requirements are met. Step 6: Use the trained policy network to control the motor model When using the action policy network for real control, the standard deviation of the normal mapping network needs to be set to 0; use the action policy network π with the highest return obtained in the fifth step θ (a t |s t ); 3. A vibration suppression reinforcement learning algorithm according to claim 2, characterized in that: In step 4, the SAC algorithm includes the following steps: 1) Initialize the strategy π network parameters θ and the Q function network parameters φ 1，2 , initialize the experience pool ; Assign the Q parameter to the target Q network parameter φ targ，1 ←φ1,φ targ，2 ←φ2; 2) Observe the state s and select actions a~π from the strategy θ (·|s), execute the action in the environment, observe the state s', benefit r, and end mark d at the next moment, and store the state transition pair (s, a, r, s', d) in the experience pool In , if s' is the end state, reset the environment; 3) After collecting K state pairs, perform the following update steps 4) to 8) L times: 4) From the experience pool Randomly collect a set of state transition pairs to form B = {(s, a, r, s′, d)}, 5) Calculate the target value of the Q function: 6) Use single-step gradient descent to update the weight parameters of the Q function, and its gradient value is: 7) Use single-step gradient ascent to update the weight parameters of the policy network, and its gradient value is in, From π θ (·|s) is the sampled value, which is made differentiable with respect to θ through the reparameterization technique. 8) Update target network parameters f tar,i ←rf tar，i +(1-r)φ i ,for i=1,2 9) Repeat steps 2) to 8) until convergence.

4. A vibration suppression reinforcement learning algorithm according to claim 2, characterized in that: In step 4, the PPO algorithm includes the following steps: 1) Define the Actor action strategy network π θ (a t |s t ), representing the state s t a t Probability distribution of Critic state value evaluation network Represents state s t The state value under the condition, Actor outputs At, Critic outputs V(St); The ultimate optimization goal of reinforcement learning is The strategy goal of using SGD to optimize the PPO algorithm is: Among them, ε is a parameter coefficient, which represents the difference between the new action strategy and the old action strategy. is the advantage function under state (s,a); 2) Initialize the action strategy network π θ (a t |s t ) State Value Assessment Network The network parameters in ; 3) Using multi-threaded parallel technology, each thread uses domain randomization to set J l 、B ml , K ml ,θ tar , determine the simulation environment used by the current thread, and set N environments in total; 4) In each thread, starting from the initial state s0, the action policy network π θ (a t |s t ) continuously according to the current state s t Select action a t , use normal mapping to maintain the exploratory nature of the action, and act on the environment to obtain the state s at the next moment t+1 , and the process is repeated until the trajectory in the thread ends; forming a single sampling trajectory τ=(s0,a0,r0,s1,a1,r1,…s t , a t , r t ,…,s T , a T , r T ), a total of N trajectories of length T are collected; 5) Combine the trajectories generated in each thread into (s t ,a t ,r t ) experience pool; for each state s in the experience pool t , calculate the state value of the current state 6) Based on the data in the experience pool and the current state value function Estimate each state-action pair (s t ,a t ) has an advantage of 7) Update the strategy by maximizing the PPO-Clip target. The gradient direction of the strategy optimization is Where i represents the i-th trajectory, a total of N trajectories are sampled, and t represents the current time Update the action policy network π using policy gradient ascent θ (a t |s t ) to optimize the network parameters Where α is the update step size, which is obtained using the Adam algorithm; 8) According to the principle of minimum mean square error, the state value function is regressed and fitted, and its goal is defined as Among them, V(s t ) To obtain the new true value of the state, use the gradient descent method to update the state value network The network parameters φ in ; 9) Repeat the process from 3) to 8) until the training times meet certain conditions and the training is terminated.

Images