Musculoskeletal robot motion control method and device based on recurrent neural network

Abstract

The application provides a kind of musculoskeletal robot motion control method and device based on recurrent neural network, the method comprises: the initial state element of the recurrent neural network is determined based on the first initial state of the recurrent neural network and the bias state corresponding to the first initial state;Determine the combination coefficient of initial state element based on multilayer perceptron neural network;Muscle control signal is determined based on initial state element and combination coefficient;Muscle control signal is used to control musculoskeletal robot to move to target position.The musculoskeletal robot motion control method based on recurrent neural network provided by the application can reshape the initial state of recurrent neural network by adjusting the combination coefficient of initial state element to adapt to new motion target, improve the motion learning efficiency of complex musculoskeletal robot system, realize the motion learning of multiple motion targets and the direct generalization of new motion target.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method and apparatus for motion control of musculoskeletal robots based on recurrent neural networks. Background Technology

[0002] With the continuous development of robotics technology, people expect humanoid robots to possess high precision and rapid response when performing tasks. Furthermore, flexibility is crucial for ensuring safe and reliable interaction between humanoid robots and their environment. Musculoskeletal robots, as a type of humanoid robot, are rigid-flexible coupled systems composed of flexible muscles as actuators and rigid skeletons as supporting structures. By adjusting antagonistic muscles, musculoskeletal robots can potentially exhibit the required compliance or stiffness in different tasks and environments. Moreover, musculoskeletal robots employ more muscle actuators than joints; redundant muscle groups can provide higher robustness, stability, and adaptability, especially under environmental uncertainty or disturbances. Therefore, its structural characteristics and actuation methods ensure the possibility of safe and reliable human-robot integration and flexible, high-precision operation. While the benefits of musculoskeletal robots are obvious, their actuation methods and the complex arrangement of muscles also present many challenges to controller design. Specifically, for the same motion, the high redundancy of muscles makes the solutions for muscle excitation infinite. Strongly nonlinear muscle dynamics rely on a large number of parameters, which are often difficult to obtain in real time, posing a challenge to obtaining sufficient feedback signals. Coupling is also manifested in the presence of cross-joint muscles and multiple muscles controlling the same joint, which makes it impossible for a single muscle to control it alone.

[0003] For these complex musculoskeletal robotic systems, model-free methods are a crucial approach for achieving precise motion control. Currently, to achieve point-to-point motion tasks with constant angular velocity, an improved deep Q-network method employs a phased target learning framework. This utilizes a fully connected neural network (FNN) to generate continuous muscle excitation and introduces biological noise sources to enhance the exploration of the solution space. To reach each target point without assuming prior knowledge of arm dynamics, a normalized radial basis function (NRBF) network can be used to implement the feedback controller and inverse static model. Combining this with the hindsight experience replay (HER) algorithm and the deep deterministic policy gradient (DDPG) algorithm enables the FNN to learn as a muscle command generator, improving the control of multi-input multi-output musculoskeletal models of the human arm.

[0004] However, these methods face significant computational and time overhead. Furthermore, to mitigate the impact of the temporal credit assignment problem in reinforcement learning algorithms, the aforementioned methods all require immediate and dense reward signals.

[0005] On the other hand, considering its inherent ability to generate time-series data for convenient control, recurrent neural networks have also been used as control signal generators for musculoskeletal robot systems. However, the learning efficiency of the controller in these methods needs improvement. Furthermore, generalization ability has not received sufficient attention; therefore, when given a new motion target, the controller needs to be readjusted, or the musculoskeletal robot system can only achieve motions extremely similar to those already learned. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a method and apparatus for motion control of a musculoskeletal robot based on a recurrent neural network.

[0007] This invention provides a motion control method for musculoskeletal robots based on recurrent neural networks, comprising:

[0008] Based on the first initial state of the recurrent neural network and the deviation state corresponding to the first initial state, the initial state primitive of the recurrent neural network is determined; the first initial state is the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches the preset motion target.

[0009] The combination coefficients of the initial state primitives are determined based on a multilayer perceptron neural network.

[0010] Based on the initial state primitives and the combination coefficients, the muscle control signal is determined;

[0011] Based on the muscle control signals, the musculoskeletal robot is controlled to move to the target position.

[0012] In some embodiments, determining the initial state primitives of the recurrent neural network based on a first initial state and a corresponding deviation state includes:

[0013] Based on the first initial state, the first signal output by the recurrent neural network is determined;

[0014] Based on the deviation state, the second signal output by the recurrent neural network is determined;

[0015] Based on the first signal and the second signal, a first loss function is determined;

[0016] Based on the first loss function, the initial state primitives of the recurrent neural network are determined.

[0017] In some embodiments, determining the muscle control signal based on the initial state primitives and the primitive combination coefficients of the recurrent neural network includes:

[0018] Based on the primitive combination coefficients and the initial primitives, the second initial state of the recurrent neural network is determined;

[0019] The second initial state is input into the recurrent neural network to determine the muscle control signal.

[0020] In some embodiments, the expression for the second initial state is as follows:

[0021] r(t=0)=a1v1+a2v2+…+a k v k

[0022] Where r(t=0) represents the second initial state, a1, a2, … a k Denotes the coefficients of the combination of basic elements, v1, v2, ..., v k This represents the initial state primitive.

[0023] In some embodiments, after controlling the musculoskeletal robot to move to the target position based on the muscle control signal, the method further includes:

[0024] Based on the target position, the actual position, velocity, and initial position of the musculoskeletal robot are used to determine the second loss function;

[0025] Based on the second loss function, a reward signal is generated;

[0026] The multilayer perceptron neural network is optimized based on the reward signal.

[0027] In some embodiments, optimizing the multilayer perceptron neural network based on the reward signal includes:

[0028] Based on the aforementioned multilayer perceptron neural network, a random beta strategy is determined;

[0029] The random beta strategy is optimized based on the reward signal to obtain a deterministic strategy;

[0030] The determination strategy is used to generate the combination coefficients.

[0031] The present invention also provides a motion control device for a musculoskeletal robot based on a recurrent neural network, comprising:

[0032] The first determining module is used to determine the initial state primitives of the recurrent neural network based on the first initial state of the recurrent neural network and the deviation state corresponding to the first initial state; the first initial state is the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches the preset motion target.

[0033] The second determining module is used to determine the combination coefficients of the initial state primitives based on a multilayer perceptron neural network.

[0034] The third determining module is used to determine the muscle control signal based on the initial state primitive and the combination coefficients;

[0035] The control module is used to control the musculoskeletal robot to move to the target position based on the muscle control signals.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the musculoskeletal robot motion control method based on recurrent neural networks as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the musculoskeletal robot motion control method based on recurrent neural networks as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the musculoskeletal robot motion control method based on recurrent neural networks as described above.

[0039] The present invention provides a musculoskeletal robot motion control method and device based on recurrent neural networks. By determining the initial state primitives according to the first initial state and the deviation state of the recurrent neural network, the solution dimension of the network corresponding to the initial state is reduced when using the recurrent neural network for musculoskeletal robot system control. By adjusting the combination coefficients of the initial state primitives, the initial state of the recurrent neural network can be reshaped to adapt to new motion targets, thereby improving the motion learning efficiency of complex musculoskeletal robot systems and enabling motion learning of multiple motion targets and direct generalization of new motion targets. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is one of the flowcharts illustrating the motion control method for musculoskeletal robots based on recurrent neural networks provided in this embodiment of the invention.

[0042] Figure 2 This is the second flowchart of the musculoskeletal robot motion control method based on recurrent neural networks provided in this embodiment of the invention.

[0043] Figure 3 This is a schematic diagram of the structure of the musculoskeletal robot motion control device based on a recurrent neural network provided in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0046] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0047] Figure 1 This is one of the flowcharts illustrating the motion control method for musculoskeletal robots based on recurrent neural networks provided in this embodiment of the invention, such as... Figure 1 As shown, the musculoskeletal robot motion control method based on recurrent neural networks provided in this embodiment of the invention includes:

[0048] Step 101: Based on the first initial state of the recurrent neural network and the deviation state corresponding to the first initial state, determine the initial state primitive of the recurrent neural network; the first initial state is the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches the preset motion target;

[0049] Step 102: Determine the combination coefficients of the initial state primitives based on the multilayer perceptron neural network;

[0050] Step 103: Determine the muscle control signal based on the initial state primitives and the combination coefficients;

[0051] Step 104: Based on the muscle control signal, control the musculoskeletal robot to move to the target position.

[0052] It should be noted that the execution subject of the musculoskeletal robot motion control method based on recurrent neural networks provided by this invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This invention does not impose specific limitations.

[0053] This invention employs a recurrent neural network (RNN) that automatically evolves from an initial state as a control signal generator for musculoskeletal robots. By modulating the initial state of the RNN, appropriate muscle control signals are generated to complete the motion task.

[0054] Specifically, a recurrent neural network (RNN) is a type of recurrent neural network based on leaky neurons, used to simulate the generation of dynamic neural coding and control signals in the motor cortex. An RNN has hidden layers and an output layer; the hidden layers consist of N interconnected leaky neurons.

[0055] Recurrent neural networks have no input vector, and their corresponding dynamic equations are as follows:

[0056]

[0057] h(t)=tanh(r(t))

[0058] z(t)=Vh(t)+b z

[0059] Where τ is the time constant of the recurrent neural network, which defines the network's time scale, and t is used to characterize the network's running time. This represents the membrane potential of neurons in the hidden layer. This represents the firing frequency vector of neurons in the hidden layer. The firing rate of neurons is expressed by a nonlinear function. It is related to membrane potential.

[0060] At adjacent time points, the hidden layer neurons pass through the weight matrix. To transmit information, matrix These are the output layer weights, which map the firing frequency vectors of the hidden layer neurons to the output signals. and It is a bias vector. It is the output vector of the recurrent neural network, which serves as the time-varying control signal for the musculoskeletal robot.

[0061] In this embodiment of the invention, the membrane potential vector of the hidden neurons in the recurrent neural network at the start of the network's dynamic evolution is referred to as the "initial state" of the recurrent neural network. At time t = 0, r(t = 0) = r0. The recurrent neural network automatically evolves from the initial state to generate control signals for the robot system.

[0062] Different initial states will result in completely different control signals, so the membrane potential vector must be correctly initialized when a musculoskeletal robot is expected to achieve a specific target-oriented motion.

[0063] In other words, this invention proposes a recurrent neural network that automatically evolves from an initial state as a control signal generator for musculoskeletal robots. The initial state determines the muscle control signal, which in turn determines the actual movement of the musculoskeletal system.

[0064] In step 101, based on the first initial state of the recurrent neural network and the deviation state corresponding to the first initial state, the initial state primitive of the recurrent neural network is determined; the first initial state is the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches the preset motion target.

[0065] In some embodiments, determining the initial state primitives of the recurrent neural network based on a first initial state and a corresponding deviation state includes:

[0066] Based on the first initial state, the first signal output by the recurrent neural network is determined;

[0067] Based on the deviation state, the second signal output by the recurrent neural network is determined;

[0068] Based on the first signal and the second signal, a first loss function is determined;

[0069] Based on the first loss function, the initial state primitives of the recurrent neural network are determined.

[0070] Specifically, we first analyze the impact of the initial state of a recurrent neural network deviating from the optimal initial state corresponding to a certain motion task on the output.

[0071] According to the RNN dynamics equations used in the embodiments of the present invention, tanh(r(t)) introduces nonlinearity to the recurrent neural network in the hidden layer neurons, but if the hidden layer neurons r (i) ∈[-1,1], where i represents the i-th neuron in the hidden layer, then tanh(r (i) )≈0.8r (i) .

[0072] By assigning appropriate weights to the weight matrices and bias vectors of each layer in a recurrent neural network, the membrane potential of most hidden layer neurons can fluctuate within the range of [-1, 1].

[0073] In this case, further restricting the subsequent analysis to the approximately linear region, the RNN dynamic equation can be further expressed as:

[0074]

[0075] Where τ represents the time constant of the recurrent neural network, and t is used to characterize the network's running time. This represents the membrane potential of neurons in the hidden layer. Represents the weight matrix. Represents the identity matrix. This represents the bias vector.

[0076] Let matrix a = 0.8WI, then the above equation can be viewed as a first-order linear differential equation with constant coefficients. This equation has an analytical solution, as follows:

[0077]

[0078] in, Let r represent the membrane potential of the hidden layer neurons, τ represent the time constant of the recurrent neural network, t represent the running time of the network, and r0 represent the initial state. Let represent the bias vector, and s represent the time used for integration.

[0079] make Under the assumption that the network is approximately linear, the output signal generated by the RNN at each time step can be further expressed by the following formula:

[0080]

[0081] in, It is the output vector of a recurrent neural network, a matrix. These are the output layer weights, τ represents the time constant of the recurrent neural network, t characterizes the network's runtime, and r0 represents the initial state. It is the bias vector.

[0082] By constructing a first loss function L, the differences in the output signal of the recurrent neural network as it evolves from different initial states are measured.

[0083] The expression for the first loss function L is as follows:

[0084]

[0085] Where L represents the first loss function, r0 represents the first initial state, which is the optimal initial state of the recurrent neural network given a motion target that a musculoskeletal robot system expects to reach. The deviation state is an initial state that deviates from r0, and z(t) represents the first signal. * z(t) represents the second signal, and z(t) and z(t) represent the second signal. * (t) represent the states when the recurrent neural network starts from the initial state r0 and r1, respectively. The muscle control signals output during automatic evolution.

[0086] Furthermore, in the above formula, let the matrix When the real part of each eigenvalue of matrix A in the formula is less than zero It is Hurwitz stable.

[0087] Matrix Q is the unique solution to the following Lyapunov equation:

[0088]

[0089] The matrix Q is symmetric and positive definite, and the numerical solution of Q can be obtained by applying the Lyapunov equations mentioned above.

[0090] According to the formula for the first loss function L, when the deviation between the initial state of the recurrent neural network and the expected optimal initial state is fixed, the impact on the subsequent output signal of the neural network is determined by the value of Q.

[0091] If we assume With the size of Q fixed, the value of Q remains unchanged. According to Lemma 2, when The function L has its maximum value when the direction of the eigenvector corresponding to the largest eigenvalue of Q is along the path of the eigenvector.

[0092] Conversely, when The first loss function L has a minimum value when the direction of the eigenvector corresponding to the minimum eigenvalue of Q is along the path of the eigenvector. At this time, when the recurrent neural network starts from the first initial state r0 and... The deviation of the muscle control signals output during automatic evolution is small.

[0093] In other words, if the initial state deviates along the direction of the eigenvector corresponding to one of the larger eigenvalues ​​in Q, the control signal output by the recurrent neural network will deviate significantly from the desired muscle control signal. This will cause a large deviation between the actual position reached by the end of the robot system and the desired motion target. When the initial state deviation is along the direction of the eigenvector corresponding to a very small eigenvalue in Q, it will hardly affect the output muscle signal.

[0094] Based on the above analysis, this embodiment of the invention proposes a method for constructing the initial state primitives of a recurrent neural network.

[0095] Let {λ1,λ2,…,λ N Let} be the set of eigenvalues ​​of matrix Q, where λ max =λ1≥λ2≥…≥λ N =λ min .

[0096] Suppose the orthogonal eigenvectors corresponding to the first k largest eigenvalues ​​are v1, v2, ..., v k You can choose these k feature vectors as the primitives of the initial state of the recurrent neural network.

[0097] The initial states of the network corresponding to different motion targets can be constructed using the same initial state primitives. This reduces the difficulty of solving the control signals of the controlled musculoskeletal robot system, because the problem of solving the time-varying muscle signals of multiple muscles is transformed into a problem of solving the combination of initial state primitives. The initial state primitives constructed in this embodiment of the invention can serve as general knowledge corresponding to different motion targets, providing a foundation for further motion learning and generalization.

[0098] In other words, this invention proposes a motion primitive based on the initial state of a recurrent neural network. The initial state of the recurrent neural network can be expressed by a set of proposed primitives, which reduces the difficulty of solving the initial state corresponding to different motions and provides a foundation for motion learning and generalization.

[0099] In step 102, the combination coefficients of the initial state primitives are determined based on the multilayer perceptron neural network.

[0100] After determining the initial state primitives of the recurrent neural network, the solution dimension of the network's initial state is reduced when using the recurrent neural network for the control of a musculoskeletal robot system. By adjusting the combination coefficients of the initial state primitives, the initial state of the recurrent neural network can be reshaped to adapt to new motion targets.

[0101] To achieve motion learning for musculoskeletal robots, this embodiment of the invention constructs a modulation model of the initial state primitives of a recurrent neural network.

[0102] Specifically, it is necessary to learn to convert the desired motion target of the robot's end effector into a mapping function of the initial state primitive combination coefficients.

[0103] Considering the good generalization ability of neural networks, this embodiment of the invention uses a multilayer perceptron neural network (MLPNN) to implement the mapping function, which can modulate the combination of primitives according to different moving targets.

[0104] Furthermore, for this modulation model, this embodiment of the invention proposes a reinforcement learning algorithm based on Proximal Policy Optimization (PPO), which assigns appropriate parameters to the initial state modulation model MLPNN in this embodiment. Based on this, motion learning of multiple moving targets and direct generalization to new moving targets can be achieved.

[0105] Reinforcement learning problems can generally be solved by state action award It is characterized by time step t. In typical related tasks, the agent interacts with the environment multiple times over time.

[0106] At each time step, the agent perceives that the environment belongs to the state space at the current moment. state s t The agent is based on policy π(·|s) t In action space Select and execute action a t The agent interacts with its environment, which then feeds back a scalar reward signal r. t and transition to the next state s t+1 Reinforcement learning aims to find a high-performing policy that enables an agent to obtain the maximum cumulative reward signal during its interaction with the environment.

[0107] In this embodiment of the invention, a reinforcement learning problem is constructed in order to enable the musculoskeletal robot system to learn multiple target-oriented motions simultaneously.

[0108] Specifically, the target vectors q1, q2, ..., q m It represents the spatial position information of multiple moving targets that are expected to be reached, that is, the direction and distance relative to the starting position of the end effector of the musculoskeletal robot system at the start of the movement.

[0109] In each step of this reinforcement learning problem, the current motion target of the robot system is considered as the environment state. If the current motion target vector of the robot system is q... d ∈{q1, q2, ..., q m}, then the current environmental state can be defined as: s n =q d .

[0110] The initial state primitive combination coefficient mapping model MLPNN in this embodiment of the invention can be regarded as an agent in a reinforcement learning problem, receiving the current environment state s. n And generate the combination coefficient vector of the initial state primitives. Where a n = (a1, a2, ... a k ).

[0111] In step 103, the muscle control signal is determined based on the initial state primitive and the combination coefficients.

[0112] In some embodiments, determining the muscle control signal based on the initial state primitives and the primitive combination coefficients of the recurrent neural network includes:

[0113] Based on the primitive combination coefficients and the initial primitives, the second initial state of the recurrent neural network is determined;

[0114] The second initial state is input into the recurrent neural network to determine the muscle control signal.

[0115] In some embodiments, the expression for the second initial state is as follows:

[0116] r(t=0)=a1v1+a2v2+…+a k v k

[0117] Where r (t=0) represents the second initial state, a1, a2, ... a k Represents the combination coefficients of the basic elements vv1, v2, ..., v k This represents the initial state primitive.

[0118] Specifically, based on the elementary combination coefficients a1, a2, ... a kand initial state primitives v1, v2, ..., v k This allows us to obtain the second initial state of the recurrent neural network.

[0119] That is, the second initial state of the recurrent neural network can be further calculated through the combination of primitives as follows:

[0120] r(t=0)=a1v1+a2v2+…+a k v k

[0121] Where r (t=0) represents the second initial state, a1, a2, ... a k Let v1, v2, ..., v be the coefficients of the combination of basic elements. k This represents the initial state primitive.

[0122] The second initial state is input into the recurrent neural network to determine the muscle control signal. That is, the activity of neurons in the recurrent neural network automatically evolves from the second initial state without external input and generates muscle control signals to drive the robot system to move.

[0123] In step 104, based on the muscle control signal, the musculoskeletal robot is controlled to move to the target position.

[0124] Based on the generated muscle control signals, the musculoskeletal robot can be controlled to move to the target position.

[0125] The musculoskeletal robot motion control method based on recurrent neural networks provided in this invention determines the initial state primitives according to the first initial state and the deviation state of the recurrent neural network. When using the recurrent neural network to control the musculoskeletal robot system, the solution dimension of the network's corresponding initial state is reduced. By adjusting the combination coefficients of the initial state primitives, the initial state of the recurrent neural network can be reshaped to adapt to new motion targets, thereby improving the motion learning efficiency of complex musculoskeletal robot systems. It can realize motion learning of multiple motion targets and direct generalization of new motion targets.

[0126] In some embodiments, after controlling the musculoskeletal robot to move to the target position based on the muscle control signal, the method further includes:

[0127] Based on the target position, the actual position, velocity, and initial position of the musculoskeletal robot are used to determine the second loss function;

[0128] Based on the second loss function, a reward signal is generated;

[0129] The multilayer perceptron neural network is optimized based on the reward signal.

[0130] In some embodiments, optimizing the multilayer perceptron neural network based on the reward signal includes:

[0131] Based on the aforementioned multilayer perceptron neural network, a random beta strategy is determined;

[0132] The random beta strategy is optimized based on the reward signal to obtain a deterministic strategy;

[0133] The determination strategy is used to generate the combination coefficients.

[0134] After controlling the musculoskeletal robot to move to the target position based on the muscle control signals, a second loss function can be constructed based on the actual position, speed, initial position, and target position of the musculoskeletal robot.

[0135] The completion status and performance of the robot system in this motion are evaluated based on the constructed second loss function, and the corresponding reward signal R is returned. n .

[0136] Figure 2 This is the second flowchart illustrating the motion control method for musculoskeletal robots based on recurrent neural networks provided in this embodiment of the invention. Figure 2 As shown, in this embodiment of the invention, the weights of the initial state primitive modulation model are adjusted based on the reward signal returned after evaluating the motion.

[0137] Second loss function L n The expression is as follows:

[0138]

[0139] Among them, L n Let R represent the second loss function. n Indicates a reward signal, q, and q d Let q represent the actual position, velocity, and target position of the musculoskeletal robot system's end effector at the end of the motion, respectively. o c1 and c2 are the initial positions of the musculoskeletal robot system's end effector at the start of movement, and c1 and c2 are the weighting factors of each term in the balance loss function.

[0140] The next moving target is obtained based on the state transition probability distribution P and is regarded as the state s in the next step. n+1 The state transition probability distribution P will be given later.

[0141] Therefore, each step yields a quadruple (s) n a n R n s n+1By solving the MLPNN that maximizes the cumulative reward in continuous multi-step interactions, it is possible to learn multiple target-oriented motions and directly generalize to new target-oriented motions in musculoskeletal robot systems.

[0142] In order to achieve motion learning of the target and generalization of new targets, the modulation model of the initial state primitive combination needs to be given appropriate parameters.

[0143] MLPNN was used to generate a policy π. θ (·|s n To generate the initial recurrent neural network

[0144] The combination coefficients of the initial state primitives.

[0145] The embodiments of the present invention optimize strategy π based on the following formula. θ (·|s n By adjusting the strategy parameters, the expected sum of the cumulative discounted motion performance evaluation values ​​is maximized. The expression for the adjusted strategy parameters is as follows:

[0146]

[0147] Where, θ * Represents the modified policy parameters, γ∈[0,1] is the discount constant, θ is the policy parameter including the MLPNN parameters, and R n This represents the reward signal, where n represents the nth movement, and T represents the total number of movements. This indicates that in the initial state s0 and policy π θ The expected cumulative reward.

[0148] In this embodiment of the invention, the beta distribution is used as the policy distribution. It is assumed that in this reinforcement learning problem, the action dimensions of the agent are independent of each other, therefore the covariance of actions in different dimensions is zero.

[0149] The random beta policy can be defined as follows:

[0150]

[0151]

[0152] Where Γ(·) is the gamma function, x i It is the i-th element of a k-dimensional vector x obtained by sampling the policy, and x i ∈[0,1],α i and β iThese are the i-th elements of the k-dimensional shape factor vectors α and β of the beta policy distribution, respectively. Vectors α and β are output by the output layer of the MLPNN under the current input, and the value of k is the same as the number of initial state primitives used.

[0153] During the strategy learning phase, when the input is s n At that time, the corresponding action vector x n It can be obtained by sampling using the above-mentioned random beta strategy to increase exploration capabilities.

[0154] By using action vector x n By expanding to a suitable range, the initial state primitive combination coefficient vector a of the recurrent neural network can be obtained. n .

[0155] It is worth noting that after the policy training is completed, in order to make the control of the robot system more stable, the action vector x n Instead of being obtained through sampling, it is equivalent to the expected policy distribution under the current input, i.e. At this point, the random beta policy becomes a deterministic policy after training is complete.

[0156] In order to achieve strategy π θ (·|s n To obtain the optimal initial state primitive combination coefficients for each desired motion target, a near-end policy optimization algorithm is introduced to maximize the following two-part objective function, thereby optimizing the update policy:

[0157]

[0158] Wherein, the objective function L θ The first term in the expression is used to avoid excessively large policy updates in successive iterations. In this term, ∈ is a hyperparameter. It is a probability ratio function, defined as follows:

[0159]

[0160] Where, θ old This indicates the strategy parameters before this update. In the nth step of the interaction, the dominant function A is... n The estimate, c l It is a weighting factor used to determine the proportion of the second term in the objective function, and S(·) represents the entropy reward term, used to ensure sufficient exploration.

[0161] about The advantage function is estimated using the Generalized Advantage Estimation (GAE) algorithm, as follows:

[0162]

[0163] δ n =R n +γV μ (s n+1 )-V μ (s n )

[0164] Among them, V μ (·) is the state value function, generated by a fully connected neural network with parameter μ, which can be called a Critic network. γ∈[0,1] is the discount constant, and R0... n Indicates the reward signal, δ n The TD error is represented by T, where T represents the total number of movements, l represents the index of the cumulative term, n represents the nth movement, λ represents the discount factor, and s represents the TD error. n Indicates the current state, s n+1 Indicates the next state.

[0165] To enable the Critic network to accurately estimate the state-value function, its loss function is defined as follows:

[0166]

[0167] Among them, V targ (s n ) can be considered as V μ (s n ) of the supervision items, and z1 represents the weight factor of the loss function, and μ represents the parameter vector of the Critic network. μ can be obtained by minimizing the loss function L. μ Optimize.

[0168] In each round of iterative optimization, motion data of the robot system is collected during T interactions, and the policy π is used. θ and state value function V μ Each was achieved by employing the batch stochastic gradient descent algorithm based on the objective function L. θ and loss function L μ Perform K rounds of optimization.

[0169] After every L rounds of iteration optimization, the state transition probability distribution is updated as follows:

[0170]

[0171] in, When the expected target is q k At the time, the actual position of the musculoskeletal robot system's end effector at the end of the movement, q oIt is the initial position of the end effector at the moment the motion begins, q Z This represents the Z-th desired goal. This indicates that when the desired target is q Z At the end of the movement, Z represents the actual position of the musculoskeletal robot system's end point, where Z represents the index of the cumulative term, m represents the total number of targets, and k represents the index of the current target.

[0172] Based on the aforementioned state transition probabilities, by dynamically adjusting the state transition probabilities according to the current learning performance of each target during the learning process, the arrival accuracy of each target can be made more balanced when learning multiple moving targets simultaneously.

[0173] This invention proposes a motion learning method based on the modulation of initial state primitives in a recurrent neural network. Specifically, it proposes a modulation model based on a neural network, which calculates the initial state of the RNN for different motions by modulating and combining initial state primitives. Then, a modulation model reinforcement learning algorithm based on proximal policy optimization is used for motion learning.

[0174] The method proposed in this invention has been verified on a complex musculoskeletal system simulation platform. It can effectively perform motion learning and control, and can directly generalize to target-oriented motions that have not been learned before. This provides a theoretical basis and technical support for the control and development of novel musculoskeletal robot systems.

[0175] The following describes the motion control device for musculoskeletal robots based on recurrent neural networks provided by the present invention. The motion control device for musculoskeletal robots based on recurrent neural networks described below can be referred to in correspondence with the motion control method for musculoskeletal robots based on recurrent neural networks described above.

[0176] Figure 3 This is a schematic diagram of the motion control device for a musculoskeletal robot based on a recurrent neural network provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the musculoskeletal robot motion control device based on a recurrent neural network provided in this embodiment of the invention includes:

[0177] The first determining module 310 is used to determine the initial state primitives of the recurrent neural network based on the first initial state of the recurrent neural network and the deviation state corresponding to the first initial state; the first initial state is the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches the preset motion target.

[0178] The second determining module 320 is used to determine the combination coefficients of the initial state primitives based on a multilayer perceptron neural network.

[0179] The third determining module 330 is used to determine the muscle control signal based on the initial state primitive and the combination coefficients;

[0180] The control module 340 is used to control the musculoskeletal robot to move to the target position based on the muscle control signal.

[0181] It should be noted that the musculoskeletal robot motion control device based on recurrent neural networks provided in this embodiment of the invention can realize all the method steps implemented in the above-mentioned musculoskeletal robot motion control method embodiment based on recurrent neural networks, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0182] Optionally, the first determining module 310 is specifically used for:

[0183] Based on the first initial state, the first signal output by the recurrent neural network is determined;

[0184] Based on the deviation state, the second signal output by the recurrent neural network is determined;

[0185] Based on the first signal and the second signal, a first loss function is determined;

[0186] Based on the first loss function, the initial state primitives of the recurrent neural network are determined.

[0187] Optionally, the third determining module 330 is specifically used for:

[0188] Based on the primitive combination coefficients and the initial primitives, the second initial state of the recurrent neural network is determined;

[0189] The second initial state is input into the recurrent neural network to determine the muscle control signal.

[0190] Optionally, the expression for the second initial state is as follows:

[0191] r(t=0)=a1v1+a2v2+…+a k v k

[0192] Where r (t=0) represents the second initial state, a1, a2, ... a k Let v1, v2, ..., v be the coefficients of the combination of basic elements. k This represents the initial state primitive.

[0193] Optionally, it also includes:

[0194] The fourth determining module is used for:

[0195] Based on the target position, the actual position, velocity, and initial position of the musculoskeletal robot are used to determine the second loss function;

[0196] Based on the second loss function, a reward signal is generated;

[0197] The multilayer perceptron neural network is optimized based on the reward signal.

[0198] Optionally, the fourth determining module is specifically used for:

[0199] The optimization of the multilayer perceptron neural network based on the reward signal includes:

[0200] Based on the aforementioned multilayer perceptron neural network, a random beta strategy is determined;

[0201] The random beta strategy is optimized based on the reward signal to obtain a deterministic strategy;

[0202] The determination strategy is used to generate the combination coefficients.

[0203] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a musculoskeletal robot motion control method based on a recurrent neural network. The method includes: determining initial state primitives of the recurrent neural network based on a first initial state of the recurrent neural network and a deviation state corresponding to the first initial state; the first initial state is the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches a preset motion target; determining combination coefficients of the initial state primitives based on a multilayer perceptron neural network; determining muscle control signals based on the initial state primitives and the combination coefficients; and controlling the musculoskeletal robot to move to the target position based on the muscle control signals.

[0204] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0205] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the musculoskeletal robot motion control method based on a recurrent neural network provided by the above methods. The method includes: determining initial state primitives of the recurrent neural network based on a first initial state of the recurrent neural network and a deviation state corresponding to the first initial state; the first initial state is the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches a preset motion target; determining combination coefficients of the initial state primitives based on a multilayer perceptron neural network; determining muscle control signals based on the initial state primitives and the combination coefficients; and controlling the musculoskeletal robot to move to the target position based on the muscle control signals.

[0206] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the motion control method for a musculoskeletal robot based on a recurrent neural network provided by the methods described above. The method includes: determining initial state primitives of the recurrent neural network based on a first initial state and a deviation state corresponding to the first initial state; the first initial state being the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches a preset motion target; determining combination coefficients of the initial state primitives based on a multilayer perceptron neural network; determining a muscle control signal based on the initial state primitives and the combination coefficients; and controlling the musculoskeletal robot to move to the target position based on the muscle control signal.

[0207] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0208] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A motion control method for musculoskeletal robots based on recurrent neural networks, characterized in that, include: Based on the first initial state of the recurrent neural network and the deviation state corresponding to the first initial state, the initial state primitive of the recurrent neural network is determined. The first initial state is the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches the preset motion target; The combination coefficients of the initial state primitives are determined based on a multilayer perceptron neural network. Based on the initial state primitives and the combination coefficients, the muscle control signal is determined; Based on the muscle control signals, the musculoskeletal robot is controlled to move to the target position; The determination of the initial state primitives of the recurrent neural network based on the first initial state and the corresponding deviation state of the first initial state includes: Based on the first initial state, the first signal output by the recurrent neural network is determined; Based on the aforementioned deviation state, the second signal output by the recurrent neural network is determined; Based on the first signal and the second signal, a first loss function is determined; Based on the first loss function, the initial state primitives of the recurrent neural network are determined; The determination of muscle control signals based on the initial state primitives and the combination coefficients includes: Based on the combination coefficients and the initial state primitives, the second initial state of the recurrent neural network is determined; The second initial state is input into the recurrent neural network to determine the muscle control signal.

2. The musculoskeletal robot motion control method based on recurrent neural networks according to claim 1, characterized in that, The expression for the second initial state is as follows: ； in, This represents the second initial state. Represents the combination coefficient. This represents the initial state primitive.

3. The musculoskeletal robot motion control method based on recurrent neural networks according to claim 1, characterized in that, After controlling the musculoskeletal robot to move to the target position based on the muscle control signal, the method further includes: Based on the target position, the actual position, velocity, and initial position of the musculoskeletal robot are used to determine the second loss function; Based on the second loss function, a reward signal is generated; The multilayer perceptron neural network is optimized based on the reward signal.

4. The musculoskeletal robot motion control method based on recurrent neural networks according to claim 3, characterized in that, The optimization of the multilayer perceptron neural network based on the reward signal includes: Based on the aforementioned multilayer perceptron neural network, a random beta strategy is determined; The random beta strategy is optimized based on the reward signal to obtain a deterministic strategy; The determination strategy is used to generate the combination coefficients.

5. A motion control device for a musculoskeletal robot based on a recurrent neural network, characterized in that, include: The first determining module is used to determine the initial state primitives of the recurrent neural network based on the first initial state of the recurrent neural network and the deviation state corresponding to the first initial state. The first initial state is the optimal initial state of the recurrent neural network when the musculoskeletal robot reaches the preset motion target; The second determining module is used to determine the combination coefficients of the initial state primitives based on a multilayer perceptron neural network. The third determining module is used to determine the muscle control signal based on the initial state primitive and the combination coefficients; The control module is used to control the musculoskeletal robot to move to the target position based on the muscle control signals. The first determining module is specifically used for: Based on the first initial state, the first signal output by the recurrent neural network is determined; Based on the aforementioned deviation state, the second signal output by the recurrent neural network is determined; Based on the first signal and the second signal, a first loss function is determined; Based on the first loss function, the initial state primitives of the recurrent neural network are determined; The third determining module is specifically used for: Based on the combination coefficients and the initial state primitives, the second initial state of the recurrent neural network is determined; The second initial state is input into the recurrent neural network to determine the muscle control signal.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the musculoskeletal robot motion control method based on recurrent neural networks as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the musculoskeletal robot motion control method based on recurrent neural networks as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the musculoskeletal robot motion control method based on recurrent neural networks as described in any one of claims 1 to 4.

Images