Adaptive control method, device and storage medium for intelligent prosthetic shoulder joint

Abstract

The present invention relates to an adaptive control method, device and storage medium for an intelligent prosthetic shoulder joint, the method comprising the following steps: step S1, constructing an adaptive control system based on a modular shoulder joint model; step S2, using a deep deterministic policy gradient DDPG algorithm to self-learn and self-tune the adaptive controller parameters, and according to the arm movement strategy, a reward mechanism adapted to the modular shoulder joint is given to train the network and achieve preliminary joint control; step S3, combining the adaptive controller after adjustment and training with the subject, performing multiple complete motion training according to the set requirements, and obtaining the final control strategy suitable for the current subject. Compared with the prior art, the present invention greatly reduces the training cost and workload compared to reinforcement learning trained without parameters; and enables the modular shoulder joint to achieve good dynamic response under the trained controller, greatly increasing the coordination between people and intelligent prostheses.

Description

Technical Field

[0001] The present invention relates to the technical field of bionic medical devices, and in particular to an adaptive control method, device and storage medium for an intelligent prosthetic shoulder joint. Background Art

[0002] The loss of an arm often makes it difficult for upper limb amputees to take care of themselves at work and in their daily lives. Since the development of artificial intelligence, the design, production and assembly of artificial prostheses to compensate for amputees have taken on a new development direction. However, for patients with shoulder dislocation amputations, the arm is already missing from the humerus or part of the scapula, and the function of the entire arm has been lost. Therefore, the research on the bionic design and control of the shoulder joint is very important.

[0003] The human shoulder joint is a ball-and-socket joint with three degrees of freedom, which carries the movement of the entire arm. The load capacity and control accuracy of this position determine whether the bionic arm can function normally. In order to meet the degrees of freedom of the shoulder joint, at least three motors must be equipped, which leads to the problems of low output, heavy weight and high noise of the bionic arm.

[0004] After searching, Chinese patent application CN113995561A discloses an adaptive control system for modular shoulder joints of bionic arms. The system obtains coding values ​​through an adaptive control strategy based on motor speed error, thereby driving and controlling the modular shoulder joints. However, the system only uses traditional adaptive control strategies, which makes it difficult to accurately adapt to diverse arm movements, especially to cope with rapidly changing movements and sudden changes in loads, affecting the wearing effect.

[0005] In view of the above defects, it is urgent to design a more adaptable modular shoulder joint control strategy. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a highly adaptable adaptive control method, device and storage medium for an intelligent prosthetic shoulder joint.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to a first aspect of the present invention, there is provided an adaptive control method for an intelligent prosthetic shoulder joint, the method comprising the following steps:

[0009] Step S1, constructing an adaptive control system based on a modular shoulder joint model;

[0010] Step S2: using the deep deterministic policy gradient (DDPG) algorithm to self-learn and self-tune the parameters of the adaptive controller, and according to the arm movement strategy, a reward mechanism adapted to the modular shoulder joint is given to train the network and achieve preliminary joint control;

[0011] Step S3: Combine the trained adaptive controller with the subject, perform multiple complete motion trainings according to the set requirements, and obtain a final control strategy suitable for the current subject.

[0012] Preferably, the modular shoulder joint model in step S1 is specifically expressed as:

[0013]

[0014] In the formula, ω r is the motor speed, i q is the q-axis phase current; u q 、u d is the phase voltage in the dq coordinate system; φ is the flux generated by the permanent magnet of the brushless DC motor, R s is the phase resistance of the brushless DC motor, L s is the inductance of the brushless DC motor, p is the number of pole pairs of the brushless DC motor, J is the moment of inertia of the brushless DC motor rotor, and B is the viscous friction coefficient; T o The actual torque provided by the motor.

[0015] Preferably, the adaptive control system based on the modular shoulder joint model in step S1 is specifically:

[0016] 1) Collect the motor speed ω of the modular shoulder joint r and the three-phase current signal i loaded on the modular shoulder joint a 、i b 、i c ; Set the motor speed ω r and the motor speed reference signal ω d Make a difference and get the motor speed deviation e1; the three-phase current signal i a 、i b 、i c The current signal i in the dq coordinate system is obtained by Clake transformation and Park transformation in sequence. d 、i q , the q-axis current signal i q With the q-axis current reference signal q dr Make a difference and get the motor q-axis current deviation e2;

[0017] 2) The motor speed deviation e1, the motor q-axis current deviation e2 and the load torque T o , input to the adaptive controller, output voltage control signal u d 、u q ;

[0018] 3) Voltage control signal u d 、uq After inverse Park transformation, SVPWM and three-phase inverter, the new phase current signal i is obtained. a 、i b 、i c , loading on the modular shoulder joint.

[0019] Preferably, the step S2 specifically includes the following sub-steps:

[0020] Step S21: In the initial state, according to the current Actor network and random noise, a reinforced proxy action output a is mapped. t ; in state S at time t t The controlled object performs action a t After that, the reward value r is returned t and the next state S t+1 ; Among them, the state includes the motor speed deviation and the tracking error of the q-axis phase current;

[0021] Step S22, the Actor network converts the state process (S t ,a t ,r t ,S t+1 ) is stored in the memory;

[0022] Step S23: randomly sample N state transition processes (S i ,a i ,r i ,S i+1 ) as the small round training data of the Actor-Critic network, so as to obtain the independent Critic network and Actor network to update the Critic network;

[0023] Step S24, training and updating the Actor network by minimizing the loss function L;

[0024] Step S25: Obtain appropriate reinforcement learning agent output after multiple iterations.

[0025] Preferably, the state S in step S21 t The specific expression is:

[0026] S t =[e1(t),e2(t),∫e1(t),∫e2(t),ω r (t)] T

[0027] Where, e1(t) and e2(t) are the motor speed deviation and tracking error of the q-axis phase current at time t, respectively; ∫e1(t) and ∫e2(t) are the motor speed cumulative deviation and tracking cumulative error of the q-axis phase current at time t, respectively; ω r (t) is the motor speed at time t.

[0028] Preferably, the reward value r in step S21 is t Specifically:

[0029] r t =α1m1(t)+α2m2(t)

[0030] In the formula, α1 and α2 are the reward coefficients that limit the error value range and feedback value range respectively; m1(t) and m2(t) are the error value range and feedback value range respectively, and the expressions are:

[0031] m1(t)=-c1e1(t) 2 -c2e2(t) 2

[0032]

[0033] Among them, e1 and e2 are the motor speed deviation and the tracking error of the q-axis phase current respectively; c1 and c2 are the reward parameters set according to the error value and the emphasis of the evaluation performance; The lower and upper limits of the motor speed for stable operation.

[0034] Preferably, the update expression of the Critic network in step S23 is:

[0035]

[0036] In the formula, y i is the output value of the updated Critic network; γ is the discount factor, satisfying 0<γ≤1; r t is the reward value; Q - is the state estimate of the Critic network output before the update; μ is the output value of the Actor network, μ is the output value of the Actor network, is the network parameter, S t+1 The state of the next moment.

[0037] Preferably, the loss function L in step S24 is expressed as:

[0038]

[0039] Where N is the number of sampling steps; y iis the output value of the updated Critic network; Q is the output value of the current Critic network, s i is the current state, a i is the reinforcement agent action output at the current moment, θ Q is the parameter of the current Critic network.

[0040] According to a second aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, any one of the methods described above is implemented.

[0041] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, any one of the methods described above is implemented.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] 1) Compared with reinforcement learning trained without parameters, the deep deterministic policy gradient DDPG algorithm adopted by the present invention can greatly reduce the training cost and workload, and ultimately enable the modular shoulder joint to achieve good dynamic response under the trained controller, greatly improving the coordination between people and intelligent prostheses;

[0044] 2) The modular shoulder joint structure adopted by the present invention integrates a driver, a brushless DC motor, a mechanical brake and a harmonic reducer. Compared with existing drive devices such as stepper motors and rope drives, the modular shoulder joint has a small size and high output torque, and is more suitable for bionic arms. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A control flow chart of an adaptive control system based on a modular shoulder joint model of the present invention;

[0046] Figure 2 This is a flowchart of reinforcement learning control;

[0047] Figure 3 This is a framework diagram of the reinforcement learning network;

[0048] Figure 4 It is the overall framework of the control system. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] Example

[0051] This embodiment provides an adaptive control method for an intelligent prosthetic shoulder joint. Figures 1 to 4 As shown, including:

[0052] 1. Establish a modular shoulder joint model:

[0053] The modular shoulder joint structure of this embodiment integrates the driver, brushless DC motor, mechanical brake and harmonic reducer into one. Considering the rotational inertia and viscous friction coefficient of the brushless DC motor rotor, the uncertain friction resistance caused by the harmonic reducer is included in the load, and the mathematical model of the modular shoulder joint is obtained:

[0054]

[0055] In the formula, ω r is the motor speed, i q is the q-axis phase current; u q 、u d is the phase voltage in the dq coordinate system; φ is the flux generated by the permanent magnet of the brushless DC motor, R s is the phase resistance of the brushless DC motor, L s is the inductance of the brushless DC motor, p is the number of pole pairs of the brushless DC motor, J is the moment of inertia of the brushless DC motor rotor, and B is the viscous friction coefficient; T o The actual torque provided by the motor.

[0056] 2. Build an adaptive control system based on a modular shoulder joint model:

[0057] 1) Collect the motor speed ω of the modular shoulder joint r and the three-phase current signal i loaded on the modular shoulder joint a 、i b 、i c ; Set the motor speed ω r and the motor speed reference signal ω d Make a difference and get the motor speed deviation e1; the three-phase current signal i a 、i b 、i c The current signal i in the dq coordinate system is obtained by Clake transformation and Park transformation in sequence. d 、i q , the q-axis current signal i q With the q-axis current reference signal q dr Make a difference and get the motor q-axis current deviation e2;

[0058] 2) The motor speed deviation e1, the motor q-axis current deviation e2 and the load torque To , input to the adaptive controller, output voltage control signal u d 、u q ;

[0059] 3) Voltage control signal u d 、u q After inverse Park transformation, SVPWM and three-phase inverter, the new phase current signal i is obtained. a 、i b 、i c , loading on the modular shoulder joint.

[0060] In order to meet the requirement that the system can still achieve the expected performance while being subject to unknown resistance, the estimated Replace the load T0. Redefine the error signal containing the desired variable and combine it with the Lyapunov function to obtain the required adaptive controller.

[0061] 3. Deep Deterministic Policy Gradient (DDPG): Build a reinforcement learning environment based on the input and output of the system's mathematical model, use the combination of deep networks and Actor-Critic learning to build a reinforcement learning network, and consider the reward and punishment mechanism of the system operation; design and analyze the controller for input and output data, use batch data to train the simulation agent, and finally train a suitable initial reinforcement learning agent and introduce subjects to perform prescribed action training, so that the reinforcement agent updates its strategy according to environmental changes. Figure 2 and Figure 3 shown.

[0062] 1) In the initial state, according to the current Actor network and random noise, a reinforced proxy action output a is mapped t , represents the motor acceleration output at time t; in state S at time t t The controlled object performs action a t After that, the reward value r is returned t and the next state S t+1 ;

[0063] Status S t The specific formula is:

[0064] S t =[e1(t),e2(t),∫e1(t),∫e2(t),ω r (t)] T

[0065] Considering the influence of system error and feedback value range on system control performance, the reward value r t Set to:

[0066] r t=α1m1(t)+α2m2(t)

[0067] In the formula, α1 and α2 are the reward coefficients that limit the error value range and feedback value range respectively; m1(t) and m2(t) are the error value range and feedback value range respectively, and the expressions are:

[0068] m1(t)=-c1e1(t) 2 -c2e2(t) 2

[0069]

[0070] Where, e1(t) and e2(t) are the motor speed deviation and tracking error of the q-axis phase current at time t, respectively; ∫e1(t) and ∫e2(t) are the motor speed cumulative deviation and tracking cumulative error of the q-axis phase current at time t, respectively; ω r (t) is the motor speed at time t.

[0071] 2) The Actor network converts the state into t ,a t ,r t ,S t+1 ) is stored in the memory;

[0072] 3) Randomly sample N state transition processes (S i ,a i ,r i ,S i+1 ) as the small round training data of the Actor-Critic network, thereby obtaining independent Critic network and Actor network;

[0073] By minimizing the loss function L, the Critic network is trained and updated:

[0074]

[0075] Where N is the number of sampling steps; Q is the output value of the current Critic network, s i is the current state, a i is the reinforcement agent action output at the current moment, θ Q is the parameter of the current Critic network; y i is the output value of the Critic network, satisfying:

[0076]

[0077] In the formula, y i is the updated output value of the Critic network; γ is the discount factor, satisfying 0<γ≤1; rt is the reward value; Q - is the state estimate output by the Critic network; μ is the output value of the Actor network, is the network parameter, S t+1 The state of the next moment;

[0078] 4) Continuously update the Actor target network according to the loss gradient;

[0079] 5) Obtain appropriate reinforcement learning agent output after multiple iterations.

[0080] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0081] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0082] The processing unit performs the various methods and processes described above, such as methods S1 to S3. For example, in some embodiments, methods S1 to S3 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S3 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S3 in any other appropriate manner (e.g., by means of firmware).

[0083] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0084] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0085] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0086] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An adaptive control method for an intelligent prosthetic shoulder joint, characterized in that: The method comprises the following steps: Step S1, constructing an adaptive control system based on a modular shoulder joint model; Step S2: using the deep deterministic policy gradient (DDPG) algorithm to self-learn and self-tune the parameters of the adaptive controller, and according to the arm movement strategy, a reward mechanism adapted to the modular shoulder joint is given to train the network and achieve preliminary joint control; Step S3, combining the trained adaptive controller with the subject, performing multiple complete motion trainings according to the set requirements, and obtaining a final control strategy suitable for the current subject; The modular shoulder joint model in step S1 is specifically expressed as: In the formula, ω r is the motor speed, i q is the q-axis phase current; u q 、u d is the phase voltage in the dq coordinate system; φ is the flux generated by the permanent magnet of the brushless DC motor, R s is the phase resistance of the brushless DC motor, L s is the inductance of the brushless DC motor, p is the number of pole pairs of the brushless DC motor, J is the moment of inertia of the brushless DC motor rotor, and B is the viscous friction coefficient; T o The torque actually provided to the motor; The adaptive control system based on the modular shoulder joint model in step S1 is specifically: 1) Collect the motor speed ω of the modular shoulder joint r and the three-phase current signal i loaded on the modular shoulder joint a 、i b 、i c ; Set the motor speed ω r and the motor speed reference signal ω d Make a difference and get the motor speed deviation e1; the three-phase current signal i a 、i b 、i c The current signal i in the dq coordinate system is obtained by Clake transformation and Park transformation in sequence. d 、i q , the q-axis current signal i q With the q-axis current reference signal q dr Make a difference and get the motor q-axis current deviation e2; 2) The motor speed deviation e1, the motor q-axis current deviation e2 and the load torque T o , input to the adaptive controller, output voltage control signal u d 、u q ; 3) Voltage control signal u d 、u q After inverse Park transformation, SVPWM and three-phase inverter, the new phase current signal i is obtained. a 、i b 、i c , loading on the modular shoulder joint; The step S2 specifically includes the following sub-steps: Step S21: In the initial state, according to the current Actor network and random noise, a reinforced proxy action output a is mapped. t ; in state S at time t t The controlled object performs action a t After that, the reward value r is returned t and the next state S t+1 ; Among them, the state includes the motor speed deviation and the tracking error of the q-axis phase current; Status S t The specific expression is: S t =[e1(t),e2(t),∫e1(t),∫e2(t),ω r (t)] T Where, e1(t) and e2(t) are the motor speed deviation and motor q-axis current deviation at time t, respectively; ∫e1(t) and ∫e2(t) are the motor speed cumulative deviation and motor q-axis current cumulative deviation at time t, respectively; ω r (t) is the motor speed at time t; Reward value r t Specifically: r t =α1m1(t)+α2m2(t) In the formula, α1 and α2 are the reward coefficients that limit the error value range and feedback value range respectively; m1(t) and m2(t) are the error value range and feedback value range respectively, and the expressions are: m1(t)=-c1e1(t) 2 -c2e2(t) 2 Among them, c1 and c2 are reward parameters set according to the error value and the emphasis of evaluation performance; To set the lower and upper limits of the motor speed for stable operation; Step S22, the Actor network converts the state process (S t ,a t ,r t ,S t+1 ) is stored in the memory; Step S23: randomly sample N state transition processes (S i ,a i ,r i ,S i+1 ) as the small round training data of the Actor-Critic network, so as to obtain the independent Critic network and Actor network to update the Critic network; Step S24, training and updating the Actor network by minimizing the loss function L; Step S25: Obtain appropriate reinforcement learning agent output after multiple iterations.

2. The adaptive control method for an intelligent prosthetic shoulder joint according to claim 1, characterized in that: The update expression of the Critic network in step S23 is: In the formula, y i Output value of the updated Critic network; γ is the discount factor, satisfying 0<γ≤1; r t is the reward value; Q - is the state estimate output by the Critic network; μ is the output value of the Actor network, is the network parameter, S t+1 The state of the next moment.

3. The adaptive control method for an intelligent prosthetic shoulder joint according to claim 2, characterized in that: The loss function L in step S24 is expressed as: Where N is the number of sampling steps; y i is the updated output value of the Critic network; Q is the state estimate output by the Critic network; s i is the current state, a i is the reinforcement agent action output at the current moment, θ Q is the Critic network parameter.

4. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images