Action-behavior-evaluation neural network learning control method in human-computer interaction process
Through the action-behavior-critical neural network learning control method, the behavior force estimation accuracy and nonlinear control problems in human-computer interaction are solved, and more efficient control accuracy and system learning effect are achieved.
Patent Information
- Application Number
- CN202411255416.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-09
AI Technical Summary
The prior art has limitations in improving the accuracy of behavioral force estimation during human-computer interaction, and traditional control methods cannot effectively deal with the nonlinear and dynamic coupling problems of tandem force tactile feedback devices.
Using the action-behavior-critical neural network learning control method, the human-computer interaction dynamics model and multi-layer RBF neural network are constructed, combined with the augmented system representation and radial basis function, the weights of the behavior network and the action network are updated to approximate the behavior. Torque and output torque improve system control accuracy.
It realizes the learning of human-computer interaction system, improves control accuracy, can more accurately estimate behavioral torque and output torque, and improves the dynamic programming performance of the system.
Smart Images

Figure CN119065504A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of human-computer interaction collaborative teleoperation system, and in particular to an action-behavior-criticism neural network learning control method for a human-computer interaction process. Background Art
[0002] In the process of studying the behavioral stability problem of human interaction with serial force tactile feedback devices in teleoperation systems, human behavior in the human-computer interaction process is taken as the research object, where human behavior is described by the interaction force exerted by humans on the end of the force tactile feedback device. Since most force tactile feedback devices do not have force sensors, it is impossible to directly obtain human interaction behavior force from the force tactile feedback device, resulting in confusion between human interaction behavior force and the control force output by the device.
[0003] The existing processing method of using observers to estimate behavioral forces is effective, but since the observer needs to set observer gain parameters, and this parameter is not adaptive, the observer method has limitations in further improving the estimation accuracy. Secondly, since the serial force tactile feedback device used has the characteristics of nonlinearity, dynamic coupling and model parameter uncertainty, the traditional control method cannot achieve the desired effect.
[0004] Therefore, it is necessary to provide an action-behavior-criticism neural network learning control method for the human-computer interaction process to solve the above problems. Summary of the invention
[0005] The present invention provides an action-behavior-criticism neural network learning control method for a human-computer interaction process to solve the problem that the existing observer method has limitations in further improving the estimation accuracy. Secondly, due to the characteristics of nonlinearity, dynamic coupling and model parameter uncertainty of the serial force tactile feedback device used, the traditional control method cannot achieve the desired effect.
[0006] The action-behavior-criticism neural network learning control method of a human-computer interaction process of the present invention adopts the following technical scheme, including:
[0007] According to the joint angle position, joint angular velocity, joint angular acceleration, inertia matrix, centripetal force and Coriolis force matrix, gravity matrix and output torque of each joint in the joint space of the force tactile feedback device during the human-computer interaction process, and the behavioral torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device on each joint, a human-computer interaction dynamics model is constructed; the human-computer interaction dynamics model is represented by an augmented system, and the state vector of the augmented system is obtained according to the joint angle position, joint angular velocity and expected trajectory of the force tactile feedback device in the joint space during the human-computer interaction process;
[0008] The Gaussian function is used as the radial basis function and the behavior network is constructed according to the multi-layer RBF neural network. According to the system state observer, radial basis function and human-computer interaction dynamics model in the human-computer interaction process, the weight estimation vector update law of the behavior network is obtained. The behavior network is updated based on the weight estimation vector update law, and the updated behavior network is used to obtain the behavior torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device on each joint during the human-computer interaction process;
[0009] According to the output torque of each joint of the force tactile feedback device in the joint space during the human-computer interaction process, the behavior torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device on each joint, and the state vector of the augmented system, the performance index function of the system during the human-computer interaction process is obtained;
[0010] Constructing an action network and a criticism network, obtaining an update law for the weight estimation vectors of the action network and the criticism network according to the performance indicator function, the activation vectors and the weight estimation vectors corresponding to the action network and the criticism network, and the behavior torque output by the behavior network, updating the action network according to the update law for the weight estimation vectors of the action network and the criticism network, and obtaining the output torque of each joint of the force tactile feedback device according to the updated action network;
[0011] According to the updated behavior network, the behavior torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device in the human-computer interaction process is obtained, and the output torque of each joint of the force tactile feedback device is obtained according to the updated action network, and the state observation error term and joint angular velocity observation value of the state observer are updated, as well as the joint angle position and joint angular velocity of the human-computer interaction dynamics model.
[0012] Preferably, the expression of the human-computer interaction dynamics model is:
[0013]
[0014] Where q represents the joint angle position of the force tactile feedback device in the joint space during human-computer interaction; It represents the joint angular velocity of the force tactile feedback device in the joint space during human-computer interaction; represents the joint angular acceleration of the force tactile feedback device in the joint space during the human-computer interaction process; M represents the inertia matrix of the force tactile feedback device in the joint space during the human-computer interaction process; C represents the centripetal force and Coriolis force matrix of the force tactile feedback device in the joint space during the human-computer interaction process; G represents the gravity matrix of the force tactile feedback device in the joint space during the human-computer interaction process; τ s represents the output torque of each joint of the force tactile feedback device in the joint space during human-computer interaction; τ hIt represents the behavioral torque on each joint of the interactive behavioral force applied by the operator at the end of the force tactile feedback device.
[0015] Preferably, the expression of the augmented system is:
[0016]
[0017]
[0018]
[0019] In the formula, represents the state vector of the augmented system at time t, and e represents the joint angle position tracking error of the system; represents the time derivative of the state vector r(t) of the augmented system at time t, represents a Lipschitz continuous function Transpose; Represents the derivative of the joint angle position tracking error of the system; q d represents the desired joint angle position of each joint; represents the derivative of the desired joint angle position of each joint with respect to time, where the subscript d represents expectation; F(r(t)) represents the nonlinear dynamic function; H(r(t)) represents the gain function; τ s represents the output torque of each joint of the force tactile feedback device in the joint space during human-computer interaction; τ h It represents the behavioral torque on each joint of the interactive behavioral force applied by the operator at the end of the force tactile feedback device; represents the transpose of the system state function; h T (e+q d ) represents the transpose of the system output gain function; represents a row vector of length n consisting of zeros.
[0020] Preferably, the expression of the system state observer in the human-computer interaction process is:
[0021]
[0022] In the formula, represents the derivative of the joint angular velocity observation; M -1 represents the inverse matrix of the inertia matrix of the force tactile feedback device in the joint space during the human-computer interaction process; C represents the centripetal force and Coriolis force matrix of the force tactile feedback device in the joint space during the human-computer interaction process; G represents the gravity matrix of the force tactile feedback device in the joint space during the human-computer interaction process; τ s It represents the output torque of each joint of the force tactile feedback device in the joint space during the human-computer interaction process; represents the estimated behavioral torque; σ represents the observed value of joint angular velocity; represents the state observation error; D represents the positive diagonal matrix; ζ represents the state observation error term; L represents the error correction matrix.
[0023] Preferably, the update law of the weight estimation vector of the behavior network is expressed as:
[0024]
[0025] In the formula, the behavior network includes the initial behavior network and the j-layer extended behavior network; Represents the weight estimation vector update law of the initial behavior network in the behavior network; represents the update law of the weight estimation vector of the i-th layer extended behavior network in the behavior network; represents the update law of the weight estimation vector of the j-th extended behavior network in the behavior network; Represents the weight estimation vector of the i+1th layer extended behavior network in the behavior network; A weight estimation vector representing an initial behavior network in the behavior network; Represents the weight estimation vector of the i-th layer extended behavior network in the behavior network; Represents the weight estimation vector of the j-th extended behavior network in the behavior network; Represents the weight estimation vector of the first layer of the extended behavior network in the behavior network; represents the activation vector of the behavior network; ζ represents the state observation error term; M -1 represents the inverse matrix of the inertia matrix of the force tactile feedback device in the joint space during human-computer interaction; δ h represents the initial behavior network learning rate in the behavior network, δ hi represents the learning rate of the i-th layer extended behavior network in the behavior network, δ hj represents the learning rate of the j-th extended behavior network in the behavior network; k h is the adjustable parameter of the initial behavior network, k hi represents the adjustable parameter of the i-th extended behavior network in the behavior network, k hj Represents the adjustable parameters of the j-th extended behavior network in the behavior network.
[0026] Preferably, the expression of the performance indicator function of the system during the human-computer interaction process is:
[0027]
[0028] Where V represents the performance index function of the system in the human-computer interaction process; α>0 represents the discount factor; Q=(Q1,0 2n×2n ; 0 2n×2n ,0 2n×2n) is a positive semidefinite matrix, is a positive definite matrix; represents a set of real matrices of dimension 2n×2n; R and K are both positive definite matrices; τ s represents the output torque of each joint of the force tactile feedback device in the joint space during human-computer interaction; τ h represents the behavioral torque of the interactive behavioral force applied by the operator at the end of the force tactile feedback device on each joint; r(ι) represents the state vector of the augmented system at time ι, where ι is the integral variable on the time interval [t,∞].
[0029] Preferably, the expressions for constructing the action network and the criticism network are:
[0030] The expression of the action network is:
[0031]
[0032] Critical network expressions:
[0033]
[0034] In the formula, It represents the estimated output torque of each joint of the force tactile feedback device in the joint space during the human-computer interaction process; It represents the estimated value of the performance index function of the system during human-computer interaction; β represents the output peak limit; represents the weight estimation vector of the action network; represents the activation vector of the action network; represents the weight estimation vector of the critic network; Represents the activation vector of the critic network.
[0035] Preferably, the update law of the weight estimation vector of the action network and the criticism network is expressed as:
[0036] Define the weight estimation vectors for the action network and the critic network Then we have:
[0037]
[0038] In the formula, Represents the update law of the weight estimation vector of the action network and the critic network; represents the weight estimation vector of the action network; The weight estimation vector of the critic network; δ represents the learning rate of the action network and the critic network; ψ represents the residual error caused by the action network and the critic network in the performance indicator function; φ represents the residual error caused by the action network and the critic network in the rest of the function.
[0039] Preferably, the expression of ψ is:
[0040]
[0041] Where ψ represents the residual error caused by the action network and the criticism network in the performance index function; Q = (Q1,02n ×2 n;02n ×2 n,02n ×2 n) is a semi-positive definite matrix, Q1 is a positive definite matrix; the superscript T represents the transpose of the matrix or function output; δ represents the learning rate; K is a positive definite matrix; α represents an adjustable parameter; β represents the output peak limit; represents the weight estimation vector of the initial behavior network; represents the transpose of the activation vector of the behavior network; r(ι) represents the state vector of the augmented system at time ι, where ι is the time interval [tT p ] is the integral variable; κ is the output interval The integral variable on .
[0042] Preferably, the expression of φ is:
[0043]
[0044]
[0045]
[0046] In the formula, φ c represents the residual error caused by the critical network; φ a represents the residual error caused by the action network; e represents the joint angle position tracking error of the system; R is a positive definite matrix; r(ι) represents the state vector of the augmented system at time ι, where ι is the time interval [tT p ,t]; T p Indicates the operation interval time; represents the activation vector of the action network; represents the activation vector of the critical network; α represents an adjustable parameter; β represents the output peak limit; represents the matrix Kronecker product operator; r(tT p ) indicates tT p The state vector of the augmented system at time t; r(t) represents the state vector of the augmented system at time t.
[0047] The beneficial effects of the present invention are:
[0048] The present invention approximates the behavioral torque of human interaction behavior force at each joint in the process of human-computer interaction through a multi-layer behavior network, that is, according to the system state observer, radial basis function and human-computer interaction dynamics model in the process of human-computer interaction, obtains the weight estimation vector update law of the behavior network, updates the behavior network based on the weight estimation vector update law, and respectively adopts the action network and the criticism network to approximate the output torque and the performance index function, and uses the performance index function to evaluate the current system state, the action network strategy and the behavior network strategy, and guides the action network to learn, that is, according to the behavior network, the performance index of the adaptive dynamic programming process is designed, so that the action network approaches the optimal solution direction corresponding to the system under the premise of considering the behavior force, thereby obtaining the weight estimation vector update law of the action network and the criticism network, and then, using the weight estimation vector update law to update the corresponding action network, and finally, using the output values of the updated action network and behavior network to update the state observation error term and joint angular velocity observation value of the state observer, as well as the joint angle position and joint angular velocity of the human-computer interaction dynamics model, thereby realizing the learning of the human-computer interaction system and improving the control accuracy of the system after learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0050] Figure 1 A flow chart of an action-behavior-criticism neural network learning control method for a human-computer interaction process of the present invention;
[0051] Figure 2 A schematic diagram of the structure of an action-behavior-criticism neural network in an embodiment of the present invention;
[0052] Figure 3 It is a schematic diagram of the error trajectory of the behavior network and the system state observer estimating the behavior torque during the simulation process of this embodiment;
[0053] Figure 4 Schematic diagram of the behavior torque and estimated behavior torque trajectory of joint 1 during the simulation process of this embodiment;
[0054] Figure 5 Schematic diagram of the behavior torque and estimated behavior torque trajectory of joint 2 during the simulation process of this embodiment;
[0055] Figure 6Schematic diagram of the behavior torque and estimated behavior torque trajectory of the joint 3 during the simulation process of this embodiment;
[0056] Figure 7 This is a schematic diagram of the weight trajectory of the behavior network nodes during the simulation process of this embodiment;
[0057] Figure 8 A schematic diagram of the joint angle position tracking error trajectory during the simulation process of this embodiment;
[0058] Fig. 9 A schematic diagram of the trajectory of the output torque estimation value estimated by the action network during the simulation process of this embodiment;
[0059] Fig.10 This is a schematic diagram of the weight trajectory of the action network during the simulation process of this embodiment;
[0060] Fig.11 This is a schematic diagram of the critical network weight trajectory during the simulation process of this embodiment. DETAILED DESCRIPTION
[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0062] An embodiment of the action-behavior-criticism neural network learning control method of a human-computer interaction process of the present invention is as follows: Figure 1 As shown, including:
[0063] S1. Build a human-computer interaction dynamics model and obtain the state vector of the augmented system;
[0064] According to the joint angle position, joint angular velocity, joint angular acceleration, inertia matrix, centripetal force and Coriolis force matrix, gravity matrix and output torque of each joint in the joint space of the force tactile feedback device during the human-computer interaction process, as well as the behavioral torque of the interactive behavioral force applied by the operator at the end of the force tactile feedback device on each joint, a human-computer interaction dynamics model is constructed; the human-computer interaction dynamics model is represented by the augmented system, and the state vector of the augmented system is obtained according to the joint angle position, joint angular velocity and expected trajectory of the force tactile feedback device in the joint space during the human-computer interaction process.
[0065] Among them, the expression of the human-computer interaction dynamics model is:
[0066]
[0067] Where q represents the joint angle position of the force tactile feedback device in the joint space during human-computer interaction. It represents the joint angular velocity of the force tactile feedback device in the joint space during human-computer interaction. Represents the joint angular acceleration of the force tactile feedback device in the joint space during human-computer interaction M represents the inertia matrix of the force tactile feedback device in the joint space during human-computer interaction. C represents the centripetal force and Coriolis force matrix of the force tactile feedback device in the joint space during human-computer interaction. G represents the gravity matrix of the force tactile feedback device in the joint space during human-computer interaction. τ s It represents the output torque of each joint of the force tactile feedback device in the joint space during human-computer interaction. τ h It represents the behavioral torque of the interactive behavioral force applied by the operator at the end of the force tactile feedback device on each joint, n is the number of joints; Represents a set of vectors of dimension n; Represents a collection of matrices of dimension n×n.
[0068] The human-computer interaction system can be described by an uncertain model as follows:
[0069]
[0070] in, represents the unknown nonlinear dynamic function of the system, h(q) is the unknown output gain function,
[0071] The joint angle position tracking error of the control system can be expressed as e = qq d , whose derivative is expressed as in, represents the desired joint angle position of each joint, is the desired joint angle position q d The derivative with respect to time, is the desired joint angle position q d The second derivative with respect to time. Assume that there exists a Lipschitz continuous function Make And the function is equal to 0 at zero point, which is expressed as m(0) = 0. Then the human-computer interaction dynamics model under the trajectory tracking problem can be described by the augmented system. Specifically, the expression of the augmented system is:
[0072]
[0073]
[0074]
[0075] In the formula, r(t) represents the state vector of the augmented system at time t, represents a set of vectors with a dimension of 4n; e represents the joint angle position tracking error of the system; represents the time derivative of the state vector r(t) of the augmented system at time t, represents a Lipschitz continuous function Transpose; Represents the derivative of the joint angle position tracking error of the system; q d represents the desired joint angle position of each joint; represents the derivative of the desired joint angle position of each joint with respect to time, where the subscript d represents expectation; F(r(t)) represents the nonlinear dynamic function; H(r(t)) represents the gain function; τ s represents the output torque of each joint of the force tactile feedback device in the joint space during human-computer interaction; τ h It represents the behavioral torque on each joint of the interactive behavioral force applied by the operator at the end of the force tactile feedback device; represents the transpose of the system state function; h T (e+q d ) represents the transpose of the system output gain function; represents a row vector of length n consisting of 0s; it should be noted that (·) T Represents the transpose of a matrix vector. Taking the inertia matrix M as an example, its matrix transpose is M T , and this expression is used uniformly in the matrix-vector transposition in the following text.
[0076] S2. construct a behavior network, obtain the weight estimation vector update law of the behavior network and update the behavior network, and use the updated behavior network to obtain the behavior torque;
[0077] Specifically, a Gaussian function is used as a radial basis function and a behavior network is constructed according to a multi-layer RBF neural network. According to the system state observer, radial basis function and human-computer interaction dynamics model in the human-computer interaction process, the weight estimation vector update law of the behavior network is obtained. The behavior network is updated based on the weight estimation vector update law, and the updated behavior network is used to obtain the behavioral torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device on each joint in the human-computer interaction process.
[0078] In this embodiment, a multi-layer RBF neural network is used to approximate the behavior moment. The RBF neural network radial basis function uses a Gaussian function. The Gaussian function of the behavior network is expressed as:
[0079]
[0080] In the formula, is the activation function of the behavior network; represents a set of vectors of dimension c; c represents the number of nodes in the hidden layer of the behavioral neural network; is the center point of the Gaussian function, represents a set of vectors of dimension 2n; λ is the width of the Gaussian function; exp is a natural constant; Input vector for the Gaussian function.
[0081] Therefore, in the subsequent design, the behavior moment τ h The RBF neural network is described as follows:
[0082]
[0083] in, is the ideal weight vector of the behavior network; represents the set of real matrices of dimension n×r; is the approximation error of the behavior network, and n is the dimension of the basis function. Since the ideal weight vector of the behavior network cannot be directly obtained, the ideal weight vector is estimated as Represents the weight estimation vector of the behavior network, and the estimated behavior moment can be obtained In the present invention, it is assumed that the ideal weight vector of the behavior network is Activation Vector and the neural network estimation error ε h is bounded, and the derivative of the ideal weight vector of the behavior network is In traditional methods, it is generally believed that is equal to 0, but this method will bring additional estimation errors. In order to reduce the estimation error, the behavior network is expanded here. On the basis of the original behavior network, The derivative of the ideal weight matrix of the layer neural network used to represent the behavior network represents a set of real numbers, the original behavior network is collectively referred to as the initial behavior network, and the introduced neural network is collectively referred to as the extended behavior network; then the ideal weight matrix of the extended behavior network is expressed as υ=1,2,...,j, defined as follows:
[0084]
[0085] In the formula, The ideal weight estimation vector for the first layer of the behavior network; The ideal weight estimation vector for the extended behavior network at layer i+1 in the behavior network; The derivative of the ideal weight estimate vector of the extended behavior network for the jth layer of the behavior network; The derivative of the ideal weight estimate vector of the extended behavior network for the i-th layer of the behavior network; is the derivative of the ideal weight estimation vector of the initial behavior network; introduces the weight estimation vector for the extended behavior network To estimate the ideal weight vector At this time, the behavior network has a total of j+1 layers, including the initial behavior network, and the weight estimation vector of the initial behavior network is expressed as And the j-layer extended behavior network, whose weight estimation vector is expressed as The problem then turns into designing the weight estimation matrix of the initial behavior network And the weight estimation matrix of the extended behavior network and The update law of the system state observer is introduced to obtain the update law of the weight estimation vector of the behavior network. Among them, the system state observer in the human-computer interaction process is in the following form:
[0086]
[0087] In the formula, represents the derivative of the joint angular velocity observation, And there are represents the state observation error, ζ represents the state observation error term, is the state observation error feedback correction term, is a positive diagonal matrix. By adjusting the matrix D parameters, the accumulation of state observation errors in the state observer during the observation process can be suppressed. -1 represents the inverse matrix of the inertia matrix of the force tactile feedback device in the joint space during the human-computer interaction process; C represents the centripetal force and Coriolis force matrix of the force tactile feedback device in the joint space during the human-computer interaction process; G represents the gravity matrix of the force tactile feedback device in the joint space during the human-computer interaction process; τ s It represents the output torque of each joint of the force tactile feedback device in the joint space during the human-computer interaction process; represents the estimated behavior torque; σ represents the observed value of the joint angular velocity; D represents the positive diagonal matrix; L represents the error correction matrix, I n represents the identity matrix of dimension n×n; represents a set of real vectors of dimension n, represents a set of real matrices of dimension n×n. In order to avoid redundant information estimated by the behavior network in combination with the system state observer, X in formula (6) is selected as
[0088] At this point, according to formula (1), formula (6) and formula (9), the weight estimation vector update law of the multi-layer behavior network is designed as:
[0089]
[0090] In the formula, the behavior network includes the initial behavior network and the j-layer extended behavior network; Represents the weight estimation vector update law of the initial behavior network in the behavior network; represents the update law of the weight estimation vector of the i-th layer extended behavior network in the behavior network; represents the update law of the weight estimation vector of the j-th extended behavior network in the behavior network; Represents the weight estimation vector of the i+1th layer extended behavior network in the behavior network; A weight estimation vector representing an initial behavior network in the behavior network; Represents the weight estimation vector of the i-th layer extended behavior network in the behavior network; Represents the weight estimation vector of the j-th extended behavior network in the behavior network; Represents the weight estimation vector of the first layer of the extended behavior network in the behavior network; represents the activation vector of the behavior network; ζ represents the state observation error term; M -1 represents the inverse matrix of the inertia matrix of the force tactile feedback device in the joint space during human-computer interaction; δ h represents the initial behavior network learning rate in the behavior network, δ h >0;δ hi represents the learning rate of the i-th layer extended behavior network in the behavior network, δ hi >0,i=1,2,L,j-1;δ hj represents the learning rate of the j-th extended behavior network in the behavior network, δ hj >0; by adjusting the learning rate, the convergence speed of the behavior network is changed; k h is the adjustable parameter of the initial behavior network, k h >0;k hi represents the adjustable parameter of the i-th extended behavior network in the behavior network, k hi >0;k hj represents the adjustable parameter of the j-th extended behavior network in the behavior network, k hj >0.
[0091] At this point, the behavior network is updated using the weight estimation vector update law, and the updated behavior network can be used to obtain the behavior torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device on each joint during the human-computer interaction process.
[0092] S3, obtaining the performance index function of the system during human-computer interaction;
[0093] According to the output torque of each joint of the force tactile feedback device in the joint space during the human-computer interaction process, the behavioral torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device on each joint and the state vector of the augmented system, the performance index function of the system in the human-computer interaction process is obtained.
[0094] In this embodiment, after completing the behavior network design, it is necessary to obtain the output torque τ s The value of is used to design the control law, and the adaptive dynamic programming control method is used to design the control law. First, the performance index function in this method is designed. The traditional performance index function only contains the system state quadratic function and the control output quadratic function. However, in the process of human-computer interaction, the human interaction behavior will also affect the system performance. The use of traditional performance index functions may affect the performance of the control algorithm. Therefore, the performance index function of the system designed in the human-computer interaction process of the present invention is:
[0095]
[0096] Where V represents the performance index function of the system in the human-computer interaction process; α>0 represents the discount factor; Q=(Q1,0 2n×2n ; 0 2n×2n ,0 2n×2n ) is a positive semidefinite matrix, is a positive definite matrix; represents a set of real matrices of dimension 2n×2n; R and K are both positive definite matrices; τ s represents the output torque of each joint of the force tactile feedback device in the joint space during human-computer interaction; τ h represents the behavioral torque of the interactive behavioral force applied by the operator at the end of the force tactile feedback device on each joint; r(ι) represents the state vector of the augmented system at time ι, where ι is the integral variable on the time interval [t,∞].
[0097] S4, constructing an action network and a critical network, obtaining the weight estimation vector update law of the action network and the critical network and updating the action network, and obtaining the output torque according to the updated action network;
[0098] Specifically, an action network and a criticism network are constructed, and based on the performance indicator function, the activation vectors and weight estimation vectors corresponding to the action network and the criticism network, and the behavior torque output by the behavior network, the weight estimation vector update law of the action network and the criticism network is obtained; the action network is updated according to the weight estimation vector update law of the action network and the criticism network, and the output torque of each joint of the force tactile feedback device is obtained based on the updated action network.
[0099] Step 41: The steps for constructing the action network and the criticism network are:
[0100] Since the performance index function of formula (11) is complex, it is impossible to obtain the performance index function value by direct integration. The present invention introduces a neural network to solve this problem, that is, introduces an action network to approximate τ s The critical network is introduced to approximate V. The hyperbolic tangent function tanh(·) is introduced to describe the output constraint problem, which is described as follows:
[0101]
[0102] in, is the ideal weight vector of the action network; is the ideal weight vector of the criticism network, l is the number of hidden layer nodes of the action network, and m is the number of hidden layer nodes of the criticism network; is the activation vector of the action network; is the activation vector of the critique network; is the approximation error of the action network, To criticize the approximation error of the network; represents a set of real matrices of dimension l×n; represents a set of real matrices of dimension m×n; represents a set of real vectors of dimension l; represents a set of real vectors of dimension m; represents a set of real vectors of dimension n; β is the output peak limit, assuming ε a and ε c is bounded, since the ideal weight vector cannot be obtained, the weight estimation vector of the action network is introduced To estimate the ideal weight vector of the action network, the weight estimation vector of the critical network is introduced To estimate the ideal weight vector of the critic network At this time, formula (12) is rewritten as:
[0103] The expression of the action network is:
[0104]
[0105] Critical network expressions:
[0106]
[0107] In the formula, It represents the estimated output torque of each joint of the force tactile feedback device in the joint space during the human-computer interaction process; It represents the estimated value of the performance index function of the system during human-computer interaction; β represents the output peak limit; represents the weight estimation vector of the action network; represents the activation vector of the action network; represents the weight estimation vector of the critic network; Represents the activation vector of the critic network.
[0108] Step 42: Obtain the weight estimation vector update law of the action network and the criticism network:
[0109] Based on formulas (13) and (14), the problem is transformed into designing the weight estimation vector of the action network and the weight estimate vector of the critical network The update law is denoted as t for the operation time and T for the operation interval. p , in the time interval [tT p ,t], the performance index function of formula (11) is applied with the model-free integral reinforcement learning algorithm, and the equation is as follows:
[0110]
[0111] In the formula, r(tT p ) indicates tT p The state vector of the augmented system at time t; r(t) represents the state vector of the augmented system at time t; r(ι) represents the state vector of the augmented system at time ι, where ι is the time interval [tT p ,t] is the integral variable.
[0112] In the present invention, since it is impossible to directly obtain the output torque τ s The ideal value and behavior moment τ h The ideal value of the output torque is thus estimated using the estimated value Behavioral moment estimate And the performance indicator function estimate Corresponding to the replacement τ s , τ h And V. And there is an error between the estimated value and the ideal value, then formula (15) can be rewritten as:
[0113]
[0114] Among them, ε represents the error term generated by replacing the ideal value with the estimated value. When ε = 0, the estimated value can be considered to be the ideal value; α>0 is an adjustable parameter; Indicates tT p The weight estimation vector of the action network at the moment; r(tT p ) represents tT p The augmented system state vector at time T p Indicates the operation interval time; represents the matrix Kronecker product operation; vec(·) represents the function of converting the matrix into a vector according to the row; r(ι) represents the state vector of the augmented system at time ι, where ι is the time interval [tT p ] is the integral variable; κ is the output interval The integral variable on .
[0115] With ε approaching 0 as the design goal, the gradient descent method is used to design the weight estimation vector update law of the action network and the criticism network for formula (16), and the weight estimation vector of the action network and the criticism network is defined as Contains the weight estimation vectors of the action network and the criticism network. Then the update law of the weight estimation vectors of the action network and the criticism network is expressed as:
[0116]
[0117] In the formula, Represents the update law of the weight estimation vector of the action network and the critic network; represents the weight estimation vector of the action network; The weight estimation vector of the critic network; δ represents the learning rate of the action network and the critic network; ψ represents the residual error caused by the action network and the critic network in the performance indicator function; φ represents the residual error caused by the action network and the critic network in the rest of the function.
[0118] Among them, δ>0 represents the learning rate, and the expression of ψ is:
[0119]
[0120] Where ψ represents the residual error caused by the action network and the criticism network in the performance index function; Q = (Q1,02n ×2 n;02n ×2 n,02n ×2 n) is a semi-positive definite matrix, Q1 is a positive definite matrix; the superscript T represents the transpose of the matrix or function output; δ represents the learning rate; K is a positive definite matrix; α represents an adjustable parameter; β represents the output peak limit; represents the weight estimation vector of the initial behavior network; represents the transpose of the activation vector of the behavior network; r(ι) represents the state vector of the augmented system at time ι.
[0121] The expression of φ is:
[0122]
[0123]
[0124]
[0125] In the formula, φ c represents the residual error caused by the critical network; φ a represents the residual error caused by the action network; e represents the joint angle position tracking error of the system; R is a positive definite matrix; r(ι) represents the state vector of the augmented system at time ι, where ι is the time interval [tT p ,t]; T p Indicates the operation interval time; represents the activation vector of the action network; represents the activation vector of the critical network; α represents an adjustable parameter; β represents the output peak limit; represents the matrix Kronecker product operator; r(tT p ) indicates tT p The state vector of the augmented system at time t; r(t) represents the state vector of the augmented system at time t.
[0126] S5, updating the state observation error term and joint angular velocity observation value of the state observer, as well as the joint angle position and joint angular velocity of the human-computer interaction dynamics model;
[0127] Specifically, the behavioral torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device during the human-computer interaction is obtained according to the updated behavior network, and the output torque of each joint of the force tactile feedback device is obtained according to the updated action network, and the state observation error term and joint angular velocity observation value of the state observer are updated, as well as the joint angle position and joint angular velocity of the human-computer interaction dynamics model.
[0128] The present embodiment is described below in conjunction with the accompanying drawings:
[0129] like Figure 2 As shown, in step 1, the initial time is recorded as t0, the system is initialized, and the initial parameters of the system are recorded as Taking the joint angle position as an example, the initial joint angle position is
[0130] The steps of system initialization are:
[0131] 1.1. Parameter initialization:
[0132] Initialization operation interval time T p , the desired trajectory of the system, the joint angle position Joint angular velocity Observer Output And neural network parameters. Among them, the neural network parameters include activation function parameters, learning rate parameters and initial weight matrix parameters. At this time, the system state observer outputs are equal to the initial joint angle positions
[0133] and joint angular velocity And the parameters M, C, and G of the human-computer interaction dynamics model are known.
[0134] 1.2. Behavior network initialization: according to the initial joint angle position Joint angular velocity as well as Get the state observation error term According to formula (10), we can obtain So as to obtain the behavior network output
[0135] 1.3. Action network and criticism network initialization: according to the initial joint angle position Joint angular velocity And the expected trajectory obtains the state vector r(t0) of the augmented system, and combines the activation function corresponding to the action network and the criticism network, the initial weight vector and the initial output of the behavior network According to formula (17), we can obtain So as to obtain the output of the action network
[0136] 1.4. State observer and human-computer interaction dynamics model update: based on the output of the behavior network And the output of the action network Update the state observation error term of the state observer and joint angular velocity observations And the joint angle position of the human-computer interaction dynamics model Joint angular velocity
[0137] Step 2: The current time is recorded as t. Assume that there is a matrix A, A t represents the matrix A at the current moment, Represents the matrix A at the next moment, and this expression method will be used uniformly in the future. The state observation error term ζ of the system state observer is obtained through the state observation value and the current joint angle position and angular velocity. t Secondly, the behavior network weight update law is determined based on the state observation error and system dynamics. The behavior network weight estimation update law is obtained by recursion And calculate the estimated behavioral moment
[0138] Step 3: Based on the current trajectory tracking error information and current behavior network output The network estimated torque output of the last action Get the update law of action and critical neural network weight estimation. Finally, calculate the output of the action network and add the output constraint of the system to get the output torque estimate.
[0139] Step 4: Estimation of the output torque of the action network Behavioral moment estimates with behavioral networks The data are input into the system together to realize the system's updated learning.
[0140] The system is simulated, and the simulation results are as follows: Figures 3 to 7 As shown, Figure 3 Schematic diagram of the error trajectory of the behavior network and the system state observer estimating the behavior torque. Figure 3 It is found that the behavior network estimates the behavior moment with a small error, can maintain fluctuations within a small range and be stable for a long time. Figure 3 The estimation of the behavior moment error trajectory using the system state observer is described. It can be seen that the behavior moment method approximated by the behavior network is better, and it has higher approximation accuracy. Figure 4 is a schematic diagram of the behavior torque of joint 1 and the estimated behavior torque trajectory. Figure 5 is a schematic diagram of the behavior torque of joint 2 and the estimated behavior torque trajectory. Figure 6 is the behavior torque of joint 3 and the estimated behavior torque trajectory diagram, from Figures 4 to 6 It can be seen that Figure 7 is a schematic diagram of the behavior network node weight trajectory, from Figure 7 It can be seen that the weight values of the behavior network nodes can converge within a limited time.
[0141] Figure 8 This is a schematic diagram of the joint angle position tracking error trajectory. Fig. 9 Schematic diagram of the trajectory of the output torque estimate estimated by the action network; Fig.10 Schematic diagram of the action network weight trajectory; Fig.11 To criticize the schematic diagram of the network weight trajectory, from Figures 8 to 11The four figures describe the joint angle trajectory tracking error under the adaptive dynamic programming control method based on action, behavior, and critical neural networks, the action network estimated torque trajectory under output constraints, and the weight trajectory of the action network and critical network. It can be found that the state error can converge quickly and remain stable. And the weights of the action network and the critical network can converge. At the same time Figure 8 Added PID method comparison group, from Figure 8 It can be seen that the proposed control method has a smaller trajectory tracking error than the PID control method in terms of trajectory tracking. Figure 3 as well as Figure 8 It can be seen that the method of this embodiment has better behavior torque estimation capability and trajectory tracking performance than the traditional method.
[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. An action-behavior-criticism neural network learning control method for human-computer interaction process, It is characterized in that include: According to the joint angle position, joint angular velocity, joint angular acceleration, inertia matrix, centripetal force and Coriolis force matrix, gravity matrix and output torque of each joint in the joint space of the force tactile feedback device during the human-computer interaction process, and the behavioral torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device on each joint, a human-computer interaction dynamics model is constructed; the human-computer interaction dynamics model is represented by an augmented system, and the state vector of the augmented system is obtained according to the joint angle position, joint angular velocity and expected trajectory of the force tactile feedback device in the joint space during the human-computer interaction process; The Gaussian function is used as the radial basis function and the behavior network is constructed according to the multi-layer RBF neural network. According to the system state observer, radial basis function and human-computer interaction dynamics model in the human-computer interaction process, the weight estimation vector update law of the behavior network is obtained. The behavior network is updated based on the weight estimation vector update law, and the updated behavior network is used to obtain the behavior torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device on each joint during the human-computer interaction process; According to the output torque of each joint of the force tactile feedback device in the joint space during the human-computer interaction process, the behavior torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device on each joint, and the state vector of the augmented system, the performance index function of the system during the human-computer interaction process is obtained; Constructing an action network and a criticism network, obtaining an update law for the weight estimation vectors of the action network and the criticism network according to the performance indicator function, the activation vectors and the weight estimation vectors corresponding to the action network and the criticism network, and the behavior torque output by the behavior network, updating the action network according to the update law for the weight estimation vectors of the action network and the criticism network, and obtaining the output torque of each joint of the force tactile feedback device according to the updated action network; According to the updated behavior network, the behavior torque of the interactive behavior force applied by the operator at the end of the force tactile feedback device in the human-computer interaction process is obtained, and the output torque of each joint of the force tactile feedback device is obtained according to the updated action network, and the state observation error term and joint angular velocity observation value of the state observer are updated, as well as the joint angle position and joint angular velocity of the human-computer interaction dynamics model.
2. The action-behavior-criticism neural network learning control method for a human-computer interaction process according to claim 1, It is characterized in that The expression of the human-computer interaction dynamics model is: Where q represents the joint angle position of the force tactile feedback device in the joint space during human-computer interaction; It represents the joint angular velocity of the force tactile feedback device in the joint space during human-computer interaction; represents the joint angular acceleration of the force tactile feedback device in the joint space during the human-computer interaction process; M represents the inertia matrix of the force tactile feedback device in the joint space during the human-computer interaction process; C represents the centripetal force and Coriolis force matrix of the force tactile feedback device in the joint space during the human-computer interaction process; G represents the gravity matrix of the force tactile feedback device in the joint space during the human-computer interaction process; τ s represents the output torque of each joint of the force tactile feedback device in the joint space during human-computer interaction; τ h It represents the behavioral torque on each joint of the interactive behavioral force applied by the operator at the end of the force tactile feedback device.
3. The action-behavior-criticism neural network learning control method for a human-computer interaction process according to claim 1, It is characterized in that The expression of the augmented system is: In the formula, represents the state vector of the augmented system at time t, and e represents the joint angle position tracking error of the system; represents the time derivative of the state vector r(t) of the augmented system at time t, represents a Lipschitz continuous function Transpose; Represents the derivative of the joint angle position tracking error of the system; q d represents the desired joint angle position of each joint; represents the derivative of the desired joint angle position of each joint with respect to time; F(r(t)) represents the nonlinear dynamics function; H(r(t)) represents the gain function; τ s represents the output torque of each joint of the force tactile feedback device in the joint space during human-computer interaction; τ h represents the behavioral torque on each joint of the interactive behavioral force applied by the operator at the end of the force tactile feedback device; represents the transpose of the system state function; h T (e+q d ) represents the transpose of the system output gain function; represents a row vector of length n consisting of zeros.
4. The action-behavior-criticism neural network learning control method for a human-computer interaction process according to claim 1, It is characterized in that The expression of the system state observer in the human-computer interaction process is: In the formula, represents the derivative of the joint angular velocity observation; M -1 represents the inverse matrix of the inertia matrix of the force tactile feedback device in the joint space during the human-computer interaction process; C represents the centripetal force and Coriolis force matrix of the force tactile feedback device in the joint space during the human-computer interaction process; G represents the gravity matrix of the force tactile feedback device in the joint space during the human-computer interaction process; τ s It represents the output torque of each joint of the force tactile feedback device in the joint space during the human-computer interaction process; represents the estimated behavioral torque; σ represents the observed value of joint angular velocity; represents the state observation error; D represents the positive diagonal matrix; ζ represents the state observation error term; L represents the error correction matrix.
5. The action-behavior-criticism neural network learning control method for a human-computer interaction process according to claim 1, It is characterized in that The update law of the weight estimation vector of the behavior network is expressed as: In the formula, the behavior network includes the initial behavior network and the j-layer extended behavior network; Representing the weight estimation vector update law of the initial behavior network in the behavior network; represents the update law of the weight estimation vector of the i-th layer extended behavior network in the behavior network; represents the update law of the weight estimation vector of the j-th extended behavior network in the behavior network; Represents the weight estimation vector of the i+1th layer extended behavior network in the behavior network; A weight estimation vector representing an initial behavior network in the behavior network; Represents the weight estimation vector of the i-th layer extended behavior network in the behavior network; Represents the weight estimation vector of the j-th extended behavior network in the behavior network; Represents the weight estimation vector of the first layer of the extended behavior network in the behavior network; represents the activation vector of the behavior network; ζ represents the state observation error term; M -1 represents the inverse matrix of the inertia matrix of the force tactile feedback device in the joint space during human-computer interaction; δ h represents the initial behavior network learning rate in the behavior network, δ hi represents the learning rate of the i-th layer extended behavior network in the behavior network, δ hj represents the learning rate of the j-th extended behavior network in the behavior network; k h is the adjustable parameter of the initial behavior network, k hi represents the adjustable parameter of the i-th extended behavior network in the behavior network, k hj Represents the adjustable parameters of the j-th extended behavior network in the behavior network.
6. The action-behavior-criticism neural network learning control method for a human-computer interaction process according to claim 1, It is characterized in that The expression of the performance index function of the system in the human-computer interaction process is: Where V represents the performance index function of the system in the human-computer interaction process; α>0 represents the discount factor; Q=(Q 1 ,0 2n×2n ; 0 2n×2n ,0 2n×2n ) is a positive semidefinite matrix, is a positive definite matrix; represents a set of real matrices of dimension 2n×2n; R and K are both positive definite matrices; τ s represents the output torque of each joint of the force tactile feedback device in the joint space during human-computer interaction; τ h represents the behavioral torque of the interactive behavioral force applied by the operator at the end of the force tactile feedback device on each joint; r(ι) represents the state vector of the augmented system at time ι, where ι is the integral variable on the time interval [t,∞].
7. The action-behavior-criticism neural network learning control method for a human-computer interaction process according to claim 1, It is characterized in that The expressions for constructing the action network and the criticism network are: The expression of the action network is: Critical network expressions: In the formula, It represents the estimated output torque of each joint of the force tactile feedback device in the joint space during the human-computer interaction process; It represents the estimated value of the performance index function of the system during human-computer interaction; β represents the output peak limit; represents the weight estimation vector of the action network; represents the activation vector of the action network; represents the weight estimation vector of the critic network; Represents the activation vector of the critic network.
8. The action-behavior-criticism neural network learning control method for a human-computer interaction process according to claim 1, It is characterized in that The update law of the weight estimation vector of the action network and the criticism network is expressed as: Define the weight estimation vectors for the action network and the critic network Then we have: In the formula, Represents the update law of the weight estimation vector of the action network and the critic network; represents the weight estimation vector of the action network; represents the weight estimation vector of the critic network; δ represents the learning rate of the action network and the critic network; ψ represents the residual error caused by the action network and the critic network in the performance indicator function; φ represents the residual error caused by the action network and the critic network in the rest of the function.
9. The action-behavior-criticism neural network learning control method for a human-computer interaction process according to claim 8, It is characterized in that The expression of ψ is: Where ψ represents the residual error caused by the action network and the criticism network in the performance index function; Q = (Q 1 ,0 2 n ×2 n; 0 2 n ×2 n,0 2 n ×2 n) is a semi-positive definite matrix, Q 1 is a positive definite matrix; the superscript T represents the transpose of the matrix or function output; δ represents the learning rate; K is a positive definite matrix; α represents an adjustable parameter; β represents the output peak limit; represents the weight estimation vector of the initial behavior network; represents the transpose of the activation vector of the behavior network; r(ι) represents the state vector of the augmented system at time ι, where ι is the time interval [tT p ] is the integral variable; κ is the output interval The integral variable on .
10. The action-behavior-criticism neural network learning control method for a human-computer interaction process according to claim 8, It is characterized in that The expression of φ is: In the formula, φ c represents the residual error caused by the critical network; φ a represents the residual error caused by the action network; e represents the joint angle position tracking error of the system; R is a positive definite matrix; r(ι) represents the state vector of the augmented system at time ι, where ι is the time interval [tT p ,t]; T p Indicates the operation interval time; represents the activation vector of the action network; represents the activation vector of the critical network; α represents an adjustable parameter; β represents the output peak limit; represents the matrix Kronecker product operator; r(tT p ) represents tT p The state vector of the augmented system at time t; r(t) represents the state vector of the augmented system at time t.
Citation Information
Patent Citations
Human-robot cooperative control method based on human body dynamic arm strength estimation model
CN113059570A
Reconfigurable mechanical arm cooperative force / motion control system and method based on terminal task assignment
CN113276114A
Cooperative game optimal tracking control method and system for input-limited mechanical arm
CN114406996A
Remote operation-based bionic manipulator man-machine interaction system and method
CN118238147A
Robot external contact force estimation method based on artificial neural network
WO2022257185A1
