A networked servo system control method based on A3C and event triggering

Abstract

A control method for a networked servo system based on A3C and event triggering belongs to the technical field of networked motor control. It includes the following steps: 1. Establishing a system model of a networked servo control system; 2. Designing an event triggering strategy considering discrete systems; 3. Designing a control scheme for a networked servo system based on event triggering; 4. Designing a controller; 5. Designing a reinforcement learning A3C scheme. The present invention provides a mathematical model of a networked servo system, and uses an analysis method of a time-delay system to propose a closed-loop time-delay system with an event triggering mechanism and state feedback control; derives a design method for a controller; designs an event-triggered state feedback control algorithm, and uses a linear matrix inequality toolbox to obtain controller parameters; finally, through the optimization of the A3C scheme, a dynamic threshold state feedback control algorithm based on A3C and event triggering is designed; a dynamic event triggering strategy that can autonomously adjust trigger parameters.

Description

Technical Field

[0001] The invention belongs to the technical field of networked motor control, and in particular relates to a networked servo system control method based on A3C and event triggering. Background Art

[0002] In recent years, digital computers have developed rapidly and have become an important tool for control system design, and the development of microprocessors has also added new possibilities to control systems. With the continuous expansion of industrial application needs, network control systems (NCS) provide an effective solution for remote control, and have certain applications in some working environments with certain dangers. Compared with traditional point-to-point wiring systems, the communication channels in network control systems can greatly reduce costs, simplify system structure, and have the advantages of easy debugging and maintenance of the system. At the same time, network control systems are widely used in various fields due to their high cost-effectiveness, low power consumption requirements, and high reliability.

[0003] In network control systems, the communication resources and computing resources in the network are often limited. Most traditional control methods adopt periodic trigger control methods. This triggering method will send many unnecessary signals through the network, increase the pressure on the communication bandwidth, and reduce the utilization of communication resources. In order to effectively deal with the above problems, Professor D. Yue's team proposed a control method based on event triggering, which can reduce the number of unnecessary data transmissions in network communication while ensuring system performance. Although the event triggering mechanism can improve the utilization of communication resources and reduce the pressure on the communication bandwidth, the trigger parameters in most existing event triggering conditions are fixed. This makes the trigger frequency of the signal very mechanical during the operation of the system and will not respond to interference in the system. If the trigger parameters can be automatically adjusted as the system changes, the system will become more flexible, so that the system can have the best possible performance and the least possible number of triggers. Therefore, it is particularly important to design a dynamic event triggering strategy that can autonomously adjust the trigger parameters. Summary of the invention

[0004] In view of the above problems existing in the prior art, the purpose of the present invention is to consider the problem of limited communication resources and computing resources in network control and propose a dynamic threshold control method combining reinforcement learning and event triggering.

[0005] The present invention provides the following technical solutions:

[0006] A networked servo system control method based on A3C and event triggering comprises the following steps:

[0007] 1) Establish the system model of networked servo control system:

[0008] Define x(k)∈R p represents the state vector of the system, u(k)∈R m represents the control input of the system, ω(k)∈R q represents the disturbance of the system, y(k)∈R p represents the system output, where p, m, and q represent the dimensions of x(k), u(k), and ω(k) respectively, and A, B, F, and C are all system parameter matrices that meet the dimensionality requirements. The system model is:

[0009] x(k+1)=Ax(k)+Bu(k)+Fω(k)

[0010] y(k)=Cx(k)

[0011] Design a servo system controller based on the state feedback signal:

[0012] u(k)=Kx(k), where K represents the controller gain.

[0013] 2) Design event triggering strategy considering discrete systems:

[0014] In the network system, due to the limited communication resources and long-distance transmission, there is a transmission delay τ j ,in is a real number greater than 0, and a zero-order holder is applied. The dynamic model of the system under the event generator is expressed as:

[0015]

[0016] Among them, k j represents the sampling time of the system, and the data x(k j ) is sent to update the controller based on the following conditions:

[0017] [x(k)-x(k j )] T Φ j [x(k)-x(k j )]

[0018] ≤ε j x T (k)Φ j x(k)

[0019] By deducing the formula, the delay function is:

[0020]

[0021] Define the maximum sampling time get:

[0022]

[0023] in,

[0024]

[0025] 3) Design of networked servo system control scheme based on event triggering:

[0026] The state error between the sampling time of the previous moment and the next moment is defined as:

[0027]

[0028] The judgment condition for event triggering can be rewritten as:

[0029] e k (k) T Φ j e(k) k ≤ε j x T (kd(k))Φ j x(kd(k))

[0030] The event-triggered networked servo control system model can be described as:

[0031]

[0032] in Indicates the initial state of the system;

[0033] 4) Controller design:

[0034] Given 0≤ε j ≤1,γ>0,τ M > 0 and a matrix of appropriate dimension

[0035] P j >0, Q j >0, R j >0, Z j >0,Φ j >0 and K make the following inequality hold:

[0036]

[0037] Then the event-triggered networked servo control system model is exponentially stable in the mean square sense and has a given H ∞ Performance level γ, the controller gain K can be obtained as:

[0038] K=(τ M *P j -1 *B*(B T *B)-1 )

[0039] 5) Reinforcement learning A3C solution design:

[0040] The threshold is optimized by the reinforcement learning method based on A3C, and a dynamic threshold with fewer trigger times is obtained while ensuring system performance. The reinforcement learning problem can usually be transformed into a dynamic programming problem, so the dynamic threshold problem is transformed into a multi-dimensional dynamic programming problem and a five-tuple (S r , A r , P r ,λ r , r) indicates that, among which, S r Design state space, A r To design the action space, P r is the probability transfer matrix, λ r is the designed discount function and r is the designed reward function.

[0041] In the networked servo system control method based on A3C and event triggering, in step 2), the derivation process of the delay function is as follows:

[0042] Due to the long delay and short delay in the network, two delay situations are considered:

[0043] 2.1. When , define the delay function:

[0044] d(k)=kk j , k∈[k j +τ j , k j+1 +τ j+1 )

[0045] It can be seen that:

[0046]

[0047] 2.1. When hour,

[0048] Two time intervals are defined:

[0049]

[0050] in l ≥ l, get:

[0051]

[0052] Therefore, when l = 1, 2, 3, ... d-1, x(k j ) and x(kj +l)Satisfy:

[0053] [x(k j +l)-x(k j )] T Φ j [x(k j +l)-x(k j )]≤ε j x T (k j +l)Φ j x(k j +l)

[0054] definition:

[0055]

[0056] get:

[0057]

[0058] By adopting the above technology, compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] The present invention provides a mathematical model of a networked servo system, and uses the analysis method of a time-delay system to propose a closed-loop time-delay system with an event-triggered mechanism and state feedback control; then, the Lyapunov theory is used to analyze the asymptotic stability and robustness of the system, and a controller design method is derived; subsequently, an event-triggered state feedback control algorithm is designed, and the controller parameters are obtained using a linear matrix inequality toolbox; finally, through the optimization of the A3C scheme, a dynamic threshold state feedback control algorithm based on A3C and event triggering is designed; a dynamic event triggering strategy that can autonomously adjust the trigger parameters is proposed. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a structural schematic diagram of the networked event-triggered servo system model of the present invention;

[0061] Figure 2 It is a structural diagram of the reinforcement learning A3C scheme of the present invention;

[0062] Figure 3 The experimental results of the servo system platform of the present invention are shown in the figure. The horizontal axis represents time in seconds, and the vertical axis represents the speed of the motor in centimeters. Figure 3-1 Meaning: System performance with a fixed threshold of 0.01; Figure 3-2 Indicates: System performance with a fixed threshold of 0.1; Figure 3-3 It means: the system performance of the threshold after the A3C method optimization; Figure 3-2 for Figure 3-1 The enlarged picture of Figure 3 -4 is Figure 3-3 The enlarged picture of Figure 3 -6 Figure 3 -5 enlarged image. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments of the specification. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0064] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0065] Reference Figure 1-2 , a dynamic threshold servo system control scheme based on A3C and event triggering is designed, including the following steps:

[0066] 1) Establish the system model of networked servo control system:

[0067] Define x(k)∈R p represents the state vector of the system, u(k)∈R m represents the control input of the system, ω(k)∈R q represents the disturbance of the system, y(k)∈R p represents the system output, where p, m, q represent x(k), u(k), ω(k) respectively, and A, B, F, C are all system parameter matrices that meet the dimension requirements. Then the system model is:

[0068]

[0069] Design a servo system controller based on the state feedback signal:

[0070] u(k)=Kx(k)

[0071] 2) Design event triggering strategy considering discrete systems:

[0072] In the network system, due to the limited communication resources and long-distance transmission, there is a transmission delay τ j ,in is a real number greater than 0. Applying the zero-order holder, the dynamic model of the system under the event generator is expressed as:

[0073]

[0074] Among them, k j represents the sampling time of the system. The data x(k j ) is sent to update the controller based on the following conditions:

[0075]

[0076] Due to the long delay and short delay in the network, two delay situations are considered:

[0077] Situation 1: When , define the delay function:

[0078] d(k)=kk j , k∈[k j +τ j , k j+1 +τ j+1 ) (4)

[0079] It can be seen that:

[0080]

[0081] Situation 2: When When , two time intervals are defined:

[0082]

[0083] in l ≥ l, get:

[0084]

[0085] Therefore, x(k j ) and x(k j +l) When l = 1, 2, 3, ... d-1, it satisfies:

[0086]

[0087] definition:

[0088]

[0089] get:

[0090]

[0091] Define the maximum sampling time get:

[0092]

[0093] 3) Design of networked servo system control scheme based on event triggering:

[0094] According to formula (9), we define:

[0095]

[0096] Combining formula (10) and formula (12), the judgment condition of event triggering can be rewritten as:

[0097] e k (k) T Φ j e(k) k ≤ε j x T (kd(k))Φ j x(kd(k)) (13)

[0098] According to formulas (2), (10) and (12), the event-triggered networked servo control system model can be described as:

[0099]

[0100] in Represents the initial state of the system.

[0101] 4) Controller design:

[0102] If there exists a given positive number 0≤ε j ≤1,γ>0,τ M > 0 and a matrix P of appropriate dimension j >0, Q j >0, R j >0, Z j >0,Φ j >0 and K make the following inequality hold:

[0103]

[0104] Then the system model formula (14) is exponentially stable in the mean square sense and has a given H ∞ Performance level γ, the controller gain K can be obtained as:

[0105] K=(τ M *P j -1 *B*(B T *B) -1 )

[0106] 5) Reinforcement learning A3C solution design:

[0107] The threshold is optimized by the reinforcement learning method based on A3C, and a dynamic threshold with fewer trigger times is obtained while ensuring system performance. The reinforcement learning problem can usually be transformed into a dynamic programming problem, so the dynamic threshold problem is transformed into a multi-dimensional dynamic programming problem and a five-tuple (S r , A r , P r ,λ r , r) indicates.

[0108] 5.1) Design state space S r , S r It is used to represent the state set of the system. In order to make the training threshold enable the system to reduce the number of triggers as much as possible while maintaining good performance, the total number of triggers in the nth period is recorded as t n , and the average output value of the system during this period is recorded as y n Therefore, define s r，n =[X n y n t n ] T , where X n Indicates the state of the last output of the system within the nth period;

[0109] 5.2) Design action space A r , A r It is composed of a group of agents that execute in the environment. Each time the agent executes, the environment is updated. The output of the agent is the threshold of the required event triggering condition. r，n =ε j，n , a r，n ≤ε j , where ε j，n represents the threshold value in the nth period of time, ε j Indicates the upper limit of the threshold. The threshold in the nth period is expanded, so a is defined r，n，i =ε j，n，i =ε j，n +0.001*h, h∈[-5, 4], i∈[l, 2,..., 10];

[0110] 5.3) Probability transfer matrix P r :S r *A r →Λ(S r ) means to execute the current action a in the current environment r，n After that, the next environment state s r，n+1 The distribution of , that is, the state space equation of the servo system based on event triggering is derived.

[0111] 5.4) Design discount function λ r :λ r The size of determines the impact on long-term rewards, r ∈[0, 1], where λ r =0.9.

[0112] 5.5) Design reward function r(s) r，n , a r，n ): r(s r，n , a r，n ) is a function that maps a set of corresponding state-actions to a scalar, which represents the current action a r，n Applied in s r，n is the immediate reward obtained after the current environment of the state. At the same time, a dual reward function is used. First, when the Agent is expanded, the immediate reward of each sub-thread is calculated and defined. Among them, S r，n，i Indicates the final environment state of the i-th subthread in the n-th period of time. And selects the smallest reward Define the total reward function,

[0113]

[0114] where s r，n，b Represents ε j，n，i =ε j，n The threshold for child threads. Rewards are The environmental status. b Indicates the state is s r，n，b Instant rewards. Indicates the weight of trigger frequency and convergence speed, which can be set manually.

[0115] Combination Figure 3 , the threshold parameters in event triggering are selected as follows:

[0116] Through system identification, the servo system state equation parameter matrix is ​​as follows:

[0117]

[0118] C j =[1 0]

[0119] Among them, the sampling time of the system is T s = 0.005s, interference ω(k) = sin(x(k)), select τ M =3,ε j = 0.1, solve LMI, and get the control gain K and parameter matrix Φ j, the initial state of the system

[0120] Figure 3-1 : Fixed threshold 0.1.

[0121] Figure 3-2 : Fixed threshold 0.01.

[0122] Figure 3-3 : Dynamic threshold, with 2000 sampling cycles as a training process, the optimal dynamic threshold is trained through the A3C algorithm. After more than 2000 samples, the system has stabilized, and the threshold at this time is the 2000th threshold in the dynamic threshold. Definition:

[0123]

[0124] The training threshold ε3∈[0.01, 0.1].

[0125] According to the experimental results, from the performance point of view, under the fixed threshold, when the threshold is 0.01, the motor speed is stable at about 0.1, and when the threshold is 0.1, the motor speed is stable at about 1.3. According to the dynamic threshold obtained by the A3C method, the motor speed is stable at about 0.3. It is concluded that the motor performance is not much different between the threshold obtained by the A3C algorithm and the fixed threshold of 0.01, which shows that the dynamic threshold optimized by the A3C algorithm can keep the system in good performance.

[0126] In terms of the number of event triggers, this experiment only counts the number of triggers within the first 1000 sampling times. Under a fixed threshold, when the threshold is 0.01, the number of triggers is about 100 within 1000 sampling times, and when the threshold is 0.1, the number of triggers is about 10 within 1000 sampling times. According to the dynamic threshold obtained by the A3C algorithm, the number of triggers is about 30 within 1000 sampling times. It is concluded that the threshold obtained based on the A3C algorithm has a significant decrease in the number of triggers compared with the fixed threshold of 0.01.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A networked servo system control method based on A3C and event triggering, characterized in that: The steps include: 1) Establish the system model of networked servo control system: Define x(k)∈R p represents the state vector of the system, u(k)∈R m represents the control input of the system, ω(k)∈R q represents the disturbance of the system, y(k)∈R p represents the system output, where p, m, and q represent the dimensions of x(k), u(k), and ω(k) respectively, and A, B, F, and C are all system parameter matrices that meet the dimensionality requirements. The system model is: x(k+1)=Ax(k)+Bu(k)+Fω(k) y(k)=Cx(k) Design a servo system controller based on the state feedback signal: u(k)=Kx(k), K represents the controller gain; 2) Design event triggering strategy considering discrete systems: In the network system, due to the limited communication resources and long-distance transmission, there is a transmission delay τ j ,in is a real number greater than 0, and a zero-order holder is applied. The dynamic model of the system under the event generator is expressed as: Among them, k j represents the sampling time of the system, and the data x(k j ) is sent to update the controller based on the following conditions: [x(k)-x(k j )] T Φ j [x(k)-x(k j )] ≤ε j x T (k)Φ j x(k) Among them, Φ j is a positive definite matrix, and then the delay function is derived by the formula: Define the maximum sampling time get: in, 3) Design of networked servo system control scheme based on event triggering: The state error between the sampling time of the previous moment and the next moment is defined as: The judgment condition for event triggering can be rewritten as: e k (k) T Φ j e(k) k ≤ε j x T (k-d(k))Φ j x(k-d(k)) The event-triggered networked servo control system model can be described as: in Represents the initial state of the system; 4) Controller design: Given 0≤ε j ≤1,γ>0,τ M > 0 and a matrix P of appropriate dimension j >0, Q j >0, R j >0, Z j >0,Φ j >0 and K make the following inequality hold: Then the event-triggered networked servo control system model is exponentially stable in the mean square sense and has a given H ∞ Performance level γ, the controller gain K can be obtained: 5) Reinforcement learning A3C solution design: The threshold is optimized by the reinforcement learning method based on A3C, and a dynamic threshold with fewer trigger times is obtained while ensuring system performance. The reinforcement learning problem can usually be transformed into a dynamic programming problem, so the dynamic threshold problem is transformed into a multi-dimensional dynamic programming problem and a five-tuple (S r , A r , P r ,λ r , r) indicates that, among which, S r Design state space, A r To design the action space, P r is the probability transfer matrix, λ r is the design discount function, r is the design reward function; 5.1) Design state space S r , S r It is used to represent the state set of the system. In order to make the training threshold enable the system to reduce the number of triggers as much as possible while maintaining good performance, the total number of triggers in the nth period is recorded as t n , and the average output value of the system during this period is recorded as y n , so, define s r，n =[X n y n t n ] T , where X n Indicates the state of the last output of the system within the nth period; 5.2) Design action space A r , A r It is composed of a group of agents executed in the environment. Each execution of the agent will update the environment. The output of the agent is the threshold of the required event triggering condition. The agent is expanded and defined as a r，n =ε j，n a r，n ≤ε j , where ε j，n represents the threshold value in the nth period of time, ε j Indicates the upper limit of the threshold, and expands the threshold within the nth period of time, so define a r，n，i =ε j，n，i =ε j，n +0.001*h, h∈[-5, 4], i∈[1, 2,..., 10]; 5.3) Probability transfer matrix P r :S r *A r →Λ(S r ) means to execute the current action a in the current environment r，n After that, the next environment state s r，n+1 The distribution of , i.e. the derived state space equation of the servo system based on event triggering; 5.4) Design discount function λ r :λ r The size of determines the impact on long-term rewards, r ∈[0, 1]; 5.5) Design reward function r(s) r，n , a r，n ): r(s r，n , a r，n ) is a function that maps a set of corresponding state-actions to a scalar, which represents the current action a r，n Applied in s r，n The instant reward obtained after the current environment of the state, and the dual reward function is used at the same time. First, when the Agent is expanded, the instant reward of each sub-thread is calculated and defined Among them, s r，n，i Indicates the final environment state of the i-th subthread in the n-th period of time, and selects the smallest reward Define the total reward function, where s r，n，b Represents ε j，n，i =ε j，n The threshold of the child thread, Rewards are The environmental state, r b Indicates the state is s r，n，b Instant rewards, Indicates the weighting of trigger frequency and convergence speed.

2. According to the A3C and event-triggered networked servo system control method of claim 1, it is characterized in that in the step 2), the derivation process of the delay function is as follows: Due to the long delay and short delay in the network, two delay situations are considered: 2.

1. When , define the delay function: d(k)=kk j ,k∈[k j +t j ,k j+1 +t j+1 ) It can be seen that: 2.

2. When hour, Two time intervals are defined: in get: Therefore, when l = 1, 2, 3, ... d-1, x(k j ) and x(k j +l)Satisfy: [x(k j +l)-x(k j )] T Φ j [x(k j +l)-x(k j )] ≤e j x T (k j +l)Φ j x(k j +l) definition: get:

Images