Control method and system of single-cylinder electro-hydraulic position servo system based on reinforcement learning

Abstract

The present application relates to the technical field of electro-hydraulic servo system, in particular to a control method and system of single-cylinder electro-hydraulic position servo system based on reinforcement learning, the method comprising: establishing a single-cylinder electro-hydraulic position servo system model; constructing a DDPG algorithm architecture, defining a reward function R; constructing a target future trajectory prediction model based on dynamic recursive prediction and multi-modal information fusion, generating a target future trajectory; training an agent of the DDPG algorithm; the observation data of the agent includes: current position, control error of the system 、 cumulative error of the system, and target future trajectory; based on the trained agent, the control output action can obtain continuous action signals within the set threshold range, completing the control of the single-cylinder electro-hydraulic position servo system. Through the control method and system, the highly nonlinear dynamic characteristics in the single-cylinder electro-hydraulic position servo system can be fully addressed, and the control accuracy, stability and robustness of the agent under complex working conditions can be improved.

Description

Technical Field

[0001] This invention relates to the field of electro-hydraulic servo system technology, and in particular to a control method and system for a single-cylinder electro-hydraulic position servo system based on reinforcement learning. Background Technology

[0002] Single-cylinder electro-hydraulic position servo systems inherently suffer from significant nonlinearity, uncertainty, and external disturbances. These issues severely limit the improvement of the dynamic and steady-state performance of single-cylinder systems. Currently, DUAN et al. proposed a control strategy combining adjacent cross-coupling and adaptive robust control. The single-cylinder control employs an adaptive robust controller, and a PID coupled controller is designed based on the adjacent cross-coupling concept. Zhang, based on a dual-cylinder parallel synchronous control structure, uses fuzzy PID to control a single cylinder.

[0003] In recent years, reinforcement learning has demonstrated strong potential in solving complex control problems. Compared to traditional control methods, reinforcement learning does not rely on precise system models, possesses strong adaptability and high control accuracy, and is particularly suitable for scenarios with strong nonlinearity and high uncertainty, such as electro-hydraulic position servo systems. Existing research has achieved certain results in controlling electro-hydraulic position servo systems using reinforcement learning methods. For example, Xu Baochang et al. used the DQN reinforcement learning algorithm to control a proportional servo valve, thereby achieving precise control of a hydraulic throttle valve. Experimental results show that the agent's control strategy for the valve position can meet the requirements of accuracy and response speed. Yin Fengyuan used the TD3 reinforcement learning algorithm to control a single-cylinder electro-hydraulic servo valve system, verifying the advantages of this method in terms of control performance and applicability.

[0004] However, in existing research, the observer dimensions of the aforementioned reinforcement learning methods are low, typically containing only error signals or a combination of error signals and cumulative error signals, and the output actions are mostly discrete, making it difficult to fully cope with the highly nonlinear dynamic characteristics of electro-hydraulic position servo systems.

[0005] In addition, a Chinese invention patent document with publication number CN117348403A and publication date of January 5, 2024, has been proposed in the prior art. The technical solution disclosed in this patent document is as follows: A control method for a dual-valve hydraulic system based on reinforcement learning adaptive robust control, using the reinforcement learning DDPG algorithm as the upper-level controller and the ARC control method as the lower-level controller. In the upper-level controller, the first intelligent agent is used to adjust the relevant parameters of the adaptive robust control algorithm, thereby enhancing the stability and anti-interference ability of the control system, and thus improving the robustness of the control system. The second intelligent agent is used to adjust the flow distribution strategy, intelligently adjusting the flow distributed by the servo valve and the proportional valve, reducing the coupling of mutual influence between parallel valves, thereby correcting the tracking error of the system in real time.

[0006] Although the agent reward function of the above technical solution also considers several factors, including the negative reward of the piston rod tracking error of the hydraulic cylinder, the comparison between the piston rod tracking error of the hydraulic cylinder at the current time t and the piston rod tracking error of the hydraulic cylinder at the previous time t-1, and the negative reward of the absolute value of the piston rod speed change, it is still relatively simple and cannot fully reflect the multi-dimensional control requirements of the electro-hydraulic position servo system in actual use. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a control method and system for a single-cylinder electro-hydraulic position servo system based on reinforcement learning. This method can effectively address the highly nonlinear dynamic characteristics of the single-cylinder electro-hydraulic position servo system and improve the control accuracy, stability, and robustness of the intelligent agent under complex working conditions.

[0008] This invention is achieved by adopting the following technical solution:

[0009] The control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning includes the following steps:

[0010] Step S1. Establish a model of a single-cylinder electro-hydraulic position servo system;

[0011] Step S2. Construct the DDPG algorithm architecture and define the reward function R:

[0012] R = R e +R de +R stop ,

[0013]

[0014] R de =-|e t -e t-1 |,

[0015]

[0016] In the formula, R e R is the reward given based on the error between the target position and the actual position. de R is the reward given based on the error change between adjacent sampling times. stop Rewards given based on security; e t Let b be the control error of the system at the current moment, and e be a very small positive number. t-1 X represents the control error of the system at the previous moment; emax represents the maximum allowable control error. p For the position output of the servo system, xpmax is the maximum allowed position output, and xpmin is the minimum allowed position output.

[0017] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion, and generate the target future trajectory S1, S2, ..., S N ;

[0018] Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, train the agent of the DDPG algorithm; wherein, the observation data of the agent includes: the current position Xp, the system control error e t The cumulative error of the system ∑e t And the target future trajectory S1, S2, ..., S N ;

[0019] Step S5. Based on the trained agent, control the output action to obtain continuous action signals within the set threshold range, and complete the control of the single-cylinder electro-hydraulic position servo system.

[0020] Step S1 specifically includes the following steps:

[0021] Step S 11 Construct a mathematical model of a single-cylinder electro-hydraulic position servo system and obtain the open-loop transfer function;

[0022] Step S 12 In the simulation platform, an external disturbance is added to the open-loop transfer function to construct a closed-loop transfer function model.

[0023] The open-loop transfer function G(s) is:

[0024]

[0025] In the formula, K a For the proportional gain of the servo amplifier, K q K represents the flow gain of the hydraulic cylinder spool valve. v K represents the servo valve gain. f For the displacement sensor gain, A p K represents the piston area of ​​a hydraulic cylinder. c The flow-pressure coefficient of the slide valve is represented by m. l For load quality, w v ξ represents the natural frequency. v This indicates the damping ratio of the servo valve.

[0026] Step S3 specifically includes the following steps:

[0027] Step S 31 By utilizing a long short-term memory neural network and combining historical target signals, hydraulic cylinder speed, and external disturbance data, a target future trajectory prediction model is constructed and trained.

[0028] Step S32 An adaptive time window mechanism is introduced to dynamically adjust the prediction time range and step size based on real-time operating conditions;

[0029] Step S 33 Based on the trained target future trajectory prediction model and the adjusted prediction time range and step size, the target future trajectory is generated.

[0030] The real-time operating conditions include load changes or linear disturbance intensity.

[0031] A priority experience replay mechanism is also set up during the training process of the intelligent agent.

[0032] The priority experience replay mechanism dynamically adjusts the sample sampling probability based on the TD error.

[0033] The agent includes an actor network and a Critic network. The training of the agent for the DDPG algorithm in step S4 specifically includes the following steps:

[0034] Step S 41 Initialize network parameters;

[0035] Step S 42 Determine if the iteration is complete. If not, proceed to step S. 43 If so, end the training;

[0036] Step S 43 Initialize the random process, receive the initial state of the single-cylinder electro-hydraulic position servo system, and predict the target's position in the next N steps;

[0037] Step S 44 Determine if position control has ended; if so, proceed to step S. 42 If not, proceed to step S. 45 ;

[0038] Step S 45 Based on strategy Π and a random process, action A is selected, and reward R is obtained;

[0039] Step S 46 .Transition to the next state S', observe the state of the single-cylinder electro-hydraulic position servo system and update the target's position for the next N steps;

[0040] Step S 47 Store (S, A, R, S') in the priority experience replay pool and update the corresponding TD error;

[0041] Step S 48 The sampling probability is dynamically adjusted based on the TD error, and K samples are sampled from the priority experience replay pool.

[0042] Step S49 Train the actor network and critic network using K samples and update the network parameters; proceed to the next control cycle and step S. 44 .

[0043] It also includes setting different types of target trajectories, including trapezoidal curves and sine curves.

[0044] The control system of a single-cylinder electro-hydraulic position servo system based on reinforcement learning includes:

[0045] A single-cylinder electro-hydraulic position servo system model building unit is used to construct a single-cylinder electro-hydraulic position servo system model;

[0046] DDPG algorithm framework building blocks, used to construct DDPG algorithm frameworks containing the following reward function R:

[0047] R = R e +R de +R stop ,

[0048]

[0049] R de =-|e t -e t-1 |,

[0050]

[0051] In the formula, R e R is the reward given based on the error between the target position and the actual position. de R is the reward given based on the error change between adjacent sampling times. stop Rewards given based on security; e t Let b be the control error of the system at the current moment, and e be a very small positive number. t-1 X represents the control error of the system at the previous moment; emax represents the maximum allowable control error. p For the position output of the servo system, xpmax is the maximum allowed position output, and xpmin is the minimum allowed position output.

[0052] The target future trajectory prediction model is used to generate the target future trajectory S1, S2, ..., S based on dynamic recursive prediction and multimodal information fusion. N ;

[0053] The agent training unit for the DDPG algorithm is used to train the agent for the DDPG algorithm. The observation data used during training includes: the current position Xp, the system control error e. t The cumulative error of the system ∑e tAnd the target future trajectory S1, S2, ..., S N ;

[0054] The execution unit is used to control the single-cylinder electro-hydraulic position servo system based on the trained intelligent agent.

[0055] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0056] 1. First, in the observer design, target prediction and state information at key time points are added to enhance the system's ability to track dynamic targets. Second, in terms of action output, a continuous action space design is adopted, ensuring that the system outputs continuous action signals within a threshold range, thus improving system control accuracy. Simultaneously, combined with an error differential control strategy, the reward function incorporates an error differential term as a penalty mechanism, penalizing agents that cause large changes in system error, ensuring smooth action and consistency with the control objective. Furthermore, in the reward function design, a nonlinear reward function is introduced to optimize the agent's learning efficiency and control stability through a smoothing reward mechanism.

[0057] Experimental results show that, through the combination of the above schemes, this method is superior to the traditional method in terms of control accuracy, response speed and robustness of single-cylinder electro-hydraulic position servo system.

[0058] Specifically, the reward function of this invention further optimizes the control effect by comprehensively evaluating multi-dimensional performance indicators. Specifically, the reward function not only includes negative rewards for tracking error and velocity changes, but also introduces positive rewards for motion smoothness to reduce oscillations during control, combines negative rewards for energy consumption to improve system energy efficiency, and considers prediction errors at key time points to enhance the system's responsiveness to dynamic targets. Furthermore, in practical industrial applications, when the position output of the electro-hydraulic position servo system exceeds the allowable range, or the control error is too large, the system needs to stop urgently to prevent device damage and accidents. Therefore, this reward function also considers system safety.

[0059] Experiments show that the proposed integrated reward function significantly improves the control accuracy, stability, and robustness of the agent under complex conditions. Specifically, the reward function employs a nonlinear design to ensure smooth rewards and exhibits higher sensitivity to small errors, thereby avoiding reward explosion. By combining it with an error differential control strategy, the smoothness of actions is further guaranteed, ensuring that the system state remains consistent with the target state.

[0060] 2. In this invention, the observer has a high dimension, which can fully cope with the highly nonlinear dynamic characteristics of the electro-hydraulic position servo system.

[0061] 3. By adding external disturbances to the transfer function of the electro-hydraulic position servo system, the uncertainties and interferences that occur in the control process of the electro-hydraulic position servo system in a real environment can be simulated.

[0062] 4. This invention dynamically adjusts the prediction time range and step size according to real-time working conditions, which facilitates further improvement in the tracking capability of dynamic targets and makes it more adaptable to the complex working environment of electro-hydraulic position servo systems.

[0063] 5. The priority experience replay mechanism and dynamic adjustment of sample sampling probability based on TD error ensure that the model focuses more on learning high-value samples, improving control accuracy and stability. Combined with the reward function, the reward signal R is fed back to the DDPG network. By prioritizing the learning of samples with larger errors, policy bias can be corrected more effectively, greatly improving the control accuracy and convergence speed of the DDPG algorithm for nonlinear systems in complex environments.

[0064] 6. By setting different types of target trajectories, this invention can comprehensively consider the characteristics of linear and nonlinear signals in the selection of training signals, so as to meet the performance index requirements of projects for a combination of linear and nonlinear signals such as sinusoidal signals and trapezoidal signals. Attached Figure Description

[0065] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments, wherein:

[0066] Figure 1 This is a schematic diagram of the training process of the intelligent agent in this invention;

[0067] Figure 2 This is a schematic diagram of the single-cylinder electro-hydraulic position servo system in this invention;

[0068] Figure 3 This is a schematic diagram of the closed-loop transfer function model in this invention;

[0069] Figure 4 This is a schematic diagram of the semi-trapezoidal curve in this invention;

[0070] Figure 5 This is a schematic diagram of the training reward curve for the intelligent agent in this invention;

[0071] Figure 6 This is a schematic diagram of the agent's position output curve in this invention;

[0072] Figure 7 The image shows a magnified comparison curve between the DDPG algorithm framework in this invention and the conventional PID algorithm. Detailed Implementation

[0073] Example 1

[0074] As a basic embodiment of the present invention, the present invention includes a control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising the following steps:

[0075] Step S1. Establish a model of a single-cylinder electro-hydraulic position servo system.

[0076] Step S2. Construct the DDPG algorithm architecture and define the reward function R:

[0077] R = R E +R de +R stop ,

[0078]

[0079] R de =-|e t -e t-1 |,

[0080]

[0081] In the formula, R e R is the reward given based on the error between the target position and the actual position. de R is the reward given based on the error change between adjacent sampling times. stop Rewards given based on security; e t Let b be the control error of the system at the current moment, and e be a very small positive number. t-1 K represents the control error of the system at the previous time step; emax represents the maximum allowable control error; and K represents the control error at the previous time step. p This represents the position output of the servo system. xpmax is the maximum allowed position output, and xpmin is the minimum allowed position output.

[0082] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion, and generate the target future trajectory S1, S2, ..., S N .

[0083] Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, train the agent of the DDPG algorithm. The agent's observation data includes: current position Xp, system control error e. t The cumulative error of the system ∑e t And the target future trajectory S1, S2, ..., S N .

[0084] Step S5. Based on the trained agent, control the output action to obtain continuous action signals within the set threshold range, and complete the control of the single-cylinder electro-hydraulic position servo system.

[0085] Example 2

[0086] As a preferred embodiment of the present invention, the present invention includes a control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising the following steps:

[0087] Step S1. Establish a model of a single-cylinder electro-hydraulic position servo system. This includes the following steps:

[0088] Step S 11 Construct a mathematical model of a single-cylinder electro-hydraulic position servo system and obtain the open-loop transfer function G(s);

[0089]

[0090] In the formula, K a For the proportional gain of the servo amplifier, K q K represents the flow gain of the hydraulic cylinder spool valve. v K represents the servo valve gain. f For the displacement sensor gain, A p K represents the piston area of ​​a hydraulic cylinder. c The flow-pressure coefficient of the slide valve is represented by m. l For load quality, w v ξ represents the natural frequency. v This indicates the damping ratio of the servo valve.

[0091] Step S 12 In the simulation platform, an external disturbance is added to the open-loop transfer function to construct a closed-loop transfer function model.

[0092] Step S2. Construct the DDPG algorithm architecture and define the reward function R:

[0093] R = R e +R de +R stop ,

[0094]

[0095] R de =-|e t -e t-1 |,

[0096]

[0097] In the formula, R e R is the reward given based on the error between the target position and the actual position. de R is the reward given based on the error change between adjacent sampling times. stop Rewards given based on security; e tLet b be the control error of the system at the current moment, and e be a very small positive number. t-1 X represents the control error of the system at the previous moment; emax represents the maximum allowable control error. p This represents the position output of the servo system. xpmax is the maximum allowed position output, and xpmin is the minimum allowed position output.

[0098] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion, and generate the target future trajectory S1, S2, ..., S N Specifically, it includes the following steps:

[0099] Step S 31 By utilizing a long short-term memory neural network and combining historical target signals, hydraulic cylinder speed, and external disturbance data, a target future trajectory prediction model is constructed and trained.

[0100] Step S 32 An adaptive time window mechanism is introduced to dynamically adjust the prediction time range and step size based on real-time operating conditions.

[0101] Step S 33 Based on the trained target future trajectory prediction model and the adjusted prediction time range and step size, the target future trajectory is generated.

[0102] Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, train the agent of the DDPG algorithm. The agent's observation data includes: current position Xp, system control error e. t The cumulative error of the system ∑e t And the target future trajectory S1, S2, ..., S N .

[0103] Step S5. Based on the trained agent, control the output action to obtain continuous action signals within the set threshold range, and complete the control of the single-cylinder electro-hydraulic position servo system.

[0104] Example 3

[0105] In another preferred embodiment of the present invention, the present invention includes a control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising the following steps:

[0106] Step S1. Establish a model of a single-cylinder electro-hydraulic position servo system.

[0107] Step S2. Construct the DDPG algorithm architecture and define the reward function R:

[0108] R = R e +R de +R stop,

[0109]

[0110] R de =-|e t -e t-1 |,

[0111]

[0112] In the formula, R e R is the reward given based on the error between the target position and the actual position. de R is the reward given based on the error change between adjacent sampling times. stop Rewards given based on security; e t Let b be the control error of the system at the current moment, and e be a very small positive number. t-1 X represents the control error of the system at the previous moment; emax represents the maximum allowable control error. p This represents the position output of the servo system. xpmax is the maximum allowed position output, and xpmin is the minimum allowed position output.

[0113] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion, and generate the target future trajectory S1, S2, ..., S N .

[0114] Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, train the agent of the DDPG algorithm. The agent's observation data includes: current position Xp, system control error e. t The cumulative error of the system ∑e t And the target future trajectory S1, S2, ..., S N The agent comprises an actor network and a Critic network. Training the agent for the DDPG algorithm specifically includes the following steps:

[0115] Step S 41 Initialize network parameters;

[0116] Step S 42 Determine if the iteration is complete. If not, proceed to step S. 43 If so, end the training;

[0117] Step S 43 Initialize the random process, receive the initial state of the single-cylinder electro-hydraulic position servo system, and predict the target's position in the next N steps;

[0118] Step S 44 Determine if position control has ended; if so, proceed to step S.42 If not, proceed to step S. 45 ;

[0119] Step S 45 Based on strategy Π and a random process, action A is selected, and reward R is obtained;

[0120] Step S 46 .Transition to the next state S', observe the state of the single-cylinder electro-hydraulic position servo system and update the target's position for the next N steps;

[0121] Step S 47 Store (S, A, R, S') in the priority experience replay pool and update the corresponding TD error;

[0122] Step S 48 The sampling probability is dynamically adjusted based on the TD error, and K samples are sampled from the priority experience replay pool.

[0123] Step S 49 Train the actor network and critic network using K samples and update the network parameters; proceed to the next control cycle and step S. 44 .

[0124] Step S5. Based on the trained agent, control the output action to obtain continuous action signals within the set threshold range, and complete the control of the single-cylinder electro-hydraulic position servo system.

[0125] Example 4

[0126] As a preferred embodiment of the present invention, the present invention includes a control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising the following steps:

[0127] Step S1. Establish a model of a single-cylinder electro-hydraulic position servo system. Refer to the attached instruction manual. Figure 2 A single-cylinder electro-hydraulic position servo system mainly consists of command, control, execution, feedback, power, and hydraulic cylinder components. In the system, the command element sends commands and compares them with the feedback element, obtaining an error signal which is then processed by the controller. Finally, a signal converter and signal amplifier convert this signal into an electrical signal, continuously controlling and monitoring the system. This closed-loop system allows the system to ultimately reach the desired position.

[0128] Specifically, step S1 includes the following steps:

[0129] Step S 11 A mathematical model of a single-cylinder electro-hydraulic position servo system is constructed, and the open-loop transfer function G(s) is obtained.

[0130] When the natural frequency of the system's actuator is higher than 50Hz, the servo valve can be considered as an oscillating element, typically of second order, and its mathematical model can be expressed as follows:

[0131]

[0132] In the formula, K ν ω represents the servo valve gain. v ξ represents the natural frequency. ν This indicates the damping ratio of the servo valve.

[0133] The hydraulic energy of the hydraulic cylinder is converted into mechanical energy for linear motion, resulting in continuous reciprocating linear motion. The hydraulic output is the displacement of the piston rod, and its transfer function is:

[0134]

[0135] In the formula, K q A represents the flow gain of the hydraulic cylinder spool valve. p K represents the piston area of ​​a hydraulic cylinder. c The flow-pressure coefficient of the slide valve is represented by m. l For load quality.

[0136] The amplifier and sensor transfer functions are treated as proportional elements in the model, and their transfer functions are as follows:

[0137] G3 = K a H = K f ,

[0138] In the formula, K a For the proportional gain of the servo amplifier, K f This is the gain of the displacement sensor.

[0139] Based on the above transfer function, the open-loop transfer function of the system is obtained as G(s):

[0140]

[0141] In the formula, K a For the proportional gain of the servo amplifier, K q K represents the flow gain of the hydraulic cylinder spool valve. v K represents the servo valve gain. f For the displacement sensor gain, A p K represents the piston area of ​​a hydraulic cylinder. c The flow-pressure coefficient of the slide valve is represented by m. l For load quality, w v ξ represents the natural frequency. v This indicates the damping ratio of the servo valve.

[0142] The servo amplifier's proportional gain is K a=5A / V, servo valve gain is K v =1.96×10 -3 m 3 / (A·s), displacement feedback coefficient K f =1V / m, the piston area of ​​the hydraulic cylinder is A p =1.3×10 -2 m 2 The flow gain of the hydraulic cylinder spool valve is K. q =2.01m 2 / s, the flow-pressure coefficient of the slide valve is K c =4.6×10 -10 m 5 / (W), the natural frequency of the servo valve is w v =100Hz, damping coefficient is ξ v =0.65.

[0143] Step S 12 In the MATLAB / Simulink simulation platform, an external perturbation is added to the open-loop transfer function to construct a simulation as shown in the attached manual. Figure 3 The closed-loop transfer function model is shown.

[0144] Step S2. Construct the DDPG algorithm architecture. The DDPG algorithm architecture is an improvement on the traditional Actor-Critic algorithm, incorporating a target Actor network and a target Critic network. The target network is used to perform soft updates on the current Actor-Critic network, avoiding instability issues during training. The DDPG algorithm observes the current environmental state S. t As input to the network, the output a of the current state is obtained by calculating the deterministic policy function μ. t And add random noise N to increase the probability of exploring unknown states:

[0145] α t =μ(S) t |θ μ )+N,

[0146] The Actor network updates the neural network parameters θ in the policy function μ using a gradient policy:

[0147]

[0148] The Critic network takes the action output by the Actor network as input to obtain the current (S) t ,α t The Q value under ( ) guides the Actor network parameters θ u Select high-value actions. The update formula for the Critic network parameters is:

[0149] y i =r i +γQ′(S i+1 ,μ′(S i+1 |θ μ′ ))|θ Q ,

[0150]

[0151] In the formula, y i S represents the actual evaluation value calculated by the target network. i Indicates the environmental state, a i Indicates in S i The next choice operation, μ represents the deterministic policy function, γ represents the reward decay rate, which indicates the impact of future reward values ​​on the current reward value, and L represents the loss function, which is the actual value y. i The sum of squared errors between the estimated and the actual values.

[0152] Based on this DDPG algorithm architecture, a reward function is defined. Specifically,

[0153] The primary performance indicator of an agent-controlled electro-hydraulic position servo system is the error between the target position and the actual position. If the error is less than a certain threshold during control, the agent receives a positive reward; conversely, the smaller the error, the greater the reward value. Its reward function R... e as follows:

[0154]

[0155] In the formula, R e e is the reward given based on the error between the target position and the actual position. t Let b be the control error of the system at the current moment, and b be a very small positive number to prevent errors in logarithmic calculations when the error is 0.

[0156] In controlling an electro-hydraulic position servo system, it is crucial to ensure the smoothness of the system's output position changes to minimize impact and wear on mechanical components. Therefore, large error variations between adjacent sampling times will result in severe penalties for the agent. Its reward function R... de as follows:

[0157] R de =-|e t -e t-1 |

[0158] In the formula, R de e is the reward given based on the error change between adjacent sampling times. t-1 This represents the control error of the system at the previous moment.

[0159] In addition, in practical industrial applications, when the position output of an electro-hydraulic servo system exceeds the allowable range or the control error is too large, the system needs to stop immediately to prevent damage to components and accidents. Therefore, the system's safety must also be considered when designing the reward function. The reward R given based on safety... stop The design is as follows:

[0160]

[0161] In the formula, emax is the maximum allowable control error, X p This represents the position output of the servo system. xpmax is the maximum allowed position output, and xpmin is the minimum allowed position output.

[0162] When R stop When the value of is 50, the entire system should stop running immediately.

[0163] Therefore, the reward function R of the DDPG algorithm architecture is:

[0164] R = R e +R de +R stop .

[0165] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion, and generate the target future trajectory S1, S2, ..., S N Specifically, it includes the following steps:

[0166] Step S 31 By utilizing a long short-term memory neural network and combining historical target signals, hydraulic cylinder speed, and external disturbance data, a target future trajectory prediction model is constructed and trained.

[0167] Step S 32 An adaptive time window mechanism is introduced to dynamically adjust the prediction time range and step size within 2 to 5 seconds based on real-time operating conditions, thereby optimizing prediction accuracy. The real-time operating conditions include load changes or linear disturbance intensity.

[0168] Step S 33 Based on the trained target future trajectory prediction model and the adjusted prediction time range and step size, the target trajectory within the next 2 to 5 seconds is generated.

[0169] Step S4. Based on the single-cylinder electro-hydraulic position servo system model and DDPG algorithm architecture, train the agent of the DDPG algorithm.

[0170] Since conventional dimensions are insufficient for accurate observation of the corresponding agent in electro-hydraulic position servo systems, it is necessary to expand the dimensionality. The observation data of the agent includes: current position Xp, system control error e. t The cumulative error of the system ∑e t And the target future trajectory S1, S2, ..., S N .

[0171] During the training of the agent, a priority experience replay mechanism is implemented. This mechanism dynamically adjusts the sample sampling probability based on the TD error. For details, please refer to the appendix of the instruction manual. Figure 1 The training of the agent in the DDPG algorithm specifically includes the following steps:

[0172] Step S 41 Initialize network parameters;

[0173] Step S 42 Determine if the iteration is complete. If not, proceed to step S. 43 If so, end the training;

[0174] Step S 43 Initialize the random process, receive the initial state of the single-cylinder electro-hydraulic position servo system, and predict the target's position in the next N steps;

[0175] Step S 44 Determine if position control has ended; if so, proceed to step S. 42 If not, proceed to step S. 45 ;

[0176] Step S 45 Based on strategy Π and a random process, action A is selected, and reward R is obtained;

[0177] Step S 46 .Transition to the next state S', observe the state of the single-cylinder electro-hydraulic position servo system and update the target's position for the next N steps;

[0178] Step S 47 Store (S, A, R, S') in the priority experience replay pool and update the corresponding TD error;

[0179] Step S 48 The sampling probability is dynamically adjusted based on the TD error, and K samples are sampled from the priority experience replay pool.

[0180] Step S 49 Train the actor network and critic network using K samples and update the network parameters; proceed to the next control cycle and step S. 44 .

[0181] More specifically, in the selection of training signals, the characteristics of both linear and nonlinear signals are taken into account to meet the project's performance requirements for a combination of linear and nonlinear signals, such as sinusoidal signals and trapezoidal signals.

[0182] For details, please refer to the instruction manual appendix. Figure 4 Since the trajectory of a single-cylinder electro-hydraulic position servo system completing a single stroke is a semi-trapezoidal curve, there are two critical time points: t1 is the time node when the curve is about to rise, and t2 is the time node when the curve is about to flatten. Using the semi-trapezoidal curve as input command, a single intelligent agent is pre-trained, referring to the appendix of the instruction manual. Figure 5 The simulation time was 35 seconds, with t1 = 5 seconds and t2 = 15 seconds. After 117 iterations, the average reward value reached 1700.

[0183] Step S5. Based on the trained agent, the control output action can obtain continuous action signals within a set threshold range, thus completing the control of the single-cylinder electro-hydraulic position servo system. Different types of target trajectories can be set, including trapezoidal curves and sine curves, enabling the single-cylinder electro-hydraulic position servo system to track the displacement signals of trapezoidal and sine curve targets.

[0184] Specifically, using the intelligent agent trained in step S4, the agent's control is incorporated into the disturbed single-cylinder electro-hydraulic position servo system, resulting in the position output curve shown in the attached instruction manual. Figure 6 As shown.

[0185] Refer to the instruction manual appendix Figure 7 Compared with the traditional PID control algorithm, the DDPG control strategy of this application can significantly improve the robustness and accuracy of the algorithm. The maximum control error and the average control error are reduced by 14.06% and 24.93% respectively compared with the traditional PID algorithm; at the same time, the overshoot is also reduced by 14.06%.

[0186] Example 5

[0187] In another preferred embodiment of the present invention, the present invention includes a control system for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising:

[0188] The single-cylinder electro-hydraulic position servo system model building unit is used to construct a single-cylinder electro-hydraulic position servo system model.

[0189] DDPG algorithm framework building blocks, used to construct DDPG algorithm frameworks containing the following reward function R:

[0190] R = R e +R de +R stop ,

[0191]

[0192] R de =-|e t -e t-1 |,

[0193]

[0194] In the formula, R e R is the reward given based on the error between the target position and the actual position. de R is the reward given based on the error change between adjacent sampling times. stop Rewards given based on security; e t Let b be the control error of the system at the current moment, and e be a very small positive number. t-1 X represents the control error of the system at the previous moment; emax represents the maximum allowable control error. p This represents the position output of the servo system. xpmax is the maximum allowed position output, and xpmin is the minimum allowed position output.

[0195] The target future trajectory prediction model is used to generate the target future trajectory S1, S2, ..., S based on dynamic recursive prediction and multimodal information fusion. N .

[0196] The agent training unit for the DDPG algorithm is used to train the agent for the DDPG algorithm. The observation data used during training includes: the current position Xp, the system control error e. t The cumulative error of the system ∑e t And the target future trajectory S1, S2, ..., S N .

[0197] The execution unit is used to control the single-cylinder electro-hydraulic position servo system based on the trained intelligent agent.

[0198] In summary, any other corresponding modifications made by those skilled in the art after reading this invention document, without requiring creative mental effort, based on the technical solutions and concepts of this invention, are all within the scope of protection of this invention.

Claims

1. A control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, characterized in that: Includes the following steps: Step S1. Establish a model of a single-cylinder electro-hydraulic position servo system; Step S2. Construct the DDPG algorithm architecture and define the reward function R: ， ， ， ， In the formula, The reward is based on the error between the target location and the actual location. The reward is based on the error change between adjacent sampling times. Rewards are given based on security. The control error of the system at the current moment. It is a very small positive number. This represents the control error of the system at the previous moment; To allow for the maximum control error, For the position output of the servo system, To allow the maximum position output, To allow the minimum position to be output; Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion to generate the target future trajectory. ; Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, train the agent of the DDPG algorithm; wherein, the agent's observation data includes: current position System control error 、 The cumulative error of the system and the target future trajectory The agent comprises an actor network and a Critic network. Training the agent for the DDPG algorithm specifically includes the following steps: Step S 41 Initialize network parameters; Step S 42 Determine if the iteration is complete. If not, proceed to step S. 43 If so, end the training; Step S 43 Initialize the random process, receive the initial state of the single-cylinder electro-hydraulic position servo system, and predict the target's position in the next N steps; Step S 44 Determine if position control has ended; if so, proceed to step S. 42 If not, proceed to step S. 45 ; Step S 45 According to the strategy And a random process selects action A and obtains reward R; Step S 46 Transition to the next state S', observe the state of the single-cylinder electro-hydraulic position servo system, and update the target's position for the next N steps; Step S 47 Store (S, A, R, S') in the priority experience replay pool and update the corresponding TD error; Step S 48 The sampling probability is dynamically adjusted based on the TD error, and K samples are sampled from the priority experience replay pool. Step S 49 Train the actor network and critic network using K samples and update the network parameters; proceed to the next control cycle and step S. 44 ; Step S5. Based on the trained agent, control the output action to obtain continuous action signals within the set threshold range, and complete the control of the single-cylinder electro-hydraulic position servo system.

2. The control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 1, characterized in that: Step S1 specifically includes the following steps: Step S 11 A mathematical model of a single-cylinder electro-hydraulic position servo system is constructed, and the open-loop transfer function is obtained. Step S 12 In the simulation platform, an external disturbance is added to the open-loop transfer function to construct a closed-loop transfer function model.

3. The control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 2, characterized in that: The open-loop transfer function for: ， In the formula, For the proportional gain of the servo amplifier, Indicates the flow gain of the hydraulic cylinder spool valve. Indicates the servo valve gain. For displacement sensor gain, Indicates the piston area of ​​the hydraulic cylinder. Indicates the flow-pressure coefficient of the slide valve. For load quality, Indicates the natural frequency. This indicates the damping ratio of the servo valve.

4. The control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 1, characterized in that: Step S3 specifically includes the following steps: Step S 31 By utilizing a long short-term memory neural network and combining historical target signals, hydraulic cylinder speed, and external disturbance data, a target future trajectory prediction model is constructed and trained. Step S 32 An adaptive time window mechanism is introduced to dynamically adjust the prediction time range and step size based on real-time operating conditions; Step S 33 Based on the trained target future trajectory prediction model and the adjusted prediction time range and step size, the target future trajectory is generated.

5. The control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 4, characterized in that: The real-time operating conditions include load changes or linear disturbance intensity.

6. The control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 1, characterized in that: It also includes setting different types of target trajectories, including trapezoidal curves and sine curves.

7. A control system for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, characterized in that: include: A single-cylinder electro-hydraulic position servo system model building unit is used to construct a single-cylinder electro-hydraulic position servo system model; DDPG algorithm framework building blocks, used to construct DDPG algorithm frameworks containing the following reward function R: ， ， ， ， In the formula, The reward is based on the error between the target location and the actual location. The reward is based on the error change between adjacent sampling times. Rewards are given based on security. The control error of the system at the current moment. It is a very small positive number. This represents the control error of the system at the previous moment; To allow for the maximum control error, For the position output of the servo system, To allow the maximum position output, To allow the minimum position to be output; The target future trajectory prediction model is used to generate the target future trajectory based on dynamic recursive prediction and multimodal information fusion. ; The agent training unit for the DDPG algorithm is used to train the agent for the DDPG algorithm. The observation data used during training includes: current position. System control error 、 The cumulative error of the system and the target future trajectory The agent comprises an actor network and a Critic network. Training the agent for the DDPG algorithm specifically includes the following steps: Step S 41 Initialize network parameters; Step S 42 Determine if the iteration is complete. If not, proceed to step S. 43 If so, end the training; Step S 43 Initialize the random process, receive the initial state of the single-cylinder electro-hydraulic position servo system, and predict the target's position in the next N steps; Step S 44 Determine if position control has ended; if so, proceed to step S. 42 If not, proceed to step S. 45 ; Step S 45 According to the strategy And a random process selects action A and obtains reward R; Step S 46 Transition to the next state S', observe the state of the single-cylinder electro-hydraulic position servo system, and update the target's position for the next N steps; Step S 47 Store (S, A, R, S') in the priority experience replay pool and update the corresponding TD error; Step S 48 The sampling probability is dynamically adjusted based on the TD error, and K samples are sampled from the priority experience replay pool. Step S 49 Train the actor network and critic network using K samples and update the network parameters; proceed to the next control cycle and step S. 44 ; The execution unit is used to control the single-cylinder electro-hydraulic position servo system based on the trained intelligent agent.

Images