Method and system for controlling single-cylinder electro-hydraulic position servo system based on reinforcement learning

Abstract

The invention relates to the technical field of electro-hydraulic servo systems, in particular to a control method and system for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, and the method comprises the steps: building a single-cylinder electro-hydraulic position servo system model; constructing a DDPG algorithm architecture, and defining a reward function R; constructing a target future trajectory prediction model based on dynamic recursive prediction and multi-modal information fusion, and generating a target future trajectory; training an intelligent agent of the DDPG algorithm; the observation data of the intelligent agent comprises a current position, a control error of the system, an accumulated error of the system and a future trajectory of the target; and based on the trained intelligent agent, the output action is controlled to obtain continuous action signals within a set threshold range, and control over the single-cylinder electro-hydraulic position servo system is completed. Through the control method and the control system, the highly nonlinear dynamic characteristic in the single-cylinder electro-hydraulic position servo system can be fully dealt with, and the control precision, the stability and the robustness of the intelligent agent under the complex working condition are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electro-hydraulic servo systems, and in particular to a control method and system for a single-cylinder electro-hydraulic position servo system based on reinforcement learning. Background Art

[0002] Single-cylinder electro-hydraulic position servo systems inherently suffer from numerous nonlinearities, uncertainties, and external disturbances. These issues severely restrict the dynamic and steady-state performance of single-cylinder systems. Duan et al. proposed a control strategy combining adjacent cross-coupling with adaptive robust control. Single-cylinder control employs an adaptive robust controller, while a PID coupling controller is designed based on adjacent cross-coupling. Zhang et al. also employed fuzzy PID control for single-cylinder control based on a dual-cylinder parallel synchronous control structure.

[0003] In recent years, reinforcement learning has shown great potential in solving complex control problems. Compared with traditional control methods, reinforcement learning does not rely on precise system models and has strong adaptability and high control accuracy. It is particularly suitable for scenarios with strong nonlinearity and high uncertainty, such as electro-hydraulic position servo systems. Existing studies have used reinforcement learning methods to control electro-hydraulic position servo systems and have achieved certain results. For example, Xu Baochang et al. used the DQN reinforcement learning algorithm to control the proportional servo valve, thereby achieving precise control of the hydraulic throttle valve. The experimental results show that the intelligent agent's control strategy for the valve position can meet the requirements of accuracy and response speed. Yin Fengyuan used the TD3 reinforcement learning algorithm to control the electro-hydraulic servo valve single-cylinder system, verifying the advantages of this method in terms of control performance and applicability.

[0004] However, in existing research, the observer dimension of the above reinforcement learning method is low, usually only containing an error signal or a combination of an error signal and an accumulated error signal, and the output action is mostly discrete, which makes it difficult to fully cope with the highly nonlinear dynamic characteristics of the electro-hydraulic position servo system.

[0005] In addition, in the prior art, a Chinese invention patent document with publication number CN117348403A and publication date January 5, 2024, has been proposed. The technical solution disclosed in this patent document is as follows: a control method for a dual-valve hydraulic system based on reinforcement learning adaptive robust control, using the reinforcement learning DDPG algorithm as the upper-level controller and the ARC control method as the lower-level controller. In the upper-level controller, the first intelligent agent is used to adjust the relevant parameters of the adaptive robust control algorithm, thereby enhancing the stability and anti-interference ability of the control system, thereby improving the robustness of the control system. The second intelligent agent is used to adjust the flow distribution strategy, intelligently adjust the flow distributed by the servo valve and the proportional valve, reduce the coupling of the mutual influence of the parallel valves, and thus correct the tracking error of the system in real time.

[0006] Although the agent reward function of the above technical solution also takes into account several factors, including the negative reward for the piston rod tracking error of the hydraulic cylinder, the comparison between the piston rod tracking error of the hydraulic cylinder at the current time t and the piston rod tracking error of the hydraulic cylinder at the previous time t-1, and the negative reward for the absolute value of the piston rod speed change, it is still relatively simple. In actual use, it cannot fully reflect the multi-dimensional control requirements of the electro-hydraulic position servo system. Summary of the Invention

[0007] In order to solve the above technical problems, the present invention proposes a control method and system for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, which can fully cope with the highly nonlinear dynamic characteristics of the single-cylinder electro-hydraulic position servo system and improve the control accuracy, stability and robustness of the intelligent body under complex working conditions.

[0008] The present invention is achieved by adopting the following technical solutions:

[0009] A control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning includes the following steps:

[0010] Step S1. Establishing a single-cylinder electro-hydraulic position servo system model;

[0011] Step S2. Build the DDPG algorithm architecture and define the reward function R:

[0012] R=R e +R de +R stop ,

[0013]

[0014] R de =-|e t -e t-1 |,

[0015]

[0016] Where R e is the reward based on the error between the target position and the actual position, R de is the reward based on the error change between adjacent sampling times, R stop Rewards based on safety; t is the control error of the system at the current moment, b is a very small positive number, e t-1 is the control error of the system at the last moment; emax is the maximum control error allowed, X p is the position output of the servo system, xpmax is the maximum position output allowed, and xpmin is the minimum position output allowed;

[0017] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion to generate target future trajectories S1, S2, ..., S N ;

[0018] Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, the DDPG algorithm agent is trained; wherein the agent's observation data includes: the current position Xp, the system's control error e t , the cumulative error of the system ∑e t and the target future trajectory S1, S2, ..., S N ;

[0019] Step S5. Based on the trained intelligent agent, the output action is controlled to obtain a continuous action signal within the set threshold range, thereby completing the control of the single-cylinder electro-hydraulic position servo system.

[0020] The step S1 specifically includes the following steps:

[0021] Step S 11 .Construct a mathematical model of a single-cylinder electro-hydraulic position servo system and obtain the open-loop transfer function;

[0022] Step S 12 .In the simulation platform, external disturbances are added to the open-loop transfer function to construct a closed-loop transfer function model.

[0023] The open-loop transfer function G(s) is:

[0024]

[0025] Where K a is the servo amplifier proportional gain, K q Indicates the flow gain of the hydraulic cylinder slide valve, K v Indicates the servo valve gain, K f is the displacement sensor gain, A p Indicates the piston area of ​​the hydraulic cylinder, K c Indicates the flow pressure coefficient of the slide valve, m l is the load mass, w v represents the natural frequency, ξ v Indicates the servo valve damping ratio.

[0026] The step S3 specifically includes the following steps:

[0027] Step S 31 Using a long short-term memory neural network, combined with historical target signals, hydraulic cylinder velocity, and external disturbance data, a model for predicting the target's future trajectory is constructed and trained.

[0028] Step S32 Introducing an adaptive time window mechanism to dynamically adjust the prediction time range and step size based on real-time operating conditions;

[0029] Step S 33 Generate the target's future trajectory based on the trained target future trajectory prediction model and the adjusted prediction time range and step size.

[0030] The real-time working condition includes load change or linear disturbance intensity.

[0031] During the training of the intelligent agent, a priority experience replay mechanism is also set up.

[0032] The priority experience replay mechanism dynamically adjusts the sample sampling probability according to the TD error.

[0033] The agent includes an actor network and a critic network. Step S4 of training the agent of the DDPG algorithm specifically includes the following steps:

[0034] Step S 41 .Initialize network parameters;

[0035] Step S 42 . Determine whether the iteration is complete, if not, go to step S 43 ,If so, end the training;

[0036] Step S 43 Initialize the random process, receive the initial state of the single-cylinder electro-hydraulic position servo system, and predict the target position N steps in the future;

[0037] Step S 44 . Determine whether the position control is completed. If so, go to step S 42 If not, go to step S 45 ;

[0038] Step S 45 Select action A based on strategy π and random process, and get reward R;

[0039] Step S 46 .Transfer to the next state S', observe the state of the single-cylinder electro-hydraulic position servo system and update the target position for the next N steps;

[0040] Step S 47 Store (S, A, R, S') into the priority experience replay pool and update the corresponding TD error;

[0041] Step S 48 Dynamically adjust the sample sampling probability based on the TD error and sample K samples from the priority experience replay pool;

[0042] Step S49 . Use K samples to train the actor network and the critic network and update the network parameters; enter the next control cycle and enter step S 44 .

[0043] It also includes setting different types of target trajectories, including trapezoidal curves and sin curves.

[0044] The control system of a single-cylinder electro-hydraulic position servo system based on reinforcement learning includes:

[0045] Single-cylinder electro-hydraulic position servo system model building unit, used to build a single-cylinder electro-hydraulic position servo system model;

[0046] DDPG algorithm framework building unit, used to construct the DDPG algorithm framework including the following reward function R:

[0047] R=R e +R de +R stop ,

[0048]

[0049] R de =-|e t -e t-1 |,

[0050]

[0051] Where R e is the reward based on the error between the target position and the actual position, R de is the reward based on the error change between adjacent sampling times, R stop Rewards based on safety; t is the control error of the system at the current moment, b is a very small positive number, e t-1 is the control error of the system at the last moment; emax is the maximum control error allowed, X p is the position output of the servo system, xpmax is the maximum position output allowed, and xpmin is the minimum position output allowed;

[0052] The target future trajectory prediction model is used to generate the target future trajectory S1, S2, ..., S based on dynamic recursive prediction and multimodal information fusion N ;

[0053] The agent training unit of the DDPG algorithm is used to train the agent of the DDPG algorithm. The observation data used during training include: the current position Xp, the control error of the system e t , the cumulative error of the system ∑e tand the target future trajectory S1, S2, ..., S N ;

[0054] The execution unit is used to complete the control of the single-cylinder electro-hydraulic position servo system based on the trained intelligent agent.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. First, in the observer design, target prediction and state information of key time nodes are added to enhance the system's ability to track dynamic targets. Second, in terms of action output, a continuous action space design is adopted. The system output action can obtain continuous action signals within the threshold range, improving the system's control accuracy. At the same time, combined with the error differential control strategy, the reward function introduces an error differential term as a penalty mechanism, penalizing agents that cause large changes in system error to ensure smooth action and consistency with the control objective. In addition, in the reward function design, a nonlinear reward function is introduced to optimize the agent's learning efficiency and control stability through a smoothed reward mechanism.

[0057] Experimental results show that through the mutual cooperation of the above schemes, the proposed method is superior to traditional methods in terms of control accuracy, response speed and robustness of the single-cylinder electro-hydraulic position servo system.

[0058] Specifically, the reward function of the present invention further optimizes the control effect by comprehensively evaluating multi-dimensional performance indicators. Specifically, the reward function not only includes negative rewards for tracking error and speed change, but also introduces positive rewards for motion smoothness to reduce oscillations in the control process, combined with negative rewards for energy consumption to improve system energy efficiency, while considering the prediction error of key time nodes to enhance the system's responsiveness to dynamic targets. In addition, due to actual industrial applications, when the position output of the electro-hydraulic position servo system exceeds the allowable range, or the control error is too large, the system needs to stop urgently to prevent device damage and accidents. Therefore, this reward function also takes the safety of the system into consideration.

[0059] Experiments show that this integrated reward function significantly improves the control accuracy, stability, and robustness of the intelligent agent under complex working conditions. Specifically, the reward function adopts a nonlinear design to ensure reward smoothness and show greater sensitivity to small errors, thereby avoiding reward explosion. By combining it with an error differential control strategy, it further ensures the smoothness of the action and keeps the system state consistent with the target state.

[0060] 2. In the present invention, the observer has a high dimension and can fully cope with the highly nonlinear dynamic characteristics of the electro-hydraulic position servo system.

[0061] 3. Adding external disturbances to the transfer function of the electro-hydraulic position servo system can simulate the uncertainty and interference that occur in the control process of the electro-hydraulic position servo system in a real environment.

[0062] 4. The present invention dynamically adjusts the prediction time range and step size according to the real-time working conditions, so as to further enhance the tracking capability of dynamic targets and make it more adaptable to the complex working environment of the electro-hydraulic position servo system.

[0063] 5. The prioritized experience replay mechanism and dynamic adjustment of sample sampling probability based on TD error ensure that the model focuses on learning high-value samples, improving control accuracy and stability. Combined with the reward function, the reward signal R is fed back to the DDPG network. By prioritizing samples with large learning errors, it can more effectively correct policy deviations, significantly improving the control accuracy and convergence speed of the DDPG algorithm for nonlinear systems in complex environments.

[0064] 6. By setting different types of target trajectories, the present invention can comprehensively consider the characteristics of linear and nonlinear signals in the selection of training signals to meet the project's performance index requirements for the combination of linear and nonlinear signals such as sinusoidal signals and trapezoidal signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, wherein:

[0066] Figure 1 Schematic diagram of the training process of the intelligent agent in the present invention;

[0067] Figure 2 Schematic diagram of the structure of the single-cylinder electro-hydraulic position servo system of the present invention;

[0068] Figure 3 is a schematic diagram of a closed-loop transfer function model in the present invention;

[0069] Figure 4 Schematic diagram of a semi-trapezoidal curve in the present invention;

[0070] Figure 5 This is a schematic diagram of the agent training reward curve in the present invention;

[0071] Figure 6 Schematic diagram of the intelligent agent position output curve in the present invention;

[0072] Figure 7 This is the enlarged comparison position curve of the DDPG algorithm framework in the present invention and the conventional PID algorithm. DETAILED DESCRIPTION

[0073] Example 1

[0074] As a basic embodiment of the present invention, the present invention includes a control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising the following steps:

[0075] Step S1: Establish a single-cylinder electro-hydraulic position servo system model.

[0076] Step S2. Build the DDPG algorithm architecture and define the reward function R:

[0077] R=R E +R de +R stop ,

[0078]

[0079] R de =-|e t -e t-1 |,

[0080]

[0081] Where R e is the reward based on the error between the target position and the actual position, R de is the reward based on the error change between adjacent sampling times, R stop Rewards based on safety; t is the control error of the system at the current moment, b is a very small positive number, e t-1 is the control error of the system at the last moment; emax is the maximum control error allowed, K p is the position output of the servo system, xpmax is the maximum position output allowed, and xpmin is the minimum position output allowed.

[0082] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion to generate target future trajectories S1, S2, ..., S N .

[0083] Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, the DDPG algorithm agent is trained. The observation data of the agent include: the current position Xp, the control error of the system e t , the cumulative error of the system ∑e t and the target future trajectory S1, S2, ..., S N .

[0084] Step S5. Based on the trained intelligent agent, the output action is controlled to obtain a continuous action signal within the set threshold range, thereby completing the control of the single-cylinder electro-hydraulic position servo system.

[0085] Example 2

[0086] As a preferred embodiment of the present invention, the present invention includes a control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising the following steps:

[0087] Step S1. Establish a single-cylinder electro-hydraulic position servo system model. Specifically, the following steps are included:

[0088] Step S 11 Construct a mathematical model of a single-cylinder electro-hydraulic position servo system and obtain the open-loop transfer function G(s);

[0089]

[0090] Where K a is the servo amplifier proportional gain, K q Indicates the flow gain of the hydraulic cylinder slide valve, K v Indicates the servo valve gain, K f is the displacement sensor gain, A p Indicates the piston area of ​​the hydraulic cylinder, K c Indicates the flow pressure coefficient of the slide valve, m l is the load mass, w v represents the natural frequency, ξ v Indicates the servo valve damping ratio.

[0091] Step S 12 .In the simulation platform, external disturbances are added to the open-loop transfer function to construct a closed-loop transfer function model.

[0092] Step S2. Build the DDPG algorithm architecture and define the reward function R:

[0093] R=R e +R de +R stop ,

[0094]

[0095] R de =-|e t -e t-1 |,

[0096]

[0097] Where R e is the reward based on the error between the target position and the actual position, R de is the reward based on the error change between adjacent sampling times, R stop Rewards based on safety; tis the control error of the system at the current moment, b is a very small positive number, e t-1 is the control error of the system at the last moment; emax is the maximum control error allowed, X p is the position output of the servo system, xpmax is the maximum position output allowed, and xpmin is the minimum position output allowed.

[0098] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion to generate target future trajectories S1, S2, ..., S N The specific steps include:

[0099] Step S 31 .Using long short-term memory neural network, combined with historical target signals, hydraulic cylinder speed and external disturbance data, a target future trajectory prediction model is constructed and trained.

[0100] Step S 32 .Introduce an adaptive time window mechanism to dynamically adjust the prediction time range and step size according to real-time working conditions.

[0101] Step S 33 Generate the target's future trajectory based on the trained target future trajectory prediction model and the adjusted prediction time range and step size.

[0102] Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, the DDPG algorithm agent is trained. The observation data of the agent include: the current position Xp, the control error of the system e t , the cumulative error of the system ∑e t and the target future trajectory S1, S2, ..., S N .

[0103] Step S5. Based on the trained intelligent agent, the output action is controlled to obtain a continuous action signal within the set threshold range, thereby completing the control of the single-cylinder electro-hydraulic position servo system.

[0104] Example 3

[0105] As another preferred embodiment of the present invention, the present invention includes a control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising the following steps:

[0106] Step S1: Establish a single-cylinder electro-hydraulic position servo system model.

[0107] Step S2. Build the DDPG algorithm architecture and define the reward function R:

[0108] R=R e +R de +R stop,

[0109]

[0110] R de =-|e t -e t-1 |,

[0111]

[0112] Where R e is the reward based on the error between the target position and the actual position, R de is the reward based on the error change between adjacent sampling times, R stop Rewards based on safety; t is the control error of the system at the current moment, b is a very small positive number, e t-1 is the control error of the system at the last moment; emax is the maximum control error allowed, X p is the position output of the servo system, xpmax is the maximum position output allowed, and xpmin is the minimum position output allowed.

[0113] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion to generate target future trajectories S1, S2, ..., S N .

[0114] Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, the DDPG algorithm agent is trained. The observation data of the agent include: the current position Xp, the control error of the system e t , the cumulative error of the system ∑e t and the target future trajectory S1, S2, ..., S N The agent includes an actor network and a critic network. Training the agent of the DDPG algorithm specifically includes the following steps:

[0115] Step S 41 .Initialize network parameters;

[0116] Step S 42 . Determine whether the iteration is complete, if not, go to step S 43 ,If so, end the training;

[0117] Step S 43 Initialize the random process, receive the initial state of the single-cylinder electro-hydraulic position servo system, and predict the target position N steps in the future;

[0118] Step S 44 . Determine whether the position control is completed. If so, go to step S42 If not, go to step S 45 ;

[0119] Step S 45 Select action A based on strategy π and random process, and get reward R;

[0120] Step S 46 .Transfer to the next state S', observe the state of the single-cylinder electro-hydraulic position servo system and update the target position for the next N steps;

[0121] Step S 47 Store (S, A, R, S') into the priority experience replay pool and update the corresponding TD error;

[0122] Step S 48 Dynamically adjust the sample sampling probability based on the TD error and sample K samples from the priority experience replay pool;

[0123] Step S 49 . Use K samples to train the actor network and the critic network and update the network parameters; enter the next control cycle and enter step S 44 .

[0124] Step S5. Based on the trained intelligent agent, the output action is controlled to obtain a continuous action signal within the set threshold range, thereby completing the control of the single-cylinder electro-hydraulic position servo system.

[0125] Example 4

[0126] As the best embodiment of the present invention, the present invention includes a control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising the following steps:

[0127] Step S1. Establish a single cylinder electro-hydraulic position servo system model. Figure 2 A single-cylinder electro-hydraulic position servo system primarily consists of command, control, execution, feedback, power, and cylinder components. The command element in the system sends a command, compares it with the feedback element, and generates an error signal that is then passed to the controller for processing. Finally, a signal converter and amplifier convert this error signal into an electrical signal, which continuously controls and monitors the system. This closed-loop system ultimately enables the system to reach the desired position.

[0128] Specifically, step S1 includes the following steps:

[0129] Step S 11 .Construct the mathematical model of the single-cylinder electro-hydraulic position servo system and obtain the open-loop transfer function G(s).

[0130] When the natural frequency of the system actuator is higher than 50Hz, the servo valve can be regarded as an oscillation link, which is generally second-order. Its mathematical model is expressed as:

[0131]

[0132] Where K ν represents the servo valve gain, ω v represents the natural frequency, ξ ν Indicates the servo valve damping ratio.

[0133] The hydraulic energy of the hydraulic cylinder oil is converted into mechanical energy of linear motion, which continuously performs linear reciprocating motion. The hydraulic output is the displacement of the piston rod, and its transfer function is:

[0134]

[0135] Where K q Indicates the flow gain of the hydraulic cylinder slide valve, A p Indicates the piston area of ​​the hydraulic cylinder, K c Indicates the flow pressure coefficient of the slide valve, m l is the load mass.

[0136] The transfer functions of the amplifier and sensor are considered as proportional links in the model, and their transfer functions are:

[0137] G3=K a ,H=K f ,

[0138] Where K a is the servo amplifier proportional gain, K f is the displacement sensor gain.

[0139] According to the above transfer function, the open-loop transfer function of the system is G(s):

[0140]

[0141] Where K a is the servo amplifier proportional gain, K q Indicates the flow gain of the hydraulic cylinder slide valve, K v Indicates the servo valve gain, K f is the displacement sensor gain, A p Indicates the piston area of ​​the hydraulic cylinder, K c Indicates the flow pressure coefficient of the slide valve, m l is the load mass, w v represents the natural frequency, ξ v Indicates the servo valve damping ratio.

[0142] The proportional gain of the servo amplifier is K a=5A / V, servo valve gain is K v =1.96×10 -3 m 3 / (A·s), displacement feedback coefficient K f =1V / m, the piston area of ​​the hydraulic cylinder is A p =1.3×10 -2 m 2 , the flow gain of the hydraulic cylinder slide valve is K q =2.01m 2 / s, the flow pressure coefficient of the slide valve is K c =4.6×10 -10 m 5 / (W), the natural frequency of the servo valve is w v =100Hz, the damping coefficient is ξ v =0.65.

[0143] Step S 12 In the MATLAB / Simulink simulation platform, add external disturbances to the open-loop transfer function and construct the Figure 3 The closed-loop transfer function model is shown.

[0144] Step S2. Construct the DDPG algorithm architecture. The DDPG algorithm architecture is an improvement on the traditional actor-critic algorithm. It adds a target actor network and a target critic network, and uses the target network to soft-update the current AC network to avoid the problem of poor stability during algorithm training. The DDPG algorithm observes the current environment state S t As the input of the network, the output a of the current state is obtained by calculating the deterministic policy function μ t , and superimpose random noise N to increase the probability of exploring unknown states:

[0145] α t =μ(S t |θ μ )+N,

[0146] The Actor network uses the gradient strategy to update the neural network parameters θ in the policy function μ:

[0147]

[0148] The Critic network takes the action output by the Actor network as input and obtains the current (S t ,α t ) under the Q value, thereby guiding the Actor network parameters θ u To select high-value actions. The update formula of the critic network parameters is:

[0149] y i =r i +γQ′(S i+1 ,μ′(S i+1 |θ μ′ ))|θ Q ,

[0150]

[0151] Where y i represents the actual evaluation value calculated by the target network, S i Indicates the environmental state, a i Indicates that in S i The action selected under , μ represents the deterministic policy function, γ represents the reward decay rate, which represents the impact of the reward value of the future step on the reward value of the current step, and L represents the loss function, which is the actual value y i The sum of squared errors between the estimated value and the

[0152] Based on the DDPG algorithm architecture, define the reward function. Specifically,

[0153] The primary performance indicator of an agent-controlled electro-hydraulic position servo system is the error between the target position and the actual position. If the error is less than a certain threshold during the control process, the agent is given a positive reward. At the same time, the smaller the error, the greater the reward value given to the agent. Its reward function R e as follows:

[0154]

[0155] Where R e is the reward based on the error between the target position and the actual position, e t is the control error of the system at the current moment, and b is a very small positive number to prevent errors in logarithmic operations when the error is 0.

[0156] In the process of controlling the electro-hydraulic position servo system, the smoothness of the system output position change should be ensured as much as possible to reduce the impact and wear on the mechanical components. Therefore, if the error changes greatly between adjacent sampling times, the agent will be severely punished. Its reward function R de as follows:

[0157] R de =-|e t -e t-1 |.

[0158] Where R de is the reward based on the error change within adjacent sampling times, e t-1 is the control error of the system at the previous moment.

[0159] In addition, in industrial applications, when the position output of the electro-hydraulic servo system exceeds the allowable range or the control error is too large, the system needs to be stopped immediately to prevent damage to the device and accidents. Therefore, when designing the reward function, the safety of the system must also be considered. The reward R given based on safety stop The design is as follows:

[0160]

[0161] Where, emax is the maximum allowable control error, X p is the position output of the servo system, xpmax is the maximum position output allowed, and xpmin is the minimum position output allowed.

[0162] When R stop When the value is 50, the entire system should stop running immediately.

[0163] Therefore, the reward function R of the DDPG algorithm architecture is:

[0164] R=R e +R de +R stop .

[0165] Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion to generate target future trajectories S1, S2, ..., S N The specific steps include:

[0166] Step S 31 .Using long short-term memory neural network, combined with historical target signals, hydraulic cylinder speed and external disturbance data, a target future trajectory prediction model is constructed and trained.

[0167] Step S 32 An adaptive time window mechanism is introduced to dynamically adjust the prediction time range and step size from 2 to 5 seconds based on real-time operating conditions, such as load changes or linear disturbance intensity, to optimize prediction accuracy.

[0168] Step S 33 Based on the trained target future trajectory prediction model and the adjusted prediction time range and step size, the target trajectory within the next 2 to 5 seconds is generated.

[0169] Step S4: Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, the DDPG algorithm agent is trained.

[0170] Since it is difficult to observe the corresponding intelligent agent well in conventional dimensions for the electro-hydraulic position servo system, the dimensions need to be expanded. The observation data of the intelligent agent include: the current position Xp, the control error of the system e t , the cumulative error of the system ∑e t and the target future trajectory S1, S2, ..., S N .

[0171] During the training of the agent, a priority experience replay mechanism is set up. The priority experience replay mechanism dynamically adjusts the sample sampling probability according to the TD error. Figure 1 , the DDPG algorithm agent training specifically includes the following steps:

[0172] Step S 41 .Initialize network parameters;

[0173] Step S 42 . Determine whether the iteration is complete, if not, go to step S 43 ,If so, end the training;

[0174] Step S 43 Initialize the random process, receive the initial state of the single-cylinder electro-hydraulic position servo system, and predict the target position N steps in the future;

[0175] Step S 44 . Determine whether the position control is completed. If so, go to step S 42 If not, go to step S 45 ;

[0176] Step S 45 Select action A based on strategy π and random process, and get reward R;

[0177] Step S 46 .Transfer to the next state S', observe the state of the single-cylinder electro-hydraulic position servo system and update the target position for the next N steps;

[0178] Step S 47 Store (S, A, R, S') into the priority experience replay pool and update the corresponding TD error;

[0179] Step S 48 Dynamically adjust the sample sampling probability based on the TD error and sample K samples from the priority experience replay pool;

[0180] Step S 49 . Use K samples to train the actor network and the critic network and update the network parameters; enter the next control cycle and enter step S 44 .

[0181] More specifically, in the selection of training signals, the linear and nonlinear signal characteristics are comprehensively considered to meet the project's performance indicator requirements for the combination of linear and nonlinear signals such as sinusoidal signals and trapezoidal signals.

[0182] For details, please refer to the attached manual. Figure 4 Since the trajectory of a single cylinder electro-hydraulic position servo system to complete a single stroke is a semi-trapezoidal curve, there are two key time points: t1 is the time point when the curve is about to rise, and t2 is the time point when the curve is about to flatten. Using the semi-trapezoidal curve as the input instruction, pre-train a single agent, refer to the instructions attached. Figure 5 , simulation time 35s, t1 = 5s, t2 = 15s. 117 iterations, average reward value reaches 1700.

[0183] Step S5. Based on the trained agent, the output action is controlled to generate a continuous action signal within a set threshold range, thereby completing the control of the single-cylinder electro-hydraulic position servo system. Different types of target trajectories can be set, including trapezoidal and sinusoidal curves, so that the single-cylinder electro-hydraulic position servo system can track the target displacement signals of trapezoidal and sinusoidal curves.

[0184] Specifically, the intelligent agent trained in step S4 is used to control the single-cylinder electro-hydraulic position servo system with disturbance, and the position output curve is obtained as shown in the attached manual. Figure 6 shown.

[0185] Refer to the instruction manual Figure 7 Compared with the traditional PID control algorithm, the DDPG control strategy of this application can greatly improve the robustness and accuracy of the algorithm. The maximum control error and average control error are reduced by 14.06% and 24.93% compared with the traditional PID algorithm; at the same time, the overshoot is also reduced by 14.06%.

[0186] Example 5

[0187] As another preferred embodiment of the present invention, the present invention includes a control system of a single-cylinder electro-hydraulic position servo system based on reinforcement learning, comprising:

[0188] The single-cylinder electro-hydraulic position servo system model building unit is used to build a single-cylinder electro-hydraulic position servo system model.

[0189] DDPG algorithm framework building unit, used to construct the DDPG algorithm framework including the following reward function R:

[0190] R=R e +R de +R stop ,

[0191]

[0192] R de =-|e t -e t-1 |,

[0193]

[0194] Where R e is the reward based on the error between the target position and the actual position, R de is the reward based on the error change between adjacent sampling times, R stop Rewards based on safety; t is the control error of the system at the current moment, b is a very small positive number, e t-1 is the control error of the system at the last moment; emax is the maximum control error allowed, X p is the position output of the servo system, xpmax is the maximum position output allowed, and xpmin is the minimum position output allowed.

[0195] The target future trajectory prediction model is used to generate the target future trajectory S1, S2, ..., S based on dynamic recursive prediction and multimodal information fusion N .

[0196] The agent training unit of the DDPG algorithm is used to train the agent of the DDPG algorithm. The observation data used during training include: the current position Xp, the control error of the system e t , the cumulative error of the system ∑e t and the target future trajectory S1, S2, ..., S N .

[0197] The execution unit is used to complete the control of the single-cylinder electro-hydraulic position servo system based on the trained intelligent agent.

[0198] In summary, after reading the present invention document, ordinary technicians in this field can make various other corresponding transformation schemes based on the technical solutions and technical concepts of the present invention without creative mental work, which all fall within the scope of protection of the present invention.

Claims

1. A control method for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, characterized by: The following steps are involved: Step S1. Establishing a single-cylinder electro-hydraulic position servo system model; Step S2. Build the DDPG algorithm architecture and define the reward function R: R=R e +R de +R stop , R de =-|e t -And t-1 |, Where R e is the reward based on the error between the target position and the actual position, R de is the reward based on the error change between adjacent sampling times, R stop Rewards based on safety; t is the control error of the system at the current moment, b is a very small positive number, e t-1 is the control error of the system at the last moment; emax is the maximum control error allowed, X p is the position output of the servo system, xpmax is the maximum position output allowed, and xpmin is the minimum position output allowed; Step S3. Construct a target future trajectory prediction model based on dynamic recursive prediction and multimodal information fusion to generate target future trajectories S1, S2, ..., S N ; Step S4. Based on the single-cylinder electro-hydraulic position servo system model and the DDPG algorithm architecture, the DDPG algorithm agent is trained; wherein the agent's observation data includes: the current position Xp, the system's control error e t , the cumulative error of the system ∑e t and the target future trajectory S1, S2, ..., S N ; Step S5. Based on the trained intelligent agent, the output action is controlled to obtain a continuous action signal within the set threshold range, thereby completing the control of the single-cylinder electro-hydraulic position servo system.

2. The control method of a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 1, characterized in that: The step S1 specifically includes the following steps: Step S 11 .Construct a mathematical model of a single-cylinder electro-hydraulic position servo system and obtain the open-loop transfer function; Step S 12 .In the simulation platform, external disturbances are added to the open-loop transfer function to construct a closed-loop transfer function model.

3. The control method of a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 2, characterized in that: The open-loop transfer function G(s) is: Where K a is the servo amplifier proportional gain, K q Indicates the flow gain of the hydraulic cylinder slide valve, K v Indicates the servo valve gain, K f is the displacement sensor gain, A p Indicates the piston area of ​​the hydraulic cylinder, K c Indicates the flow pressure coefficient of the slide valve, m l is the load mass, w v represents the natural frequency, ξ v Indicates the servo valve damping ratio.

4. The control method of a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 1, characterized in that: The step S3 specifically includes the following steps: Step S 31 Using a long short-term memory neural network, combined with historical target signals, hydraulic cylinder velocity, and external disturbance data, a model for predicting the target's future trajectory is constructed and trained. Step S 32 Introducing an adaptive time window mechanism to dynamically adjust the prediction time range and step size based on real-time operating conditions; Step S 33 Generate the target's future trajectory based on the trained target future trajectory prediction model and the adjusted prediction time range and step size.

5. The control method of a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 4, characterized in that: The real-time working condition includes load change or linear disturbance intensity.

6. The control method of a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 1, characterized in that: During the training of the intelligent agent, a priority experience replay mechanism is also set up.

7. The control method of a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 6, characterized in that: The priority experience replay mechanism dynamically adjusts the sample sampling probability according to the TD error.

8. The control method of a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 7, characterized in that: The agent includes an actor network and a critic network. Step S4 of training the agent of the DDPG algorithm specifically includes the following steps: Step S 41 .Initialize network parameters; Step S 42 . Determine whether the iteration is complete, if not, go to step S 43 ,If so, end the training; Step S 43 Initialize the random process, receive the initial state of the single-cylinder electro-hydraulic position servo system, and predict the target position N steps in the future; Step S 44 . Determine whether the position control is completed. If so, go to step S 42 If not, go to step S 45 ; Step S 45 Select action A based on strategy π and random process, and get reward R; Step S 46 .Transfer to the next state S', observe the state of the single-cylinder electro-hydraulic position servo system and update the target position for the next N steps; Step S 47 Store (S, A, R, S') into the priority experience replay pool and update the corresponding TD error; Step S 48 Dynamically adjust the sample sampling probability based on the TD error and sample K samples from the priority experience replay pool; Step S 49 . Use K samples to train the actor network and the critic network and update the network parameters; enter the next control cycle and enter step S 44 .

9. The control method of a single-cylinder electro-hydraulic position servo system based on reinforcement learning according to claim 1, characterized in that: It also includes setting different types of target trajectories, including trapezoidal curves and sin curves.

10. A control system for a single-cylinder electro-hydraulic position servo system based on reinforcement learning, characterized by: include: The single-cylinder electro-hydraulic position servo system model building unit is used to build a single-cylinder electro-hydraulic position servo system model; the DDPG algorithm framework building unit is used to construct a DDPG algorithm framework that includes the following reward function R: R = R e +R de +R stop , R de =-|e t -And t-1 |, Where R e is the reward based on the error between the target position and the actual position, R de is the reward based on the error change between adjacent sampling times, R stop Rewards based on safety; t is the control error of the system at the current moment, b is a very small positive number, e t-1 is the control error of the system at the last moment; emax is the maximum control error allowed, X p is the position output of the servo system, xpmax is the maximum position output allowed, and xpmin is the minimum position output allowed; The target future trajectory prediction model is used to generate the target future trajectory S1, S2, ..., S based on dynamic recursive prediction and multimodal information fusion N ; The agent training unit of the DDPG algorithm is used to train the agent of the DDPG algorithm. The observation data used during training include: the current position Xp, the control error of the system e t , the cumulative error of the system ∑e t and the target future trajectory S1, S2, ..., S N ; The execution unit is used to complete the control of the single-cylinder electro-hydraulic position servo system based on the trained intelligent agent.

Images