Control system for mobile engineering equipment work mechanism based on reinforcement learning PID controller and use method

Abstract

A control system and method for mobile engineering equipment working mechanism based on reinforcement learning PID controller, aiming to realize the switching management of remote driving and automatic driving two working modes, the control of instruction flow transmission, and the rapid switching and control of two joint control modes; through the reinforcement learning algorithm, the agent automatically adjusts the parameters of the PID controller, so that the controller can adaptively optimize the control effect under different working conditions.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of unmanned mobile engineering equipment, and particularly relates to a control system and use method of a mobile engineering equipment working mechanism based on a reinforcement learning PID controller, to realize more accurate and efficient control of the working mechanism of mobile engineering equipment such as a pusher, a shovel, and a loader. BACKGROUND

[0002] Mobile engineering equipment, such as pushers, shovels, and loaders, usually includes working mechanisms such as booms and buckets, which are usually driven by hydraulic rods. For unmanned mobile engineering equipment, it is usually necessary to meet various working modes such as remote driving and automatic driving, and the operator needs to switch the working mode according to the actual production conditions during actual work. In the operation mode of the working mechanism, joint speed control and joint position control have their own advantages. Speed control is suitable for situations where the speed control requirement is high and the position accuracy requirement is low, such as digging and loading; position control is suitable for situations where the position accuracy requirement is high, such as flattening work. If the operation mode can be switched according to the operation needs during actual work, the operation complexity can be reduced and the operation efficiency can be improved.

[0003] On the other hand, the control performance of these devices is directly related to work efficiency, safety, and energy consumption. In the control system, the PID controller is widely used in various industrial automation scenarios due to its simple structure, easy understanding, and implementation. However, when facing complex and variable working environments and nonlinear system characteristics, the fixed parameters of the traditional PID controller often cannot meet the requirements of high-precision control, especially in terms of dynamic response speed, anti-interference ability, and stability. SUMMARY

[0004] The present application proposes a control system and use method of a mobile engineering equipment working mechanism based on a reinforcement learning PID controller, aiming to realize the switching management of remote driving and automatic driving two working modes, the control of instruction flow transmission, and the switching and control of the two joint control modes; through the reinforcement learning algorithm, the agent automatically adjusts the parameters of the PID controller, so that the controller can adaptively optimize the control effect under different working conditions.

[0005] A control system of a mobile engineering equipment working mechanism based on a reinforcement learning PID controller is composed of a remote control platform, an automatic driving control unit, a vehicle-mounted computer, a virtual debugging system, a TCP-CAN protocol conversion unit, an electro-hydraulic proportional valve, a joint driving hydraulic rod, and a stroke sensor. The stroke sensor receives the actual position and actual speed of the joint driving hydraulic rod and transmits them to the TCP-CAN protocol conversion unit.

[0006] The remote management platform and the autonomous driving control unit send control instructions of the working mechanism to the vehicle-mounted computer through Ethernet, and the content of the control instructions includes the working mode, the control mode, and the joint target speed or joint target position of the working mechanism.

[0007] The vehicle-mounted computer includes a data communication server, which receives control instructions from the remote management platform and the autonomous driving control unit, and transmits the control instructions to the working mode switching unit, respectively. Meanwhile, the data communication server receives actual position and actual speed information from the TCP-CAN protocol conversion unit, and transmits the actual position to the position PID controller in the position controller, and transmits the actual speed to the speed PID controller in the speed controller.

[0008] The working mode switching unit selects control instructions from the data communication server according to the working mode and sends them to the joint control mode switching unit. The joint control mode switching unit determines the selected joint control mode, and if it is a speed control mode, sends the target speed data to the speed PID controller; if it is a position mode, sends the target position to the position PID controller.

[0009] The position PID controller obtains the actual position of the joint driving hydraulic rod from the data communication server, calculates the position error through the difference between the actual position and the target position, and transmits the position error to the position control agent. The position control agent outputs a set of PID parameter increments to the position PID controller according to the current position error, the position PID controller calculates the target speed for controlling the joint driving hydraulic rod according to the new PID parameters and the current position error, and sends the target speed data to the speed PID controller. The speed PID controller obtains the actual speed data of the joint hydraulic rod in the data communication server according to the received target speed, calculates the speed error, and transmits the speed error to the speed control agent. The speed control agent outputs a set of PID parameter increments to the speed PID controller according to the current speed error, the speed PID controller calculates the target valve opening of the electro-hydraulic proportional valve for controlling the joint driving hydraulic rod according to the new PID parameters and the current speed error, and sends it to the electro-hydraulic proportional valve through the TCP-CAN protocol conversion unit, realizing the control of the joint driving hydraulic rod.

[0010] More specifically, the data communication server includes a "management platform control section", an "autonomous driving control section", and a "vehicle state feedback section". The "management platform control section" is used to store control data from the remote management platform; the "autonomous driving control section" is used to store control instructions from the intelligent control module; and the "vehicle state feedback section" is used to store vehicle state and sensor data fed back by the vehicle controller.

[0011] More specifically, the job mode switching unit polls the "job mode register" in the "system control area" of the "management and control platform control section" at a frequency of 10 Hz, with "1" defining the remote driving mode and "2" defining the automatic job mode.

[0012] If the job mode is "1", it is determined that the job mode is the remote driving mode, and the data in the "management and control platform control section" is sent to the joint control mode switching unit as control data;

[0013] If the job mode is "2", it is determined that the job mode is the automatic job mode, and the data in the "automatic driving control section" is sent to the joint control mode switching unit as control data;

[0014] The joint control mode switching unit determines the control mode selected by the remote management and control platform or the automatic driving control unit according to the data of the "joint control mode register" in the "management and control platform section" or the "automatic driving control section" data output by the "job mode switching unit";

[0015] If the control mode is "speed mode", the data of the "target speed register" of each joint in the control data is taken as the reference input of the speed PID controller of each joint, and this reference input is sent to the speed controller;

[0016] If the control mode is "position control", the data of the "target position register" of each joint in the corresponding control section data is taken as the reference input of the position PID controller of each joint, and this reference input is sent to the position controller. The position PID controller calculates the expected speed of the controlled joint hydraulic rod, and then sends this expected speed as the reference input of the joint speed PID controller to the speed controller.

[0017] More specifically, the use method of the above-mentioned mobile engineering equipment job mechanism control system based on reinforcement learning PID controller is as follows:

[0018] A. Speed mode;

[0019] A1. Define the state space s of the speed control intelligent agent as the current speed error e v (t) of the joint drive hydraulic rod

[0020] e v (t)=S set -S actual (t) (1)

[0021] Where S set is the target speed, and S actual is the actual speed;

[0022] A2. Define the action space, define the action space as the increment of the three gain values of the PID controller, i.e. ΔK p , ΔK i , ΔK d ;

[0023] A3. Design the reward function:

[0024] R t = -ω1|e t | + ω2I(|e t | < ε) + ω3(-|I t |) + ω4(-|D t |)

[0025] + ω5(-(|ΔK p,t | + |ΔK i,t | + |ΔK d,t |)) (2)

[0026] Where e t is the control error at step t, for the speed controller it is the speed error e v (t), for the position controller it is the position error e p (t); I(|e t | < ε) is an indicator function, which is 1 when |e t | < ε, otherwise 0; ε represents the acceptable error range; I t is the integral term at step t; D t is the differential term at step t; ΔK p,t , ΔK i,t , ΔK d,t are the incremental adjustments to the PID gains at step t; ω1, ω2, ω3, ω4, ω5 are non-negative weight parameters, which are adjusted according to specific application requirements;

[0027] A4. Construct the reinforcement learning agent:

[0028] Initialize the action network, use the ReLU function as the activation function, at each time step, sample the specific action a t according to the Gaussian distribution output by the action network:

[0029] a t = μ t + σ t · ∈ (3)

[0030] Where a t is the sampled action; μ t and σ t are the mean and standard deviation of the action distribution output by the action network; ∈ is a random number sample drawn from the standard normal distribution;

[0031] Initialize the value network, the activation function uses the ReLU function, and the last layer outputs a scalar representing the value of the state V(s t );

[0032] Initialize the optimizer and hyperparameters: use the Adam optimizer and set the learning rate a, usually a is set to a small value to ensure the stability of the training process; set the number of steps collected before each update (n_steps), the discount factor γ, the maximum number of steps per episode, the batch size, and the truncation constant ε;

[0033] A5. Train the constructed agent:

[0034] A5.1 The remote control platform and the autonomous driving control unit send target speed and speed mode instructions, which are transmitted to the joint control mode switching unit through the data communication server and the job mode switching unit in turn; the joint control mode switching unit switches to the speed mode and transmits the target speed to the speed PID controller; at the same time, the speed PID controller obtains the actual speed of the joint driving hydraulic rod from the data communication server, and calculates the current speed error e v (t) to get the state s t of the speed control agent, input the state into the action network of the speed control agent, and sample a specific action a t according to the action distribution output by the action network of the speed control agent; the sampled action a t is the increment of the speed PID controller parameters, and is added to the speed PID controller parameters at the last time to get the new PID parameters; the speed PID controller calculates the control amount u v (t) according to formula (4), and u v (t) is the opening percentage of the electro-hydraulic proportional valve in this system;

[0035]

[0036] A5.2 Interactive training of the speed control agent;

[0037] A5.2.1 If the joint driving hydraulic rod is selected to train the speed control agent, the speed PID controller inputs u v (t) to the electro-hydraulic proportional valve through the TCP-CAN protocol conversion unit, and then controls the joint driving hydraulic rod; the joint driving hydraulic rod outputs new speed data, which is transmitted back to the speed PID controller through the stroke sensor, TCP-CAN protocol conversion unit and data communication server; the speed PID controller calculates the new state s t+1 in combination with the current given target control speed, and calculates the reward r t, the advantage function is calculated, the experience (s t , a t , r t , s t+1 ) is stored in the experience pool of the speed control agent, and one interaction between the speed control agent and the environment is completed;

[0038] A5.2.2 If the virtual debugging system is selected to train the speed control agent, the speed PID controller inputs u v (t) into the virtual debugging system; the virtual debugging system outputs new speed data, which is transmitted back to the speed PID controller by the data communication server; the speed PID controller calculates new state s t+1 in combination with the currently given target control speed, and calculates reward r t , the advantage function is calculated, the experience (s t , a t , r t , s t+1 ) is stored in the experience pool of the speed control agent, and one interaction between the speed control agent and the environment is completed;

[0039] A5.3 Repeat the interaction training of the selected speed control agent in A5.2 for n_steps times, and use generalized advantage estimation to calculate the advantage function

[0040]

[0041] δ t+l = r t+l + γV(s t+l+1 ) - V(s t+l ) (6)

[0042] Where t is the current time step. L is the offset from the current time step t. γ is the discount factor, indicating the importance of future rewards; λ is the smoothing parameter of GAE, controlling the time range of advantage estimation; δ t+l is the TD error of time step t+l, indicating the difference between the immediate reward and the value estimate; r t+l is the reward of time step t+l, calculated by equation (2);

[0043] At each time step, the generalized advantage estimate is calculated and stored in the experience pool of the speed control agent together with the experience, for guiding the update of the action network;

[0044] A5.4 The speed control agent updates the action network according to the objective function of the action network: the update objective of the action network is to maximize the expected cumulative reward while ensuring that the new policy does not deviate too far from the old policy, and the objective function of the action network is:

[0045]

[0046] wherein, is the policy ratio, and ε is the clipping ratio, is the mathematical expectation;

[0047] According to the above objective function, the action network is updated multiple times in small batches to maximize the objective function; as the speed control agent updates the action network parameters, the PID parameters generated by the action network can make the control performance of the speed PID controller tend to the optimal response speed and control error;

[0048] A5.5 The speed control agent updates the value network according to the loss function of the value network: the value network samples in small batches according to the data in the experience pool; the loss function L VF (θ v ) of the value network is calculated, and the gradient descent adjusts the parameters of the value network to reduce the estimation error of the value network for the reward value that can be obtained by the current policy of the action network; the loss function L VF (θ v ) of the value network is the mean square error, which is used to minimize the difference between the predicted value and the target value:

[0049]

[0050] wherein, V(s t ) is the value estimate output by the value network, is the generalized advantage estimate, and θ v is the parameter of the value network; V θv (s t ) is the predicted value output by the value network in state s t ; represents the expected value of the variance of all predicted values and target values in time step t; V t target is the target value, which is calculated using the generalized advantage estimate:

[0051]

[0052] V(s t ) is the value estimate output by the value network, is the generalized advantage estimate;

[0053] Finally, the value network estimates the expected cumulative reward that can be obtained by following the current policy of the action network starting from a given state s, calculates the advantage function, and calculates the objective function of the action network to update the action network;

[0054] A5.6 Repeating the training steps of A5.4-A5.5 for the speed control agent, constantly updating the action network and the value network until the average reward tends to be stable, or a predetermined maximum number of iterations is reached; after completing the update, stop training, and save the parameters of the action network and the value network, at this time the PID control parameters output by the action network can make the speed tracking performance of the speed PID controller achieve the best response speed and the smallest control error as possible;

[0055] A5.7 If A5.2.2 is used to train the speed control agent in A5.2, additionally perform A5.2.1 to train the speed control agent using the joint-driven hydraulic rod, and repeat the training process of A5.2-A5.6 to further adaptively train the speed control agent to optimize the performance loss caused by the mathematical model error between the virtual debugging system and the real system;

[0056] A6. Put the speed mode into practical application, the remote control platform and the automatic driving control unit send the target speed and the speed mode instruction, which is transmitted to the joint control mode switching unit through the data communication server and the operation mode switching unit in turn; the joint control mode switching unit switches to the speed mode, and transmits the target speed to the speed PID controller;

[0057] The action network of the speed control agent generates a set of PID parameter increments according to the current speed error, and adds the parameter increments to the original PID controller, and the speed PID controller calculates the target opening degree of the electro-hydraulic proportional valve according to the control error between the current target speed and the actual speed to control the joint-driven hydraulic rod; the action network outputs different PID parameter increments according to different input states, so that the speed controller adopts the best PID parameters to control the joint-driven hydraulic rod according to the current input state, so as to achieve the best speed tracking control effect;

[0058] B. Position mode;

[0059] B1. Define the state space s as the current speed error e p (t) of the joint-driven hydraulic rod p (t) = L set -L actual (t) (9) where L set is the target position, and L actual is the actual position;

[0060] Perform A2-A4 on the position PID controller to construct a position control agent and initialize the parameter settings; B2. Train the constructed position control agent:

[0061] B2.1 The position controller interacts with the controlled object to obtain experience: the remote control platform and the automatic driving control unit send target position and position mode instructions, which are transmitted to the joint control mode switching unit through the data communication server and the operation mode switching unit in turn; the joint control mode switching unit switches to the position mode and transmits the target position to the position PID controller; at the same time, the position PID controller obtains the actual position of the joint driving hydraulic rod from the data communication server, and calculates the current position error e p (t) as the state s of the position control agent t , the action distribution output by the action network of the position control agent is sampled to obtain a specific action a t ; the sampled action a t , that is, the parameter increment of the position PID controller, and the position PID controller parameters of the last moment are accumulated to obtain new position PID controller parameters, and the control amount u is calculated according to formula (4) p (t), u p (t) is input to the speed PID controller as the reference input of the speed controller, and the speed PID controller calculates the opening of the electro-hydraulic proportional valve;

[0062] B2.2.1 If the joint driving hydraulic rod is selected to train the position control agent, the speed PID controller inputs u v (t) to the electro-hydraulic proportional valve through the TCP-CAN protocol conversion unit, and then controls the joint driving hydraulic rod; the joint driving hydraulic rod outputs new position data, which is transmitted back to the position PID controller through the stroke sensor, the TCP-CAN protocol conversion unit and the data communication server; the speed PID controller calculates the new state s t+1 in combination with the currently given target position, and calculates the reward r t , the advantage function is calculated, and the experience (s t , a t , r t , s t+1 ) is stored in the experience pool of the position control agent, and the interaction between the position control agent and the environment is completed once;

[0063] B2.2.2 If the virtual debugging system is selected to train the position control agent, the speed PID controller inputs u v (t) to the virtual debugging system; the virtual debugging system outputs new position data, which is transmitted back to the position PID controller through the data communication server; the position PID controller calculates the new state s t+1 in combination with the currently given target position, and calculates the reward r t , the advantage function is calculated, and the experience (s t , at r t s t+1 ) stored in the experience pool of the position control agent, completing one interaction of the position control agent with the environment;

[0064] B2.3 After the position control agent interacts with the environment for n_steps times, the generalized advantage estimate is calculated according to formula (5) For each time step t, the generalized advantage estimate is calculated and stored together with the experience in the experience pool of the position controller, for guiding the update of the action network of the position control agent;

[0065] B2.4 The action network objective function of the position control agent is calculated according to formula (7), and the gradient ascent method is performed according to the objective function to update the parameters of the action network of the position control agent, so as to maximize its objective function;

[0066] B2.5 The loss function of the value network of the position control agent is calculated according to formula (8), and the gradient descent method is performed on the loss function of the value network of the position control agent to adjust the parameters of the value network of the position control agent, so as to reduce the estimation error of the value network of the position control agent for the reward value that can be achieved by the current policy of the action network in a given state, thereby more accurately evaluating the pros and cons of the current action network of the position control agent and further guiding the update of the action network of the position control agent;

[0067] B2.6 The training steps of B2.1 to B2.5 are performed on the position controller, and the action and value networks of the position control agent are continuously adjusted until the average reward tends to be stable or reaches a predetermined maximum number of iterations; after the update is completed, the PID control parameters output by the action network can achieve the best response speed and the smallest control error for the position tracking performance of the position PID controller; when the performance of the position control agent reaches the expected target, the training is stopped, and the parameters of the action network and the value network are saved;

[0068] B2.7 If the virtual debugging system is used to train the position control agent in B2.2.2 of B2.2, the speed control agent is additionally trained using the joint-driven hydraulic rod in B2.2.1, and the training process of B2.2 to B2.5 is repeated to further adaptively train the position control agent to optimize the loss of agent performance caused by the mathematical model error between the virtual debugging system and the real system;

[0069] B3. The position mode is put into practical application, the remote control platform and the automatic driving control unit send the target position and the position mode instruction, and the target position and the position mode instruction are transmitted to the joint control mode switching unit through the data communication server and the operation mode switching unit in turn; the joint control mode switching unit is switched to the position mode, and the target position is transmitted to the position PID controller;

[0070] The action network of the position control agent generates a set of PID parameter increments according to the error between the actual position and the target position, and the parameter increments are accumulated to the position PID controller; the position PID controller calculates the target speed of the joint driving hydraulic rod according to the control error between the target position and the actual position by using the new PID parameters; the action network of the position PID controller outputs different position PID parameter increments according to different input states, so that the position PID controller can adopt the best PID parameters to control the joint driving hydraulic rod according to the current input state.

[0071] The present application has the following advantages:

[0072] 1. By designing the data communication server, the operation mode switching unit, the joint control mode switching unit, the position controller and the speed controller, the operation mode switching control and the operation mode switching control of the operation mechanism of the unmanned mobile engineering equipment are realized, and the operation personnel can quickly and conveniently switch the operation mode and the operation mode according to the actual operation condition.

[0073] 2. The joint hydraulic servo control method based on the PID controller of the reinforcement learning is proposed, the speed control agent and the position control agent are designed and trained, so that the speed controller and the position controller can adaptively adjust the PID controller parameters under different control error states, and the best control effect of the controller can be achieved without the participation of personnel, and the adaptability of the hydraulic servo system based on the PID controller to the complex environmental changes is improved. DETAILED DESCRIPTION

[0074] Figure 1 is the structure diagram of the control system of the mobile engineering equipment operation mechanism based on the PID controller of the reinforcement learning of the present application.

[0075] Figure 2 is the register partitioning schematic diagram of the data communication server of the present application.

[0076] Figure 3 is the operation mode and control mode switching control flow chart of the present application.

[0077] Figure 4 is the learning principle diagram of the speed control agent and the position control agent of the present application. DETAILED DESCRIPTION

[0078] The application is further described below in conjunction with the drawings.

[0079] In this embodiment, Komatsu D65E pusher machine is adopted, the working mechanism of the pusher machine has two degrees of freedom, respectively driven by hydraulic rods, the opening of the hydraulic circuit is controlled through the electro-hydraulic proportional valve, and the control is carried out through the CAN bus. A high-performance on-board computer is equipped at the end of the vehicle, communicates with the remote management and control platform and the automatic driving control unit through Ethernet, the on-board computer controls the electro-hydraulic proportional valve through the TCP-CAN protocol converter, drives the hydraulic rod on which a stroke sensor is installed to measure the current length of the hydraulic rod, and feeds back data to the on-board computer at a frequency of 100 Hz through the TCP-CAN protocol converter.

[0080] As shown in Figure 1 The control system involved in the mobile engineering equipment working mechanism control method based on the PID controller of the reinforcement learning proposed by the application is composed of a remote management and control platform, an automatic driving control unit, an on-board computer, a virtual debugging system, a TCP-CAN protocol conversion unit, an electro-hydraulic proportional valve, a joint driving hydraulic rod and a stroke sensor.

[0081] The remote management and control platform and the automatic driving control unit send the control instructions of the working mechanism to the on-board computer through Ethernet, including the working mode, the control mode and the joint target speed or the joint target position of the working mechanism. The on-board computer is a computing carrier for realizing the control of the working mechanism of the mobile engineering equipment, receives the control instructions from the remote management and control platform and the automatic driving unit, acquires the data of the stroke sensor, calculates the opening of the electro-hydraulic proportional valve of the joint driving hydraulic rod according to the current state and the instructions, and sends the opening data of the electro-hydraulic proportional valve to the virtual debugging system or the TCP-CAN protocol conversion unit. The virtual debugging system is used to replace the real controlled object, receives the opening instructions of the electro-hydraulic proportional valve from the on-board computer, outputs the current position of the joint driving hydraulic rod through the simulation calculation in the internal thereof, and is used to train the PID controller based on the reinforcement learning. The TCP-CAN protocol conversion unit is used to convert the TCP / IP communication protocol used by the on-board computer and the CAN communication protocol used by the stroke sensor and the electro-hydraulic proportional valve, and forwards the data. The electro-hydraulic proportional valve is used for the opening control of the hydraulic circuit of the joint driving hydraulic rod, and adjusts the valve opening according to the received target opening value of the valve. The joint driving hydraulic rod is used to drive the mobile engineering equipment working mechanism to act. The stroke sensor is installed in parallel with the joint driving hydraulic rod, and is used to measure the actual length of the joint driving hydraulic rod in real time, and sends the measured length data of the joint hydraulic rod to the on-board computer through the TCP-CAN protocol conversion unit.

[0082] The vehicle-mounted computer comprises a data communication server, a working mode switching unit, a joint control mode switching unit, and a position controller and a speed controller based on reinforcement learning.

[0083] The data communication server receives working mechanism working mode data, control mode data, and joint target speed or joint target position instruction data from a remote management and control platform and an automatic driving control unit, and receives travel sensor data forwarded by a TCP-CAN protocol conversion unit; the working mode switching unit selects control instructions from the data communication server according to the working mode and sends them to the joint control mode switching unit; the joint control mode switching unit determines the selected joint control mode, and if it is a speed control mode, sends target speed data to the speed controller. If it is a position mode, sends target position to the position controller; the position controller obtains the actual position of the joint driving hydraulic rod from the data communication server, calculates the target speed of the joint driving hydraulic rod in combination with the target position data, and sends the target speed data to the speed controller; the speed controller obtains the actual speed data of the joint hydraulic rod in the data communication server according to the received target speed, calculates the target valve opening of the electro-hydraulic proportional valve for controlling the joint driving hydraulic rod, and sends it to the electro-hydraulic proportional valve through the TCP-CAN protocol conversion unit, thereby realizing the control of the joint driving hydraulic rod.

[0084] The construction steps of the data communication server are as follows: first, a memory is opened in the vehicle-mounted computer as a data register, and the register is divided into a “management and control platform control section”, an “automatic driving control section”, and a “vehicle state feedback section” according to the source of the control instructions. The “management and control platform control section” is used to store control data from the remote management and control platform; the “automatic driving control section” is used to store control instructions from the intelligent control module; and the “vehicle state feedback section” is used to store vehicle state and sensor data fed back by the vehicle controller. The data communication server communicates with the remote management and control platform, the automatic driving control unit, and the TCP-CAN protocol conversion unit through TCP / IP protocol respectively.

[0085] Further, the “management and control platform control section” is divided into a “motion control area” and a “system control area”. The “target position register” and the “target speed register” of each joint of the working mechanism are allocated and defined in the “motion control area”; and the “working mode register” and the “joint control mode register” are allocated and defined in the “system control area”.

[0086] The “automatic driving control section” is divided into a “motion control area” and a “control mode area”. The “target position register” and the “target speed register” of each joint of the working mechanism are allocated and defined in the “motion control area”; and the “joint control mode register” is allocated and defined in the “control mode area”.

[0087] The job mode switching unit reads the job mode data in the data communication server in real time, and according to the job mode, selects the control mode and motion control data from the "platform driving section" or the "automatic driving control section" to send to the joint control mode switching unit. The specific judgment process is as follows:

[0088] The job mode switching unit polls the "job mode register" in the "system control area" of the "platform control section" at a frequency of 10 Hz, defines "1" as the remote driving mode, and "2" as the automatic job mode.

[0089] If the job mode is "1", it is judged that the job mode is the remote driving mode, and the data in the "platform control section" is sent to the joint control mode switching unit as control data;

[0090] If the job mode is "2", it is judged that the job mode is the automatic job mode, and the data in the "automatic driving control section" is sent to the joint control mode switching unit as control data.

[0091] The joint control mode switching unit judges the control mode selected by the remote platform control or the automatic driving control unit according to the data of the "joint control mode register" in the "platform section" or the "automatic driving control section" data output by the "job mode switching unit".

[0092] If the control mode is "speed mode", the "target speed register" data of each joint in the control data is taken as the reference input of the speed PID controller of each joint, and the reference input is sent to the speed controller.

[0093] If the control mode is "position control", the data of the "target position register" of each joint in the corresponding control section data is taken as the reference input of the position PID controller of each joint, and the reference input is sent to the position controller. The position PID controller calculates the expected speed of the controlled joint hydraulic rod, and then sends the expected speed as the reference input of the joint speed PID controller to the speed controller.

[0094] The actual position of the driving hydraulic rod is obtained from the stroke sensor through the TCP-CAN protocol conversion unit, the actual speed of the driving hydraulic rod is obtained by differentiating the actual position, and is stored in the "vehicle state feedback section" to provide state feedback to the PID controller.

[0095] The use method of the above Komatsu D65E rake machine control system is as follows:

[0096] A. Speed mode;

[0097] A1. Define the state space s of the speed control intelligent agent as the current speed error e of the joint driving hydraulic rodv (t),

[0098] e v (t) = S set - S actual (t) (1)

[0099] where S set is the target speed, S actual is the actual speed;

[0100] A2. Define the action space, which is defined as the increments of the three gain values of the PID controller, i.e., ΔK p , ΔK i , ΔK d ;

[0101] A3. Design the reward function:

[0102] R t = -ω1|e t | + ω2I(|e t | < ε) + ω3(-|I t |) + ω4(-|D t |)

[0103] + ω5(-(|ΔK p,t | + |ΔK i,t | + |ΔK d,t |)) (2

[0104] where e t is the control error at step t, which is the speed error e v (t) for the speed controller and the position error e p (t) for the position controller; I(|e t | < ε) is an indicator function, which is 1 when |e t | < ε and 0 otherwise; ε represents the acceptable error range; I t is the integral term at step t; D t is the derivative term at step t; ΔK p,t , ΔK i,t , ΔK d,t are the incremental adjustments to the PID gains at step t, respectively; ω1, ω2, ω3, ω4, ω5 are non-negative weight parameters, which are adjusted according to specific application requirements;

[0105] A4. Construct the reinforcement learning agent:

[0106] Initialize the action network, using the ReLU function as the activation function. At each time step, sample a specific action a t from the Gaussian distribution output by the action network:

[0107] a t = μ t + σ t · ∈ (3)

[0108] where a t is the sampled action; μ t and σ t are the mean and standard deviation of the action distribution output by the action network; ∈ is a random number sample drawn from the standard normal distribution;

[0109] Initialize the value network, and use the ReLU function as the activation function. The last layer outputs a scalar, which represents the value of the state V(s t );

[0110] Initialize the optimizer and hyperparameters: use the Adam optimizer, and set the learning rate α. Usually, α is set to a small value to ensure the stability of the training process; set the number of steps collected before each update (n_steps), the discount factor γ, the maximum number of steps per episode, the batch size, and the truncation constant ε;

[0111] A5. Train the constructed agent:

[0112] A5.1 The remote control platform and the autonomous driving control unit send the target speed and speed mode instructions, which are transmitted to the joint control mode switching unit through the data communication server and the operation mode switching unit in turn; the joint control mode switching unit switches to the speed mode, and transmits the target speed to the speed PID controller; at the same time, the speed PID controller obtains the actual speed of the joint driving hydraulic rod from the data communication server, and calculates the current speed error e v (t) to obtain the state s t of the speed control agent, inputs the state into the action network of the speed control agent, samples a specific action a t according to the action distribution output by the action network of the speed control agent; the sampled action a t is the increment of the speed PID controller parameters, and is added to the speed PID controller parameters at the last time to obtain new PID parameters; the speed PID controller calculates the control amount u v (t) according to formula (4), and u v (t) is the opening percentage of the electro-hydraulic proportional valve in the system;

[0113]

[0114] A5.2 Interactive training of the speed control agent;

[0115] A5.2.1 If the joint-driven hydraulic rod training speed control agent is selected, the speed PID controller will input u v (t) to the electro-hydraulic proportional valve through the TCP-CAN protocol conversion unit, thereby controlling the joint-driven hydraulic rod; the joint-driven hydraulic rod outputs new speed data, and returns the speed PID controller through the stroke sensor, TCP-CAN protocol conversion unit and data communication server; the speed PID controller calculates the new state s t+1 in combination with the currently given target control speed, and calculates the reward r t , calculates the advantage function, and stores the experience (s t , a t , r t , s t+1 ) in the experience pool of the speed control agent, completing an interaction between the speed control agent and the environment;

[0116] A5.2.2 If the virtual debugging system is selected to train the speed control agent, the speed PID controller will input u v (t) to the virtual debugging system; the virtual debugging system outputs new speed data, and the data communication server returns the speed PID controller; the speed PID controller calculates the new state s t+1 in combination with the currently given target control speed, and calculates the reward r t , calculates the advantage function, and stores the experience (s t , a t , r t , s t+1 ) in the experience pool of the speed control agent, completing an interaction between the speed control agent and the environment;

[0117] A5.3 Repeat the interaction training of the selected speed control agent in A5.2 n_steps times, and use generalized advantage estimation to calculate the advantage function

[0118]

[0119] δ t+l = r t+l + γV(s t+l+1 ) - V(s t+l ) (6)

[0120] Where t is the current time step. l is the offset from the current time step t. γ is the discount factor, indicating the importance of future rewards; λ is the smoothing parameter of GAE, controlling the time range of advantage estimation; δ t+l is the TD error of time step t+l, indicating the difference between immediate reward and value estimation; r t+lis the reward at time step t+1, which is calculated by equation (2);

[0121] At each time step, the generalized advantage estimate is calculated and stored in the experience pool of the speed control agent together with the experience for guiding the update of the action network;

[0122] A5.4 The speed control agent updates the action network according to the objective function of the action network: the update target of the action network is to maximize the expected cumulative reward while ensuring that the new policy does not deviate too far from the old policy, and the objective function of the action network is:

[0123]

[0124] wherein, is the policy ratio, and ε is the truncation ratio, is the mathematical expectation;

[0125] Perform multiple small-batch updates on the action network according to the above objective function to maximize the objective function; as the action network parameters of the speed control agent are updated, the PID parameters generated by the action network can make the control performance of the speed PID controller tend to the optimal response speed and control error;

[0126] A5.5 The speed control agent updates the value network according to the loss function of the value network: the value network performs small-batch sampling according to the data in the experience pool; the loss function L VF (θ v ) of the value network is calculated, and the gradient descent adjusts the parameters of the value network to reduce the estimation error of the value network on the reward value that can be achieved by the current policy of the action network; the loss function L VF (θ v ) of the value network is the mean square error, which is used to minimize the difference between the predicted value and the target value:

[0127]

[0128] wherein, V(s t ) is the value estimate output by the value network, is the generalized advantage estimate, θ v is the parameter of the value network; V θv (s t ) is the predicted value output by the value network at state s t ; represents the expected value of the variance of all predicted values and target values in time step t; V t target is the target value, which is calculated using the generalized advantage estimate:

[0129]

[0130] V(s t ) is the value estimate output by the value network, is the generalized advantage estimate;

[0131] Finally, the value network estimates the expected cumulative reward that can be obtained by following the policy of the current action network starting from a given state s, and computes the advantage function, which is used to compute the objective function of the action network to update the action network;

[0132] A5.6 repeats the training steps of A5.4-A5.5 for the speed control agent, constantly updating the action network and the value network, until the average reward tends to be stable, or a predetermined maximum number of iterations is reached; after completing the update, the training is stopped, and the parameters of the action network and the value network are saved, at this time the PID control parameters output by the action network can make the speed tracking performance of the speed PID controller achieve the best response speed and the smallest control error as possible;

[0133] A5.7 If A5.2.2 is used to train the speed control agent using the virtual debugging system in A5.2.2, additionally perform A5.2.1 to train the speed control agent using the joint driven hydraulic rod, and repeat the training process of A5.2-A5.6 to further adaptively train the speed control agent to optimize the performance loss caused by the mathematical model error between the virtual debugging system and the real system;

[0134] A6. Put the speed mode into practical application, the remote control platform and the automatic driving control unit send the target speed and the speed mode instruction, which are transmitted to the joint control mode switching unit through the data communication server and the operation mode switching unit in turn; the joint control mode switching unit switches to the speed mode, and transmits the target speed to the speed PID controller;

[0135] The action network of the speed control agent generates a set of PID parameter increments according to the current speed error, and adds the parameter increments to the original PID controller; the speed PID controller calculates the target opening degree of the electro-hydraulic proportional valve according to the control error between the current target speed and the actual speed, and controls the joint driven hydraulic rod; the action network outputs different PID parameter increments according to different input states, so that the speed controller adopts the best PID parameters to control the joint driven hydraulic rod according to the current input state, so as to achieve the best speed tracking control effect;

[0136] B. Position mode;

[0137] B1. Define the state space s as the current speed error e p (t) of the joint driven hydraulic rod, p (t) = Lset -L actual (t) (9) wherein L set is the target position, L actual is the actual position;

[0138] A2-A4 are performed on the position PID controller, a position control agent is constructed, and parameter settings are initialized; B2. The constructed position control agent is trained:

[0139] B2.1 The position controller interacts with the controlled object to obtain experience: the remote management and control platform and the automatic driving control unit send target position and position mode instructions, which are transmitted to the joint control mode switching unit in turn through the data communication server and the operation mode switching unit; the joint control mode switching unit switches to the position mode and transmits the target position to the position PID controller; at the same time, the position PID controller obtains the actual position of the joint driving hydraulic rod from the data communication server, and calculates the current position error e p (t) as the state s t of the position control agent, inputs the state into the action network of the position control agent, samples a specific action a t according to the action distribution output by the action network of the position control agent; t , which is the parameter increment of the position PID controller, and is accumulated with the position PID controller parameter at the last time to obtain new position PID controller parameters, and the control amount u p (t) is calculated according to formula (4), u p (t) is taken as the reference input of the speed controller and input to the speed PID controller, and the speed PID controller calculates the opening of the electro-hydraulic proportional valve;

[0140] B2.2.1 If the joint driving hydraulic rod is selected to train the speed position control agent, the speed PID controller inputs u v (t) to the electro-hydraulic proportional valve through the TCP-CAN protocol conversion unit, and then controls the joint driving hydraulic rod; the joint driving hydraulic rod outputs new position data, which is transmitted back to the position PID controller through the stroke sensor, TCP-CAN protocol conversion unit and data communication server; the speed PID controller calculates the new state s t+1 in combination with the currently given target position, and calculates the reward r t , the advantage function is calculated, the experience (s t , a t , r t , s t+1 ) is stored in the experience pool of the position control agent, and one interaction between the position control agent and the environment is completed;

[0141] B2.2.2 If the virtual commissioning system is selected to train the position control agent, the velocity PID controller will output u v (t) Input the virtual commissioning system; the virtual commissioning system outputs new position data, which is transmitted back to the position PID controller through the data communication server; the position PID controller calculates a new state s t+1 according to the current given target position t , and calculates the reward r t , and stores the experience (s t , a t , r t+1 ) in the experience pool of the position control agent, completing an interaction between the position control agent and the environment;

[0142] B2.3 After the position control agent interacts with the environment n_steps times, the generalized advantage estimate is calculated according to formula (5) For each time step t, the generalized advantage estimate is calculated and is stored together with the experience in the experience pool of the position controller, which is used to guide the update of the action network of the position control agent;

[0143] B2.4 The action network objective function of the position control agent is calculated according to formula (7), and the gradient ascent method is performed according to the objective function to update the parameters of the action network of the position control agent, so that its objective function tends to be maximized;

[0144] B2.5 The loss function of the value network of the position control agent is calculated according to formula (8), and the gradient descent method is performed on the loss function of the value network of the position control agent to adjust the parameters of the value network of the position control agent, so as to reduce the estimation error of the value network of the position control agent for the reward value that can be achieved by the current policy of the action network in a given state, thereby more accurately evaluating the pros and cons of the current action network of the position control agent and further guiding the update of the action network of the position control agent;

[0145] B2.6 The training steps of B2.1 to B2.5 are performed on the position controller, and the action and value networks of the position control agent are continuously adjusted until the average reward tends to be stable or the predetermined maximum iteration number is reached; after the update is completed, the PID control parameters output by the action network can achieve the best response speed and the smallest control error in the position tracking performance of the position PID controller; when the performance of the position control agent reaches the expected target, the training is stopped, and the parameters of the action network and the value network are saved;

[0146] B2.7 If B2.2.2 is performed in B2.2, the position control agent is trained using a virtual debugging system, then additionally perform B2.2.1 to train the speed control agent using the joint driven hydraulic rod, and repeat the training process of B2.2-B2.5 to further adaptively train the position control agent to optimize the loss of agent performance caused by the mathematical model error between the virtual debugging system and the real system;

[0147] B3. The position mode is put into practical application, the remote management and control platform and the automatic driving control unit send the target position and the position mode instruction, which is transmitted to the joint control mode switching unit through the data communication server and the operation mode switching unit in turn; the joint control mode switching unit switches to the position mode and transmits the target position to the position PID controller;

[0148] The action network of the position control agent generates a set of PID parameter increments according to the error between the actual position and the target position, and adds the parameter increments to the position PID controller; the position PID controller calculates the target speed of the joint driven hydraulic rod according to the control error between the target position and the actual position using the new PID parameters, and the target speed is tracked and controlled by the speed PID controller; the action network of the position PID controller outputs different position PID parameter increments according to different input states, so that the position PID controller adopts the optimal PID parameters to control the joint driven hydraulic rod according to the current input state.

[0149] It should be emphasized that the embodiments described in the present application are illustrative rather than limiting, and therefore the present application includes but is not limited to the embodiments described in the specific embodiments, and any other embodiments derived by those skilled in the art according to the technical solutions of the present application also belong to the protection scope of the present application.

Claims

1. A control system for a mobile engineering equipment work implement based on a reinforcement learning PID controller, characterized by: The remote control platform, the automatic driving control unit, the vehicle-mounted computer, the virtual debugging system, the TCP-CAN protocol conversion unit, the electro-hydraulic proportional valve, the joint driving hydraulic rod and the stroke sensor are connected to form a system; the stroke sensor receives the actual position and actual speed of the joint driving hydraulic rod and transmits them to the TCP-CAN protocol conversion unit; The remote control platform and the automatic driving control unit send the control instructions of the working mechanism to the vehicle-mounted computer through Ethernet, and the content of the control instructions includes the working mode, the control mode and the joint target speed or joint target position of the working mechanism; The vehicle-mounted computer includes a data communication server, which receives the control instructions from the remote control platform and the automatic driving control unit and transmits the control instructions to the working mode switching unit; at the same time, the data communication server receives the actual position and actual speed information from the TCP-CAN protocol conversion unit, transmits the actual position to the position PID controller in the position controller and transmits the actual speed to the speed PID controller in the speed controller; The working mode switching unit selects the control instructions from the data communication server according to the working mode and sends them to the joint control mode switching unit; If the selected joint control mode is the speed mode, the joint control mode switching unit sends the target speed data to the speed PID controller; If the selected joint control mode is the position mode, the joint control mode switching unit sends the target position to the position PID controller; The position PID controller obtains the actual position of the joint driving hydraulic rod from the data communication server, calculates the position error through the difference between the actual position and the target position, and transmits the position error to the position control agent; the position control agent outputs a set of PID parameter increments to the position PID controller according to the current position error, the position PID controller calculates the target speed for controlling the joint driving hydraulic rod according to the new PID parameters and the current position error, and sends the target speed data to the speed PID controller; the speed PID controller obtains the actual speed data of the joint hydraulic rod in the data communication server according to the received target speed, calculates the speed error, and transmits the speed error to the speed control agent; the speed control agent outputs a set of PID parameter increments to the speed PID controller according to the current speed error, the speed PID controller calculates the target valve opening of the electro-hydraulic proportional valve for controlling the joint driving hydraulic rod according to the new PID parameters and the current speed error, and sends the target valve opening to the electro-hydraulic proportional valve through the TCP-CAN protocol conversion unit, thereby realizing the control of the joint driving hydraulic rod; The working steps of the speed mode are as follows: A1. Define the state space s of the velocity control agent as the current velocity error of the joint driven hydraulic rod , wherein is the target speed, is the actual speed; A2. Define the action space, define the action space as the increment of the three gain values of the PID controller, that is, ΔK p , ΔK i , ΔK d ; A3. Design the reward function: wherein, is the control error at step t, for a speed controller the speed error , for a position controller the position error ; is an indicator function, which is 1 when , and 0 otherwise; denotes the acceptable error range; is the integral term at step t; D t is the derivative term at step t; ΔK p,t , ΔK i,t , ΔK d,t are the incremental adjustments to the PID gains at step t, respectively; ω1, ω2, ω3, ω4, ω5 are non-negative weight parameters; A4. Construct the reinforcement learning agent: Initialize the action network with ReLU activation function, at each time step, sample a specific action according to the Gaussian distribution output by the action network : where, is the action sampled from the action distribution; and are the mean and standard deviation of the action distribution output by the action network, respectively; is a random sample drawn from a standard normal distribution. Initialize the value network, the activation function uses the ReLU function, and the last layer outputs a scalar representing the value of the state ; Initialize the optimizer and hyperparameters: Use the Adam optimizer and set the learning rate ; Set the number of steps to collect before each update, the discount factor γ, the maximum number of steps per episode, the batch size, and the truncation constant ε; A5. Train the constructed agent; A6. Put the speed mode into practical application.

2. The control system for mobile engineering equipment work implement based on reinforcement learning PID controller of claim 1, wherein: The data communication server comprises a "management and control platform control section", an "automatic driving control section", and a "vehicle state feedback section". The "management and control platform control section" is used to store control data from a remote management and control platform. The "automatic driving control section" is used to store control instructions from an intelligent control module. The "vehicle state feedback section" is used to store vehicle state and sensor data fed back by a vehicle controller.

3. The control system for a mobile work equipment implement based on a reinforcement learning PID controller of claim 2, wherein: The job mode switching unit polls the "job mode register" of the "system control area" in the "management and control platform control section" at a frequency of 10 Hz, and defines "1" as a remote driving mode and "2" as an automatic job mode. If the job mode is "1", it is determined that the job mode is a remote driving mode, and the data in the "management and control platform control section" is sent to the joint control mode switching unit as control data. If the job mode is "2", it is determined that the job mode is an automatic job mode, and the data in the "automatic driving control section" is sent to the joint control mode switching unit as control data. The joint control mode switching unit determines the control mode selected by the remote management and control platform or the automatic driving control unit according to the data of the "joint control mode register" in the "management and control platform section" or the "automatic driving control section" data output by the "job mode switching unit". If the control mode is "speed mode", the data of the "target speed register" of each joint in the control data is taken as the reference input of the speed PID controller of each joint, and the reference input is sent to the speed controller. If the control mode is "position control", the data of the "target position register" of each joint in the corresponding control section data is taken as the reference input of the position PID controller of each joint, and the reference input is sent to the position controller. The position PID controller calculates the expected speed of the controlled joint hydraulic rod, and then sends the expected speed as the reference input of the joint speed PID controller to the speed controller.

4. The use method of the mobile engineering equipment job mechanism control system based on the reinforcement learning PID controller according to claim 1, comprising the following steps: The specific process of A5 is as follows: A5.1 The remote management and control platform and the automatic driving control unit send target speed and speed mode instructions, which are transmitted to the joint control mode switching unit in turn through the data communication server and the operation mode switching unit; the joint control mode switching unit switches to the speed mode and transmits the target speed to the speed PID controller; at the same time, the speed PID controller obtains the actual speed of the joint driving hydraulic rod from the data communication server, and calculates the current speed error in combination with the given target control speed obtaining the state of the speed control agent , inputting the state into the action network of the speed control agent, sampling a specific action according to an action distribution output by the action network of the speed control agent ; the sampled action , that is, the speed PID controller parameter increment, and the new PID parameter is obtained by accumulating the speed PID controller parameter at the last moment; the speed PID controller calculates the control amount according to formula (4) , In the system, it is the opening percentage of the electro-hydraulic proportional valve A5.2 Speed control agent interactive training; A5.2.1 If the joint driving hydraulic rod training speed control agent is selected, the speed PID controller will The joint driving hydraulic rod is controlled by inputting the data to the electro-hydraulic proportional valve through the TCP-CAN protocol conversion unit; the joint driving hydraulic rod outputs new speed data, which is transmitted back to the speed PID controller through the stroke sensor, TCP-CAN protocol conversion unit and data communication server; the speed PID controller calculates a new state according to the current given target control speed t , calculates the advantage function, and stores the experience (s t ,a t ,r t ,s t+1 ) in the experience pool of the speed control agent, completing an interaction between the speed control agent and the environment; A5.2.2 If the virtual debugging system is selected to train the speed control agent, the speed PID controller will Input the virtual debugging system; the virtual debugging system outputs new speed data, and the data communication server returns the speed PID controller; the speed PID controller calculates a new state in combination with the currently given target control speed , and calculates the reward r t , calculates the advantage function, and stores the experience (s t , a t , r t , s t+1 ) in the experience pool of the speed control agent, completing one interaction between the speed control agent and the environment; A5.3 Repeat the selected speed-controlled agent interaction training in A5.2 n_steps times, using generalized advantage estimation to compute the advantage function where, t is the current time step; l is the offset from the current time step t ; is the discount factor, representing the importance of future rewards; is the smoothing parameter for GAE, controlling the time horizon of advantage estimates; is the TD error for time step t + l , representing the difference between the immediate reward and the value estimate; is the reward for time step t + l , computed by equation (2); At each time step, a generalized advantage estimate is computed and stored with the experience in the experience pool of the speed control agent for guiding the update of the action network; A5.4 The speed control agent updates the action network according to the target function of the action network: the update target of the action network is to maximize the expected cumulative reward while ensuring that the new strategy does not deviate too far from the old strategy, and the target function of the action network is: wherein, is the strategy ratio, is the cutoff ratio, is the mathematical expectation; According to the above target function, perform gradient ascent on the action network for multiple small batch updates to maximize the target function; as the action network parameters of the speed control agent are updated, the PID parameters generated by the action network can make the control performance of the speed PID controller tend to the optimal response speed and control error; A5.5 The speed control agent updates the value network according to the loss function of the value network: the value network is sampled in small batches according to the data in the experience pool; the loss function of the value network is calculated The gradient descent adjusts the parameters of the value network to reduce the estimation error of the value network on the reward value that can be achieved by the action network taking the current policy; the loss function of the value network is the mean square error, which is used to minimize the difference between the predicted value and the target value: where, is the value estimate output by the value network, is the generalized advantage estimate, is a parameter of the value network; is the predicted value output by the value network in state at time step t; denotes the expected value of the variance of all predicted values and the target value in time step t; is the target value, calculated using the generalized advantage estimate: value estimate for the value network output, generalized advantage estimate; Finally, the value network estimates the expected cumulative reward that can be obtained by following the current action network strategy from a given state s, calculates the advantage function, and uses it to calculate the target function of the action network to update the action network. A5.6 repeats the training steps of A5.4-A5.5 for the speed control agent, constantly updating the action network and the value network until the average reward tends to be stable, or a predetermined maximum number of iterations is reached; after the update is completed, the training is stopped, and the parameters of the action network and the value network are saved, at this time the PID control parameters output by the action network can make the speed tracking performance of the speed PID controller achieve the best response speed and the smallest control error; A5.7 if A5.2.2 is executed in A5.2 to train the speed control agent using the virtual debugging system, additionally execute A5.2.1 to train the speed control agent using the joint driven hydraulic rod, and repeat the training process of A5.2-A5.6 to further adaptively train the speed control agent to optimize the performance loss caused by the mathematical model error between the virtual debugging system and the real system; The specific steps of A6 are as follows: The remote management and control platform and the automatic driving control unit send the target speed and the speed mode instruction, which are transmitted to the joint control mode switching unit through the data communication server and the operation mode switching unit in turn; the joint control mode switching unit switches to the speed mode and transmits the target speed to the speed PID controller; The action network of the speed control agent generates a set of PID parameter increments according to the current speed error, and adds the parameter increments to the original PID controller; the speed PID controller calculates the target opening degree of the electro-hydraulic proportional valve according to the control error between the current target speed and the actual speed, and controls the joint driven hydraulic rod; the action network outputs different PID parameter increments according to different input states, so that the speed controller adopts the best PID parameters to control the joint driven hydraulic rod according to the current input state, so as to achieve the best speed tracking control effect; The working steps of the position mode are as follows: B1. Define a state space s as the velocity error of the joint driven hydraulic rod current , wherein is the target position, is the actual position; A2-A4 are executed for the position PID controller to construct the position control agent and initialize the parameter setting; B2. Train the constructed position control agent: B2.1 The position controller interacts with the controlled object to gain experience: The remote control platform and the automatic driving control unit send target position and position mode commands, which are transmitted sequentially through the data communication server and the operation mode switching unit to the joint control mode switching unit; the joint control mode switching unit switches to position mode and transmits the target position to the position PID controller; simultaneously, the position PID controller obtains the actual position of the joint drive hydraulic rod from the data communication server and calculates the current position error based on the given target position. The state of the position control agent The state is input into the action network of the position control agent. Based on the action distribution output by the action network, a specific action is sampled. ; sampling action That is, the parameter increment of the position PID controller is added to the position PID controller parameter of the previous time step to obtain the new position PID controller parameter, and the control quantity is calculated according to equation (4). ,Will As a reference input for the speed controller, it is input to the speed PID controller, which calculates the opening degree of the electro-hydraulic proportional valve. B2.2.1 If the joint driving hydraulic rod training speed position control agent is selected to be used, the speed PID controller will The joint driving hydraulic rod is controlled by inputting to the electro-hydraulic proportional valve through the TCP-CAN protocol conversion unit; the joint driving hydraulic rod outputs new position data and returns the position PID controller through the stroke sensor, TCP-CAN protocol conversion unit and data communication server; the speed PID controller calculates a new state in combination with the currently given target position t , calculates the advantage function, and stores the experience (s t , a t , r t , s t+1 ) in the experience pool of the position control agent, completing an interaction between the position control agent and the environment; B2.2.2 If the virtual debugging system is selected to train the position control agent, the speed PID controller will input the virtual debugging system; the virtual debugging system outputs new position data, which is transmitted back to the position PID controller through the data communication server; the position PID controller calculates a new state according to the current given target position t , calculates the advantage function, and stores the experience (s t , a t , r t , s t+1 ) in the experience pool of the position control agent, completing an interaction between the position control agent and the environment; B2.3 After the position control agent interacts with the environment for n_steps times, the generalized advantage estimate is calculated according to equation (5) ; for each time step t, the generalized advantage estimate is calculated and stored in the experience pool of the position controller together with the experience for guiding the update of the action network of the position control agent; B2.4 calculates the action network target function of the position control agent according to formula (7), and updates the parameters of the position control agent action network according to the gradient ascent method to maximize its target function; B2.5 calculates the loss function of the value network of the position control agent according to formula (8), and adjusts the parameters of the value network of the position control agent to reduce the estimation error of the reward value that the value network can achieve under the given state, so as to more accurately evaluate the current position control agent action network and further guide the update of the position control agent action network; B2.6 The position control agent continuously adjusts the action and value networks until the average reward tends to be stable or reaches a predetermined maximum number of iterations, and the PID control parameters output by the action network can achieve the best response speed and the smallest control error for the position tracking performance of the position PID controller; after the update is completed, the training is stopped when the performance of the position control agent reaches the expected target, and the parameters of the action network and the value network are saved; B2.7 If the virtual debugging system is used to train the position control agent in B2.2.2, additionally perform B2.2.1 to train the speed control agent using the joint-driven hydraulic rod, and repeat the training process of B2.2-B2.5 to further adaptively train the position control agent to optimize the loss of agent performance caused by the mathematical model error between the virtual debugging system and the real system; B3. The position mode is put into practical application, and the remote management and control platform and the automatic driving control unit send the target position and the position mode instruction, which are transmitted to the joint control mode switching unit through the data communication server and the operation mode switching unit in turn; the joint control mode switching unit switches to the position mode and transmits the target position to the position PID controller; The action network of the position control agent generates a set of PID parameter increments according to the error between the actual position and the target position, and adds the parameter increments to the position PID controller; the position PID controller calculates the target speed of the joint-driven hydraulic rod according to the control error between the target position and the actual position using the new PID parameters, and the target speed is tracked and controlled by the speed PID controller; the action network of the position PID controller outputs different position PID parameter increments according to different input states, so that the position PID controller can adopt the best PID parameters to control the joint-driven hydraulic rod according to the current input state.

Images