Adaptive single-joint servo PID control method and system based on deep reinforcement learning
By constructing an adaptive SCA_PID hierarchical controller and combining the SAC and PID algorithms, the PID controller parameters are dynamically adjusted, solving the problems of poor adaptability and insufficient control accuracy of traditional PID control methods in complex systems and multiple environments. This achieves high-precision and robust adaptive single-joint servo control.
Patent Information
- Application Number
- CN202411231866.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-04
AI Technical Summary
In the existing technology, traditional PID control methods are difficult to achieve adaptive control when facing complex nonlinear systems, and existing deep reinforcement learning methods have insufficient control accuracy and poor adaptability in multiple environments.
An adaptive single-joint servo PID control method based on deep reinforcement learning is adopted. By constructing an adaptive SCA_PID hierarchical controller and combining the SAC algorithm and the PID algorithm, the PID controller parameters are dynamically adjusted. The improved experience playback pool is constructed by training with multi-factor state parameters to achieve adaptive control.
It improves the accuracy and adaptability of single-joint servo control, enhances the robustness and scalability of the system, and enables rapid parameter adjustment and updates in multiple environments.
Smart Images

Figure CN118795780B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of servo control technology, and in particular to an adaptive single-joint servo PID control method and system based on deep reinforcement learning. Background Art
[0002] With the continuous development of science and technology, adaptive control technology has been widely used in various fields. In the field of servo control, traditional PID control methods often find it difficult to achieve ideal control effects when facing complex nonlinear systems. In order to improve the performance of servo control, researchers have begun to explore the application of deep reinforcement learning in servo control. Deep reinforcement learning can learn the optimal control strategy through interaction with the environment, and has strong adaptability and robustness. However, how to combine deep reinforcement learning with traditional PID control to achieve adaptive single-joint servo control is still a technical problem. In addition, the traditional PID controller has fixed parameters and has poor adaptability to different environmental parameters and system parameter changes. When the system parameters change, the traditional PID controller needs to adjust the parameters manually, and adaptive control cannot be achieved. In addition, the existing PID control method based on reinforcement learning cannot achieve real-time adaptive control updates when facing different working environments, making the existing control method less adaptable.
[0003] For example, a Chinese patent with the authorization announcement number CN113093526B discloses a method for parameter tuning of a non-overshoot PID controller based on reinforcement learning. This method constructs a learning agent and trains the constructed agent. Finally, the agent outputs the optimal PID parameters according to the current state data and provides them to the PID controller, so that the state value of the controlled object reaches the set value at a faster speed without overshoot, thereby achieving improvements in the non-overshoot setting of the controller PID parameters.
[0004] Through the analysis of the above-mentioned prior art, it is found that most of the prior art only considers the adjustment control of PID parameters from a single perspective when constructing reinforcement learning control methods, such as only considering the data of the actual angles of each joint. However, when controlling the controlled object, it is also affected by many other factors, such as speed, acceleration, load force and other factors; it is difficult to achieve high-precision control of the controlled object only from the perspective of unilateral factor errors. In addition, the existing methods only consider the control methods of a single environment, resulting in insufficient control accuracy of the controller under multiple environments. In view of the above problems, the present invention provides an adaptive single-joint servo PID control method and system based on deep reinforcement learning. Summary of the invention
[0005] The technical problem to be solved by the present invention is to address the deficiencies in the prior art, and to provide an adaptive single-joint servo PID control method and system based on deep reinforcement learning, which solves the problem of fixed parameters and poor adaptability of PID controller in the prior art by means of dynamic adjustment of the upper decision maker, and improves the accuracy of single-joint servo control. And through the combination of deep reinforcement learning algorithm and PID controller, adaptive single-joint servo control is realized, and the parameters of PID controller can be adaptively adjusted according to different environmental parameters and system parameters, improving the adaptability and robustness of the system. In addition, the control system also has good scalability and portability, and can be widely used in various single-joint servo control systems.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] The adaptive single-joint servo PID control method based on deep reinforcement learning includes the following steps:
[0008] S1: Build a single-joint servo PID system, including the controlled object, sensor, and actuator, and use the sensor to collect parameter data of the controlled object and actuator operation;
[0009] S2: Based on the SAC and PID algorithms, an adaptive SCA_PID hierarchical controller is constructed. The input state is constructed using the collected parameter data. The constructed hierarchical controller is optimized and trained in a virtual environment using the input state to obtain a single-joint servo controller.
[0010] S3: deploying the trained single-joint servo controller to different real operating environments to control specific controlled objects to perform specific tasks, and uploading different environmental parameter data obtained by performing specific tasks in different environments to the virtual environment through sensors for preprocessing and storage;
[0011] S4: Use the different environmental parameter data obtained in S3 to construct a new state input to the trained adaptive SCA_PID hierarchical controller for fine-tuning, obtain the secondary optimized single-joint servo controller, and use the secondary optimized single-joint servo controller to perform specific control tasks;
[0012] The sensor collects the parameter data of the controlled object and the actuator operation, including the actual position data corresponding to each coordinate component of the controlled object in the o-xyz coordinate system. and set position data , component data of the actual angular velocity of the controlled object in the three coordinate directions And the expected angular velocity component data , angular acceleration data , and the servo system load force N;
[0013] Input Status Including the position coordinate error, angular velocity coordinate error, angular acceleration and load force N; the position coordinate error includes: , , ,in , and It represents the error between the actual position data of the controlled object and the set position data in the coordinate axis x, y, and z directions respectively;
[0014] Angular velocity coordinate errors include: , , ,in , and The errors between the actual angular velocity component data of the controlled object and the expected angular velocity component data in the directions of the coordinate axes x, y, and z are represented in turn;
[0015] The adaptive SCA_PID hierarchical controller includes an adaptive SCA upper-layer decision maker and a lower-layer controller based on the PID algorithm. The adaptive SCA upper-layer decision maker includes a strategy network , target strategy network , First Rating Network , the first target evaluation network , Second Evaluation Network , Second target evaluation network and improved experience replay pool B; the strategy network and target strategy network The first evaluation network is composed of a first neural network with the same structure, wherein the first neural network includes a first fully connected layer, a ReLU layer, a Gaussian distribution layer, and a Tanh layer; , the first target evaluation network , Second Evaluation Network , Second target evaluation network The second neural network is composed of a second neural network with the same structure, wherein the second neural network includes a first fully connected layer, a ReLU layer, a second fully connected layer, a ReLU layer, and a second fully connected layer; the first evaluation network With the first target evaluation network Correspondingly, the second evaluation network With the second target evaluation network correspond;
[0016] The adaptive SCA upper-level decision maker is used to generate the dynamic adjustment amount of the output action of the lower-level controller according to the state parameter data generated by the controlled object;
[0017] The lower controller based on the PID algorithm is used to control the controlled object according to its own input parameters and the input dynamic adjustment amount;
[0018] Policy Network , used to input the current state s t , calculate the probability of all actions being executed ;
[0019] Target Policy Network , used as the parameters of the delay policy network, and based on the next time step state extracted from the improved experience replay pool , generating the execution action for the next time step ; The next time step state Sampling is performed from the improved experience replay pool B;
[0020] First Rating Network , used to input state s according to time t t And the probability of all actions, calculate the current action state evaluation value;
[0021] Second evaluation network , used to calculate the next time step state and the execution action at the next time step Calculate and obtain the action state evaluation value at the next moment;
[0022] First target evaluation network With the second target evaluation network They are used to update the corresponding evaluation network parameters and fix the target value function to speed up the convergence;
[0023] Improved experience replay pool B is used to store experience sample data and store the stored historical experience sample data in layers; improved experience replay pool B includes ordinary experience replay pool B 1 and Special Experience Replay Pool B 2 ;
[0024] Normal Experience Replay Pool B 1 , used to store low-quality experience sample data; special experience replay pool B 2 , used to store high-quality experience sample data; the quality of the experience sample data is expressed by the mixed priority of the experience sample data, as follows: ,in represents the mixing priority of the i-th experience sample data, represents the target priority of the i-th experience sample data, represents the TD error of the i-th experience sample data; the experience sample data target priority represents the absolute value of the target priority difference between the current state and the next state of the controlled object, as follows: , , ;
[0025] in, represents the target priority of the i-th experience sample data, Indicates the target priority of the i-th experience sample data extracted at the next moment. Indicates the target priority of the i-th experience sample data extracted at the current state; It represents the ratio of the distance between the controlled object and the target point and the initial distance in the next state of the i-th experience sample data; It represents the ratio of the distance between the controlled object and the target point and the initial distance at the current state of the i-th experience sample data;
[0026] Normal Experience Replay Pool B 1 and Special Experience Replay Pool B 2 The classification criteria are: ,in, Indicates the mixed priority threshold of empirical sample data set based on experience. The experience sample data is stored in the special experience replay pool B 2 , otherwise stored in the normal experience replay pool B 1 ;
[0027] The specific process of hierarchical controller optimization training in S2 includes:
[0028] S501: Randomly initialize the network parameters corresponding to the strategy network , the network parameters corresponding to the target strategy network , the network parameters corresponding to the first evaluation network, the network parameters corresponding to the first target evaluation network, the network parameters corresponding to the second evaluation network, the network parameters corresponding to the second target evaluation network, and the initial parameters in the PID algorithm , and ;
[0029] S502: Use the randomly initialized network to execute the strategy network n times to initialize the improved experience replay pool B, and save the initialized experience sample data in the ordinary experience replay pool B according to the mixed priority of the experience sample data 1 and Special Experience Replay Pool B 2 ;
[0030] S503: Input status of the build , input to the policy network , calculate and select a three-dimensional dynamic action adjustment value for the current single-joint servo PID system a ' t = [ U p , U i , U d ] ;
[0031] S504: Input the acquired three-dimensional dynamic motion adjustment amount into the lower controller based on the PID algorithm and the initial parameter value [ B p , B i , B d ] Dynamic fusion obtains the dynamic adjustment parameters of the single-joint servo PID system, and uses the obtained dynamic adjustment parameters and the position error of the controlled object , calculate the output control quantity of the servo PID system , as follows:
[0032] ,
[0033] ,
[0034] ,
[0035] in, represents the output control quantity of the servo PID system at time t, , , They represent the final proportional coefficient dynamic parameters, integral time dynamic parameters and differential time dynamic parameters of the lower controller based on the PID algorithm, respectively. They represent the initialization proportional coefficient parameter, integral time parameter and differential time parameter of the lower controller based on the PID algorithm, respectively. They represent the proportional coefficient dynamic action adjustment amount, the integral time dynamic action adjustment amount and the differential time dynamic action adjustment amount output by the upper decision maker of the adaptive SCA in turn; In order to indicate the corresponding The offset of Indicates the length of a cycle of control system operation;
[0036] S505: Use the control variable to control the controlled object and obtain the next state in the corresponding working environment And the corresponding combination reward , and the obtained [ s t , a t , R t , s t + 1 ] The group is stored in the corresponding improved experience replay pool B;
[0037] S506: Replay from normal experience pool B 1 and Special Experience Replay Pool B 2 A batch of empirical sample data is extracted in a ratio of 7:3 [ s i , a i , R i , s i + 1 ] , by minimizing the time difference from the current state to the action pair To the next moment state action pair The loss function of value estimation updates the first evaluation network and the second evaluation network. The specific process is as follows:
[0038] J φ Q 1 = minE ( s i , a i , R i , s i + 1 , a i + 1 ) [ 1 2 ( φ Q 1 ( s i , a i ) − ( R t ( s i , a i ) + γ Q Target ( s i + 1 , a i + 1 )) ) 2 ] ,
[0039] ,
[0040] in, Represents the loss function of the value estimation from the current state-action pair to the next state-action pair, represents the estimate of the action at the next moment, Indicates the reward from the current state-action pair to the next state-action pair The minimum expected loss under the condition of Represents the current state action value evaluation value extracted from the experience sample data, It represents the reward value obtained by the agent when executing the current state action extracted from the experience sample data. represents the discount factor; Represents the minimum action value evaluation value selected by the agent for the next time step; represents the action value evaluation value of the next time step extracted from the experience sample data, Indicates the action value evaluation value corresponding to the next moment at the current moment;
[0041] The policy network is updated by maximizing the future reward value estimated by the evaluation network, and its loss function can be expressed as:
[0042] L π ( θ ) = E s i ∈ B , ε ∈ π θ [ log π θ ( ε , a t | s t ) − φ Q 1 ( s t , a t ) ] ,
[0043] in, represents the policy network update loss function, represents the input noise, An evaluation value representing the value of the current state and action state;
[0044] Combined Rewards The specific construction process includes:
[0045] ,
[0046] ,
[0047] in, represents the position error reward, It represents the reward for comparing the error between time t-1 and time t. represents the angular velocity error reward, represents the angular acceleration reward, represents the load capacity reward, Indicates internal rewards; In turn, they represent the negative gain coefficients of the corresponding rewards; Indicates the maximum weight set for the controlled object; Represents the error of angular velocity in space;
[0048] The specific steps of extracting a batch of experience sample data in S506 include:
[0049] S701: Use random sampling method to replay from special experience pool B 2 Extract empirical sample data from
[0050] S702: Use the priority probability sampling method to replay from the common experience pool B 1 The empirical sample data is extracted as follows:
[0051] ,
[0052] ,
[0053] in, represents the discreteness of the i-th empirical sample data, represents the mixing priority of the i-th experience sample data, Indicates normal experience replay pool B 1 The mean of the mixed priority of all samples in , Represents a positive number that is smaller than the preset value to ensure that the priority of each sample in the experience pool is not 0; represents the sampling probability of the i-th experience sample data, and K represents the common experience replay pool B 1 The total number of empirical sample data, An adjustment factor representing the mixing priority of the empirical sample data;
[0054] The secondary optimization of the single-joint servo controller adopts a federation strategy. The specific process includes:
[0055] S801: Opening n different virtual sub-processes in a virtual environment, and uploading different environmental parameter data to the n virtual sub-processes through sensors for storage; each of the n virtual sub-processes includes a corresponding unique environment, and the n virtual sub-processes are independent of each other;
[0056] S802: The adaptive SCA_PID hierarchical controller constructed and trained in S2 is used as the main process, and the adaptive SCA_PID hierarchical controller constructed and trained in the main process is copied to n virtual sub-processes, and the trained adaptive SCA_PID hierarchical controller is fine-tuned using the environmental parameter data stored in the sub-processes;
[0057] S803: Upload the gradient parameters of the hierarchical controllers in the n sub-processes after fine-tuning to the hierarchical controller of the main process. The hierarchical controller in the main process uses the gradient parameters uploaded by the n sub-processes to update the parameters and weights in the network; obtain the adaptive SCA_PID hierarchical controller after the secondary optimization of the single-joint servo controller;
[0058] S804: When the single-joint servo controller adds new environmental parameter data, the collected new environmental parameter data is used to overwrite the parameter data saved in the sub-process with the lowest utilization rate among the original n sub-processes, and the new environmental parameter data is used to complete the continuous update of the single-joint servo controller.
[0059] An adaptive single-joint servo PID control system based on deep reinforcement learning includes a sensor parameter processing module, an adaptive servo PID control module, and a controller deployment and update optimization module; the adaptive servo PID control module includes an adaptive SAC decision unit and a PID dynamic control unit; the controller deployment and update optimization module includes a controller deployment unit and a controller update optimization unit;
[0060] Sensor parameter processing module, used for collecting and preprocessing parameter data of the controlled object in different environments;
[0061] Adaptive servo PID control module, used to generate output control command signals corresponding to the environment and control the specific working tasks of the single-joint servo system;
[0062] Controller deployment and update optimization module, used for deployment of single-joint servo controller and update and adjustment of control parameters.
[0063] Adaptive SAC decision unit, used for the construction, optimization training, and generation and output of dynamic action adjustment of SAC decision controller;
[0064] PID dynamic control unit, used to dynamically control and adjust the specific tasks of the controlled object according to the adjustment amount output by the adaptive SAC decision unit;
[0065] The controller deployment unit is used to deploy the trained single-joint servo controller to different specific environments and generate different numbers of child processes for specific environments;
[0066] The controller update optimization unit is used for secondary update optimization of single-joint servo controllers and continuous optimization of subsequent tasks.
[0067] A storage medium of the present invention stores instructions, and when a computer reads the instructions, the computer executes the adaptive single-joint servo PID control method based on deep reinforcement learning as described above.
[0068] An electronic device of the present invention comprises a processor and the above-mentioned storage medium, wherein the processor executes instructions in the storage medium.
[0069] Compared with the prior art, the present invention has the following beneficial effects:
[0070] 1. Aiming at the shortcomings of the traditional servo PID control method, the present invention uses the adaptive SCA upper-layer decision maker constructed by the SAC algorithm to adjust the parameters of the lower-layer controller based on PID in real time, and uses multi-factor state parameters to train the adaptive SCA upper-layer decision maker, which not only improves the overall accuracy of the servo PID control system, but also further improves the accuracy of the output decision of the adaptive SCA upper-layer decision maker by using the combined reward; in addition, the present invention uses secondary optimization to further improve the adaptability and robustness of the system for different environments;
[0071] 2. The present invention aims at the shortcomings of slow training and low precision of the upper-layer decision maker of adaptive SCA. By improving the experience replay pool, the decision maker can learn more prior knowledge from high-quality experience sample data more efficiently and quickly during training, thereby improving the speed and precision of the upper-layer decision maker training.
[0072] 3. By constructing multiple parallel sub-processes, the present invention enables the control system to quickly adjust and update parameters when facing multiple different environments and working scenarios, thereby improving the scalability and portability of the control method and system. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0074] Figure 1 This is a flow chart of an adaptive single-joint servo PID control method based on deep reinforcement learning according to Embodiment 1 of the present invention;
[0075] Figure 2 This is a structural diagram of an adaptive single-joint servo PID control method based on deep reinforcement learning in Example 1 of the present invention;
[0076] Figure 3 A neural network structure diagram of a strategy network and a target strategy network in Example 1 of the present invention;
[0077] Figure 4 A neural network structure diagram of the first evaluation network, the first target evaluation network, the second evaluation network, and the second target evaluation network in Example 1 of the present invention;
[0078] Figure 5 This is a secondary optimization structure diagram of a single-joint servo controller according to Embodiment 1 of the present invention;
[0079] Figure 6 This is a module structure diagram of an adaptive single-joint servo PID control system based on deep reinforcement learning according to Example 2 of the present invention. DETAILED DESCRIPTION
[0080] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0081] Embodiment 1:
[0082] When constructing reinforcement learning control methods, traditional servo system control technology mostly only considers the adjustment control of PID parameters from a single perspective. However, when controlling the controlled object, it is also affected by many other factors, such as speed, acceleration, load force, etc. It is difficult to achieve high-precision control of the controlled object by only considering the error of one factor. In addition, the existing methods only consider the control method of a single environment, resulting in insufficient control accuracy of the controller in multiple environments and relatively poor adaptability.
[0083] To this end, the present invention provides an embodiment: an adaptive single-joint servo PID control method based on deep reinforcement learning, please refer to Figure 1 , the specific steps include:
[0084] S1: Build a single-joint servo PID system, including a controlled object, a sensor, and an actuator, and use the sensor to collect parameter data of the controlled object and the actuator; the controlled object is a joint of a rotary robot arm, the actuator is a motor and a reducer, and the sensor is a rotary encoder;
[0085] S2: Based on the SAC and PID algorithms, an adaptive SCA_PID hierarchical controller is constructed. The input state is constructed using the collected parameter data. The constructed hierarchical controller is optimized and trained in a virtual environment using the input state to obtain a single-joint servo controller.
[0086] S3: deploying the trained single-joint servo controller to different real operating environments to control specific controlled objects to perform specific tasks, and uploading different environmental parameter data obtained by performing specific tasks in different environments to the virtual environment through sensors for preprocessing and storage;
[0087] S4: Use the different environmental parameter data obtained in S3 to construct a new state input to the trained adaptive SCA_PID hierarchical controller for fine-tuning, obtain the secondary optimized single-joint servo controller, and use the secondary optimized single-joint servo controller to perform specific control tasks.
[0088] Furthermore, the sensor collects parameter data of the controlled object and the actuator operation, including the actual position data corresponding to each coordinate component of the controlled object in the o-xyz coordinate system. and set position data , component data of the actual angular velocity of the controlled object in the three coordinate directions And the expected angular velocity component data , angular acceleration data , and the servo system load force N.
[0089] Furthermore, the input state Including the position coordinate error, angular velocity coordinate error, angular acceleration and load force N; the position coordinate error includes: , , ,in , and It represents the error between the actual position data of the controlled object and the set position data in the coordinate axis x, y, and z directions respectively;
[0090] Angular velocity coordinate errors include: , , ,in , and It represents the error between the actual angular velocity component data of the controlled object and the expected angular velocity component data in the directions of the coordinate axes x, y, and z.
[0091] The input state constructed by position error, velocity error, acceleration and load force can more accurately control the controlled object from different aspects, and have stronger adaptability when facing more emergencies.
[0092] For further information, see Figure 2-Figure 4The adaptive SCA_PID hierarchical controller includes an adaptive SCA upper-layer decision maker and a lower-layer controller based on the PID algorithm. The adaptive SCA upper-layer decision maker includes a strategy network , target strategy network , First Rating Network , the first target evaluation network , Second Evaluation Network , Second target evaluation network and improved experience replay pool B;
[0093] Policy Network and target strategy network It is composed of the first neural network with the same structure, which includes the first fully connected layer, ReLU layer, Gaussian distribution layer, and Tanh layer; for details, please refer to Figure 3 , the input state s t , input to the first fully connected layer with a parameter of 32 and the ReLU layer, get the mean of the policy network and the standard deviation of the policy, and get the Gaussian distribution of the policy mean and the policy standard deviation in the fourth layer, and output the action after the Gaussian distribution and tanh activation function ;
[0094] First Rating Network , the first target evaluation network , Second Evaluation Network , Second target evaluation network The second neural network is composed of a second neural network with the same structure, the second neural network includes a first fully connected layer, a ReLU layer, a second fully connected layer, a ReLU layer, and a second fully connected layer; the first evaluation network With the first target evaluation network Correspondingly, the second evaluation network With the second target evaluation network For details, see Figure 4 , the input is from state s t and actions The output is the state action evaluation Q value. There are three first fully connected layers, second fully connected layers, and second fully connected layers with parameter sizes of 32, 1, and 1 respectively. In order to improve the training speed and prevent the gradient from disappearing, the nonlinear activation function ReLu is introduced after the first two fully connected layers;
[0095] The adaptive SCA upper-level decision maker is used to generate the dynamic adjustment amount of the output action of the lower-level controller according to the state parameter data generated by the controlled object;
[0096] The lower controller based on the PID algorithm is used to control the controlled object according to its own input parameters and the input dynamic adjustment amount;
[0097] Policy Network , used to input the current state s t , calculate the probability of all actions being executed ;
[0098] Target Policy Network , used as the parameters of the delay policy network, and based on the next time step state extracted from the improved experience replay pool , generating the execution action for the next time step ; The next time step state Sampling is performed from the improved experience replay pool B;
[0099] First Rating Network , used to input state s according to time t t And the probability of all actions, calculate the current action state evaluation value;
[0100] Second evaluation network , used to calculate the next time step state and the execution action at the next time step Calculate and obtain the action state evaluation value at the next moment;
[0101] First target evaluation network With the second target evaluation network They are used to update the corresponding evaluation network parameters and fix the target value function to speed up the convergence;
[0102] Improved experience replay pool B is used to store experience sample data and store the stored historical experience sample data in layers; improved experience replay pool B includes ordinary experience replay pool B 1 and Special Experience Replay Pool B 2 ; Specifically, improve the experience replay pool B, the normal experience replay pool B 1 and Special Experience Replay Pool B 2 The capacity is , and ;
[0103] Normal Experience Replay Pool B 1 , used to store low-quality experience sample data; special experience replay pool B 2 , used to store high-quality experience sample data; the quality of the experience sample data is expressed by the mixed priority of the experience sample data, as follows: ,in represents the mixing priority of the i-th experience sample data, represents the target priority of the i-th experience sample data, represents the TD error of the i-th experience sample data; the experience sample data target priority represents the absolute value of the target priority difference between the current state and the next state of the controlled object, as follows: , , ;
[0104] in, represents the target priority of the i-th experience sample data, Indicates the target priority of the i-th experience sample data extracted at the next moment. Indicates the target priority of the i-th experience sample data extracted at the current state; It represents the ratio of the distance between the controlled object and the target point and the initial distance in the next state of the i-th experience sample data; It represents the ratio of the distance between the controlled object and the target point and the initial distance at the current state of the i-th experience sample data;
[0105] Normal Experience Replay Pool B 1 and Special Experience Replay Pool B 2 The classification criteria are: ,in, Indicates the mixed priority threshold of empirical sample data set based on experience. The experience sample data is stored in the special experience replay pool B 2 , otherwise stored in the normal experience replay pool B 1 , among which Set it according to the specific working environment and experimental parameters. By dividing the improved experience replay pool into two parts, the efficient experience sample data is stored separately, and the experience replay and sampling processes of the experience replay pool are optimized; and the speed and accuracy of model training are improved.
[0106] For further information, see Figure 2 ,The specific process of hierarchical controller optimization training in S2 includes:
[0107] S501: Randomly initialize the network parameters corresponding to the strategy network , the network parameters corresponding to the target strategy network , the network parameters corresponding to the first evaluation network, the network parameters corresponding to the first target evaluation network, the network parameters corresponding to the second evaluation network, the network parameters corresponding to the second target evaluation network, and the initial parameters in the PID algorithm , and Specifically, the PID algorithm is initialized by manually inputting the desired input position signal x into the PID-based lower-level controller. p , get the tracking response position signal xt ; Obtain the initial position error through the expected input signal and the tracking response signal, and use the position error to initialize the initial parameters in the PID algorithm;
[0108] S502: Execute the strategy network n times using the randomly initialized network to initialize the improved experience replay pool B, and save the initialized experience sample data in the common experience replay pool B according to the mixed priority of the experience sample data. 1 and Special Experience Replay Pool B 2 ;
[0109] S503: Input status of the build , input to the policy network , calculate and select a three-dimensional dynamic action adjustment value for the current single-joint servo PID system a ' t = [ U p , U i , U d ] ; Specifically, the policy network in the SAC algorithm outputs the probability scores of all actions, and then calculates the entropy of each action based on these probability scores. Next, the agent will select actions based on the size of the entropy. If the entropy value of an action is large, it means that the action has a large exploration space, and the agent is more inclined to choose this action. At the same time, the agent will also consider the expected return value of the action in order to strike a balance between exploration and utilization. The specific calculation process is as follows:
[0110] π * max = arg <m> max < / m> E ( s t , a t ) [ R t ( s t , a t ) + α H ( π θ ( • | s t ) ) ]
[0111] in, Indicates the current state Execute an action to obtain the maximum reward and entropy value; arg represents the inverse function, It means to obtain the maximum value from the sum of the reward and entropy obtained from all actions performed by the agent, that is, to calculate the action with the maximum entropy by the probability scores of all actions obtained. Indicates that the agent is in the input state , and perform actions The expected value obtained, Indicates the current state Next action The maximum cumulative reward obtained; represents the temperature coefficient, specifically 0.200, Indicates the current state Perform any action The entropy value obtained; Indicates the current state The probability score of any action among all actions obtained under
[0112] S504: Input the acquired three-dimensional dynamic motion adjustment amount into the lower controller based on the PID algorithm and the initial parameter value [ B p , B i , B d ] Dynamic fusion obtains the dynamic adjustment parameters of the single-joint servo PID system, and uses the obtained dynamic adjustment parameters and the position error of the controlled object , calculate the output control quantity of the servo PID system , as follows:
[0113] ,
[0114] ,
[0115] in, represents the output control quantity of the servo PID system at time t, , , They represent the final proportional coefficient dynamic parameters, integral time dynamic parameters and differential time dynamic parameters of the lower controller based on the PID algorithm, respectively. They represent the initialization proportional coefficient parameter, integral time parameter and differential time parameter of the lower controller based on the PID algorithm, respectively. They represent the proportional coefficient dynamic action adjustment amount, the integral time dynamic action adjustment amount and the differential time dynamic action adjustment amount output by the upper decision maker of the adaptive SCA in turn; In order to indicate the corresponding The offset of Indicates the length of a cycle of control system operation;
[0116] S505: Utilizing Control Volume Control the controlled object and obtain the next state in the corresponding working environment And the corresponding combination reward , and the obtained [ s t , a t , R t , s t + 1 ] The group is stored in the corresponding improved experience replay pool B;
[0117] S506: Replay from normal experience pool B 1 and Special Experience Replay Pool B 2 A batch of empirical sample data is extracted in a ratio of 7:3 [ s i , a i , R i , s i + 1 ] , by minimizing the time difference from the current state to the action pair To the next moment state action pair The loss function of value estimation updates the first evaluation network and the second evaluation network. The specific process is as follows:
[0118] J φ Q 1 = minE ( s i , a i , R i , s i + 1 , a i + 1 ) [ 1 2 ( φ Q 1 ( s i , a i ) − ( R t ( s i , a i ) + γ Q Target ( s i + 1 , a i + 1 )) ) 2 ] ,
[0119] ,
[0120] in, Represents the loss function of the value estimation from the current state-action pair to the next state-action pair, represents the estimate of the action at the next moment, Indicates the reward from the current state-action pair to the next state-action pair The minimum expected loss under the condition of Represents the current state action value evaluation value extracted from the experience sample data, It represents the reward value obtained by the agent when executing the current state action extracted from the experience sample data. represents the discount factor; Represents the minimum action value evaluation value selected by the agent for the next time step; represents the action value evaluation value of the next time step extracted from the experience sample data, Indicates the action value evaluation value corresponding to the next moment at the current moment;
[0121] The policy network is updated by maximizing the future reward value estimated by the evaluation network, and its loss function can be expressed as:
[0122] L π ( θ ) = E s i ∈ B , ε ∈ π θ [ log π θ ( ε , a t | s t ) − φ Q 1 ( s t , a t ) ] ,
[0123] in, represents the policy network update loss function, represents the input noise, Indicates the evaluation value of the current state and action state value, Indicates that the agent has input noise In the state, execute the current state action The entropy value calculated when
[0124] The target policy network parameters are updated as: ,in is the update coefficient, represents the policy network parameters, represents the target strategy network parameter, and the update coefficient in this embodiment is 0.01; the update process of the remaining network weight parameters is the same as the original SAC network weight parameter update process, which will not be repeated here.
[0125] Through the above training process, the adaptive SAC upper-level decision maker and the lower-level controller can be trained simultaneously and reach the optimal state, thereby reducing the difficulty of model training and enabling the overall controller to converge faster.
[0126] Furthermore, the specific steps of extracting a batch of experience sample data in S506 include:
[0127] S701: Use random sampling method to replay from special experience pool B 2 Specifically, the random sampling method of this embodiment adopts a stratified sampling method, which divides all the experience sample data into 10 layers, and extracts 5% of the experience sample data from each layer;
[0128] S702: Use the priority probability sampling method to replay from the common experience pool B 1 The empirical sample data is extracted as follows:
[0129] ,
[0130] ,
[0131] in, represents the discreteness of the i-th empirical sample data, represents the mixing priority of the i-th experience sample data, Indicates normal experience replay pool B 1 The mean of the mixed priority of all samples in , Represents a positive number that is smaller than the preset value to ensure that the priority of each sample in the experience pool is not 0. The specific preset value is determined according to specific application requirements in a specific environment; represents the sampling probability of the i-th experience sample data, and K represents the common experience replay pool B 1 The total number of empirical sample data, Indicates the adjustment factor of the mixing priority of the empirical sample data. If it is 1, full priority sampling is used to calculate the sample sampling probability.
[0132] Furthermore, the reward function is crucial in the upper decision maker because it determines the behavior of the controller and guides the learning process to achieve the desired goal. The reward function must take into account several factors in the upper controller to ensure effective tracking performance. First, the function should prioritize minimizing the tracking position error when tracking the input signal. Second, it should minimize the tracking position error by comparing the current position error and the position error at the previous moment To encourage the reduction of tracking position error. Third, consider the response speed and stability of the control system to minimize the angular velocity error. Fourth, the function should prevent over-acceleration to avoid oscillation, which is achieved by giving angular acceleration rewards. Fifth, it should be considered whether the load force of the controlled object exceeds the preset load force, and rewards should be given to avoid overload; finally, internal rewards are used to guide the overall learning of unfamiliar states by the upper-level decision maker. As the decision maker continues to converge, the internal reward finally returns to zero and returns to the other 5 rewards. Specifically, the combined reward The specific construction process includes:
[0133] ,
[0134] ,
[0135] in, represents the position error reward, It represents the reward for comparing the error between time t-1 and time t. represents the angular velocity error reward, represents the angular acceleration reward, represents the load capacity reward, Indicates internal rewards; In turn, they represent the negative gain coefficients of the corresponding rewards; Indicates the maximum weight set for the controlled object; Represents the error of angular velocity in space.
[0136] This combined reward can better promote the upper-level decision maker to learn the input environmental parameter data from multiple different aspects, thereby improving the accuracy of the overall decision output.
[0137] For further information, see Figure 5 ,The secondary optimization of the single joint servo controller adopts a federated strategy. The specific process includes:
[0138] S801: Open n different virtual sub-processes in a virtual environment, and upload different environmental parameter data to the n virtual sub-processes through sensors for storage; each of the n virtual sub-processes contains a corresponding unique environment, and the n virtual sub-processes are independent of each other; specifically, this embodiment simulates the working environment of a specific controlled object under different temperatures, humidity and load forces; and constructs three sub-processes for the above environment;
[0139] S802: The adaptive SCA_PID hierarchical controller constructed and trained in S2 is used as the main process, and the adaptive SCA_PID hierarchical controller constructed and trained in the main process is copied to three virtual sub-processes, and the trained adaptive SCA_PID hierarchical controller is fine-tuned using the environmental parameter data stored in the sub-processes;
[0140] S803: Upload the gradient parameters of the hierarchical controllers in the three sub-processes after fine-tuning to the hierarchical controller of the main process. The hierarchical controller in the main process uses the gradient parameters uploaded by the three sub-processes to update the parameters and weights in the network; obtain the adaptive SCA_PID hierarchical controller after secondary optimization of the single-joint servo controller;
[0141] S804: When the single-joint servo controller adds new environmental parameter data, the collected new environmental parameter data is used to overwrite the parameter data saved in the sub-process with the lowest utilization rate among the original three sub-processes, and the new environmental parameter data is completed to complete the continuous update of the single-joint servo controller.
[0142] By creating a subprocess in a virtual environment to fine-tune the controller, the control accuracy and robustness of the controller can be improved, the need for actual experiments can be reduced, and continuous updating and optimization can be achieved.
[0143] Embodiment 2:
[0144] See also Figure 6 , the present invention provides an embodiment: an adaptive single-joint servo PID control system based on deep reinforcement learning, comprising a sensor parameter processing module, an adaptive servo PID control module, and a controller deployment and update optimization module; the adaptive servo PID control module comprises an adaptive SAC decision unit and a PID dynamic control unit; the controller deployment and update optimization module comprises a controller deployment unit and a controller update optimization unit;
[0145] Sensor parameter processing module, used for collecting and preprocessing parameter data of the controlled object in different environments;
[0146] Adaptive servo PID control module, used to generate output control command signals corresponding to the environment and control the specific working tasks of the single-joint servo system;
[0147] Controller deployment and update optimization module, used for deployment of single-joint servo controller and update and adjustment of control parameters.
[0148] Adaptive SAC decision unit, used for the construction, optimization training, and generation and output of dynamic action adjustment of SAC decision controller;
[0149] PID dynamic control unit, used to dynamically control and adjust the specific tasks of the controlled object according to the adjustment amount output by the adaptive SAC decision unit;
[0150] The controller deployment unit is used to deploy the trained single-joint servo controller to different specific environments and generate different numbers of child processes for specific environments;
[0151] The controller update optimization unit is used for secondary update optimization of single-joint servo controllers and continuous optimization of subsequent tasks.
[0152] Furthermore, the system collects and uploads parameter data of the controlled object in different environments through the sensor parameter processing module, and the parameter data is transmitted to the adaptive servo PID control module, in which the dynamic action adjustment amount is calculated and obtained through the adaptive SAC decision unit, and the dynamic action adjustment amount is input into the PID dynamic control unit to realize real-time control of the controlled object; the single-joint servo controller in the adaptive servo PID control module is input into the controller deployment and update optimization module, in which the single-joint servo controller is deployed to different environments to perform tasks through the controller deployment unit, and parameter data of different environments is obtained, and the obtained data is input into the controller update optimization unit to realize secondary optimization update of the single-joint servo controller.
[0153] Embodiment 3:
[0154] A storage medium of the present invention stores instructions, and when a computer reads the instructions, the computer executes the adaptive single-joint servo PID control method based on deep reinforcement learning as described above.
[0155] An electronic device of the present invention comprises a processor and the above-mentioned storage medium, wherein the processor executes instructions in the storage medium.
[0156] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0157] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0158] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0160] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. An adaptive single-joint servo PID control method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Build a single-joint servo PID system, including the controlled object, sensor, and actuator, and use the sensor to collect parameter data of the controlled object and actuator operation; S2: Based on the SAC and PID algorithms, an adaptive SCA_PID hierarchical controller is constructed. The input state is constructed using the collected parameter data. The constructed hierarchical controller is optimized and trained in a virtual environment using the input state to obtain a single-joint servo controller. S3: deploying the trained single-joint servo controller to different real operating environments to control specific controlled objects to perform specific tasks, and uploading different environmental parameter data obtained by performing specific tasks in different environments to the virtual environment through sensors for preprocessing and storage; S4: Use the different environmental parameter data obtained in S3 to construct a new state input to the trained adaptive SCA_PID hierarchical controller for fine-tuning, obtain the secondary optimized single-joint servo controller, and use the secondary optimized single-joint servo controller to perform specific control tasks; The sensor collects parameter data of the controlled object and the actuator operation, including the actual position data corresponding to each coordinate component of the controlled object in the o-xyz coordinate system. and set position data , component data of the actual angular velocity of the controlled object in the three coordinate directions And the expected angular velocity component data , angular acceleration data , and the servo system load force N; The input status Including the position coordinate error, angular velocity coordinate error, angular acceleration and load force N; the position coordinate error includes: , , ,in , and It represents the error between the actual position data of the controlled object and the set position data in the coordinate axis x, y, and z directions respectively; The angular velocity coordinate error includes: , , ,in , and The errors between the actual angular velocity component data of the controlled object and the expected angular velocity component data in the directions of the coordinate axes x, y, and z are represented in turn; The adaptive SCA_PID hierarchical controller includes an adaptive SCA upper layer decision maker and a lower layer controller based on a PID algorithm. The adaptive SCA upper layer decision maker includes a strategy network , target strategy network , First Rating Network , the first target evaluation network , Second Evaluation Network , Second target evaluation network and improved experience replay pool B; the strategy network and target strategy network The first evaluation network is composed of a first neural network with the same structure, wherein the first neural network includes a first fully connected layer, a ReLU layer, a Gaussian distribution layer, and a Tanh layer; , the first target evaluation network , Second Evaluation Network , Second target evaluation network The second neural network is composed of a second neural network with the same structure, wherein the second neural network includes a first fully connected layer, a ReLU layer, a second fully connected layer, a ReLU layer, and a second fully connected layer; the first evaluation network With the first target evaluation network Correspondingly, the second evaluation network With the second target evaluation network correspond; The adaptive SCA upper-layer decision maker is used to generate a dynamic adjustment amount of the output action of the lower-layer controller according to the state parameter data generated by the controlled object; The lower controller based on the PID algorithm is used to control the controlled object according to its own input parameters and the input dynamic adjustment amount; The policy network , used to input the current state s t , calculate the probability of all actions being executed ; The target policy network , used as the parameters of the delay policy network, and based on the next time step state extracted from the improved experience replay pool , generating the execution action for the next time step ; The next time step state Sampling is performed from the improved experience replay pool B; The first evaluation network is used to input the state s at time t t And the probability of all actions, calculate the current action state evaluation value; The second evaluation network is used to evaluate the next time step state and the execution action at the next time step Calculate and obtain the action state evaluation value at the next moment; The first target evaluation network and the second target evaluation network are both used to update corresponding evaluation network parameters and fix the target value function to speed up the convergence speed; The improved experience replay pool B is used to store experience sample data and store the stored historical experience sample data in layers; the improved experience replay pool B includes a common experience replay pool B1 and a special experience replay pool B2; The common experience replay pool B1 is used to store low-quality experience sample data; the special experience replay pool B2 is used to store high-quality experience sample data; the quality of the experience sample data is represented by the mixed priority of the experience sample data, as follows: ,in represents the mixing priority of the i-th experience sample data, represents the target priority of the i-th experience sample data, represents the TD error of the i-th experience sample data; the experience sample data target priority represents the absolute value of the target priority difference between the current state and the next state of the controlled object, as follows: , , ; in, represents the target priority of the i-th experience sample data, Indicates the target priority of the i-th experience sample data extracted at the next moment. Indicates the target priority of the i-th experience sample data extracted at the current state; It represents the ratio of the distance between the controlled object and the target point and the initial distance in the next state of the i-th experience sample data; It represents the ratio of the distance between the controlled object and the target point and the initial distance at the current state of the i-th experience sample data; The standard for dividing the common experience replay pool B1 and the special experience replay pool B2 is: ,in, Represents the mixed priority threshold of the empirical sample data. When the experience sample data is stored in the special experience replay pool B2, otherwise it is stored in the ordinary experience replay pool B1; The specific process of the hierarchical controller optimization training in S2 includes: S501: Randomly initialize the network parameters corresponding to the strategy network , the network parameters corresponding to the target strategy network , the network parameters corresponding to the first evaluation network, the network parameters corresponding to the first target evaluation network, the network parameters corresponding to the second evaluation network, the network parameters corresponding to the second target evaluation network, and the initial parameters in the PID algorithm , and ; S502: Execute the strategy network n times using the randomly initialized network to initialize the improved experience replay pool B, and save the initialized experience sample data in the common experience replay pool B1 and the special experience replay pool B2 respectively according to the mixed priority of the experience sample data; S503: construct the input state , input to the policy network , calculate and select a three-dimensional dynamic action adjustment value for the current single-joint servo PID system ; S504: Input the acquired three-dimensional dynamic motion adjustment amount into the lower controller based on the PID algorithm and the initial parameter value Dynamic fusion obtains the dynamic adjustment parameters of the single-joint servo PID system, and uses the obtained dynamic adjustment parameters and the position error of the controlled object , calculate the output control quantity of the servo PID system , as follows: , , , in, represents the output control quantity of the servo PID system at time t, , , They represent the final proportional coefficient dynamic parameters, integral time dynamic parameters and differential time dynamic parameters of the lower controller based on the PID algorithm, respectively. They represent the initialization proportional coefficient parameter, integral time parameter and differential time parameter of the lower controller based on the PID algorithm, respectively. They represent the proportional coefficient dynamic action adjustment amount, the integral time dynamic action adjustment amount and the differential time dynamic action adjustment amount output by the upper decision maker of the adaptive SCA in turn; In order to indicate the corresponding The offset of Indicates the length of a cycle of control system operation; S505: Use the control variable to control the controlled object and obtain the next state in the corresponding working environment And the corresponding combination reward , and the obtained The group is stored in the corresponding improved experience replay pool B; S506: Extract a batch of experience sample data from the common experience replay pool B1 and the special experience replay pool B2 in a ratio of 7:3 , by minimizing the time difference from the current state to the action pair To the next moment state action pair The loss function of value estimation updates the first evaluation network and the second evaluation network. The specific process is as follows: , , in, Represents the loss function of the value estimation from the current state-action pair to the next state-action pair, represents the estimate of the action at the next moment, Indicates the reward from the current state-action pair to the next time step state-action pair The minimum expected value of loss under the condition of ; Represents the current state action value evaluation value extracted from the experience sample data, It represents the reward value obtained by the agent when executing the current state action extracted from the experience sample data. represents the discount factor; Represents the minimum action value evaluation value selected by the agent for the next time step; represents the action value evaluation value of the next time step extracted from the experience sample data, Indicates the action value evaluation value corresponding to the next moment at the current moment; The policy network The update is performed by maximizing the future reward value estimated by the evaluation network, and its loss function is expressed as: , in, represents the policy network update loss function, represents the input Gaussian noise, An evaluation value representing the value of the current state and action state; The combined reward The specific construction process includes: , , in, represents the position error reward, It represents the reward for comparing the error between time t-1 and time t. represents the angular velocity error reward, represents the angular acceleration reward, represents the load capacity reward, Indicates internal rewards; In turn, they represent the negative gain coefficients of the corresponding rewards; Indicates the maximum weight set for the controlled object; Represents the error of angular velocity in space; The specific steps of extracting a batch of experience sample data in S506 include: S701: extracting experience sample data from the special experience replay pool B2 using a random sampling method; S702: Extracting experience sample data from the common experience replay pool B1 using the priority probability sampling method, as follows: , , in, represents the discreteness of the i-th empirical sample data, represents the mixing priority of the i-th experience sample data, represents the mean of the mixed priority of all samples in the common experience replay pool B1, Represents a positive number that is smaller than the preset value to ensure that the priority of each sample in the experience pool is not 0; represents the sampling probability of the i-th experience sample data, K represents the total number of experience sample data in the common experience replay pool B1, An adjustment factor representing the mixing priority of the empirical sample data; The secondary optimization of the single-joint servo controller adopts a federation strategy, and the specific process includes: S801: Opening n different virtual sub-processes in a virtual environment, and uploading different environmental parameter data to the n virtual sub-processes through sensors for storage; each of the n virtual sub-processes includes a corresponding unique environment, and the n virtual sub-processes are independent of each other; S802: The adaptive SCA_PID hierarchical controller constructed and trained in S2 is used as the main process, and the adaptive SCA_PID hierarchical controller constructed and trained in the main process is copied to n virtual sub-processes, and the trained adaptive SCA_PID hierarchical controller is fine-tuned using the environmental parameter data stored in the sub-processes; S803: Upload the gradient parameters of the hierarchical controllers in the n sub-processes after fine-tuning to the hierarchical controller of the main process. The hierarchical controller in the main process uses the gradient parameters uploaded by the n sub-processes to update the parameters and weights in the network; obtain the adaptive SCA_PID hierarchical controller after the secondary optimization of the single-joint servo controller; S804: When the single-joint servo controller adds new environmental parameter data, the collected new environmental parameter data is used to overwrite the parameter data saved in the sub-process with the lowest utilization rate among the original n sub-processes, and the new environmental parameter data is used to complete the continuous update of the single-joint servo controller.
2. An adaptive single-joint servo PID control system based on deep reinforcement learning, which is implemented based on the adaptive single-joint servo PID control method based on deep reinforcement learning as described in claim 1, characterized in that: The system includes a sensor parameter processing module, an adaptive servo PID control module, and a controller deployment and update optimization module; The sensor parameter processing module is used for collecting and preprocessing parameter data of the controlled object in different environments; The adaptive servo PID control module is used to generate an output control command signal corresponding to the environment to control the specific working tasks of the single-joint servo system; The controller deployment and update optimization module is used for the deployment of a single-joint servo controller and the update and adjustment of control parameters.
3. The adaptive single-joint servo PID control system based on deep reinforcement learning according to claim 2 is characterized in that: The adaptive servo PID control module includes an adaptive SAC decision unit and a PID dynamic control unit; the controller deployment and update optimization module includes a controller deployment unit and a controller update optimization unit; The adaptive SAC decision unit is used for the construction, optimization training and generation and output of dynamic action adjustment amount of the SAC decision controller; The PID dynamic control unit is used to dynamically control and adjust the specific tasks of the controlled object according to the adjustment amount output by the adaptive SAC decision unit; The controller deployment unit is used to deploy the trained single-joint servo controller into different specific environments and generate different numbers of sub-processes for the specific environments; The controller update optimization unit is used for secondary update optimization of the single-joint servo controller and continuous optimization of subsequent tasks.
4. A storage medium, characterized in that: The storage medium stores instructions, and when a computer reads the instructions, the computer executes the adaptive single-joint servo PID control method based on deep reinforcement learning as described in claim 1.
5. An electronic device, characterized in that: It includes a processor and the storage medium as described in claim 4, and the processor executes the adaptive single-joint servo PID control method based on deep reinforcement learning as described in claim 1 in the storage medium.
