A self-optimization method of an intelligent controller based on reinforcement learning

Abstract

This invention discloses a self-optimization method for an intelligent controller based on reinforcement learning, comprising the following steps: S1, constructing the state space, action space, and environmental feedback function of the control system; S2, initializing the improved TD3 control algorithm structure; S3, forming experience tuples and storing them in an experience replay buffer; S4, randomly sampling a batch of experience tuples from the experience replay buffer; S5, if the delayed update condition is met, updating the parameters of the main policy network once; S6, synchronously updating the parameters of the target policy network and the target Q network; S7, deploying the trained main policy network to the intelligent controller and outputting the optimal control action; S8 continuously collecting interaction data and appending it to the experience replay buffer, periodically triggering the policy incremental training process. This invention constructs a reinforcement learning control structure, integrates policy evaluation and online training mechanisms, and achieves adaptive optimization of the controller.

Description

Technical Field

[0001] This invention relates to the field of adaptive control technology, and in particular to a self-optimization method for intelligent controllers based on reinforcement learning. Background Technology

[0002] In modern control systems, intelligence and adaptability have become key indicators for evaluating controller performance. Traditional control strategies often rely on manual experience or fixed parameter settings, such as PID control, fuzzy control, or model predictive control. These methods are applicable to static or linear systems to a certain extent, but their control performance is easily limited when facing complex systems with high nonlinearity, frequent disturbances, or changing operating conditions. It is difficult to achieve long-term optimal control behavior, especially in scenarios where system parameters change or the operating environment changes dynamically. Traditional controllers cannot automatically adjust the control strategy based on feedback, leading to decreased control accuracy, response lag, or even system instability.

[0003] To overcome these problems, reinforcement learning techniques have been gradually introduced into the field of control systems in recent years. Reinforcement learning enables policy optimization based on feedback through agent-environment interaction, without relying on explicit models. Among them, Deep Deterministic Policy Gradient (DDPG) and its improved algorithm, Dual-Delay Deep Deterministic Policy Gradient (TD3), have shown superior performance in continuous action spaces. The improved TD3 effectively alleviates the problems of Q-value overestimation and policy instability by introducing mechanisms such as dual-Q networks, delayed updates, and target policy perturbations. However, most existing TD3-based control methods are applied to standard simulation environments and lack systematic support for the deployment of actual embedded intelligent controllers. In particular, there is a lack of a unified adaptive closed-loop structure between the initialization mechanism of the policy network and value network, the logic of constructing experience samples, the target network synchronization strategy, and online training and inference deployment. This results in insufficient guarantees of real-time performance, stability, and training effectiveness in actual control scenarios.

[0004] Furthermore, existing technologies often fail to process the interactive data generated by the controller during operation in a timely manner or lack targeted sampling strategies, making it difficult to form a stable incremental learning mechanism for the strategy. Problems such as strategy degradation and decreased adaptability after long-term operation remain prevalent. Therefore, there is an urgent need for a controller self-optimization method that can cover the entire process of state-space modeling, policy network structure initialization, interactive data acquisition, master Q-network training, target network synchronization, and deployment execution. This method would enable automatic learning and dynamic updating of the control strategy throughout its entire lifecycle, improving the long-term stability and control performance of the control system in non-stationary environments.

[0005] Therefore, how to provide a self-optimization method for intelligent controllers based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a self-optimization method for intelligent controllers based on reinforcement learning. This invention constructs a reinforcement learning-driven controller self-optimization structure, integrates state perception, policy evaluation, and target network synchronization mechanisms, and realizes dynamic training and deployment of control policies through an improved TD3 algorithm. It supports online experience acquisition and incremental updates, forming a closed-loop process of perception-reasoning-execution-retraining, effectively improving the adaptability, stability, and long-term operating performance of the control system.

[0007] A self-optimization method for an intelligent controller based on reinforcement learning according to an embodiment of the present invention includes the following steps: S1. Construct the state space, action space, and environmental feedback function of the control system. The state space includes current sensor measurements, historical control outputs, and error information. The action space is the set of control commands that the intelligent controller can adjust. The environmental feedback function quantifies each control action and provides an immediate feedback. S2. Based on the constructed state space and action space, initialize the improved TD3 control algorithm structure, including two master policy networks, two master Q networks, and corresponding target policy networks and target Q networks. The target network uses a delayed parameter synchronization mechanism to improve the stability of policy training. S3. Utilize the interaction process between the intelligent controller and the controlled object to collect the current state, selected action, obtained reward value, and next state generated by each interaction to form an experience tuple, which is stored in the experience replay buffer. S4. Randomly sample a batch of experience tuples from the experience replay buffer and input them into the two master Q networks. Based on the current master policy network, generate the action corresponding to the next state, input it into the target Q network, and take the minimum Q value as the target action value. Update the parameters of the two master Q networks using the minimum mean square error criterion. S5. Determine if the delayed update condition is met. If so, update the parameters of the main policy network based on the current output of the main Q network, and use the target policy network to generate the policy output in the next state, adding truncated Gaussian noise to enhance the exploratory nature of the policy. S6. Synchronously update the parameters of the target policy network and the target Q network, making them gradually approach the corresponding main network parameters, thereby improving convergence stability and control accuracy during policy training. S7. Deploy the trained main policy network to the intelligent controller, receiving the current state and outputting the optimal control action in the actual control task, realizing real-time closed-loop control of the target system. S8. Continuously collect interactive data during the operation of the intelligent controller and add it to the experience playback buffer, periodically triggering the incremental policy training process to achieve online optimization of the control policy and adaptive improvement of control performance.

[0008] Optionally, the initialization of the improved TD3 control algorithm structure in S2 specifically includes: S21. Based on the structure of the state space and action space, the input dimension of the main policy network is set to the state vector dimension, and the output dimension is set to the control command dimension. The input dimension of the main Q network is set to the concatenation of the state vector and the action vector, and the output is the Q-value scalar. S22. Initialize two main policy networks and two main Q networks with identical structures but independent parameters. The main policy networks use a multilayer perceptron structure to output deterministic action values, and the main Q networks estimate action values ​​respectively. S23. Copy the two main policy networks to obtain two target policy networks, and copy the two main Q networks to obtain two target Q networks. The target network structure is consistent with the main network, but the parameters are updated independently. S24. Set a soft update coefficient and a synchronous update interval for each target network to ensure that it approximates the corresponding main network parameters at a fixed ratio during policy training. S25. Set the initial parameter values ​​of the main policy network and the main Q network. The parameters adopt a preset weight initialization scheme, and ensure that the initial parameter values ​​of the target networks are completely consistent with those of the main networks. S26. Configure the core hyperparameters of the TD3 control algorithm, including the maximum capacity of the experience replay buffer, the sampling batch size for each training round, the policy delay update steps, the reward discount factor, and the target smoothing noise amplitude; S27. Set the upper and lower limits of the control policy output to constrain the output range of the main policy network within the physical operation range allowed by the intelligent controller, and prevent illegal instruction output during the training process; S28. Start the improved TD3 control algorithm structure and keep all main networks and target networks in an initial synchronization state.

[0009] Optionally, S22 specifically includes: S221. Set the input dimension of the two master policy networks as the state vector dimension and the output dimension as the control command dimension. The master policy network maps the current state to a deterministic control action. S222. Construct a first master policy network and a second master policy network. Both have the same structure, including an input layer, multiple fully connected hidden layers, and an output layer. The output layer introduces a saturation function to limit the numerical range of the output action, satisfying the physical output boundary conditions of the controller. S223. Initialize the weights of the first master policy network and the second master policy network using different parameter initialization strategies to ensure that their parameters are independent and their learning paths are separated. S224. Construct a first master Q network and a second master Q network. Each master Q network receives a concatenated input of the current state vector and the corresponding control action vector. The output is an estimated value of the action in the current state-action pair, reflecting the long-term reward potential of the action in the current state. S225. The first and second master Q-networks adopt the same structural configuration, both containing an input layer, three hidden layers, and an output layer. The output layer outputs a single scalar value, representing the network's independent value judgment result for the same state-action pair, ensuring that the two master Q-networks do not produce parameter coupling when estimating the action value function. S226. The parameters of the first and second master Q-networks are initialized to different initial values, and each master Q-network is configured with an independent optimizer, learning rate, and regularization parameters, ensuring that they learn and estimate the action value function independently during the training phase, thereby effectively reducing the risk of overestimation of the value function. S227. Separate training scheduling modules are configured for the two master policy networks and the two master Q-networks. The training scheduling modules include parameter update strategies, gradient calculation processes, and training state buffering mechanisms, ensuring that all networks run independently and progress synchronously. S228. The initialization and construction of all master policy networks and master Q-networks are completed, establishing a data flow path for the TD3 control algorithm.

[0010] Optionally, S3 specifically includes: S31. Set the interaction cycle of the intelligent controller. In each interaction cycle, the intelligent controller outputs a control action based on the current state input through the main policy network and applies the control action to the controlled object. S32. Collect the response of the controlled object after the current action is executed, record the state before the action is applied as the current state, and the state after the action is applied as the next state. S33. Evaluate the current state and the response result after the action is executed according to a preset environmental feedback function, calculate the reward value obtained by the current control behavior, and use it to provide feedback on the controller's behavior effect. S34. Organize the current state, the selected action, the obtained reward value, and the next state into an experience tuple in the form of a four-tuple according to a unified data structure. S35. Assign a timestamp identifier and execution index number to the experience tuple to ensure the traceability of the time sequence and the controllability of the sampling distribution during the training process. S36. Write the constructed experience tuple into the experience replay buffer. The experience replay buffer is a finite-length circular queue structure. When the storage space reaches its capacity limit, it automatically overwrites the oldest stored experience tuple. S37. Set the writing frequency, writing strategy and cache update mechanism of the experience playback buffer to ensure that key state areas can be covered and the ratio of new and old experience samples can be balanced in different control stages; S38. Maintain the asynchronous execution of experience tuple acquisition and experience playback buffer writing process during the continuous operation of the intelligent controller to form a complete interactive data stream, providing continuous, diverse and behavioral feedback learning samples for the subsequent training stage.

[0011] Optionally, S4 specifically includes: S41. Randomly sample a set of empirical tuples without replacement from the empirical replay buffer according to a preset batch size; S42. Input the sampled current state and current action into two main Q networks, and calculate the estimated action value of the state-action pair under the current policy; S43. Input the sampled next state into the main policy network to generate the predicted action in the next state, as a behavior estimate for the next action; S44. Input the generated predicted action in the next state into two target Q networks, and calculate the target Q value output under the predicted action, obtaining two sets of corresponding action value estimates; S45. Compare the outputs of the two target Q networks, and select the minimum value as the current target action value.

[0012] in, The target Q-value of the master Q-network; Let be the instantaneous reward value at time t; This is a reward discount factor used to balance current and future rewards; , Two target Q-networks are used to estimate the action value in the next state; The next state; Actions generated for the target policy network; To truncate Gaussian noise perturbations and enhance policy robustness; To use the smaller target value in the dual Q-network; S46, use the difference between the selected target action value and the current action value calculated by the main Q-network as the error signal to construct the input term of the loss function of the main Q-network; S47, based on the minimum mean square error criterion, perform gradient backpropagation and optimization update on the parameters of the two main Q-networks respectively, so that their outputs gradually approach the target action value; S48, record the convergence rate, error magnitude and parameter change rate during each main Q-network parameter update process to provide training feedback for policy updates and synchronization with the target network.

[0013] Optionally, S5 specifically includes: S51. Set the delayed update frequency threshold for the main policy network and record the main policy network update counter value in the current training round; S52. Determine whether the current training round meets the delayed update condition of the main policy network. If not, skip the main policy network update step and only optimize the main Q network parameters; S53. If the delayed update condition is met, obtain the action value results output by the two main Q networks in the previous training round, and select the gradient feedback path corresponding to the first main Q network as the basis for updating the main policy network parameters; S54. Input the current state into the main policy network to obtain the control action output by the current policy, and concatenate this action with the current state as the input to the main Q network to calculate the action value under the current policy; S55. Construct the objective function of the main policy network:

[0014] in, The objective function value of the main policy network. The parameter set of the main policy network. This represents the batch sample size. The control actions generated by the main policy network. () represents the first master Q network.

[0015] Using the action value corresponding to the current policy output as the optimization direction, the parameters of the main policy network are adjusted through the backpropagation mechanism; S56, the target policy network is called with the next state in the current sampling batch as input to generate the target policy output action in the next state, which assists in updating the data stream required by the target Q network; S57, a truncated Gaussian noise perturbation is added to the output action of the target policy network. The noise parameters include mean, variance and upper and lower limits. After adding the perturbation, the action result is pruned to keep it within the action range allowed by the controller; S58, the target policy output after the noise processing is used to calculate the target Q value in the next round of training.

[0016] Optionally, S6 specifically includes: S61. Set the parameter update method for the target policy network and the target Q network to a soft update mechanism, and configure the update frequency, synchronization coefficient, and initial synchronization state for each type of target network. S62. After each round of policy training, determine whether the target network synchronization update condition is met. If it is met, perform parameter synchronization operations for the target policy network and the target Q network respectively. S63. Read the latest parameter values ​​of the current main policy network and the two main Q networks, and construct their corresponding synchronization targets. S64. Perform a weighted combination of the parameters of each layer of the target policy network and the parameters of the corresponding main policy network, and gradually adjust the target network parameters using proportional interpolation to make them slowly follow the changes of the main network. S65. Perform soft synchronization updates on the parameters in the two target Q networks in the same way, keeping them relatively consistent with but not completely equal to the parameters of the first and second main Q networks. S66. Set a parameter drift protection mechanism during the synchronization update process to limit the maximum change range of a single parameter update and avoid instability of the target network due to drastic fluctuations in the main network. S67. Record the target network parameter state after each synchronization and input it into the inference input for target action value calculation and policy evaluation in the next round of training; S68. Execute the target network synchronization update process repeatedly during the continuous training phase of the intelligent controller, so that the target policy network and the target Q network always maintain the evolution process of consistent structure with the main network, asymptotic parameters, and controlled fluctuations, thereby improving the policy convergence speed and output stability in the overall training process.

[0017] Optionally, S7 specifically includes: S71. After the main strategy network is trained and reaches the preset convergence index, export all network structures and parameter configurations to generate a deployable control strategy model file; S72. Embed the exported main strategy network into the inference module of the intelligent controller in compiled form, and configure the model loading process and interface protocol to support real-time state input and action output; S73. Construct the state perception channel of the intelligent controller, connect the sensor sampling module, actuator feedback module, and control parameter recording module to ensure that the current system state can be collected in real time and standardized into the input format of the main strategy network; S74. Start the main strategy network inference service during the control task initialization phase, read the current state vector in each control cycle and input it into the main strategy network; S75. The main strategy network outputs the optimal control action according to the current state. The optimal control action is directly transmitted to the actuator or scheduling module as a control command to realize the instruction issuance of the physical control process; S76. After the control action is executed, sample and record the actual response, and transmit the execution feedback back to the controller for subsequent control cycle state updates; S77. The intelligent controller continuously executes the state acquisition, action generation and response feedback process to construct a closed-loop control structure with the main strategy network as the core: state perception - strategy reasoning - instruction output; S78. During the entire closed-loop control operation, the execution performance indicators of the main strategy network are continuously monitored, including output delay, control error and stability fluctuation, to ensure that the controller maintains high response accuracy and stable control effect in actual application scenarios.

[0018] Optionally, S8 specifically includes: S81. During the actual operation of the intelligent controller, record the current state, selected action, reward obtained, and next state after system response for each control action, forming a new experience tuple; S82. Append the newly generated experience tuple to the experience replay buffer and arrange them in order according to timestamps to maintain the continuity and retrieval of experience samples in time; S83. When the number of new samples in the experience replay buffer reaches the set incremental threshold, automatically trigger the policy incremental training module to execute the parameter update task; S84. Extract a batch of samples containing a mixture of old and new samples from the experience replay buffer to ensure that the policy network takes into account both historical control experience and the latest interaction information during training; S85. Use the current main policy network and main Q network to train the sampled experience tuples and update the parameters to adapt to the control feedback characteristics under the latest operating environment; S86. According to the frequency and step size set in the policy training cycle, dynamically adjust the sample ratio and target value update strategy for each round of training to achieve long-term optimization of training resource scheduling; S87. After each policy incremental training is completed, determine whether to update the parameters of the target policy network and the target Q network synchronously, and record the convergence performance index of the current training cycle; S88. During the continuous operation of the intelligent controller, maintain the cyclic execution mechanism of experience acquisition, experience playback buffer writing and policy incremental training to realize online optimization of control policy and adaptive continuous improvement of controller performance.

[0019] The beneficial effects of this invention are: First, this invention constructs a state space, action space, and environmental feedback function model for practical control scenarios, explicitly incorporating sensor measurements, historical control outputs, and error information into the state representation scope, significantly improving the granularity of the control system's perception of dynamic environmental changes. By limiting the controller's output actions to an adjustable set of discrete or continuous control commands, and constructing an immediate feedback function, the system can immediately obtain environmental feedback after each control action to guide the strategy optimization process, solving the problems of traditional controllers' delayed response to external changes and static, fixed strategies.

[0020] Secondly, this invention employs an improved TD3 control algorithm structure, initializing two sets of main policy networks and main Q networks, and constructing corresponding target policy networks and target Q networks based on the main network structure. By introducing a dual-network structure, a delayed policy update mechanism, and a soft synchronization scheme for the target network, this method effectively suppresses Q-value overestimation, improving the stability of policy convergence and training efficiency. The main policy network adopts a multi-layer feedforward neural structure, possessing nonlinear action mapping capabilities, while the main Q network independently estimates the action value function, accurately evaluating the effectiveness of the current policy and providing clear guidance for policy optimization, avoiding the problems of insufficient value estimation accuracy and unclear gradient guidance direction in existing methods.

[0021] Furthermore, this invention designs a complete experience acquisition and storage mechanism. During the interaction between the intelligent controller and the controlled object, the current state, selected action, reward value, and next state are automatically recorded, forming experience tuples and appended to the experience replay buffer. This buffer adopts a circular structure, possessing capacity limiting and time series management functions, and can dynamically balance the ratio of old samples to new samples, providing timely and diverse data support for each round of training. During training, the system randomly samples batches of tuples from the experience pool, calculates the minimum action value as the target value through the target Q-network, and guides the main Q-network to update parameters in a minimum mean square error manner, thereby ensuring a clear gradient convergence direction during the training process and improving the network fitting accuracy.

[0022] Finally, after the main strategy network is trained, this invention deploys it into an embedded intelligent controller, supporting the reception of the current state and the output of control actions during real-time control, thus achieving efficient execution of the inference process. During system operation, new interactive data is continuously collected, and training trigger thresholds are set to achieve incremental policy updates and online self-optimization capabilities. The entire process forms a closed-loop mechanism from perception, inference, execution to retraining. The system can continuously adapt the strategy according to the operating state, achieving the control goal of "becoming more accurate with use," significantly improving the stability, flexibility, and robustness of the control system in long-term operation. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a self-optimization method for an intelligent controller based on reinforcement learning proposed in this invention. Figure 2 This is a schematic diagram of the improved TD3 control algorithm structure, which is based on a reinforcement learning-based intelligent controller self-optimization method proposed in this invention. Figure 3 This is a schematic diagram of the closed-loop structure of policy deployment, real-time inference, and online optimization in the controller operation process of the self-optimization method of intelligent controller based on reinforcement learning proposed in this invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0025] refer to Figure 1-3 A self-optimization method for intelligent controllers based on reinforcement learning includes the following steps: S1. Construct the state space, action space, and environmental feedback function of the control system. The state space includes current sensor measurements, historical control outputs, and error information. The action space is the set of control commands that the intelligent controller can adjust. The environmental feedback function quantifies each control action and provides an immediate feedback. S2. Based on the constructed state space and action space, initialize the improved TD3 control algorithm structure, including two master policy networks, two master Q networks, and corresponding target policy networks and target Q networks. The target network uses a delayed parameter synchronization mechanism to improve the stability of policy training. S3. Utilize the interaction process between the intelligent controller and the controlled object to collect the current state, selected action, obtained reward value, and next state generated by each interaction to form an experience tuple, which is stored in the experience replay buffer. S4. Randomly sample a batch of experience tuples from the experience replay buffer and input them into the two master Q networks. Based on the current master policy network, generate the action corresponding to the next state, input it into the target Q network, and take the minimum Q value as the target action value. Update the parameters of the two master Q networks using the minimum mean square error criterion. S5. Determine if the delayed update condition is met. If so, update the parameters of the main policy network based on the current output of the main Q network, and use the target policy network to generate the policy output in the next state, adding truncated Gaussian noise to enhance the exploratory nature of the policy. S6. Synchronously update the parameters of the target policy network and the target Q network, making them gradually approach the corresponding main network parameters, thereby improving convergence stability and control accuracy during policy training. S7. Deploy the trained main policy network to the intelligent controller, receiving the current state and outputting the optimal control action in the actual control task, realizing real-time closed-loop control of the target system. S8. Continuously collect interactive data during the operation of the intelligent controller and add it to the experience playback buffer, periodically triggering the incremental policy training process to achieve online optimization of the control policy and adaptive improvement of control performance.

[0026] This invention introduces an improved TD3 reinforcement learning structure to construct a complete self-optimizing control process, from state awareness, policy generation, Q-value evaluation, parameter updating to real-time deployment. This method breaks through the limitations of traditional controllers that rely on manual modeling and static parameter setting, enabling continuous learning and automatic evolution of control strategies in complex dynamic environments. It possesses full-process autonomy, real-time performance, and closed-loop optimization capabilities, significantly improving the response speed and control accuracy of the control system under varying operating conditions.

[0027] In this embodiment, the initialization of the improved TD3 control algorithm structure in S2 specifically includes: S21. Based on the structure of the state space and action space, the input dimension of the main policy network is set to the state vector dimension, and the output dimension is set to the control command dimension. The input dimension of the main Q network is set to the concatenation of the state vector and the action vector, and the output is the Q-value scalar. S22. Initialize two main policy networks and two main Q networks with identical structures but independent parameters. The main policy networks use a multilayer perceptron structure to output deterministic action values, and the main Q networks estimate action values ​​respectively. S23. Copy the two main policy networks to obtain two target policy networks, and copy the two main Q networks to obtain two target Q networks. The target network structure is consistent with the main network, but the parameters are updated independently. S24. Set a soft update coefficient and a synchronous update interval for each target network to ensure that it approximates the corresponding main network parameters at a fixed ratio during policy training. S25. Set the initial parameter values ​​of the main policy network and the main Q network. The parameters adopt a preset weight initialization scheme, and ensure that the initial parameter values ​​of the target networks are completely consistent with those of the main networks. S26. Configure the core hyperparameters of the TD3 control algorithm, including the maximum capacity of the experience replay buffer, the sampling batch size for each training round, the policy delay update steps, the reward discount factor, and the target smoothing noise amplitude; S27. Set the upper and lower limits of the control policy output to constrain the output range of the main policy network within the physical operation range allowed by the intelligent controller, and prevent illegal instruction output during the training process; S28. Start the improved TD3 control algorithm structure and keep all main networks and target networks in an initial synchronization state.

[0028] This invention constructs an improved TD3 control architecture with dual-network collaboration by precisely defining the input-output dimensions of the main policy network and the main Q-network, and establishing a target network system with independent parameters. Through initializing the policy boundary, optimizer parameters, and implementing a soft update mechanism, it provides stable structural support and numerical foundation for subsequent policy learning, effectively mitigating problems such as gradient explosion and unstable convergence in reinforcement learning, and ensuring that the entire control system can enter an efficient learning state early in the training process.

[0029] In this embodiment, S22 specifically includes: S221. Set the input dimension of the two master policy networks as the state vector dimension and the output dimension as the control command dimension. The master policy network maps the current state to a deterministic control action. S222. Construct a first master policy network and a second master policy network. Both have the same structure, including an input layer, multiple fully connected hidden layers, and an output layer. The output layer introduces a saturation function to limit the numerical range of the output action, satisfying the physical output boundary conditions of the controller. S223. Initialize the weights of the first master policy network and the second master policy network using different parameter initialization strategies to ensure that their parameters are independent and their learning paths are separated. S224. Construct a first master Q network and a second master Q network. Each master Q network receives a concatenated input of the current state vector and the corresponding control action vector. The output is an estimated value of the action in the current state-action pair, reflecting the long-term reward potential of the action in the current state. S225. The first and second master Q-networks adopt the same structural configuration, both containing an input layer, three hidden layers, and an output layer. The output layer outputs a single scalar value, representing the network's independent value judgment result for the same state-action pair, ensuring that the two master Q-networks do not produce parameter coupling when estimating the action value function. S226. The parameters of the first and second master Q-networks are initialized to different initial values, and each master Q-network is configured with an independent optimizer, learning rate, and regularization parameters, ensuring that they learn and estimate the action value function independently during the training phase, thereby effectively reducing the risk of overestimation of the value function. S227. Separate training scheduling modules are configured for the two master policy networks and the two master Q-networks. The training scheduling modules include parameter update strategies, gradient calculation processes, and training state buffering mechanisms, ensuring that all networks run independently and progress synchronously. S228. The initialization and construction of all master policy networks and master Q-networks are completed, establishing a data flow path for the TD3 control algorithm.

[0030] This invention employs independent initialization and separate training scheduling mechanisms for the two main policy networks and two main Q networks, ensuring that policy generation and value evaluation maintain computational path separation. The main Q networks independently estimate the value of the same state-action pair, improving the robustness of the estimation and significantly suppressing the Q-value overestimation problem common in TD3. Simultaneously, the policy network structure incorporates saturation function constraints, ensuring that control command outputs are bounded by physical boundaries, thus enhancing the controllability of control actions and the system's security.

[0031] In this embodiment, S3 specifically includes: S31. Set the interaction cycle of the intelligent controller. In each interaction cycle, the intelligent controller outputs a control action based on the current state input through the main policy network and applies the control action to the controlled object. S32. Collect the response of the controlled object after the current action is executed, record the state before the action is applied as the current state, and the state after the action is applied as the next state. S33. Evaluate the current state and the response result after the action is executed according to a preset environmental feedback function, calculate the reward value obtained by the current control behavior, and use it to provide feedback on the controller's behavior effect. S34. Organize the current state, the selected action, the obtained reward value, and the next state into an experience tuple in the form of a four-tuple according to a unified data structure. S35. Assign a timestamp identifier and execution index number to the experience tuple to ensure the traceability of the time sequence and the controllability of the sampling distribution during the training process. S36. Write the constructed experience tuple into the experience replay buffer. The experience replay buffer is a finite-length circular queue structure. When the storage space reaches its capacity limit, it automatically overwrites the oldest stored experience tuple. S37. Set the writing frequency, writing strategy and cache update mechanism of the experience playback buffer to ensure that key state areas can be covered and the ratio of new and old experience samples can be balanced in different control stages; S38. Maintain the asynchronous execution of experience tuple acquisition and experience playback buffer writing process during the continuous operation of the intelligent controller to form a complete interactive data stream, providing continuous, diverse and behavioral feedback learning samples for the subsequent training stage.

[0032] By constructing quadruple experience tuples, assigning timestamp indices, and introducing a finite-capacity circular replay buffer, this invention achieves an efficient, sequentially controllable, and broadly state-covered control interaction data acquisition mechanism. This structure avoids training imbalance caused by redundant sample accumulation while retaining the ability to rapidly update fresh behavioral data, providing policy networks with time-relevant and highly representative data support, thereby improving the generalization ability and adaptability of control policies in non-stationary systems.

[0033] In this embodiment, S4 specifically includes: S41. Randomly sample a set of empirical tuples without replacement from the empirical replay buffer according to a preset batch size; S42. Input the sampled current state and current action into two main Q networks, and calculate the estimated action value of the state-action pair under the current policy; S43. Input the sampled next state into the main policy network to generate the predicted action in the next state, as a behavior estimate for the next action; S44. Input the generated predicted action in the next state into two target Q networks, and calculate the target Q value output under the predicted action, obtaining two sets of corresponding action value estimates; S45. Compare the outputs of the two target Q networks, and select the minimum value as the current target action value.

[0034] in, The target Q-value of the master Q-network; Let be the instantaneous reward value at time t; This is a reward discount factor used to balance current and future rewards; , Two target Q-networks are used to estimate the action value in the next state; The next state; Actions generated for the target policy network; To truncate Gaussian noise perturbations and enhance policy robustness; To use the smaller target value in the dual Q-network; S46, use the difference between the selected target action value and the current action value calculated by the main Q-network as the error signal to construct the input term of the loss function of the main Q-network; S47, based on the minimum mean square error criterion, perform gradient backpropagation and optimization update on the parameters of the two main Q-networks respectively, so that their outputs gradually approach the target action value; S48, record the convergence rate, error magnitude and parameter change rate during each main Q-network parameter update process to provide training feedback for policy updates and synchronization with the target network.

[0035] This invention proposes taking the minimum output of two target Q-networks as the target action value, and combining it with the mean squared error loss of the main Q-network to construct the training target. This fundamentally suppresses the policy bias problem caused by Q-value overestimation during policy learning. The training mechanism supports independent updates of the two networks, avoiding information interference; at the same time, multi-round sampling improves the ability to approximate the expected policy value, making the control policy more conservative and stable, effectively supporting steady-state control tasks in complex environments.

[0036] In this embodiment, S5 specifically includes: S51. Set the delayed update frequency threshold for the main policy network and record the main policy network update counter value in the current training round; S52. Determine whether the current training round meets the delayed update condition of the main policy network. If not, skip the main policy network update step and only optimize the main Q network parameters; S53. If the delayed update condition is met, obtain the action value results output by the two main Q networks in the previous training round, and select the gradient feedback path corresponding to the first main Q network as the basis for updating the main policy network parameters; S54. Input the current state into the main policy network to obtain the control action output by the current policy, and concatenate this action with the current state as the input to the main Q network to calculate the action value under the current policy; S55. Construct the objective function of the main policy network:

[0037] in, The objective function value of the main policy network. The parameter set of the main policy network. This represents the batch sample size. The control actions generated by the main policy network. () represents the first master Q network.

[0038] Using the action value corresponding to the current policy output as the optimization direction, the parameters of the main policy network are adjusted through the backpropagation mechanism; S56, the target policy network is called with the next state in the current sampling batch as input to generate the target policy output action in the next state, which assists in updating the data stream required by the target Q network; S57, a truncated Gaussian noise perturbation is added to the output action of the target policy network. The noise parameters include mean, variance and upper and lower limits. After adding the perturbation, the action result is pruned to keep it within the action range allowed by the controller; S58, the target policy output after the noise processing is used to calculate the target Q value in the next round of training.

[0039] By setting a policy update delay step and using the main Q network feedback signal to update the main policy network, this invention achieves a high-quality, low-frequency policy update process, thereby reducing the uncertainty fluctuations caused by policy updates. Furthermore, the target policy output is superimposed with truncated Gaussian noise, and boundary pruning ensures output safety, further enhancing the policy's exploratory ability and robustness during the training phase, and improving the robustness and expected returns of the final deployed policy in actual execution.

[0040] In this embodiment, S6 specifically includes: S61. Set the parameter update method for the target policy network and the target Q network to a soft update mechanism, and configure the update frequency, synchronization coefficient, and initial synchronization state for each type of target network. S62. After each round of policy training, determine whether the target network synchronization update condition is met. If it is met, perform parameter synchronization operations for the target policy network and the target Q network respectively. S63. Read the latest parameter values ​​of the current main policy network and the two main Q networks, and construct their corresponding synchronization targets. S64. Perform a weighted combination of the parameters of each layer of the target policy network and the parameters of the corresponding main policy network, and gradually adjust the target network parameters using proportional interpolation to make them slowly follow the changes of the main network. S65. Perform soft synchronization updates on the parameters in the two target Q networks in the same way, keeping them relatively consistent with but not completely equal to the parameters of the first and second main Q networks. S66. Set a parameter drift protection mechanism during the synchronization update process to limit the maximum change range of a single parameter update and avoid instability of the target network due to drastic fluctuations in the main network. S67. Record the target network parameter state after each synchronization and input it into the inference input for target action value calculation and policy evaluation in the next round of training; S68. Execute the target network synchronization update process repeatedly during the continuous training phase of the intelligent controller, so that the target policy network and the target Q network always maintain the evolution process of consistent structure with the main network, asymptotic parameters, and controlled fluctuations, thereby improving the policy convergence speed and output stability in the overall training process.

[0041] A soft update strategy is employed to perform low-speed synchronization between the target policy network and the target Q network, effectively reducing policy oscillations caused by sudden changes in network parameters. This invention uses a controllable parameter interpolation method to gradually approximate the target network parameters to the main network state and sets a maximum parameter drift limit, thereby improving the stability of the training process from a mechanistic perspective. This strengthens the target network's support for policy evaluation and stable behavioral output, providing a foundation for consistently consistent policy output in high-frequency control systems.

[0042] In this embodiment, S7 specifically includes: S71. After the main strategy network is trained and reaches the preset convergence index, export all network structures and parameter configurations to generate a deployable control strategy model file; S72. Embed the exported main strategy network into the inference module of the intelligent controller in compiled form, and configure the model loading process and interface protocol to support real-time state input and action output; S73. Construct the state perception channel of the intelligent controller, connect the sensor sampling module, actuator feedback module, and control parameter recording module to ensure that the current system state can be collected in real time and standardized into the input format of the main strategy network; S74. Start the main strategy network inference service during the control task initialization phase, read the current state vector in each control cycle and input it into the main strategy network; S75. The main strategy network outputs the optimal control action according to the current state. The optimal control action is directly transmitted to the actuator or scheduling module as a control command to realize the instruction issuance of the physical control process; S76. After the control action is executed, sample and record the actual response, and transmit the execution feedback back to the controller for subsequent control cycle state updates; S77. The intelligent controller continuously executes the state acquisition, action generation and response feedback process to construct a closed-loop control structure with the main strategy network as the core: state perception - strategy reasoning - instruction output; S78. During the entire closed-loop control operation, the execution performance indicators of the main strategy network are continuously monitored, including output delay, control error and stability fluctuation, to ensure that the controller maintains high response accuracy and stable control effect in actual application scenarios.

[0043] This invention exports and embeds the master policy network into an intelligent controller after training. By constructing a real-time control closed loop of state perception, action generation, execution response, and feedback update, it achieves an efficient transition from learning-based control strategies to engineering-level deployment. The deployment strategy features structural compression, high operational efficiency, and precise action response. Combined with standardized processing interfaces for the state channel, the controller possesses plug-and-play deployment capabilities and precise execution capabilities for complex tasks, significantly improving the overall intelligence level of the system.

[0044] In this embodiment, S8 specifically includes: S81. During the actual operation of the intelligent controller, record the current state, selected action, reward obtained, and next state after system response for each control action, forming a new experience tuple; S82. Append the newly generated experience tuple to the experience replay buffer and arrange them in order according to timestamps to maintain the continuity and retrieval of experience samples in time; S83. When the number of new samples in the experience replay buffer reaches the set incremental threshold, automatically trigger the policy incremental training module to execute the parameter update task; S84. Extract a batch of samples containing a mixture of old and new samples from the experience replay buffer to ensure that the policy network takes into account both historical control experience and the latest interaction information during training; S85. Use the current main policy network and main Q network to train the sampled experience tuples and update the parameters to adapt to the control feedback characteristics under the latest operating environment; S86. According to the frequency and step size set in the policy training cycle, dynamically adjust the sample ratio and target value update strategy for each round of training to achieve long-term optimization of training resource scheduling; S87. After each policy incremental training is completed, determine whether to update the parameters of the target policy network and the target Q network synchronously, and record the convergence performance index of the current training cycle; S88. During the continuous operation of the intelligent controller, maintain the cyclic execution mechanism of experience acquisition, experience playback buffer writing and policy incremental training to realize online optimization of control policy and adaptive continuous improvement of controller performance.

[0045] By employing a continuous data acquisition and incremental strategy training mechanism during controller operation, this invention achieves online optimization of the control strategy. This mechanism supports self-updating of interactive data, dynamic judgment of training trigger conditions, and periodic synchronization of the target network, effectively addressing the problems of control strategy degradation and decreased adaptive capability over time. The controller can continuously adjust parameters based on the current environment and historical behavior, thereby maintaining control performance and enhancing the system's self-evolutionary capabilities over long-term operation.

[0046] Example 1: To verify the feasibility of this invention in practice, it was applied to the dynamic control scenario of a high-precision servo system. Traditional PID controllers, due to their fixed parameters and difficulty in adaptive adjustment, exhibit response lag and over-adjustment when faced with task changes and system model disturbances. To improve the system's response speed and control accuracy, the research team deployed the intelligent controller self-optimization method based on the improved TD3 algorithm proposed in this invention at the control end of the servo system. The system, through the installation of state sensors, actuator feedback modules, and a policy inference engine, realizes the complete process of state acquisition, policy training, online deployment, and control closed loop.

[0047] In actual deployment, the research team set the control cycle to 0.5 seconds. Each cycle, the system's current position, velocity, and control error information were read in real time and fed into the trained main policy network for inference to generate control commands. Simultaneously, the controller continuously wrote information such as the current state, control actions, and response feedback into the experience buffer. Every 300 new samples, an incremental policy training process was triggered to update the parameters of the main policy and the Q-network, and to synchronize the target network, ensuring that the policy has dynamic adjustment capabilities.

[0048] To verify the effectiveness of the method proposed in this invention, the research team conducted parallel tests on five control strategies on the same experimental platform: a traditional PID controller, a genetic algorithm-optimized controller, a deep Q-network (DQN), the original TD3 method, and the improved TD3 controller proposed in this invention. The testing period was from March to May 2025, at an industrial automation testing base in Guangdong Province. Multiple rounds of experiments were conducted under uniform operating conditions. After each controller group ran for 1000 rounds, its core indicators, such as average steady-state error, response time, maximum overshoot, required number of training samples, and controller update cycle, were statistically analyzed. The test results are shown in Table 1.

[0049] Table 1: Comparison of Test Results

[0050] As can be seen from the table, the improved TD3 controller proposed in this invention outperforms other control methods in all performance indicators. Firstly, regarding the average steady-state error, the improved TD3 method achieves only 1.27%, a significant reduction compared to traditional PID (6.12%) and genetic algorithm-optimized controllers (4.57%), indicating stronger steady-state accuracy control capability. Secondly, in terms of system response time, the method of this invention provides a faster control response, completing stable control in just 2.9 seconds, approximately 0.8 seconds shorter than the original TD3. Furthermore, the overshoot control capability is also significantly enhanced, at only 3.2%, far lower than the 12.5% ​​of the traditional method.

[0051] It is worth mentioning that, regarding the number of training samples, the improved TD3 avoids the large training burden required by the original TD3 through a soft synchronization and delay strategy mechanism, effectively reducing the sample requirement (42,000 sets vs. 50,000 sets) while maintaining performance. Finally, regarding the controller update cycle, traditional control methods all use a fixed frequency update, while the method of this invention supports adaptive strategy optimization, dynamically adjusting the update frequency to the optimal level (average 2.5 seconds), improving the strategy convergence speed while ensuring the stability of the control system.

[0052] In summary, this invention offers significant advantages in addressing the problems of traditional controllers' inability to dynamically adapt to complex system disturbances, Q-value overestimation, and control instability. Its proposed intelligent controller self-optimization mechanism is widely applicable to various highly dynamic application scenarios such as industrial automation, robot control, and new energy management. Further extensions to distributed control systems are possible, enabling collaborative learning and intelligent evolution among multiple control units.

[0053] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A self-optimization method for intelligent controllers based on reinforcement learning, characterized in that, Includes the following steps: S1. Construct the state space, action space, and environmental feedback function of the control system. The state space includes current sensor measurements, historical control outputs, and error information. The action space is the set of control commands that the intelligent controller can adjust. The environmental feedback function quantifies each control action and provides an immediate feedback. S2. Based on the constructed state space and action space, initialize the improved TD3 control algorithm structure. S3. Utilize the interaction process between the intelligent controller and the controlled object to collect the current state, selected action, obtained reward value, and next state generated by each interaction, forming an experience tuple, which is stored in the experience playback buffer. S4. Randomly sample a batch of experience tuples from the experience playback buffer and input them into the two main Q networks. Generate the action corresponding to the next state based on the current main strategy network, input it into the target Q network, and take the minimum Q value as the target value. Update the parameters of the two main Q networks using the minimum mean square error criterion. S5. Determine whether the delayed update condition is met. If it is, update the parameters of the main strategy network once based on the output of the current main Q network, and use the target strategy network to generate the strategy output in the next state, adding truncated Gaussian noise. S6. Synchronously update the parameters of the target strategy network and the target Q network. S7. Deploy the trained master policy network to the intelligent controller, receive the current state and output the optimal control action in the actual control task; S8. Continuously collect interaction data during the operation of the intelligent controller and add it to the experience playback buffer, and periodically trigger the policy incremental training process.

2. The self-optimization method for an intelligent controller based on reinforcement learning according to claim 1, characterized in that, The initialization of the improved TD3 control algorithm structure in S2 specifically includes: S21. Based on the structure of the state space and action space, the input dimension of the main policy network is set to the state vector dimension, and the output dimension is set to the control command dimension. The input dimension of the main Q network is set to the concatenation of the state vector and the action vector, and the output is the Q-value scalar. S22. Initialize two main policy networks and two main Q networks with identical structures but independent parameters. The main policy networks use a multilayer perceptron structure to output deterministic action values, and the main Q networks estimate action values ​​respectively. S23. Copy the two main policy networks to obtain two target policy networks, and copy the two main Q networks to obtain two target Q networks. The target network structures are consistent with the main networks, but the parameters are updated independently. S24. Set a soft update coefficient and a synchronous update interval for each target network. S25. Set the initial parameter values ​​for the main policy network and the main Q network. The parameters use a preset weight initialization scheme. S26. Configure the core hyperparameters of the TD3 control algorithm, including the maximum capacity of the experience replay buffer, the sampling batch size for each training round, the policy delay update steps, the reward discount factor, and the target smoothing noise amplitude; S27. Set the upper and lower limits of the control policy output to constrain the output range of the main policy network within the physical operation range allowed by the intelligent controller; S28. Start the improved TD3 control algorithm structure and keep all main networks and target networks in an initial synchronization state.

3. The self-optimization method for an intelligent controller based on reinforcement learning according to claim 2, characterized in that, S22 specifically includes: S221. Set the input dimension of the two master policy networks as the state vector dimension and the output dimension as the control command dimension. The master policy network maps the current state to a deterministic control action. S222. Construct a first master policy network and a second master policy network. Both have the same structure, including an input layer, multiple fully connected hidden layers, and an output layer. The output layer introduces a saturation function to limit the numerical range of the output action, satisfying the physical output boundary conditions of the controller. S223. Initialize the weights of the first master policy network and the second master policy network using different parameter initialization strategies. S224. Construct a first master Q network and a second master Q network. Each master Q network receives a concatenated input of the current state vector and the corresponding control action vector, and outputs an estimated value of the action for the current state-action pair. S225. The first master Q network and the second master Q network adopt the same structural configuration, both including an input layer, three hidden layers, and an output layer. The output layer outputs a single scalar value, representing the network's independent value judgment result for the same state-action pair. S226. Initialize the parameters of the first master Q network and the second master Q network to different initial values, and configure an independent optimizer, learning rate and regularization parameters for each master Q network; S227. Configure the training scheduling module for each of the two master policy networks and the two master Q networks. The training scheduling module includes a parameter update strategy, gradient calculation process and training state buffer; S228. Complete the initialization and construction of all master policy networks and master Q networks, and establish a data flow path for the TD3 control algorithm.

4. The self-optimization method for an intelligent controller based on reinforcement learning according to claim 1, characterized in that, S3 specifically includes: S31. Set the interaction cycle of the intelligent controller. In each interaction cycle, the intelligent controller outputs a control action through the main strategy network based on the current state input and applies the control action to the controlled object. S32. Collect the response of the controlled object after the current action is executed, record the state before the action is applied as the current state, and the state after the action is applied as the next state. S33. Evaluate the current state and the response result after the action is executed according to a preset environmental feedback function, and calculate the reward value obtained by the current control behavior. S34. Organize the current state, the selected action, the obtained reward value, and the next state into an experience tuple in the form of a four-tuple according to a unified data structure. S35. Assign a timestamp and execution index number to the experience tuple. S36. Write the constructed experience tuple into the experience replay buffer. The experience replay buffer is a finite-length circular queue structure. When the storage space reaches its capacity limit, it automatically overwrites the oldest stored experience tuple. S37. Set the writing frequency, writing strategy, and cache update mechanism of the experience replay buffer. S38. During the continuous operation of the intelligent controller, the asynchronous execution of the experience tuple acquisition and experience playback buffer writing process is maintained to form a complete interactive data stream.

5. The self-optimization method for an intelligent controller based on reinforcement learning according to claim 1, characterized in that, S4 specifically includes: S41. Randomly sample a set of empirical tuples without replacement from the empirical replay buffer according to a preset batch size; S42. Input the sampled current state and current action into two main Q networks, and calculate the estimated action value of the state-action pair under the current policy; S43. Input the sampled next state into the main policy network to generate the predicted action in the next state; S44. Input the generated predicted action in the next state into two target Q networks, and calculate the target Q value output under the predicted action to obtain two sets of corresponding action value estimates; S45. Compare the output results of the two target Q networks and select the minimum value as the current target action value; S46. Use the difference between the selected target action value and the current action value calculated by the main Q network as the error signal to construct the input term of the loss function of the main Q network; S47. Based on the minimum mean square error criterion, perform gradient backpropagation and optimization update on the parameters of the two main Q networks; S48. Record the convergence rate, error magnitude and parameter change rate during each main Q network parameter update process to provide training feedback for policy update and target network synchronization.

6. The self-optimization method for an intelligent controller based on reinforcement learning according to claim 1, characterized in that, S5 specifically includes: S51. Set the delayed update frequency threshold for the main policy network and record the main policy network update counter value in the current training round. S52. Determine whether the current training round meets the delayed update condition of the main policy network. If not, skip the main policy network update step and only optimize the parameters of the main Q network. S53. If the delayed update condition is met, obtain the action value results of the two main Q networks output in the previous training round and select the gradient feedback path corresponding to the first main Q network as the basis for updating the parameters of the main policy network. S54. Input the current state into the main policy network to obtain the control action output by the current policy, and concatenate the action with the current state as the input of the main Q network to calculate the action value under the current policy. S55. Construct the objective function of the main policy network, take the action value corresponding to the current policy output as the optimization direction, and adjust the parameters of the main policy network through the backpropagation mechanism. S56. Call the target policy network with the next state in the current sampling batch as input to generate the target policy output action in the next state, and assist in updating the data stream required by the target Q network. S57. Add truncated Gaussian noise perturbation to the output action of the target policy network. The noise parameters include mean, variance and upper and lower limits. After adding the perturbation, the action result is truncated. S58. Output the target policy after adding noise.

7. The self-optimization method for an intelligent controller based on reinforcement learning according to claim 1, characterized in that, S6 specifically includes: S61. Set the parameter update method for the target policy network and the target Q network as a soft update mechanism, and configure the update frequency, synchronization coefficient, and initial synchronization state for each type of target network. S62. After each round of policy training, determine whether the target network synchronization update condition is met. If it is met, perform parameter synchronization operations for the target policy network and the target Q network respectively. S63. Read the latest parameter values ​​of the current main policy network and the two main Q networks, and construct their corresponding synchronization targets. S64. Perform weighted combination of the parameters of each layer of the target policy network and the parameters of the corresponding main policy network, and gradually adjust the target network parameters using proportional interpolation. S65. Perform soft synchronization update on the parameters in the two target Q networks in the same way, keeping them relatively consistent with but not completely equal to the parameters of the first and second main Q networks. S66. Set a parameter drift protection mechanism during the synchronization update process to limit the maximum change range of a single parameter update. S67. Record the target network parameter state after each synchronization and input it into the inference input for target action value calculation and policy evaluation in the next round of training. S68. During the continuous training phase of the intelligent controller, the synchronous update process of the target network is executed cyclically to ensure that the target policy network and the target Q network always maintain an evolutionary process that is consistent with the main network structure, with asymptotic parameters and controlled fluctuations.

8. The self-optimization method for an intelligent controller based on reinforcement learning according to claim 1, characterized in that, Specifically, S7 includes: S71. After the main strategy network is trained and reaches the preset convergence index, export all network structures and parameter configurations to generate a deployable control strategy model file; S72. Embed the exported main strategy network into the inference module of the intelligent controller in compiled form, and configure the model loading process and interface protocol; S73. Construct the state perception channel of the intelligent controller and connect the sensor sampling module, actuator feedback module, and control parameter recording module; S74. Start the main strategy network inference service during the control task initialization phase, read the current state vector in each control cycle and input it into the main strategy network; S75. The main strategy network outputs the optimal control action according to the current state, and the optimal control action is directly transmitted to the actuator or scheduling module as a control command; S76. After the control action is executed, sample and record the actual response and transmit the execution feedback back to the controller; S77. The intelligent controller continuously executes the state acquisition, action generation, and response feedback process to construct a closed-loop control structure with the main strategy network as the core: state perception—strategy inference—command output; S78. During the entire closed-loop control operation, continuously monitor the execution performance indicators of the main strategy network, including output delay, control error, and stability fluctuations.

9. The self-optimization method for an intelligent controller based on reinforcement learning according to claim 1, characterized in that, S8 specifically includes: S81. During the actual operation of the intelligent controller, record the current state, selected action, reward obtained, and next state after system response for each control action to form a new experience tuple; S82. Append the newly generated experience tuple to the experience replay buffer and arrange them in order according to timestamps to maintain the continuity and retrieval of experience samples in time; S83. When the number of new samples in the experience replay buffer reaches the set incremental threshold, automatically trigger the policy incremental training module to execute the parameter update task; S84. Extract a batch of samples containing a mixture of new and old samples from the experience replay buffer; S85. Train the sampled experience tuples using the current main policy network and main Q network to update the parameters to adapt to the control feedback characteristics under the latest operating environment; S86. Dynamically adjust the sample ratio and target value update strategy for each training round according to the frequency and step size set in the policy training cycle; S87. After each policy incremental training is completed, determine whether to synchronously update the parameters of the target policy network and target Q network, and record the convergence performance index of the current training cycle; S88. During the continuous operation of the intelligent controller, a cyclic execution mechanism is maintained for experience acquisition, experience playback buffer writing, and policy incremental training.

Images