Deep reinforcement learning active control parameter adjustment method considering random time delay

Abstract

A deep reinforcement learning active control parameter adjustment method considering random time delay comprises the steps of determining a controlled structure model and an actuator installation position, and establishing a structure vibration control kinetic equation; determining a preliminary controller; random time delay is added in the actuator to simulate a real working environment, and a two-way communication environment is established; responding to the set feedback function by using the structure collected by the sensor; determining a deep reinforcement learning algorithm, a neural network framework and training parameters, and establishing an interactive interface; training a deep reinforcement learning agent in the interactive interface, and feeding back a function value through accumulated reward expectation of a deep reinforcement learning algorithm to carry out strategy updating iteration so as to obtain a mature agent; a mature intelligent agent is used as a parameter regulator in a preliminary controller, parameter values are regulated in real time, and vibration control of the structure under random time lag is achieved. According to the method, the robustness of the controller in a complex environment is ensured, and the method belongs to the field of structural vibration control.

Description

Technical Field

[0001] The present invention relates to the field of structural vibration control, and in particular to a deep reinforcement learning active control parameter adjustment method considering random time delay. Background Art

[0002] Active structural control is the best way to suppress structural response. It has the unique advantage of dynamically adjusting structural response according to user needs. Most of the existing active control methods are generally limited by their strong dependence on the accuracy of the structural dynamics model. In particular, when the service life of the building is extended and the structural parameters undergo irreversible shifts, the mathematical model will inevitably deviate from the initial design state, which poses a severe test for the controller built based on classical control theory. In addition, current methods often require the derivation of optimal control analytical solutions based on precise system descriptions, and the time delay in the control loop can easily cause phase offset problems, which not only weakens the control efficiency, but also may induce the risk of system instability. Faced with the increasing complexity and time-varying characteristics of modern building systems, traditional control frameworks also pose challenges in parameter fault tolerance and time delay compensation.

[0003] Currently, the parameters of most active control methods are fixed after the design is completed, such as the Q and R parameters in the linear quadratic Gaussian control algorithm. This parameter fixation characteristic can easily lead to a mismatch between the control gain and the system's dynamic characteristics when faced with unexpected external disturbances and random time-delay conditions of the actuator, resulting in a weakening of the system's anti-interference ability and an increased risk of instability. Conventional neural network control and adjustment methods, such as fuzzy networks, still rely on high-dimensional training data sets for network architecture optimization. In actual engineering scenarios, it is difficult to obtain complete samples. Such methods often face the severe challenge of insufficient model generalization ability, resulting in significant deviations between actual control accuracy and theoretical design indicators. Summary of the Invention

[0004] In order to improve the robustness of the active control algorithm under random time delay and external disturbance, the present invention proposes a deep reinforcement learning active control parameter adjustment method considering random time delay.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for adjusting active control parameters of deep reinforcement learning considering random time delay includes the following steps:

[0007] Step 1: Determine the controlled structure model, actuator installation position, and establish the structural vibration control dynamics equation;

[0008] Step 2: Determine the preliminary controller designed according to the precise mathematical model;

[0009] Step 3: Add random time delay to the actuator to simulate a real working environment and establish a two-way communication environment between the controlled structure and the actuator; use the structural response collected by the sensor to set an appropriate feedback function;

[0010] Step 4: Determine the deep reinforcement learning algorithm, its neural network framework, and training parameters, and build an interactive interface between the environment and the deep reinforcement learning agent.

[0011] Step 5: Train the deep reinforcement learning agent in the interactive interface constructed in step 4. Update the strategy iteratively based on the cumulative reward expectation and feedback function value of the deep reinforcement learning algorithm to obtain a mature agent that meets user needs.

[0012] Step 6: Use the mature intelligent agent obtained in step 5 as the parameter regulator in the preliminary controller designed in step 2, and adjust the parameter values in real time according to the changes in the environment, so as to achieve vibration control of the structure under random time delay conditions.

[0013] As a preferred embodiment, in step 1, the structural vibration control dynamics equation is:

[0014]

[0015] Among them, M, K, and C are the structural mass, stiffness matrix, and damping matrix respectively; and X vectors represent the acceleration, velocity, and displacement of the structure, respectively; U represents the seismic record and active control force; I and H represent the unit column vector and the active control device layout matrix, respectively.

[0016] As a preferred embodiment, in step 2, the structural vibration control dynamics equation established in step 1 is rewritten into a state space equation as shown below, and an LQG controller is designed based on the state space equation.

[0017]

[0018] in

[0019] As a preferred embodiment, in step 3, the structural dynamic response collected by the sensor is used as the observation value of the intelligent agent, and the feedback function value is calculated; the feedback function established is:

[0020]

[0021] Among them, ζ=1000 is the parameter of the feedback function.

[0022] As a preferred embodiment, in step 4, the SAC algorithm is used as the training algorithm. The SAC framework includes a policy network, an action network, an action value function and a state value function.

[0023] As a preferred method, in step five, the SAC algorithm is used as the core training algorithm for the intelligent agent; the intelligent agent in deep reinforcement learning is used as a controller to interact with the environment built in step two; during the training process, the training data is stored in the experience pool of the SAC algorithm; the intelligent agent's strategy is updated and iterated through the cumulative reward expectation and feedback function to obtain a mature intelligent agent.

[0024] As a preferred embodiment, in step six, the mature intelligent agent obtained in step five is embedded in the structural dynamics system to act as a parameter regulator of the LQG controller; when the controlled structure is excited by external loads and the actuator itself has random time delays, the intelligent agent can adjust the values of the parameters Q and R of the LQG controller in real time according to the dynamic response of the structure.

[0025] The principle of the present invention is:

[0026] A deep reinforcement learning-based active control parameter adjustment method that considers stochastic time delay utilizes a deep reinforcement learning algorithm to train a parameter adjustment agent that works in conjunction with a traditional controller. This parameter adjustment method establishes a nonlinear mapping relationship between time-varying parameters and control objectives based on real-time system observations, enabling adaptive optimization of the controller gain matrix. This significantly improves the control system's dynamic adjustment capabilities in terms of time-varying disturbance rejection and time-varying time delay.

[0027] The present invention has the following advantages:

[0028] 1. Using deep reinforcement learning algorithms to train intelligent agents does not rely on sophisticated dynamics models or require a large amount of upfront data.

[0029] 2. Using the cumulative reward expectation and feedback function value of the deep reinforcement learning algorithm to adjust the controller parameter values in real time, to a certain extent, it can ensure the robustness of the controller in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a framework diagram of the active control parameter adjustment method of the present invention.

[0031] Figure 2 It is a flow chart of the active control parameter adjustment method of the present invention.

[0032] Figure 3 It is the time history diagram of the inter-story displacement response of the first floor of the structure.

[0033] Figure 4is the time history diagram of the top acceleration response of the structure.

[0034] Figure 5 It is a time chart of control force. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below with reference to specific implementation methods.

[0036] The present invention provides a deep reinforcement learning active control parameter adjustment method considering random time delay, the control framework is as follows Figure 1 As shown, it includes a controlled structure, a sensor, an active control device, a controller, and a parameter adjuster based on deep reinforcement learning. The active control parameter adjustment method of the present invention includes the following steps 1 to 6. The specific flow chart is shown in Figure 2 The specific steps are as follows.

[0037] (1) Step 1: Determine the controlled structure model, actuator installation location, and establish the structural vibration control dynamics equation. The details are as follows.

[0038] In this embodiment, the controlled structure is a three-story frame building structure, and the specific structural parameters are shown in Table 1. An active control device is installed on the top of the structure, and the vibration control dynamic equation of the structure is established as follows:

[0039]

[0040] Among them, M, K, and C are the structural mass, stiffness matrix, and damping matrix respectively; and X vectors represent the acceleration, velocity, and displacement of the structure, respectively; U represents the seismic record and active control force; I and H represent the unit column vector and the active control device layout matrix, respectively.

[0041] Table 1 Structural information

[0042]

[0043] (2) Step 2: Determine the preliminary controller designed according to the precise mathematical model. The details are as follows.

[0044] In this embodiment, the structural vibration control dynamics equation established in step 1 is rewritten as a state-space equation as shown below; and an LQG controller is designed based on the state-space equation.

[0045]

[0046] in

[0047] (3) Step 3: Add random time delay to the actuator to simulate a real working environment, establish a two-way communication environment between the controlled structure and the actuator, and use the structural response collected by the sensor to set an appropriate feedback function. The details are as follows.

[0048] In this embodiment, a random time delay of 0-10 times the sampling time is added to the actuator working environment, and a two-way communication environment is established between the controlled structure obtained in steps 1 and 2 and the active control device actuator. The structural dynamic response collected by the sensor is used as the observation value of the intelligent agent, and the feedback function value is calculated. The feedback function established in this embodiment is:

[0049]

[0050] Among them, ζ=1000 is the parameter of the feedback function.

[0051] (4) Step 4: Determine the deep reinforcement learning algorithm, its neural network framework, and training parameters, and build an interactive interface between the environment and the deep reinforcement learning agent. The details are as follows.

[0052] In this embodiment, the Soft Actor-Critic (SAC) algorithm is used as the training algorithm. The SAC framework includes a policy network, an action network, an action value function, and a state value function. The hyperparameter settings involved are shown in Table 2.

[0053] Table 2. Hyperparameter settings of the SAC algorithm

[0054]

[0055] (5) Step 5: Train the deep reinforcement learning agent in the interactive interface constructed in Step 4. Update the strategy iteratively using the cumulative reward expectation and feedback function value of the deep reinforcement learning algorithm to obtain a mature agent that meets user needs. The details are as follows.

[0056] In this embodiment, the SAC algorithm is used as the core training algorithm for the agent. The deep reinforcement learning agent acts as a controller to interact with the environment established in step 2. During training, the training data is stored in the SAC algorithm's experience pool. The agent's strategy is iteratively updated through the cumulative reward expectation and feedback function, allowing the agent to maintain good parameter adjustment performance in complex environments, thereby achieving a mature agent.

[0057] (6) Step 6: Use the mature intelligent agent obtained in Step 5 as the parameter regulator in the preliminary controller designed in Step 2. Adjust the parameter values in real time according to environmental changes, thereby achieving vibration control of the structure under random time delay conditions. The details are as follows.

[0058] In this embodiment, according to the set step six, the mature intelligent agent obtained in step five is embedded in the structural dynamics system to act as a parameter regulator of the LQG controller; when the controlled structure is excited by external loads and the actuator itself has random time delays, the intelligent agent can adjust the values of the parameters Q and R of the LQG controller in real time according to the dynamic response of the structure, thereby ensuring the robustness and efficiency of the active control system in the working environment.

[0059] In this example, the building structure is subjected to external ground motion records from the classic 1940 El Centro earthquake. Following the above steps, a deep reinforcement learning active control parameter adjustment method considering random time delays can be developed. The following is an analysis of the control effect.

[0060] In this embodiment, compared with the traditional LQG control algorithm with fixed parameters, the structural dynamic response obtained in the case of random time lag in this embodiment is divergent and the control performance is unstable; however, the method of adjusting the LQG controller parameters Q and R using the SAC parameter adjustment method proposed in the present invention still has excellent control effects. In the uncontrolled state, the inter-story displacement peak is 0.0045m, and the maximum acceleration is 2.1683m / s; in the case of random time lag in the control device, the LQG control algorithm with fixed parameters is invalid, and its maximum control force has reached the actuator limit, and the structure is in an unstable state; after embedding the SAC parameter adjuster of the present invention, the LQG control effect is obvious, and its maximum inter-story displacement is 0.0023m, which is 48.89% of that of the uncontrolled state; and the acceleration peak is 1.6691m / s, which is 23.02% of that of the uncontrolled state. Figure 3 It shows the time history diagram of the inter-story displacement response of the first floor of the structure; Figure 4 It shows the time history diagram of the top acceleration response of the structure; Figure 5 It represents the time course of control force.

[0061] The traditional active control algorithm is based on a deterministic and precise model to build a static parameter configuration, and its control gain remains constant throughout the life cycle of the system. When faced with time-varying and time-delayed working conditions or encountering non-matching external disturbances, it is easy to cause phase margin attenuation and insufficient gain margin problems, resulting in the control effect may not meet the user's expectations, and even cause structural instability; although the traditional neural network control regulator has a certain nonlinear approximation capability, it is limited by the difficulty of obtaining sample sets and the model generalization capability. The deep reinforcement learning active control parameter adjustment method considering random time delay proposed in the present invention does not rely on a precise mathematical model. It constructs a real-time mapping relationship between state-action strategies and uses the information generated by environmental interactions to perform policy gradient updates, avoiding the strong demand of traditional methods for large-scale prior data; at the same time, it realizes real-time adjustment of the parameter prediction value of the intelligent agent (regulator), thereby ensuring the control performance of the active controller in a complex environment to a certain extent, and ensuring the robustness of the active control system.

[0062] This invention constructs a deep reinforcement learning active control parameter adjustment method that accounts for random time delay. By embedding a parameter adjuster, it achieves efficient control of active control systems in the presence of random time delay and external disturbances in actuators. This invention is not limited to random time delay or the LQG algorithm; other deep reinforcement learning active control parameter adjusters involving external disturbances, actuator failures, structural parameter errors, and other controller parameter adjustment issues are also covered by the scope of this invention.

[0063] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for adjusting active control parameters of deep reinforcement learning considering random time delay, characterized in that: The steps include: Step 1: Determine the controlled structure model, actuator installation position, and establish the structural vibration control dynamics equation; Step 2: Determine the preliminary controller designed according to the precise mathematical model; Step 3: Add random time delay to the actuator to simulate a real working environment and establish a two-way communication environment between the controlled structure and the actuator; use the structural response collected by the sensor to set an appropriate feedback function; Step 4: Determine the deep reinforcement learning algorithm, its neural network framework, and training parameters, and build an interactive interface between the environment and the deep reinforcement learning agent. Step 5: Train the deep reinforcement learning agent in the interactive interface constructed in step 4. Update the strategy iteratively based on the cumulative reward expectation and feedback function value of the deep reinforcement learning algorithm to obtain a mature agent that meets user needs. Step 6: Use the mature intelligent agent obtained in step 5 as the parameter regulator in the preliminary controller designed in step 2, and adjust the parameter values in real time according to the changes in the environment, so as to achieve vibration control of the structure under random time delay conditions.

2. A method for adjusting active control parameters of deep reinforcement learning considering random time delay according to claim 1, characterized in that: In step 1, the structural vibration control dynamic equation is: Among them, M, K, and C are the structural mass, stiffness matrix, and damping matrix respectively; and X vectors represent the acceleration, velocity, and displacement of the structure, respectively; U represents the seismic record and active control force; I and H represent the unit column vector and the active control device layout matrix, respectively.

3. The method for adjusting active control parameters of deep reinforcement learning considering random time delay according to claim 2, characterized in that: In step 2, the structural vibration control dynamics equation established in step 1 is rewritten into a state-space equation as shown below, and the LQG controller is designed based on the state-space equation. in 4. The method for adjusting active control parameters of deep reinforcement learning considering random time delay according to claim 3, characterized in that: In step 3, the structural dynamic response collected by the sensor is used as the observation value of the intelligent agent, and the feedback function value is calculated; the feedback function established is: Among them, ζ=1000 is the parameter of the feedback function.

5. The method for adjusting active control parameters of deep reinforcement learning considering random time delay according to claim 4, characterized in that: In step 4, the SAC algorithm is used as the training algorithm. The SAC framework includes policy network, action network, action value function and state value function.

6. A method for adjusting active control parameters of deep reinforcement learning considering random time delay according to claim 5, characterized in that: In step five, the SAC algorithm is used as the core training algorithm for the intelligent agent; the intelligent agent in deep reinforcement learning is used as a controller to interact with the environment built in step two; during the training process, the training data is stored in the experience pool of the SAC algorithm; the intelligent agent's strategy is updated and iterated through the cumulative reward expectation and feedback function to obtain a mature intelligent agent.

7. The method for adjusting active control parameters of deep reinforcement learning considering random time delay according to claim 6, characterized in that: In step six, the mature intelligent agent obtained in step five is embedded in the structural dynamics system to act as a parameter regulator for the LQG controller. When the controlled structure is excited by external loads and the actuator itself has random time delays, the intelligent agent can adjust the values of the LQG controller parameters Q and R in real time according to the dynamic response of the structure.

Images