A frequency-unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels
By combining the backstepping control inner loop and the Q-table reinforcement learning outer loop, the problem of frequency shift in wind tunnel forced vibration tests was solved, frequency self-tuning and vibration energy optimization were achieved, and the adaptive capability and energy utilization of the vibration system were improved.
Patent Information
- Application Number
- CN202611579483.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-13
- Publication Date
- 2026-11-10
AI Technical Summary
In wind tunnel forced vibration tests, existing technologies suffer from frequency shifts that lead to a decrease in vibration amplitude and energy utilization. Furthermore, existing reinforcement learning methods struggle to effectively find the optimal driving frequency under varying parameter conditions, resulting in evaluation bias and condition migration issues.
The inner loop of backstepping control is used to update the estimated values of equivalent mass, equivalent damping and equivalent stiffness. Combined with the frequency decision outer loop of Q-table reinforcement learning, the acceleration data is converted into the frequency-unbiased equivalent root mean square value of displacement. The Actor-Critic student decision-maker is used for autonomous closed-loop trajectory training to achieve frequency self-tuning and vibration energy optimization.
It enables unbiased comparison of vibration response and unified evaluation of mechanical energy changes under varying parameter conditions, improves the continuity and adaptability of frequency search, and ensures efficient resonance performance of the vibration system under different operating conditions.
Smart Images

Figure CN123065710A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of frequency self-tuning control of high-frequency vibration test equipment, and relates to a frequency unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels. Background Technology
[0002] In the dynamic stability analysis and aerodynamic characteristic identification of aircraft, dynamic derivatives are usually measured using wind tunnel forced vibration tests. During the test, the actuator drives the model to produce small-amplitude, high-frequency periodic motions, and the corresponding dynamic aerodynamic parameters are then identified based on the model's surface pressure, aerodynamic torque, or acceleration response.
[0003] Piezoelectric ceramic actuators offer fast response times and high output forces, making them suitable as excitation sources for high-frequency vibration devices. However, in actual wind tunnel environments, the vibration system comprised of piezoelectric ceramics, flexible springs, a vibrating head, a bow shaft, and the test model is affected by airflow pulsations, structural modal changes, temperature drift, and external disturbances. Its equivalent mass, stiffness, and damping are not constant. Once these parameters change, the system's actual resonant frequency will also shift. If the original set frequency is continued, the vibration amplitude and energy utilization rate will often decrease.
[0004] The patent "A Frequency Tracking Search Method Based on Fuzzy PI" by Lian Shengzhe et al., publication number CN111596544A and authorization announcement number CN111596544B, discloses a frequency tracking search method based on fuzzy PI, which adjusts the output frequency by adjusting the phase difference and rate of change of transducer voltage and current. This method can reduce full-band frequency sweep, but it requires pre-setting membership functions and fuzzy rules. In other words, when the controlled object or operating conditions change significantly, the original rules may no longer be accurately applicable.
[0005] Existing patent CN110045612B, "An Backstepping Adaptive Control Method for a Roll-Reducing Fin Hydraulic Servo Simulation Test Bench," discloses a backstepping adaptive control method for a roll-reducing fin hydraulic servo simulation test bench. This method achieves trajectory tracking by constructing a virtual control quantity, a Lyapunov function, and a parameter update law. While backstepping control is suitable for handling model parameter uncertainties and external disturbances, it primarily addresses "how to stably track a given trajectory" and cannot autonomously find the optimal driving frequency solely based on vibration energy feedback.
[0006] Q-learning can divide different frequencies into discrete states, treat frequency increase, hold, and decrease as actions, and update the Q value based on the acceleration response, thus possessing a certain degree of autonomous frequency optimization capability. However, if the Q-table states only contain frequency, it is difficult to determine whether the vibration response is increasing or decreasing; if acceleration, changes in adjacent windows, and action history are added, the Q-table size will increase rapidly. Ordinary Q-tables also have weak generalization ability for unvisited states, making them unsuitable for direct application to operating conditions with continuously changing parameters.
[0007] The Actor-Critic method proposed by Sutton et al. in their paper "Policy Gradient Methods for Reinforcement Learning with Function Approximation" employs parameterized policies and value functions, where the Actor is responsible for outputting action probabilities, and the Critic is responsible for evaluating the current state and guiding policy updates. The policy distillation method proposed by Rusu et al. in their paper "Policy Distillation" can transmit the action probabilities of the teacher's reinforcement learning policy to the student network. However, existing methods rarely consider vibrational energy rewards, backstepping identification parameters, frequency safety boundaries, Q-table confidence, Critic value pre-training, and teacher exit conditions simultaneously, thus their direct application to this vibration system still has limitations.
[0008] Existing reinforcement learning frequency matching methods typically use acceleration amplitude or its variation directly as the optimization criterion. However, under near-sinusoidal vibration conditions, even if the displacement amplitude remains constant, the acceleration amplitude increases with the square of the frequency. Therefore, directly rewarding a larger acceleration may only increase the misjudgment caused by frequency-induced response amplification, failing to maximize resonance performance. On the other hand, when mass, damping, and stiffness change, using absolute frequency as the learning state will cause similar resonance states under different operating conditions to behave differently, reducing the transferability of historical strategy data. Existing technologies have not yet resolved the combined effect of the aforementioned evaluation bias and operating condition transfer problems in constrained frequency closed-loop control. Summary of the Invention
[0009] To address the aforementioned technical challenges, this invention provides a frequency-unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels. The backstepping inner loop updates the control input and estimates of equivalent mass, equivalent damping, and equivalent stiffness according to the first control cycle. The frequency decision outer loop, based on Q-table reinforcement learning, calculates the theoretical resonant frequency according to the second control cycle, converting acceleration data into frequency-unbiased equivalent root mean square displacement values. Frequency search data is generated by the Q-table teacher decision-maker within a safe search interval. This frequency search data is used to pre-train the Actor policy network and Critic value network of the Actor-Critic student decision-maker. After pre-training, the Actor-Critic student decision-maker continues training using an autonomous closed-loop trajectory and exits the Q-table teacher decision-maker after reaching the independent operation requirement, thus achieving resonant frequency self-tuning and vibration energy optimization for a multi-degree-of-freedom high-frequency motion simulation system.
[0010] The technical solution of this invention: A method for frequency-unbiased energy optimization and self-tuning control of forced vibration in wind tunnels includes the following steps: Step 1: Update the equivalent mass, equivalent stiffness, and equivalent damping of the inner loop of the backstep control. This frequency-unbiased energy optimization and self-tuning control method is executed through a multi-degree-of-freedom high-frequency motion simulation system. This system mainly consists of piezoelectric ceramics, flexible springs, a vibrating head, a bow shaft, and a rigid plane for mounting the bow shaft. Using the piezoelectric ceramics as the excitation source, the torque output by the excitation source is transmitted through four flexible springs, enabling the vibrating head to achieve... shaft and Two-dimensional vibration along the axial direction; simultaneously, the fixed point where the bow shaft and the vibrating head are fixed is set as the origin of the global inertial coordinate system. When the piezoelectric ceramic actuates and drives the flexible spring to bend and deform, the axis around which the vibrating head and the bow shaft rotate relative to each other is... Axis, same The directions orthogonal to the axis are Axis; the target vibration direction is the vibration head revolving around the origin. shaft or The shaft performs high-frequency rotation on a fixed axis. For the direction of target vibration, establish the equivalent mass-spring-damping dynamic equation: In the formula, , and These are the equivalent displacement, equivalent velocity, and equivalent acceleration in the direction of target vibration, respectively. , and These are equivalent mass, equivalent damping, and equivalent stiffness, respectively. This is the equivalent control force exerted by the piezoelectric ceramic on the target vibration direction; The combined disturbance term, consisting of airflow pulsation, structural coupling, and assembly errors, is defined as the disturbance estimate value. The upper bound of the preset disturbance error is Perturbation estimation error ,satisfy ; Define the trajectory of the target vibration direction as The displacement error of the system motion trajectory of the multi-degree-of-freedom high-frequency motion simulation system is: The virtual velocity control quantity of the multi-degree-of-freedom high-frequency motion simulation system is The system velocity error of the multi-degree-of-freedom high-frequency motion simulation system is : In the formula, , is the displacement error feedback gain for the motion trajectory of a multi-degree-of-freedom high-frequency motion simulation system; Let the expected velocity of the trajectory of the target vibration direction be denoted by ; from this, we obtain: In the formula, The desired acceleration of the trajectory of the target vibration direction; This represents the rate of change of the displacement error of the system's motion trajectory; The rate of change of the system's virtual speed control quantity; make Let be the estimated values of equivalent mass, equivalent damping, and equivalent stiffness, respectively. The backstep adaptive control law is as follows: In the formula, , is the feedback coefficient of the displacement error of the system's motion trajectory; , is the feedback coefficient of the system speed error; , is the robustness compensation coefficient; , where is the boundary layer thickness, used to suppress chattering that occurs when the system velocity error of a multi-degree-of-freedom high-frequency motion simulation system approaches zero; It is a saturation function used to reduce high-frequency vibrations caused by the positive and negative changes of the backstep adaptive control law; Define equivalent dynamic parameter vector and its estimated value : Define a regression vector that matches the backstep adaptive control law. : A dynamic prediction regression vector for a multi-degree-of-freedom high-frequency motion simulation system is constructed based on equivalent displacement, equivalent velocity, and equivalent acceleration. : Define the corresponding dynamic prediction error : Update the estimates of equivalent mass, equivalent damping, and equivalent stiffness. : In the formula, for Diagonal positive definite adaptive gain matrix, containing , , and The update gains are the estimated values of equivalent mass, equivalent damping, and equivalent stiffness, respectively, which are used to adjust the autonomous correction speed of equivalent mass, equivalent damping, and equivalent stiffness. When the update gain increases, the autonomous correction speed of equivalent mass, equivalent stiffness, and equivalent damping increases. , which is the correction weight for the dynamic prediction error; The working range is set for the equivalent mass, equivalent stiffness, and equivalent damping in a multi-degree-of-freedom high-frequency motion simulation system. This is a projection operator used to constrain the estimates of equivalent mass, equivalent damping, and equivalent stiffness within the working range. Used to improve the convergence speed of equivalent mass, equivalent stiffness and equivalent damping by utilizing dynamic prediction errors; The coupling term between the vector used to offset the estimation error of equivalent dynamic parameters and the system velocity error during the differentiation of the Lyapunov function ensures the stability of the backstepping control inner loop. The backstepping control inner loop is the fast closed-loop control part of the multi-degree-of-freedom high-frequency motion simulation system, used to construct the control input based on the displacement error of the system trajectory and the system velocity error, and to update the estimated values of equivalent mass, equivalent damping, and equivalent stiffness online. When the equivalent mass is determined by weighing or structural conversion of the multi-degree-of-freedom high-frequency motion simulation system, let... And update the estimated value of equivalent damping. and the estimated value of equivalent stiffness ; Substituting the backstepping adaptive control law into the equivalent mass-spring-damped dynamics equation, we obtain the backstepping control error equation: In the formula, , which is the vector of the estimation error of the equivalent dynamic parameters; The cross term is caused by the estimation error of the equivalent dynamic parameters; The system acceleration error of a multi-degree-of-freedom high-frequency motion simulation system; The Lyapunov function is: In the formula, Lyapunov functions are constructed based on multi-degree-of-freedom high-frequency motion simulation systems; Through the backstep adaptive control law and Diagonal positive definite adaptive gain matrix This causes the displacement error of the system's motion trajectory. System speed error And the estimation error of the equivalent dynamic parameters remains bounded; the disturbance estimate Bounded and Under the condition that the displacement error and velocity error of the system trajectory converge to the value determined by the boundary layer thickness, Within a defined neighborhood; Set the control period of the inner loop of the backstep control to the first control period. The period of the frequency decision outer loop based on Q-table reinforcement learning is set as the second control period. Make a frequency decision and satisfy the following: In the formula, It is an integer multiple of the first control cycle; in each second control cycle The inner loop of the backstepping control keeps the driving frequency constant and updates the control input and the estimated values of equivalent mass, equivalent stiffness, and equivalent damping with the first control cycle; at the end of the second control cycle, the first control cycle is output to the frequency decision outer loop. Equivalent quality estimate within Equivalent damping estimate and equivalent stiffness estimate ; The frequency decision outer loop maintains the basic excitation amplitude of the piezoelectric ceramics in a consistent manner, and the equivalent mass, equivalent stiffness, and equivalent damping are limited to a preset range, which facilitates the frequency decision outer loop in finding the resonant frequency of the multi-degree-of-freedom high-frequency motion simulation system.
[0011] The second step is to use equivalent mass, equivalent damping, and equivalent stiffness to form the resonant reference frequency and the safe search range; During the second control cycle, with The collected acceleration data are integrated into a frequency decision window using a time step. According to the t Estimates of equivalent mass, equivalent damping, and equivalent stiffness obtained from each frequency decision window are used to calculate the theoretical resonant frequency of the multi-degree-of-freedom high-frequency motion simulation system under the current operating conditions. : In the formula, and These are the equivalent mass and equivalent stiffness under this frequency decision window, respectively; The theoretical resonant frequency is set as the resonant reference frequency for frequency search; the minimum and maximum allowable driving frequencies for the multi-degree-of-freedom high-frequency motion simulation system are defined as follows: and The frequency range within both ranges constitutes the safe search range, with a half-width of [missing information]. The safe search range is within the range of and Within the safe frequency boundary region, the theoretical resonant frequency is used as the resonant reference frequency. For half-width, perform a frequency search; To find the coordinate differences of the driving frequency between operating conditions corresponding to different equivalent stiffness, equivalent damping, and equivalent mass, the coordinates of the frequency difference between the driving frequency and the resonant reference frequency under the current frequency decision window are defined. for: In the formula, To prevent constants with a denominator of zero; The driving frequency under the current frequency decision window; When the estimated values of the equivalent mass, equivalent damping, or equivalent stiffness change, the theoretical resonant frequency is recalculated. Minimum drive frequency and highest drive frequency ; by The starting frequency of the safe search interval. Let be the frequency step size, and discretize it as follows: Frequency levels: .
[0012] Step 3: Conversion of frequency-unbiased physical evaluation quantities; Using the theoretical resonant frequency as the resonant reference frequency, an accelerometer positioned along the target vibration direction is used to collect data within each frequency decision window. Acceleration data collected from acceleration sampling points are converted into frequency-unbiased equivalent root mean square displacement values. The decision-maker based on Q-table reinforcement learning is defined as the Q-table teacher decision-maker, and the frequency-unbiased equivalent root mean square displacement values are set as the driving frequency for the Q-table teacher decision-maker to search for real working conditions. Each frequency decision window contains The acceleration sampling point, for the _th ... The acceleration data collected by each frequency decision window are root mean squared to obtain the frequency-unbiased equivalent displacement root mean square value. : In the formula, For the first The squared value of acceleration for each frequency decision window; For driving frequency For the approximate sinusoidal vibration of the main frequency components, the root mean square value of acceleration is converted into the root mean square value of the equivalent displacement without frequency bias: In the formula, This is the root mean square value of the equivalent displacement without frequency bias; The maximum amplitude of the equivalent root mean square value of the frequency-unbiased displacement is determined based on the collected acceleration data. and minimum amplitude ,right Perform monotonic interval normalization: In the formula, As a reference bias, ensure that the denominator is always greater than the numerator; The normalized vibration amplitude for adjacent frequency decision windows is explored within the range of 0 to 1; For amplitude limiting calculation, ensure Retrieve data within the range of 0 to 1; Define the change in the frequency unbiased equivalent shift of adjacent frequency decision windows as: ; No. Instant rewards for the teacher decision-maker in a frequency decision window. Set to: In the formula, , , , and All are non-negative weights; The Q-table represents the actions of the teacher decision-maker; This is a penalty for repeated frequency modulation when the same up-frequency or down-frequency action is performed continuously, but the root mean square value of the equivalent shift without frequency bias is not increased. The security cost of reaching or exceeding the security frequency boundary of the safe search range.
[0013] Step 4: The Q-table teacher decision-maker generates frequency search data; Set the Q-table teacher decision-maker to include a set of actions. Set of actions The actions in the text are divided into 3 frequency levels: in, 0 indicates a decrease of one frequency level, while 0 indicates maintaining the current frequency level. This indicates an increase of one frequency level; Adjust the action of the Q-table teacher decision-making device according to frequency. Candidate driving frequencies are obtained. : The actual next drive frequency is: When a candidate drive frequency exceeds the current frequency's limit, the next drive frequency actually executed is forced to be either the lowest or highest drive frequency allowed by the multi-degree-of-freedom high-frequency motion simulation system. Simultaneously, through... Actions whose candidate drive frequencies exceed the current frequency limit are negatively evaluated; Define the Q-table in the teacher decision-making system. t Q-table state of each frequency decision window : In the formula, according to discrete Each frequency level, This is a grading function for frequency difference coordinates, which divides the frequency difference coordinates into grades. A grading function for the normalized frequency-unbiased equivalent root mean square displacement value, which classifies the frequency-unbiased equivalent root mean square displacement value into levels. This is a classification function for the change in frequency-unbiased equivalent displacement, which classifies the change in frequency-unbiased equivalent displacement into levels. This is a Q-table state encoding function that performs encoding operations on the Q-table states of the Q-table teacher decision-maker. Because the Q-table state is updated iteratively over time, the Q-table state update rule for the Q-table teacher decision-maker is defined as follows: In the formula, The discount factor for the Q-table teacher decision-maker controls the impact of the predicted Q-table state on the previous Q-table state. This is the true termination flag for the Q-table teacher decision-maker's operation, indicating that Q-table state updates have stopped. The set of actions within the safe frequency boundary; To obtain the set of actions within the safe frequency boundary under the current Q-table state. Instant rewards for the largest Q-table teacher decision-making system in China The corresponding driving frequency; The learning rate decreases with each successive learning iteration and is used to control the influence of the Q-table state in the new frequency decision window on the previous Q-table state. when When, remove actions with a frequency level of -1 from the action set; when When the frequency level is +1, remove actions from the action set; keep actions always at a frequency level of +1. and The search must be conducted within the safe search range defined by the safe frequency boundary. Actions within the safe search range are considered safe actions, and the Q-table state update rules of the Q-table teacher decision-maker are used. The probability of safe actions of the teacher decision-maker is obtained from the Q-table. for: In the formula, , where is the temperature coefficient of the probability of a safe action in the Q-table teacher decision-maker, and represents the probability of converting a state in the Q-table into a safe action. The distribution concentration used to adjust the probability of safety actions does not represent the actual physical temperature; The mask for the safe actions of the teacher decision-maker in table Q; A safe action mask for traversing the action set during summation; To sum the states, the Q-table of the action set is traversed. The Actor-Critic student decision-maker consists of an Actor policy network and a Critic value network, and is used to receive the Q-table state from the Q-table teacher decision-maker. To avoid insufficient Q-table state updates due to insufficient sampling and action execution by the Q-table teacher decision-maker, which could lead to excessively strong constraints being passed to the Actor-Critic student decision-maker, the confidence level of the Q-table teacher decision-maker is determined jointly based on the number of times the Q-table teacher decision-maker accesses the Q-table state and the probability of safe actions. : In the formula, when the probability of the action at all three frequency levels is 1, the probability of the action at all three frequency levels is 1. hour, When the Q-table teacher decision-maker determines an action, Close to 1; The Q-table teacher decision-maker frequency search data should include at least: in, Search data for teacher decision-making frequency in the Q-table; For the first t The nine-dimensional input state vector of the Actor-Critic student decision-maker in a frequency decision window; For the first t The combination of Q-table states and actions under a frequency decision window; For the first t The action with the largest amplitude among all frequency levels under a frequency decision window; This flag is used to indicate whether the Q-table teacher decision-making system has ended due to the running time reaching the preset upper limit of the second cycle.
[0014] Step 5: The Actor-Critic student decision-making system obtains strategy and value initialization; The estimates of equivalent mass, equivalent damping, and equivalent stiffness are Gaussian standardized, and the results are obtained respectively. ; In the formula, The three-dimensional one-hot encoding of the action in the (t-1)th frequency decision window; t The action with the largest amplitude among all frequency levels under a frequency decision window. The labels are used only to supervise the training of the Actor-Critic student decision-maker and are not used as input states. The Actor policy network consists of a nine-dimensional input layer, a first hidden layer with 32 neurons, a second hidden layer with 32 neurons, and a three-dimensional action score output layer. The action score output by the Actor policy network is denoted as... The probability of its safe action for: in, For Actor policy network parameters; For the Actor policy network according to Output action The score; For the Actor policy network, the output action The set action score; To this action Perform natural exponentiation on fractions; The Critic value network consists of a nine-dimensional input layer, a first hidden layer with 64 neurons, a second hidden layer with 64 neurons, and a one-dimensional state value output layer, the output of which is denoted as... ,in, For Critic value network parameters; Sample weights were set based on the confidence levels of the teacher decision-making system in the Q-table. : In the formula, Set the lower bound of the sample weights for the confidence level of the Q-table teacher decision maker; Using the probability of safe actions of the teacher decision-maker in the Q-table as soft labels, construct the Actor soft probability distillation loss. : Using the safest action with the largest vibration amplitude in the Q-table teacher decision-maker as a hard label, construct the Actor's optimal action imitation loss. : We construct the pre-training loss for the Actor policy network using Actor soft probability distillation loss and Actor optimal action imitation loss. : In the formula, The number of data points for the teacher decision-maker frequency search data in table Q; For the Actor policy network, the input state vector is nine dimensions. The probability of selecting the safe action with the largest vibration amplitude from the Q-table teacher decision-maker; and These are the weights of the Actor soft probability distillation loss and the Actor optimal action imitation loss under the output action of the Actor policy network, respectively. The data generated by the Q-table teacher decision-maker is divided into complete running data sequences according to time order. Then, starting from the end of each complete running data sequence, discount factors from different Actor-Critic student decision-makers are used. Calculate the cumulative discount reward for the Actor-Critic student decision-maker at each time step. Rearrange the data; When the calculation of the cumulative discount reward is truncated only due to reaching the running time, the cumulative discount reward not yet obtained after the complete running data sequence is completed using the Q-table state of the Q-table teacher decision-maker is as follows: Constructing Critic's experience reward loss And Q table auxiliary value loss : in, The Huber loss function is used to reduce the impact of erroneous reward data on the training of the Critic value network. Losses from Critic experience And Q table auxiliary value loss Obtain the Critic pre-training loss : In the formula, For Critic experience reward loss The weight, Auxiliary value loss for Q table The weights, and The Q-table state of the teacher decision-maker is used to initialize the auxiliary value loss of the Q-table.
[0015] Step 6: Exit the Q-table teacher decision-maker based on the actual closed-loop performance.
[0016] When the Actor-Critic student decision-maker cannot make decisions independently, it needs to be trained with the help of the Q-table teacher decision-maker. The Actor-Critic student decision-maker interacts directly with a multi-degree-of-freedom high-frequency motion simulation system to generate an autonomous closed-loop trajectory for training the Actor-Critic student decision-maker. : The first Instant rewards for the teacher decision-maker in a frequency decision window. The nine-dimensional input state vector of the Actor-Critic student decision-maker in the next frequency decision window The estimated value of future cumulative discount rewards Add them together, then subtract the output of the Critic value network in the Actor-Critic student decision maker at the current frequency decision window. Value prediction bias was obtained. ; Based on value prediction bias The dominance estimate is calculated by recursively working backward from the end of the complete running data sequence: In the formula, Indicates the first t The advantage estimate for each frequency decision window; This is an adjustment coefficient for the next advantage estimate, used to control the impact of differences in advantage estimates in subsequent frequency decision windows on the current advantage estimate; the Actor policy network, based on... Increase the probability of selecting actions with a value greater than 0, and decrease the probability of selecting actions with a value less than 0; Output the current Critic value network With advantage estimate The sum is used as the fixed value objective for this training. By reducing the Huber loss between the Critic value network output and the fixed value objective, the parameters of the Critic value network are updated. In the initial stage of training the Actor-Critic student decision-maker in a multi-degree-of-freedom high-frequency motion simulation system, the Actor soft probability distillation loss is retained; n Constraint discount factor of Q-table teacher decision-maker in round training Decreasing in successive rounds; among them, The constraint coefficient for the Actor soft probabilistic distillation loss controls the proportion of influence of the Q-table teacher decision-maker on the Actor-Critic student decision-maker. The attenuation coefficient controls how quickly the Q-table teacher decision-maker exits the Actor-Critic student decision-maker; Joint Loss of the Actor-Critic Student Decision Maker : In the formula, The joint training loss of the Actor policy network is given by the autonomous reinforcement loss of the Actor policy network. Actor soft probabilistic distillation loss and average policy entropy composition; The joint training loss of the Critic value network is given by the autonomous reinforcement loss of the Critic value network. The Q table is composed of auxiliary value loss; and These are the strategy loss and value loss obtained from the autonomous closed-loop trajectory, respectively. The average policy entropy of the Actor-Critic student decision-maker represents the dispersion of the probability of safe actions output by the Actor-Critic student decision-maker. When the probabilities of decreasing the frequency level by one, maintaining the current frequency level, and increasing the frequency level by one are relatively close, it indicates that the Actor policy network of the Actor-Critic student decision-maker still retains strong exploratory capabilities. When the probability of an action at a certain frequency level is significantly higher than that of actions at other frequency levels, it indicates that the Actor policy network tends to be deterministic. The weights for the Actor's soft probabilistic distillation loss; The weights of the average policy entropy are used to adjust the strength of the effect of the average policy entropy of the Actor-Critic student decision-maker on the Q-table teacher decision-maker and the Actor-Critic student decision-maker. The larger the value, the more the Actor-Critic student decision-maker tends to generate actions randomly, avoiding prematurely fixing a particular action. The smaller the value, the more the Actor policy network focuses on the advantage estimate. Adjust the probability of safety actions at various frequencies; when At that time, the average policy entropy is not included in the calculation of the Q-table teacher decision-maker and the Actor-Critic student decision-maker; The exit of the Q-table teacher decision-maker is determined not only by the number of training iterations of the Actor-Critic student decision-maker, but also by the number of consecutive iterations. The average cumulative discount reward, the frequency-unbiased equivalent root mean square value of displacement, and the probability of unsafe actions obtained within each independent evaluation period are jointly determined. An independent evaluation period involves pausing parameter updates of the Actor policy network and the Critic value network, allowing the Q-table teacher decision-maker and the Actor-Critic student decision-maker to independently control the multi-degree-of-freedom high-frequency motion simulation system once, under the same driving frequency, the same motion trajectory, the same safe search interval, and the same preset upper limit of running time. The immediate reward, the frequency-unbiased equivalent root mean square value of displacement, and the number of unsafe actions generated during the operation are recorded respectively. Each completed operation of the multi-degree-of-freedom high-frequency motion simulation system is recorded as one independent evaluation period. The average cumulative discount reward for the Actor-Critic student decision-maker and the Q-table teacher decision-maker is set as follows: and The equivalent root mean square values of the frequency-unbiased shift are respectively and The probabilities of unsafe actions are respectively and When the average cumulative reward of the student decision-maker in the Actor-Critic table is not less than the allowed percentage of the average cumulative reward of the teacher decision-maker in the Q table. The frequency unbiased equivalent root mean square value of the student decision-maker is not less than the allowable equivalent displacement ratio of the corresponding frequency unbiased equivalent root mean square value of the teacher decision-maker in the Q table. Furthermore, the probability of unsafe actions by the Actor-Critic student decision-maker does not exceed the deviation between the probability of unsafe actions and the allowable safe probability of actions by the teacher decision-maker in the Q-table. When the summation is complete, the Actor-Critic student decision-maker achieves independent operation requirements in three aspects: control average discount cumulative reward, frequency unbiased equivalent root mean square value of displacement, and probability of unsafe actions; and after... In each independent evaluation cycle, when the above conditions are met, the teacher constraint weight... If the value is 0, stop reading the action probabilities, Q-table states, and Q-table teacher states of the Q-table teacher decision-maker, and exit the Q-table teacher decision-maker; where the teacher constraint weights are... In the Actor-Critic student decision-maker training process, it is a dimensionless coefficient used to adjust the proportion of the Q-table teacher decision-maker frequency search data in the joint training loss of the Actor policy network and the Critic value network. To allow for reward bias, To allow for equivalent displacement deviation; After the Q-table teacher decision-maker exits, the Actor-Critic student decision-maker continues training and control using its own generated autonomous closed-loop trajectory data. At the end of training, the equivalent mass, equivalent damping, equivalent stiffness, nine-dimensional input state vector, safe frequency boundary, frequency step size, and safe action mask of the Actor-Critic student decision-maker are saved. During online deployment, only the Actor-Critic student decision-maker is retained, and it outputs actions to decrease by one frequency level, maintain the current frequency level, and increase by one frequency level based on the current resonant reference frequency and the frequency-unbiased equivalent displacement.
[0017] The beneficial effects of this invention are: First, the present invention converts the root mean square value of acceleration within the frequency decision window into the root mean square value of equivalent displacement without frequency bias, thereby enabling the comparison of vibration response under different driving frequencies on a unified equivalent displacement scale, thus establishing a correspondence between the instantaneous reward and the displacement response and mechanical energy change in the target vibration direction.
[0018] Second, this invention utilizes the backstepping control inner loop to update the estimated values of equivalent mass, equivalent damping, and equivalent stiffness online, and calculates the theoretical resonant frequency, determines the safe search interval, and constructs the frequency difference coordinates accordingly. When the equivalent dynamic parameters change, the frequency search center and the nine-dimensional input state vector of the Actor-Critic student decision-maker are updated synchronously, improving the continuity and usability of frequency search data under different parameter conditions.
[0019] Third, this invention utilizes the frequency search data from the Q-table teacher decision-maker to pre-train the Actor policy network and the Critic value network, respectively, and adjusts the training weights of different frequency search data based on the confidence level of the Q-table teacher decision-maker. After the Actor-Critic student decision-maker acquires the initial policy and state value estimation capabilities, it undergoes reinforcement training through autonomous closed-loop trajectories. When it continuously meets the independent operation requirements, the teacher constraint weight is reset to zero, and the Actor-Critic student decision-maker independently completes the frequency adjustment, taking into account frequency search efficiency, control accuracy, and adaptive capability under parameter variation conditions. Attached Figure Description
[0020] Figure 1 It is a control framework for frequency-unbiased energy optimization and self-tuning control methods for forced vibration in wind tunnels. Figure 2 This is a structural diagram of the device that establishes rotation around the x-axis via an actuating mechanism; Figure 3 It is a strategy-value network structure diagram; In the diagram: 1. Bow shaft, 2. Piezoelectric ceramic, 3. Vibrating head, 4. Flexible spring. Detailed Implementation
[0021] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.
[0022] A method for frequency-unbiased energy optimization and self-tuning control of forced vibration in wind tunnels includes the following steps: Step 1: Setting the control cycle for the multi-degree-of-freedom high-frequency motion simulation system; This embodiment is executed using a multi-degree-of-freedom high-frequency motion simulation system. The multi-degree-of-freedom high-frequency motion simulation system mainly consists of piezoelectric ceramics, four flexible springs, a vibrating head, a bow shaft, a test model, and a rigid plane for mounting the bow shaft. The rigid plane is a rigid support structure used to fix the bow shaft and provide a mounting reference for it.
[0023] Using piezoelectric ceramics as the excitation source, the force generated by the piezoelectric ceramics is converted and transmitted into torque acting on the vibrating head via four flexible springs. The fixed point where the bow shaft is fixed to the vibrating head is set as the origin of the global inertial coordinate system; the axis around which the vibrating head rotates relative to the bow shaft is... Axis, and The directions orthogonal to the axis are Axis. The target vibration direction is the direction of the vibrating head around the origin. shaft or The direction of the shaft's high-frequency rotation. An accelerometer is positioned along the target vibration direction to collect acceleration data within each frequency decision window.
[0024] The control period of the inner loop of the backstepping control is set to the first control period. The control period of the frequency decision outer loop based on Q-table reinforcement learning is set as the second control period. Both conditions are met: In the formula, This means that a second control cycle contains 300 first control cycles. During the second control cycle, the drive frequency remains constant, and the backstepping control inner loop updates the control input and the estimated values of equivalent mass, equivalent damping, and equivalent stiffness according to the first control cycle.
[0025] The accelerometer sampling frequency is set to 2000 Hz, corresponding to a sampling time step of... Therefore, the number of acceleration sampling points contained in each frequency decision window is: Step 2: Update the equivalent mass, equivalent stiffness, and equivalent damping of the inner loop of the backstep control. Establish the equivalent mass-spring-damping dynamic equation for the target vibration direction: In the formula, , and These are the equivalent displacement, equivalent velocity, and equivalent acceleration in the direction of target vibration, respectively. , and These are equivalent mass, equivalent damping, and equivalent stiffness, respectively. This is the equivalent control force exerted by the piezoelectric ceramic on the target vibration direction; This is a combined disturbance term resulting from airflow pulsation, structural coupling, and assembly errors.
[0026] Define the trajectory of the target vibration direction as The displacement error of the system motion trajectory of the multi-degree-of-freedom high-frequency motion simulation system is: The virtual velocity control quantity of the multi-degree-of-freedom high-frequency motion simulation system is The system velocity error of the multi-degree-of-freedom high-frequency motion simulation system is : In the formula, , is the displacement error feedback gain for the motion trajectory of a multi-degree-of-freedom high-frequency motion simulation system; Let be the desired velocity of the trajectory of the target vibration direction. From this, we obtain: In the formula, The desired acceleration of the trajectory of the target vibration direction; This represents the rate of change of the displacement error of the system's motion trajectory; This represents the rate of change of the system's virtual speed control quantity.
[0027] make Let be the estimated values of equivalent mass, equivalent damping, and equivalent stiffness, respectively. The backstep adaptive control law is as follows: In the formula, , is the feedback coefficient of the displacement error of the system's motion trajectory; , is the feedback coefficient of the system speed error; , is the robustness compensation coefficient; , where is the boundary layer thickness; It is a saturation function that keeps the control input continuous when the system speed error is close to zero.
[0028] Define equivalent dynamic parameter vector and its estimated value : Define a regression vector that matches the backstep adaptive control law. : A dynamic prediction regression vector for a multi-degree-of-freedom high-frequency motion simulation system is constructed based on equivalent displacement, equivalent velocity, and equivalent acceleration. : Define the corresponding dynamic prediction error : Update the estimates of equivalent mass, equivalent damping, and equivalent stiffness. : In the formula, for Diagonal positive definite adaptive gain matrix; , which is the correction weight for the dynamic prediction error; The operating range for equivalent mass, equivalent damping, and equivalent stiffness; This is a projection operator used to constrain the estimates of equivalent mass, equivalent damping, and equivalent stiffness within the working range.
[0029] Define the vector of equivalent dynamic parameter estimation error as: The backstep control error equation is obtained as follows: In the formula, This represents the estimation error of the synthetic perturbation term; The cross term is caused by the estimation error of the equivalent dynamic parameters; This refers to the system acceleration error of a multi-degree-of-freedom high-frequency motion simulation system.
[0030] Among them, the Lyapunov function is selected as : like When the error exceeds the upper bound of the synthetic perturbation term estimation error, the displacement error, velocity error, and equivalent dynamic parameter estimation error of the multi-degree-of-freedom high-frequency motion simulation system remain bounded. The displacement error and system velocity error of the motion trajectory converge to the boundary layer thickness. Within a defined neighborhood.
[0031] In this embodiment, the working ranges for equivalent mass, equivalent stiffness, and equivalent damping are respectively set as follows: Under a set of calibration conditions, an estimated value of the equivalent mass is obtained. Estimated value of equivalent damping and the estimated value of equivalent stiffness At the end of each frequency decision window, three estimates are output to the frequency decision outer loop based on Q-table reinforcement learning.
[0032] Step 3: Use equivalent mass, equivalent damping, and equivalent stiffness to form the resonant reference frequency and safe search range; According to the t Estimates of equivalent mass, equivalent damping, and equivalent stiffness obtained from each frequency decision window are used to calculate the theoretical resonant frequency of the multi-degree-of-freedom high-frequency motion simulation system under the current operating conditions. : Will and Substitution ,get: The theoretical resonant frequency is set as the resonant reference frequency. The permissible drive frequency range of the device is set as... half-width of the resonant frequency search interval Set as Then the safe frequency boundary is: definition and The frequency range within the specified range is the safe search range. Calculate the frequency difference coordinates between the driving frequency and the resonant reference frequency under the current frequency decision window. for: In the formula, To prevent constants with a denominator of zero; This refers to the driving frequency within the current frequency decision window.
[0033] Frequency step size set to The number of frequency levels is: Step 4: Conversion of frequency-unbiased physical evaluation quantities; Assume an accelerometer collects data. The acceleration sample value, for the _th ... The acceleration data collected by each frequency decision window are root mean squared to obtain the frequency-unbiased equivalent displacement root mean square value. : In the formula, For the first The squared value of acceleration for each frequency decision window.
[0034] For driving frequency For the approximate sinusoidal vibration of the main frequency components, the root mean square value of acceleration is converted into the root mean square value of the equivalent displacement without frequency bias: In the formula, This is the root mean square value of the equivalent displacement without frequency bias.
[0035] The maximum amplitude of the equivalent root mean square value of the frequency-unbiased displacement is determined based on the collected acceleration data. and minimum amplitude ,right Monotonic interval normalization is performed, and the 1% quantile of the equivalent root mean square value of the unbiased frequency shift is taken as the lower boundary of the normalization. : In the formula, As a reference bias, ensure that the denominator is always greater than the numerator; The normalized vibration amplitude for adjacent frequency decision windows is explored within the range of 0 to 1; For amplitude limiting calculation, ensure Retrieve data within the range of 0 to 1; Define the change in the frequency unbiased equivalent shift of adjacent frequency decision windows as: .
[0036] No. Instant rewards for the teacher decision-maker in a frequency decision window. Set to: In the formula, , , , and All are non-negative weights; The Q-table represents the actions of the teacher decision-maker; This is a penalty for repeated frequency modulation when the same up-frequency or down-frequency action is performed continuously, but the root mean square value of the equivalent shift without frequency bias is not increased. The security cost of reaching or exceeding the security frequency boundary of the safe search range.
[0037] In this embodiment, the following is taken: Step 5: The Q-table teacher decision-maker generates frequency search data; Set the Q-table teacher decision-maker to include a set of actions. Set of actions The actions in the text are divided into 3 frequency levels: in, 0 indicates a decrease of one frequency level, while 0 indicates maintaining the current frequency level. This indicates an increase of one frequency level.
[0038] Adjust the action of the Q-table teacher decision-making device according to frequency. Candidate driving frequencies are obtained. : The actual next drive frequency is: Define the Q-table in the teacher decision-making system. t Q-table state of each frequency decision window : This embodiment places the frequency difference coordinates in The area is divided into 7 levels; The normalized frequency unbiased equivalent root mean square displacement values within the range are divided into 5 levels at 0.2 intervals; the change in the decision window between adjacent frequencies is divided into: There are 5 levels in total.
[0039] The actions in the previous frequency decision window included three possibilities: decreasing the frequency level by one, maintaining the current frequency level, and increasing the frequency level by one. Therefore, the total number of states in the Q table is: Each Q-table state corresponds to 3 actions. The number of combinations of Q-table states and actions is: Because the Q-table state is updated iteratively over time, the Q-table state update rule for the Q-table teacher decision-maker is defined as follows: In the formula, The discount factor for the Q-table teacher decision-maker controls the impact of the predicted Q-table state on the previous Q-table state. This is the true termination flag for the Q-table teacher decision-maker's operation, indicating that Q-table state updates have stopped. The set of actions within the safe frequency boundary; The learning rate decreases according to the following formula: In the formula, The initial learning rate; This is the learning rate decay coefficient; This represents the number of times the corresponding Q-table state and action combination has been updated. In this embodiment, we take... , , .
[0040] Actions within the safe search range are considered safe actions, and the Q-table state update rules of the Q-table teacher decision-maker are used. The probability of safe actions of the teacher decision-maker is obtained from the Q-table. for: In the formula, , where is the temperature coefficient of the probability of a safe action in the Q-table teacher decision-maker, and represents the probability of converting a state in the Q-table into a safe action. The distribution concentration used to adjust the probability of safety actions does not represent the actual physical temperature; The mask for the safe actions of the teacher decision-maker in table Q; A safe action mask for traversing the action set during summation; To sum the states, the Q-table of the action set is traversed. This embodiment takes... .
[0041] The confidence level of the Q-table teacher decision-maker is determined by combining the number of times the Q-table state is accessed and the probability of safe actions. : In the formula, when the probability of the action at all three frequency levels is 1, the probability of the action at all three frequency levels is 1. hour, When the Q-table teacher decision-maker determines an action, Close to 1; When the total number of states in the Q-table no longer increases during continuous checks, the executable actions at each frequency level are accessed at least twice, and the relative change of the current highest instant reward for three consecutive times does not exceed 2%, the Q-table teacher decision-maker is deemed to have completed this frequency search.
[0042] The Q-table teacher decision-maker frequency search data should include at least: in, Search data for teacher decision-making frequency in the Q-table; For the first t The nine-dimensional input state vector of the Actor-Critic student decision-maker in a frequency decision window; For the first t The combination of Q-table states and actions under a frequency decision window; For the first t The action with the largest amplitude among all frequency levels under a frequency decision window; This flag is used to indicate whether the Q-table teacher decision-making system has ended due to the running time reaching the preset upper limit of the second cycle.
[0043] Step Six: The Actor-Critic student decision-making system obtains strategies and value initialization; The nine-dimensional input state vector of the Actor-Critic student decision-maker is: In the formula, The three-dimensional one-hot encoding of the action in the (t-1)th frequency decision window; t The action with the largest amplitude among all frequency levels under a frequency decision window. The labels are used only to supervise the training of the Actor-Critic student decision-maker and are not used as input states. The Actor policy network consists of a nine-dimensional input layer, a first hidden layer with 32 neurons, a second hidden layer with 32 neurons, and a three-dimensional action score output layer. The action score output by the Actor policy network is denoted as... The probability of its safe action for: in, For Actor policy network parameters; For the Actor policy network according to Output action The score; For the Actor policy network, the output action The set action score; To this action The fraction is used for natural exponentiation.
[0044] The Critic value network consists of a nine-dimensional input layer, a first hidden layer with 64 neurons, a second hidden layer with 64 neurons, and a one-dimensional state value output layer, the output of which is denoted as... ,in, For Critic value network parameters; Sample weights were set based on the confidence levels of the teacher decision-making system in the Q-table. : In the formula, Set the lower bound of the sample weights for the confidence level of the Q-table teacher decision-maker. This embodiment takes... .when hour, ; When it approaches 1, Close to 1.
[0045] Using the probability of safe actions of the teacher decision-maker in the Q-table as soft labels, construct the Actor soft probability distillation loss. : Using the safest action with the largest vibration amplitude in the Q-table teacher decision-maker as a hard label, construct the Actor's optimal action imitation loss. : We construct the pre-training loss for the Actor policy network using Actor soft probability distillation loss and Actor optimal action imitation loss. : In the formula, The number of data points for the teacher decision-maker frequency search data in table Q; For the Actor policy network, the input state vector is nine dimensions. The probability of selecting the safe action with the largest vibration amplitude from the Q-table teacher decision-maker; and These are the weights of the Actor soft probabilistic distillation loss and the Actor optimal action imitation loss under the output action of the Actor policy network, respectively. In this embodiment, we take... , .
[0046] The data generated by the Q-table teacher decision-maker is divided into complete running data sequences according to time order. Then, starting from the end of each complete running data sequence, discount factors from different Actor-Critic student decision-makers are used. Calculate the cumulative discount reward for the Actor-Critic student decision-maker at each time step. Rearrange the data.
[0047] When the calculation of the cumulative discount reward is truncated only due to reaching the running time, the cumulative discount reward not yet obtained after the complete running data sequence is completed using the Q-table state of the Q-table teacher decision-maker is as follows: Constructing Critic's empirical reward loss And Q table auxiliary value loss : in, The Huber loss function is used to reduce the impact of erroneous reward data on the training of the Critic value network.
[0048] Losses from Critic experience And Q table auxiliary value loss Obtain the Critic pre-training loss : In the formula, For Critic experience reward loss The weight, Auxiliary value loss for Q table The weights, and The Q-table state of the teacher decision-maker is used to initialize the auxiliary value loss of the Q-table. This embodiment takes... , .
[0049] Step 7: Exit the Q-table teacher decision-maker based on the actual closed-loop performance; The Actor-Critic student decision-maker interacts directly with a multi-degree-of-freedom high-frequency motion simulation system to generate autonomous closed-loop trajectories for training the Actor-Critic student decision-maker. : The first Instant rewards for the teacher decision-maker in a frequency decision window. The nine-dimensional input state vector of the Actor-Critic student decision-maker in the next frequency decision window The estimated value of future cumulative discount rewards Add them together, then subtract the output of the Critic value network in the Actor-Critic student decision maker at the current frequency decision window. Value prediction bias was obtained. .
[0050] Based on value prediction bias The dominance estimate is calculated by recursively working backward from the end of the complete running data sequence: In the formula, Indicates the firstt The advantage estimate for each frequency decision window; This is an adjustment coefficient for the next advantage estimate, used to control the impact of differences in advantage estimates in subsequent frequency decision windows on the current advantage estimate; the Actor policy network, based on... Increase the probability of choosing actions with a value greater than 0, and decrease the probability of choosing actions with a value less than 0.
[0051] Output the current Critic value network With advantage estimate The sum is used as the fixed value objective for this training. By reducing the Huber loss between the Critic value network output and the fixed value objective, the parameters of the Critic value network are updated.
[0052] In the initial stage of training the Actor-Critic student decision-maker in a multi-degree-of-freedom high-frequency motion simulation system, the Actor soft probability distillation loss is retained; n Constraint discount factor of Q-table teacher decision-maker in round training Decreasing in successive rounds; among them, The constraint coefficient for the Actor soft probabilistic distillation loss controls the proportion of influence of the Q-table teacher decision-maker on the Actor-Critic student decision-maker. The attenuation coefficient controls how quickly the Q-table teacher decision-maker exits the Actor-Critic student decision-maker.
[0053] Joint Loss of the Actor-Critic Student Decision Maker : In the formula, The joint training loss of the Actor policy network is given by the autonomous reinforcement loss of the Actor policy network. Actor soft probabilistic distillation loss and average policy entropy composition; The joint training loss of the Critic value network is given by the autonomous reinforcement loss of the Critic value network. The Q table is composed of auxiliary value loss; and These are the strategy loss and value loss obtained from the autonomous closed-loop trajectory, respectively. Let be the average policy entropy of the Actor-Critic student decision-maker, used to represent the dispersion of the probability of the safe action output by the Actor-Critic student decision-maker; when the probabilities of decreasing by one frequency level, maintaining the current frequency level, and increasing by one frequency level are relatively close, A higher probability indicates that the Actor-Critic student decision-maker's Actor policy network still retains strong exploratory capabilities; when the probability of an action at a certain frequency level is significantly higher than that of actions at other frequency levels, A smaller value indicates that the Actor policy network tends to be deterministic; The weights for the Actor's soft probabilistic distillation loss; The weights of the average policy entropy are used to adjust the strength of the effect of the average policy entropy of the Actor-Critic student decision-maker on the Q-table teacher decision-maker and the Actor-Critic student decision-maker. The larger the value, the more the Actor-Critic student decision-maker tends to generate actions randomly, avoiding prematurely fixing a particular action. The smaller the value, the more the Actor policy network focuses on the advantage estimate. Adjust the probability of safety actions at various frequencies; when At that time, the average policy entropy is not included in the calculation of the Q-table teacher decision-maker and the Actor-Critic student decision-maker.
[0054] The exit of the Q-table teacher decision-maker is determined not only by the number of training iterations of the Actor-Critic student decision-maker, but also by the number of consecutive iterations. The average cumulative discount reward, the frequency-unbiased equivalent root mean square value of displacement, and the probability of unsafe actions obtained within each independent evaluation period are jointly determined. An independent evaluation period involves pausing parameter updates of the Actor policy network and the Critic value network, allowing the Q-table teacher decision-maker and the Actor-Critic student decision-maker to independently control the multi-degree-of-freedom high-frequency motion simulation system once, under the same driving frequency, the same motion trajectory, the same safe search interval, and the same preset upper limit of running time. The immediate reward, the frequency-unbiased equivalent root mean square value of displacement, and the number of unsafe actions generated during the operation are recorded respectively. Each completed operation of the multi-degree-of-freedom high-frequency motion simulation system is recorded as one independent evaluation period. The average cumulative discount reward for the Actor-Critic student decision-maker and the Q-table teacher decision-maker is set as follows: and The equivalent root mean square values of the frequency-unbiased shift are respectively and The probabilities of unsafe actions are respectively and When the average cumulative reward of the student decision-maker in the Actor-Critic table is not less than the allowed percentage of the average cumulative reward of the teacher decision-maker in the Q table. The frequency unbiased equivalent root mean square value of the student decision-maker is not less than the allowable equivalent displacement ratio of the corresponding frequency unbiased equivalent root mean square value of the teacher decision-maker in the Q table. Furthermore, the probability of unsafe actions by the Actor-Critic student decision-maker does not exceed the deviation between the probability of unsafe actions and the allowable safe probability of actions by the teacher decision-maker in the Q-table. When the summation is complete, the Actor-Critic student decision-maker achieves independent operation requirements in three aspects: control average discount cumulative reward, frequency unbiased equivalent root mean square value of displacement, and probability of unsafe actions; and after... In each independent evaluation cycle, when the above conditions are met, the teacher constraint weight... If the value is 0, stop reading the action probabilities, Q-table states, and Q-table teacher states of the Q-table teacher decision-maker, and exit the Q-table teacher decision-maker; where the teacher constraint weights are... It refers to the dimensionless coefficient used in the Actor-Critic student decision-maker training process to adjust the proportion of the Q-table teacher decision-maker frequency search data in the joint training loss of the Actor policy network and the Critic value network. To allow for reward bias, To allow for equivalent displacement deviation.
[0055] This embodiment takes , , ,and When this condition is met for five consecutive independent evaluation periods, let... After the Q-table teacher decision-maker exits, the Actor-Critic student decision-maker continues training and control using its own generated autonomous closed-loop trajectory data. At the end of training, the equivalent mass, equivalent damping, equivalent stiffness, nine-dimensional input state vector, safe frequency boundary, frequency step size, and safe action mask of the Actor-Critic student decision-maker are saved. During online deployment, only the Actor-Critic student decision-maker is retained, and it outputs actions to decrease by one frequency level, maintain the current frequency level, and increase by one frequency level based on the current resonant reference frequency and the frequency-unbiased equivalent displacement.
[0056] Step 8: Verify the calculation results; In a set of closed-loop response data in this embodiment, the root mean square value of acceleration is 4.2 m / s² when the driving frequency is 50 Hz. 2 At 86 Hz, it is 15.7 m / s2 The corresponding unbiased root mean square values of the equivalent displacement at the corresponding frequencies are as follows: The equivalent displacement without frequency bias is increased by approximately 26.36%, and the ratio of the displacement square energy index is: according to for Calculations show that the maximum stored energy of the spring increases from approximately 0.2644 mJ to 0.4221 mJ, an increase of approximately 59.66%. According to... for Calculations show that the single-cycle damping dissipation energy increases from approximately 8.865 μJ to 24.344 μJ, which is about 2.746 times the original value.
[0057] This embodiment uses the frequency-unbiased equivalent displacement squared after frequency conversion to construct the reward, which can eliminate the influence of the natural amplification of acceleration amplitude with increasing frequency and avoid misjudging this natural amplification as an increase in resonant energy. The highest reward frequency in the teacher decision-maker in the Q table is 86 Hz, which is basically consistent with the damped resonant frequency of 86.0026 Hz calculated based on the identification parameters. The difference between the two is 0.0026 Hz, and the relative deviation is approximately 0.0030%. This shows that the teacher search results, equivalent energy evaluation results, and system identification results have good consistency under the conditions of this implementation.
Claims
1. A frequency-unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels, characterized in that, Includes the following steps: Step 1: Update the equivalent mass, equivalent stiffness, and equivalent damping of the inner loop of the backstep control. The second step is to use equivalent mass, equivalent damping, and equivalent stiffness to form the resonant reference frequency and the safe search range; Step 3: Conversion of frequency-unbiased physical evaluation quantities; Step 4: The Q-table teacher decision-maker generates frequency search data; Step 5: The Actor-Critic student decision-making system obtains strategy and value initialization; Step 6: Exit the Q-table teacher decision-maker based on the actual closed-loop performance.
2. The frequency-unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels according to claim 1, characterized in that, The specific implementation process of the first step is as follows: This frequency-unbiased energy optimization and self-tuning control method is executed through a multi-degree-of-freedom high-frequency motion simulation system. This system mainly consists of piezoelectric ceramics, flexible springs, a vibrating head, a bow shaft, and a rigid plane for mounting the bow shaft. Using the piezoelectric ceramics as the excitation source, the torque output by the excitation source is transmitted through four flexible springs, enabling the vibrating head to achieve... shaft and Two-dimensional vibration along the axial direction; simultaneously, the fixed point where the bow shaft and the vibrating head are fixed is set as the origin of the global inertial coordinate system. When the piezoelectric ceramic actuates and drives the flexible spring to bend and deform, the axis around which the vibrating head and the bow shaft rotate relative to each other is... Axis, same The directions orthogonal to the axis are Axis; the target vibration direction is the vibration head revolving around the origin. shaft or The shaft performs high-frequency rotation on a fixed axis. For the direction of target vibration, establish the equivalent mass-spring-damping dynamic equation: In the formula, , and These are the equivalent displacement, equivalent velocity, and equivalent acceleration in the direction of target vibration, respectively. , and These are equivalent mass, equivalent damping, and equivalent stiffness, respectively. This is the equivalent control force exerted by the piezoelectric ceramic on the target vibration direction; The combined disturbance term, consisting of airflow pulsation, structural coupling, and assembly errors, is defined as the disturbance estimate value. The upper bound of the preset disturbance error is Perturbation estimation error ,satisfy ; Define the trajectory of the target vibration direction as The displacement error of the system motion trajectory of the multi-degree-of-freedom high-frequency motion simulation system is: The virtual velocity control quantity of the multi-degree-of-freedom high-frequency motion simulation system is The system velocity error of the multi-degree-of-freedom high-frequency motion simulation system is : In the formula, , is the displacement error feedback gain for the motion trajectory of a multi-degree-of-freedom high-frequency motion simulation system; Let the expected velocity of the trajectory of the target vibration direction be denoted by ; from this, we obtain: In the formula, The desired acceleration of the trajectory of the target vibration direction; This represents the rate of change of the displacement error of the system's motion trajectory; The rate of change of the system's virtual speed control quantity; make Let be the estimated values of equivalent mass, equivalent damping, and equivalent stiffness, respectively. The backstep adaptive control law is as follows: In the formula, , is the feedback coefficient of the displacement error of the system's motion trajectory; , is the feedback coefficient of the system speed error; , is the robustness compensation coefficient; , where is the boundary layer thickness, used to suppress chattering that occurs when the system velocity error of a multi-degree-of-freedom high-frequency motion simulation system approaches zero; It is a saturation function used to reduce high-frequency vibrations caused by the positive and negative changes of the backstep adaptive control law; Define the equivalent dynamic parameter vector and its estimated value : Define a regression vector that matches the backstep adaptive control law. : A dynamic prediction regression vector for a multi-degree-of-freedom high-frequency motion simulation system is constructed based on equivalent displacement, equivalent velocity, and equivalent acceleration. : Define the corresponding dynamic prediction error : Update the estimates of equivalent mass, equivalent damping, and equivalent stiffness. : In the formula, for Diagonal positive definite adaptive gain matrix, containing , , and The update gains are the estimated values of equivalent mass, equivalent damping, and equivalent stiffness, respectively, which are used to adjust the autonomous correction speed of equivalent mass, equivalent damping, and equivalent stiffness. When the update gain increases, the autonomous correction speed of equivalent mass, equivalent stiffness, and equivalent damping increases. , which is the correction weight for the dynamic prediction error; The working range is set for the equivalent mass, equivalent stiffness, and equivalent damping in a multi-degree-of-freedom high-frequency motion simulation system. This is a projection operator used to constrain the estimates of equivalent mass, equivalent damping, and equivalent stiffness within the working range. Used to improve the convergence speed of equivalent mass, equivalent stiffness and equivalent damping by utilizing dynamic prediction errors; The coupling term between the vector used to offset the equivalent dynamic parameter estimation error and the system velocity error when differentiating the Lyapunov function ensures the stability of the backstepping control inner loop; The backstepping control inner loop is the fast closed-loop control part in a multi-degree-of-freedom high-frequency motion simulation system. It is used to construct control inputs based on the displacement error and velocity error of the system's motion trajectory, and to update the estimated values of equivalent mass, equivalent damping, and equivalent stiffness online. When the equivalent mass is determined through weighing or structural conversion of a multi-degree-of-freedom high-frequency motion simulation system, let And update the estimated value of equivalent damping. and the estimated value of equivalent stiffness ; Substituting the backstepping adaptive control law into the equivalent mass-spring-damped dynamic equation, we obtain the backstepping control error equation: In the formula, , which is the vector of the estimation error of the equivalent dynamic parameters; The cross term is caused by the estimation error of the equivalent dynamic parameters; The system acceleration error of a multi-degree-of-freedom high-frequency motion simulation system; The Lyapunov function is: In the formula, Lyapunov functions are constructed based on multi-degree-of-freedom high-frequency motion simulation systems; Through the backstep adaptive control law and Diagonal positive definite adaptive gain matrix This causes the displacement error of the system's motion trajectory. System speed error And the estimation error of the equivalent dynamic parameters remains bounded; the disturbance estimate Bounded and Under the condition that the displacement error and velocity error of the system trajectory converge to the value determined by the boundary layer thickness, Within a defined neighborhood; Set the control period of the inner loop of the backstep control to the first control period. The period of the frequency decision outer loop based on Q-table reinforcement learning is set as the second control period. Make a frequency decision and satisfy the following: In the formula, It is an integer multiple of the first control cycle; in each second control cycle The inner loop of the backstepping control keeps the driving frequency constant and updates the control input and the estimated values of equivalent mass, equivalent stiffness, and equivalent damping with the first control cycle; at the end of the second control cycle, the first control cycle is output to the frequency decision outer loop. Equivalent quality estimate within Equivalent damping estimate and equivalent stiffness estimate ; The frequency decision outer loop maintains the basic excitation amplitude of the piezoelectric ceramics in a consistent manner, and the equivalent mass, equivalent stiffness, and equivalent damping are limited to a preset range, which facilitates the frequency decision outer loop in finding the resonant frequency of the multi-degree-of-freedom high-frequency motion simulation system.
3. The frequency-unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels according to claim 1, characterized in that, The specific implementation process of the second step is as follows: During the second control cycle, with The collected acceleration data are integrated into a frequency decision window using a time step. According to the t Estimates of equivalent mass, equivalent damping, and equivalent stiffness obtained from each frequency decision window are used to calculate the theoretical resonant frequency of the multi-degree-of-freedom high-frequency motion simulation system under the current operating conditions. : In the formula, and These are the equivalent mass and equivalent stiffness under the frequency decision window, respectively; The theoretical resonant frequency is set as the resonant reference frequency for frequency search; the minimum and maximum allowable driving frequencies for the multi-degree-of-freedom high-frequency motion simulation system are defined as follows: and The frequency range within both ranges constitutes the safe search range, with a half-width of [missing information]. The safe search area is within the range of and Within the safe frequency boundary region, the theoretical resonant frequency is used as the resonant reference frequency. For half-width, perform a frequency search; To find the coordinate differences of the driving frequency between operating conditions corresponding to different equivalent stiffness, equivalent damping, and equivalent mass, the coordinates of the frequency difference between the driving frequency and the resonant reference frequency under the current frequency decision window are defined. for: In the formula, To prevent constants with a denominator of zero; The driving frequency under the current frequency decision window; When the estimated values of the equivalent mass, equivalent damping, or equivalent stiffness change, the theoretical resonant frequency is recalculated. Minimum drive frequency and highest drive frequency ; by The starting frequency of the safe search interval. Let be the frequency step size, and discretize it as follows: Frequency levels: 。 4. The frequency-unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels according to claim 1, characterized in that, The specific implementation process of the third step is as follows: Using the theoretical resonant frequency as the resonant reference frequency, an accelerometer positioned along the target vibration direction is used to collect data within each frequency decision window. Acceleration data collected from acceleration sampling points are converted into frequency-unbiased equivalent root mean square displacement values. The decision-maker based on Q-table reinforcement learning is defined as the Q-table teacher decision-maker, and the frequency-unbiased equivalent root mean square displacement values are set as the driving frequency for the Q-table teacher decision-maker to search for real working conditions. Each frequency decision window contains The acceleration sampling point, for the _th ... The acceleration data collected by each frequency decision window are root mean squared to obtain the frequency-unbiased equivalent displacement root mean square value. : In the formula, For the first The squared value of acceleration for each frequency decision window; For driving frequency For the approximate sinusoidal vibration of the main frequency components, the root mean square value of acceleration is converted into the root mean square value of the equivalent displacement without frequency bias: In the formula, This is the root mean square value of the equivalent displacement without frequency bias; The maximum amplitude of the equivalent root mean square value of the frequency-unbiased displacement is determined based on the collected acceleration data. and minimum amplitude ,right Perform monotonic interval normalization: In the formula, As a reference bias, ensure that the denominator is always greater than the numerator; The normalized vibration amplitude for adjacent frequency decision windows is explored within the range of 0 to 1; For amplitude limiting calculation, ensure Retrieve data within the range of 0 to 1; Define the change in the frequency unbiased equivalent shift of adjacent frequency decision windows as: ; No. Instant rewards for the teacher decision-maker in a frequency decision window. Set to: In the formula, , , , and All are non-negative weights; The Q-table represents the actions of the teacher decision-maker; This is a penalty for repeated frequency modulation when the same up-frequency or down-frequency action is performed continuously, but the root mean square value of the equivalent shift without frequency bias is not increased. The security cost of reaching or exceeding the security frequency boundary of the safe search range.
5. The frequency-unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels according to claim 1, characterized in that, The specific implementation process of the fourth step is as follows: Set the Q-table teacher decision-maker to include a set of actions. Set of actions The actions in the text are divided into 3 frequency levels: in, 0 indicates a decrease of one frequency level, while 0 indicates maintaining the current frequency level. This indicates an increase of one frequency level; Adjust the action of the Q-table teacher decision-making device according to frequency. Candidate driving frequencies are obtained. : The actual next drive frequency is: When a candidate drive frequency exceeds the current frequency's limit, the next drive frequency actually executed is forced to be either the lowest or highest drive frequency allowed by the multi-degree-of-freedom high-frequency motion simulation system. Simultaneously, through... Actions whose candidate drive frequencies exceed the current frequency limit are negatively evaluated; Define the Q-table in the teacher decision-making system. t Q-table state of each frequency decision window : In the formula, according to discrete Each frequency level, This is a grading function for frequency difference coordinates, which divides the frequency difference coordinates into grades. A grading function for the normalized frequency-unbiased equivalent root mean square displacement value, which classifies the frequency-unbiased equivalent root mean square displacement value into levels. This is a classification function for the change in frequency-unbiased equivalent displacement, which classifies the change in frequency-unbiased equivalent displacement into levels. This is a Q-table state encoding function that performs encoding operations on the Q-table states of the Q-table teacher decision-maker. Because the Q-table state is updated iteratively over time, the Q-table state update rule for the Q-table teacher decision-maker is defined as follows: In the formula, The discount factor for the Q-table teacher decision-maker controls the impact of the predicted Q-table state on the previous Q-table state. This is the true termination flag for the Q-table teacher decision-maker's operation, indicating that Q-table state updates have stopped. The set of actions within the safe frequency boundary; To obtain the set of actions within the safe frequency boundary under the current Q-table state. Instant rewards for the largest Q-table teacher decision-making system in China The corresponding driving frequency; The learning rate decreases with each successive learning iteration and is used to control the influence of the Q-table state in the new frequency decision window on the previous Q-table state. when When, remove actions with a frequency level of -1 from the action set; when When the frequency level is +1, remove actions from the action set; keep actions always at a frequency level of +1. and The search shall be conducted within the safe search range defined by the safe frequency boundary. Actions within the safe search range are considered safe actions, and the Q-table state update rules of the Q-table teacher decision-maker are used. The probability of safe actions of the teacher decision-maker is obtained from the Q-table. for: In the formula, , where is the temperature coefficient of the probability of a safe action in the Q-table teacher decision-maker, and represents the probability of converting a state in the Q-table into a safe action. The distribution concentration used to adjust the probability of safety actions does not represent the actual physical temperature; The mask for the safe actions of the teacher decision-maker in table Q; A safe action mask for traversing the action set during summation; To sum the states, the Q-table of the action set is traversed. The Actor-Critic student decision-maker consists of an Actor policy network and a Critic value network, and is used to receive the Q-table state from the Q-table teacher decision-maker. To avoid insufficient Q-table state updates due to insufficient sampling and action execution by the Q-table teacher decision-maker, which could lead to excessively strong constraints being passed to the Actor-Critic student decision-maker, the confidence level of the Q-table teacher decision-maker is determined jointly based on the number of times the Q-table teacher decision-maker accesses the Q-table state and the probability of safe actions. : In the formula, when the probability of the action at all three frequency levels is 1, the probability of the action at all three frequency levels is 1. hour, When the Q-table teacher decision-maker determines an action, Close to 1; The Q-table teacher decision-maker frequency search data should include at least: in, Search data for teacher decision-making frequency in the Q-table; For the first t The nine-dimensional input state vector of the Actor-Critic student decision-maker in a frequency decision window; For the first t The combination of Q-table states and actions under a frequency decision window; For the first t The action with the largest amplitude among all frequency levels under a frequency decision window; This flag is used to indicate whether the Q-table teacher decision-making system has ended due to the running time reaching the preset upper limit of the second cycle.
6. The frequency-unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels according to claim 1, characterized in that, The specific implementation process of step five is as follows: The estimates of equivalent mass, equivalent damping, and equivalent stiffness are Gaussian standardized, and the results are obtained respectively. ; In the formula, The three-dimensional one-hot encoding of the action in the (t-1)th frequency decision window; t The action with the largest amplitude among all frequency levels under a frequency decision window. The labels are used only to supervise the training of the Actor-Critic student decision-maker and are not used as input states. The Actor policy network consists of a nine-dimensional input layer, a first hidden layer with 32 neurons, a second hidden layer with 32 neurons, and a three-dimensional action score output layer. The action score output by the Actor policy network is denoted as... The probability of its safe action for: in, For Actor policy network parameters; For the Actor policy network according to Output action The score; For the Actor policy network, the output action The set action score; To the action Perform natural exponentiation on fractions; The Critic value network consists of a nine-dimensional input layer, a first hidden layer with 64 neurons, a second hidden layer with 64 neurons, and a one-dimensional state value output layer, the output of which is denoted as... ,in, For Critic value network parameters; Sample weights were set based on the confidence levels of the teacher decision-making system in the Q-table. : In the formula, Set the lower bound of the sample weights for the confidence level of the Q-table teacher decision maker; Using the probability of safe actions of the teacher decision-maker in the Q-table as soft labels, construct the Actor soft probability distillation loss. : Using the safest action with the largest vibration amplitude in the Q-table teacher decision-maker as a hard label, construct the Actor's optimal action imitation loss. : We construct the pre-training loss for the Actor policy network using Actor soft probability distillation loss and Actor optimal action imitation loss. : In the formula, The number of data points for the teacher decision-maker frequency search data in table Q; For the Actor policy network, the input state vector is nine dimensions. The probability of selecting the safe action with the largest vibration amplitude from the Q-table teacher decision-maker; and These are the weights of the Actor soft probability distillation loss and the Actor optimal action imitation loss under the output action of the Actor policy network, respectively. The data generated by the Q-table teacher decision-maker is divided into complete running data sequences according to time order. Then, starting from the end of each complete running data sequence, discount factors from different Actor-Critic student decision-makers are used. Calculate the cumulative discount reward for the Actor-Critic student decision-maker at each time step. Rearrange the data; When the calculation of the cumulative discount reward is truncated only due to reaching the running time, the cumulative discount reward not yet obtained after the complete running data sequence is completed using the Q-table state of the Q-table teacher decision-maker is as follows: Constructing Critic's experience reward loss And Q table auxiliary value loss : in, The Huber loss function is used to reduce the impact of erroneous reward data on the training of the Critic value network. Losses from Critic experience And Q table auxiliary value loss Obtain the Critic pre-training loss : In the formula, For Critic experience reward loss The weight, Auxiliary value loss for Q table The weights, and The Q-table state of the teacher decision-maker is used to initialize the auxiliary value loss of the Q-table.
7. The frequency-unbiased energy optimization and self-tuning control method for forced vibration in wind tunnels according to claim 1, characterized in that, The specific implementation process of step six is as follows: When the Actor-Critic student decision-maker cannot make decisions independently, it needs to be trained with the help of the Q-table teacher decision-maker. The Actor-Critic student decision-maker interacts directly with a multi-degree-of-freedom high-frequency motion simulation system to generate an autonomous closed-loop trajectory for training the Actor-Critic student decision-maker. : The first Instant rewards for the teacher decision-maker in a frequency decision window. The nine-dimensional input state vector of the Actor-Critic student decision-maker in the next frequency decision window The estimated value of future cumulative discount rewards Add them together, then subtract the output of the Critic value network in the Actor-Critic student decision maker at the current frequency decision window. Value prediction bias was obtained. ; Based on value prediction bias The dominance estimate is calculated by recursively working backward from the end of the complete running data sequence: In the formula, Indicates the first t The advantage estimate for each frequency decision window; This is an adjustment coefficient for the next advantage estimate, used to control the impact of differences in advantage estimates in subsequent frequency decision windows on the current advantage estimate; the Actor policy network, based on... Increase the probability of selecting actions with a value greater than 0, and decrease the probability of selecting actions with a value less than 0; Output the current Critic value network With advantage estimate The sum is used as the fixed value objective for this training. By reducing the Huber loss between the Critic value network output and the fixed value objective, the parameters of the Critic value network are updated. In the initial stage of training the Actor-Critic student decision-maker in a multi-degree-of-freedom high-frequency motion simulation system, the Actor soft probability distillation loss is retained; n Constraint discount factor of Q-table teacher decision-maker in round training Decreasing in successive rounds; among them, The constraint coefficient for the Actor soft probabilistic distillation loss controls the proportion of the influence of the teacher decision-maker on the Actor-Critic student decision-maker in the Q table. The attenuation coefficient controls how quickly the Q-table teacher decision-maker exits the Actor-Critic student decision-maker; Joint Loss of the Actor-Critic Student Decision Maker : In the formula, The joint training loss of the Actor policy network is given by the autonomous reinforcement loss of the Actor policy network. Actor soft probabilistic distillation loss and average policy entropy composition; The joint training loss of the Critic value network is given by the autonomous reinforcement loss of the Critic value network. The Q table is composed of auxiliary value loss; and These are the strategy loss and value loss obtained from the autonomous closed-loop trajectory, respectively. The average policy entropy of the Actor-Critic student decision-maker represents the dispersion of the probability of safe actions output by the Actor-Critic student decision-maker. When the probabilities of decreasing the frequency level by one, maintaining the current frequency level, and increasing the frequency level by one are relatively close, it indicates that the Actor policy network of the Actor-Critic student decision-maker still retains strong exploratory capabilities. When the probability of an action at a certain frequency level is significantly higher than that of actions at other frequency levels, it indicates that the Actor policy network tends to be deterministic. The weights for the Actor's soft probabilistic distillation loss; The weights of the average policy entropy are used to adjust the strength of the effect of the average policy entropy of the Actor-Critic student decision-maker on the Q-table teacher decision-maker and the Actor-Critic student decision-maker. The larger the value, the more the Actor-Critic student decision-maker tends to generate actions randomly, avoiding prematurely fixing a particular action. The smaller the value, the more the Actor policy network focuses on the advantage estimate. Adjust the probability of safety actions at various frequencies; when At that time, the average policy entropy is not included in the calculation of the Q-table teacher decision-maker and the Actor-Critic student decision-maker; The exit of the Q-table teacher decision-maker is determined not only by the number of training iterations of the Actor-Critic student decision-maker, but also by the number of consecutive iterations. The average cumulative discount reward, the frequency-unbiased equivalent root mean square value of displacement, and the probability of unsafe actions obtained within each independent evaluation period are jointly determined. An independent evaluation period involves pausing parameter updates of the Actor policy network and the Critic value network, allowing the Q-table teacher decision-maker and the Actor-Critic student decision-maker to independently control the multi-degree-of-freedom high-frequency motion simulation system once, under the same driving frequency, the same motion trajectory, the same safe search interval, and the same preset upper limit of running time. The immediate reward, the frequency-unbiased equivalent root mean square value of displacement, and the number of unsafe actions generated during the operation are recorded respectively. Each completed operation of the multi-degree-of-freedom high-frequency motion simulation system is recorded as one independent evaluation period. The average cumulative discount reward for the Actor-Critic student decision-maker and the Q-table teacher decision-maker is set as follows: and The equivalent root mean square values of the frequency-unbiased shift are respectively and The probabilities of unsafe actions are respectively and When the average cumulative reward of the student decision-maker in the Actor-Critic table is not less than the allowed percentage of the average cumulative reward of the teacher decision-maker in the Q table. The frequency unbiased equivalent root mean square value of the student decision-maker is not less than the allowable equivalent displacement ratio of the corresponding frequency unbiased equivalent root mean square value of the teacher decision-maker in the Q table. Furthermore, the probability of unsafe actions by the Actor-Critic student decision-maker does not exceed the deviation between the probability of unsafe actions and the allowable safe probability of actions by the teacher decision-maker in the Q-table. When the summation is complete, the Actor-Critic student decision-maker achieves independent operation requirements in three aspects: control average discount cumulative reward, frequency unbiased equivalent root mean square value of displacement, and probability of unsafe actions; and after... In each independent evaluation cycle, when the above conditions are met, the teacher constraint weight... If the value is 0, stop reading the action probabilities, Q-table states, and Q-table teacher states of the Q-table teacher decision-maker, and exit the Q-table teacher decision-maker; where the teacher constraint weights are... In the Actor-Critic student decision-maker training process, it is a dimensionless coefficient used to adjust the proportion of the Q-table teacher decision-maker frequency search data in the joint training loss of the Actor policy network and the Critic value network. To allow for reward bias, To allow for equivalent displacement deviation; After the Q-table teacher decision-maker exits, the Actor-Critic student decision-maker continues training and control using its own generated autonomous closed-loop trajectory data. At the end of training, the equivalent mass, equivalent damping, equivalent stiffness, nine-dimensional input state vector, safe frequency boundary, frequency step size, and safe action mask of the Actor-Critic student decision-maker are saved. During online deployment, only the Actor-Critic student decision-maker is retained, and it outputs actions to decrease by one frequency level, maintain the current frequency level, and increase by one frequency level based on the current resonant reference frequency and the frequency-unbiased equivalent displacement.
Citation Information
Patent Citations
A backstepping adaptive control method for a rock-damping fin hydraulic servo simulation test bench
CN110045612B
Frequency tracking search method based on fuzzy PI
CN111596544A
A frequency tracking search method based on fuzzy PD
CN111596544B
