A Helicopter Multi-Dynamic Vibration Damping Method Based on Meta-Reinforcement Learning

Abstract

The present invention discloses a multi-dynamic vibration reduction method for helicopters based on meta-reinforcement learning, and relates to the technical field of active vibration reduction control for helicopters. The present invention provides a multi-dynamic vibration reduction method for helicopters based on meta-reinforcement learning, including defining different rotational speeds as different task levels based on a Markov decision process to establish a multi-dynamic helicopter vibration response model; obtaining a random perturbation strategy based on a policy perturbation mechanism, evaluating the strategy's performance through policy fitness, and updating the strategy to obtain a meta-strategy learning algorithm for vibration control in a dynamic environment; and obtaining a rapidly adaptable data set through a sample fusion method, and performing model optimization. Therefore, the above method can achieve autonomous intelligent optimization of the helicopter vibration control strategy, ensuring stability and effectiveness under multi-dynamic flight conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of active vibration reduction control of helicopters, and in particular to a multi-dynamic vibration reduction method for helicopters based on meta-reinforcement learning. Background Art

[0002] During multi-mission helicopter flight, speed variations are a common operational requirement. Different missions, such as rapid forward flight, precise hovering, and low-speed cruising, require different speeds. This variable speed requirement not only affects the helicopter's flight performance but also significantly influences its vibration characteristics. In a multi-speed environment, the helicopter's vibration response can vary significantly due to speed variations, potentially leading to fatigue damage to the airframe and onboard equipment. Therefore, designing a vibration control strategy that can adapt to varying speed conditions is key to ensuring safe and efficient helicopter operation in multi-mission environments.

[0003] Vibration control strategies for variable speed conditions help improve a helicopter's mission flexibility and flight stability. Under varying mission requirements, helicopter vibration control strategies must rapidly adapt to speed changes to ensure smooth flight and enhance pilot comfort. Furthermore, by designing a universal vibration control strategy, the control system can recognize and adapt to different vibration modes, eliminating the need for complex parameter adjustments before each mission. This not only saves flight preparation time but also reduces potential errors from human intervention, thereby extending the helicopter's service life. Therefore, researching and developing vibration control strategies that can adapt to simultaneous changes in speed and forward ratio is crucial for optimizing helicopter performance and enabling rapid adaptation to various flight conditions in practical applications.

[0004] In addition, the speed and advance ratio changes of the helicopter belong to two different change modes and need to be analyzed separately. Moreover, the speed and advance ratio changes greatly affect the vibration response relationship of the active control, especially the time-varying characteristics of the unsteady aerodynamic force introduced by the trailing edge flap and the nonlinear characteristics of the material. These factors are often difficult to accurately describe through mathematical models. At present, although the linear parameter time-varying method and the traditional high-order harmonic method can adapt to the changes in the flight environment to a certain extent, they are mainly the continuous changes of the advance ratio rather than the speed change. The H∞ method can ensure good control performance in the worst case by minimizing the H∞ norm of the system to a certain extent. However, for the problem of helicopter vibration control, the existing theoretical methods have some shortcomings when designing vibration reduction controllers for helicopters with multiple speeds: (1) The existing algorithms cannot be effectively applied in the mode where both the speed and advance ratio change. In this multi-dynamic environment, the vibration response of the helicopter becomes more complex and dynamic. The existing control methods are not adaptable enough to deal with such variable flight conditions. (2) In the flight state where the helicopter speed and advance ratio change simultaneously, the vibration characteristics change with the changes in the flight conditions, which requires the controller to frequently adjust parameters in different states. The controller parameter tuning process of traditional methods is usually complex and time-consuming, making it difficult to adapt to changes in flight in real time. (3) Existing control methods, such as H∞ control, often sacrifice certain vibration reduction performance in order to enhance system robustness. Although the system remains stable under the worst-case scenario, the vibration reduction performance may not reach the expected optimal level under variable speeds and complex flight environments.

[0005] Therefore, it is urgent to design an intelligent vibration control method for helicopters in multi-dynamic flight environments, which can cope with the complex vibration responses caused by the simultaneous changes of speed and forward ratio, obtain more flexible control capabilities through comprehensive learning iteration in multiple environments, and realize vibration reduction control with wide application, autonomous adjustment, and optimization effect, so as to solve the problems of insufficient adaptability of vibration reduction control methods, difficulty in parameter adjustment, and poor vibration control effect in dynamic conditions, and make up for the shortcomings of existing methods. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-dynamic vibration reduction method for helicopters based on meta-reinforcement learning, which can autonomously adjust parameters, maintain efficient vibration suppression performance, is suitable for various environments, and has a wide range of applications.

[0007] To achieve the above objectives, the present invention provides a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, comprising the following steps:

[0008] S1. Establish a multi-dynamic helicopter vibration response model based on Markov decision process;

[0009] S2. Based on the policy perturbation mechanism, a meta-strategy learning algorithm for vibration control in dynamic environments is obtained;

[0010] S3. Obtain a rapidly adapted data set through sample fusion method and perform model optimization.

[0011] Preferably, step S1 includes using a Markov decision process to describe the helicopter vibration problem, defining different speeds as different tasks, and keeping them constant in the same Markov decision process. The specific expression is as follows:

[0012] MDP(S,A,P,R,Ω)

[0013] Where MDP represents Markov decision process, S represents state space, A represents action space, P represents state transition, and R represents state s t Take action a t The expected reward obtained, Ω represents the rotation speed;

[0014] Among them, the state s t is defined as:

[0015]

[0016] Y t =[F zct F zst M xct M xst M yct M yst ] T

[0017] U t =[η N-1,ct η N-1,st η N,ct η N,st η N+1,ct η N+1,st ]

[0018] Where, Represents Y t and U t The derivative of N is the number of blades, F z 、M x 、M y They represent the N-order vertical force, rolling moment, and pitching moment, respectively. The subscripts ct and st represent the cosine and sine components at time t, respectively. N-1 ,、η N ,、η N+1 , represent the N-1, N and N+1 order trailing edge flap deflection harmonics respectively;

[0019] The reward function for helicopter vibration control is defined as:

[0020]

[0021] Where, Q, Q d and R d are all positive definite matrices, and d represents the differential sign.

[0022] Preferably, the method includes assigning an initial strategy to each task based on a meta-strategy, evaluating the performance of the strategy using strategy fitness, and updating the strategy through a strategy perturbation mechanism;

[0023] Among them, the strategy fitness is defined as:

[0024]

[0025] Where f is the strategy fitness, P e is the time step, P n For P e The number of poorly performing samples.

[0026] Preferably, updating the strategy through the strategy perturbation mechanism includes establishing an inner loop framework of strategy adaptive learning for the same speed environment, specifically as follows:

[0027] First, based on the calculated strategy fitness, a random strategy is obtained through random perturbation:

[0028]

[0029] α e =λ 1-f

[0030] Where, is the optimal strategy at time t The parameter θ r Initialization strategy The parameter α e is the parameter factor, λ is the hyperparameter;

[0031] Second, we interact the random policy with the changing environment to obtain sample episodes, and then expand the sample using episode quality, speed, and advance ratio:

[0032]

[0033] Where, For the expanded sample, w(τ i ) represent the samples in episode τ i The state, action, reward, state at time t+1 and episode quality in , ρ is the regularization constant, num(τ i ) represents the episode τ iThe number of samples contained in r(τ i ) is the reward for the sample episode, are the speed and forward ratio respectively;

[0034] Then, we build a target gradient function based on the episode quality and optimize the strategy through gradient descent:

[0035]

[0036] Where, is the objective function J relative to the policy parameters The gradient, is in state s t Next take action a t The probability of strategy The logarithmic gradient of , γ represents the discount factor, represents the total discounted reward starting from time t, represents the optimization strategy parameter at time t+1, and α represents the learning rate of the inner loop.

[0037] Preferably, a general strategy learning algorithm is established for different speed environments to obtain the optimal meta-strategy, including:

[0038] Utilize optimized strategy parameters Update meta parameters

[0039]

[0040] Where, represents the updated meta-parameters in the outer loop, Represents the current meta parameters, represents the gradient of the objective function J with respect to the policy parameters of task n, represents the objective function value of task n, β is the learning rate of the outer loop, and N is the total number of tasks.

[0041] Preferably, the method includes obtaining a new sample set by using an optimal meta-strategy; calculating the similarity between the current sample and the experience sample to obtain a high-quality sample set; obtaining a sample subset based on the sampling probability of each sample in the high-quality sample set; fusing the sample subset with the new sample set to update the meta-strategy;

[0042] Among them, the similarity ζ between the current sample and the experience sample is:

[0043]

[0044] Where, v current is the speed and forward ratio label of the current sample, are the speed and advance ratio labels of the empirical samples, || || represents the Euclidean norm;

[0045] The sampling probability is obtained according to the quality of the episodes corresponding to the sample:

[0046]

[0047] Where p i It represents the probability of the i-th sample being selected, and the subscript j represents the high-quality sample.

[0048] Therefore, the present invention adopts the above-mentioned helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, which has the following technical effects:

[0049] (1) By designing an inner-outer loop architecture, the complex vibration response of the helicopter under different changes in speed and forward ratio is described by defining discrete changes in speed as different tasks and continuously changing forward ratio as the dynamic environment in a specific task.

[0050] (2) Under changing flight conditions, the adaptive adjustment mechanism greatly reduces the complexity of manual adjustments, enabling the system to automatically optimize control parameters according to real-time flight status, quickly respond to environmental changes, and improve the convenience and efficiency of operation.

[0051] (3) Through continuous learning and optimization, the algorithm can maintain efficient vibration suppression performance in various flight environments, ensuring the safety and durability of the equipment. It can not only maintain the stability of the system, but also achieve excellent vibration reduction effects under variable speed and forward ratio conditions.

[0052] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 In an embodiment of a multi-dynamic vibration reduction method for a helicopter based on meta-reinforcement learning;

[0054] Figure 2 This is a graph of an average reward function when training in a multi-dynamic environment in an embodiment of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning;

[0055] Figure 3 This is a graph of a reward function for a change in forward ratio at 720 rpm in an embodiment of a multi-dynamic vibration reduction method for a helicopter based on meta-reinforcement learning;

[0056] Figure 4 In an embodiment of a multi-dynamic vibration reduction method for a helicopter based on meta-reinforcement learning, the trailing edge flap angle input is calculated when the speed is 720 rpm and the forward ratio changes. Figure 4(a) is the cosine and sine components of the trailing edge flap deflection angle, Figure 4 Middle (b) is the waveform of the trailing edge flap deflection angle change;

[0057] Figure 5 This is a vertical force control diagram for a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning at a speed of 720 rpm under a change in the forward ratio, wherein: Figure 5 (a) is the cosine and sine components of the vertical force, Figure 5 (b) is the waveform of vertical force variation;

[0058] Figure 6 This is a roll moment control diagram for a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning at a speed of 720 rpm under a change in the forward ratio, wherein: Figure 6 (a) is the cosine and sine components of the rolling moment, Figure 6 (b) is the rolling moment change waveform;

[0059] Figure 7 This is a pitch moment control diagram for a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning at a speed of 720 rpm under a change in the forward ratio, wherein: Figure 7 (a) is the cosine and sine components of the pitching moment, Figure 7 (b) is the pitching moment change waveform;

[0060] Figure 8 This is a graph of a reward function for a change in forward ratio at 960 rpm in an embodiment of a multi-dynamic vibration reduction method for a helicopter based on meta-reinforcement learning;

[0061] Figure 9 The embodiment of the multi-dynamic vibration reduction method of a helicopter based on meta-reinforcement learning is an input of the trailing edge flap angle when the speed is 1032 rpm and the forward ratio changes, wherein, Figure 9 (a) is the cosine and sine components of the trailing edge flap deflection angle, Figure 9 Middle (b) is the waveform of the trailing edge flap deflection angle change;

[0062] Figure 10 This is a vertical force control diagram for a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning at a speed of 1032 rpm when the forward ratio changes, wherein: Figure 10 (a) is the cosine and sine components of the vertical force control, Figure 10 Middle (b) is the vertical force control change waveform;

[0063] Figure 11 This is a roll moment control diagram under the condition of a change in the forward ratio at a speed of 1032 rpm in an embodiment of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, wherein: Figure 11 (a) is the cosine and sine components of the rolling moment control, Figure 11 (b) is the rolling moment control change waveform;

[0064] Figure 12 The present invention is a pitch moment control diagram of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning at a speed of 1032 rpm under a change in the forward ratio, wherein: Figure 12 (a) is the cosine and sine components of the pitch moment control, Figure 12 (b) is the pitch moment control change waveform. DETAILED DESCRIPTION

[0065] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0066] like Figure 1 As shown, the present invention provides a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, comprising the following steps:

[0067] S1. In order to establish a general helicopter vibration controller framework that can be applied at multiple speeds, the helicopter vibration problem is converted into a Markov process, and different speeds are defined as different tasks, thereby achieving a description of the helicopter vibration problem at multiple speeds and laying the foundation for the subsequent design of a meta-learning controller with stronger generalization capabilities.

[0068] Due to factors such as coordinate system conversion and hub filtering in the transmission process of helicopter vibration, its vibration presents typical harmonic order characteristics. Therefore, the transmission process of helicopter vibration is specifically expressed as follows:

[0069] First, define the speed Ω as 1 / (rmin -1 ), for the N blades of a helicopter, they are usually expressed in a rotating coordinate system by N-1 / (rmin -1 )、N / (rmin -1 ) and N+1 / (rmin -1 ) order trailing edge flap harmonics are reasonably coordinated to generate effective additional aerodynamic forces and moments, thereby reducing the hub vibration load in the non-rotating coordinate system.

[0070] When the helicopter speed changes, N / (rmin -1 ) is converted into a vertical force in a non-rotating system, N / (rmin -1 ) rolling moment M x , pitching moment M y They come from the rotating system N-1 / (r·min -1 ) and N+1 / (r·min-1 )’s deformation moment.

[0071] When N=4, the transfer function G(s) from the trailing edge flap input to the vibration response of the multi-output helicopter can be expressed as:

[0072]

[0073] Among them, Y s =[F z M x M y ] T , represents the vertical force, rolling moment and pitching moment at the helicopter hub, U s =[η3 η4 η5] T represents the trailing edge flap deflection angle 3 / (rmin -1 ), 4 / (rmin -1 ) and 5 / (rmin -1 ), F1, F2, F3, F4, F5 represent the dynamic relationship function according to the change of speed and forward ratio, as well as They represent the corresponding transfer functions respectively.

[0074] In order to use reinforcement learning to solve the helicopter vibration control problem, it is necessary to express it as a Markov decision process (MDP). In the context of reinforcement learning, the interaction between the agent and its environment is described by a Markov decision process, usually represented by (S, A, P, R, γ), where S is the state space, A is the action space, P is the state transition, and R is the state in state s. t Take action a t The expected reward obtained, γ∈[0,1) is the discount factor.

[0075] In order to construct the MDP of the helicopter vibration control problem, this embodiment transforms the state s t Defined as:

[0076]

[0077] Among them, U t =[η 3ct η 3st η 4ct η 4st η 5ct η 5st ] represents the trailing edge flap deflection angle 3 / (rmin at the current time t -1 ), 4 / (rmin -1 ) and 5 / (rmin -1 )’s cosine and sine components; Y t =[Fzct F zst M xct M xst M yct M yst ] T Indicates the vertical force, rolling moment and pitching moment in N / (r·min -1 ) order cosine and sine components; d(·) represents the differential of (·), such as dY t Y t The differential of represents the rate of change of the vibration output.

[0078] against Figure 1 The dynamic vibration model of the helicopter rotor with trailing edge flaps is established in the figure. The harmonic components calculated by the controller will act on the rotor system through the harmonic synthesis module to achieve vibration control. t Defined as U t , which can be directly used as the harmonic input of the controller to adjust the trailing edge flap.

[0079] At this point, the Markov decision can be described as:

[0080] s t+1 ~P(s t+1 ∣s t ,a t )

[0081] Where, P(s t+1 ∣s t ,a t ) means in state s t Next take action a t Then transfer to state s t+1 For the vibration control problem of the helicopter, the reward function r(s t ,a t ) is defined as:

[0082]

[0083] Where, Q, Q d and R d are all positive definite matrices; the reward function takes into account the vibration output, the rate of change of the vibration output, and the rate of change of the trailing edge flap, and calculates the value of the reward function through Q, Q d and R d Adjust the weight.

[0084] After the above process, a traditional Markov problem has been constructed. When the rotor speed Ω changes, not only the vibration model of the helicopter with trailing edge flaps is changed, but also the need to reduce the basic vibration load D is changed. t, which increases the complexity of classifying the vibration control problem into the traditional MDP framework (S, A, P, R). Therefore, a new type of MDP (S, A, P, R, Ω) is proposed, which introduces an additional parameter Ω to describe the dynamic changes.

[0085] For the helicopter vibration model with different rotor speeds Ω, the vibration control task is further subdivided into multiple specific tasks, each with fixed time-varying dynamics, forming a series of specific MDPMs i :(S,A,P,R,Ω i ), i = 1,...,n. Each MDP assumes that Ω remains constant. The meta-reinforcement learning method with this setting aims to learn a general strategy that can effectively operate between these MDPs to deal with vibration control problems in dynamic environments and achieve autonomous adjustment and adaptation in new environments.

[0086] S2. Based on the established multi-dynamic helicopter vibration response model, a meta-strategy learning framework in a dynamic environment is constructed as follows:

[0087] For each individual mission, the forward ratio μ is in real-time change. The working environment of the helicopter is actually dynamic. In this embodiment, the dynamic environment Defined as:

[0088]

[0089] Where, [C1,...,C t-1 ,C t ,…] represents a set of different dynamic environments; since the disturbance changes are continuous, the dynamic environment The definition of only means that the environment is changing in real time, and does not mean that the disturbance change can be clearly divided into multiple discrete stages, because this is inconsistent with the actual situation.

[0090] Furthermore, during helicopter operation, it should not be assumed that the controller itself is fully aware of the dynamic situation, i.e. the changing environment to which it is assigned. The specific circumstances of the environment are unknown to the meta-RL controller. To reduce the number of algorithm iterations and improve learning efficiency, the meta-RL controller should have a performance evaluation mechanism. Only when environmental changes affect the learned optimal policy to a certain extent should the learned optimal policy be fine-tuned. In other words, when environmental changes do not affect the learned optimal policy, the learned optimal policy remains highly applicable even after the environmental changes.

[0091] In order to adapt to the vibration control problem of the helicopter, the strategy fitness f is defined as:

[0092]

[0093] In the formula, the strategy fitness f is used as an indicator to evaluate the strategy, P e is the evaluation time step, P n is defined as e The number of samples that perform poorly for the given sample type.

[0094] Specifically, at time t, calculate from (tP e ,…,t) of the total P e From the definition of the reward function, we can see that the reward for the helicopter vibration control problem is composed of the vibration control level, the control amount, and the change in the control amount. When r(a t ,s t )≤r ε When , it means that the strategy learned in the previous environment performs poorly in the current environment, and this sample is included in P n And calculate the fitness through the definition of strategy fitness.

[0095] When the strategy fitness f≤f min When P e There are enough samples in the sample that perform poorly, and it is necessary to make incremental adjustments to the strategy and ensure that the strategy is adaptable to the new environment.

[0096] When the environment changes, the agent tends to take the previously optimal action, which may lead to a local optimum. Furthermore, in a new environment, using a randomly initialized policy for training requires the agent to take longer to learn the optimal policy. On the other hand, when using importance sampling, the difference between the policy distribution used for sampling and the policy distribution used for updating increases, and the variance tends to infinity, making it impossible to effectively improve the distribution of the learned optimal policy. To address these issues, the meta-reinforcement learning control algorithm designs a policy perturbation mechanism, which learns the parameters of the optimal policy based on environmental changes. This not only reduces training time but also prevents the agent from falling into a local optimum.

[0097] Specifically, based on the calculated strategy fitness f, a new strategy is obtained through random perturbation:

[0098]

[0099] Where, is the previously learned optimal strategy The parameter θ r is a random initialization strategy The parameter α e It is a parameter factor calculated based on the strategy fitness f:

[0100] α e=λ 1-f

[0101] Where λ is a hyperparameter that determines the degree of policy perturbation; the larger λ is, the smaller the degree of policy perturbation is.

[0102] Through the above method, a new strategy with randomness can be obtained Through new strategies and changing environment By interacting, a collection of new sample episodes can be obtained. Since the helicopter vibration control problem is different from other problems with clear end conditions, such as obstacle avoidance problems and target search problems, in order to simplify the problem, a sample episode τ i It is defined as the time from the beginning of the episode to the time when P e Samples appear r t ≤r ε When τ i End resets the environment and starts the next episode τ i+1 .

[0103] This definition can measure the adaptability of an episode to a new environment that has changed. If an episode reaches the termination condition only after interacting with the current environment and obtaining multiple samples, it means that the episode is of high quality. The episode quality is defined as:

[0104]

[0105] Where ρ is the regularization constant, num(τ i ) represents the episode τ i The specific number of samples included in ; From this definition, it can be seen that the quality of the episode is determined by the average value of the reward and of the entire episode, so for the episode τ i For each sample in the example, the experience tuple of the sample can be expanded by combining the current speed and forward ratio information. Expand to in, and w(τ i ) represent the state and action in the episode, the corresponding reward, the next moment state, and the corresponding episode quality, respectively. represents the set of speed and forward ratio. Since the episode quality remains constant for all samples in the entire episode, this expanded It does not affect the complexity of the algorithm. On the contrary, this extension will better record the performance of samples from different episodes when the environment changes, and provide high-quality sample selection for subsequent strategy optimization or online rapid adaptation.

[0106] According to the definition of episode quality, the objective gradient function based on episode quality weighting is rewritten as:

[0107]

[0108] Where, Represents the objective function J relative to the policy parameter gradient; Represents the trajectory τ i It is parameterized as Strategy Generated,trajectory τ i Contains a series of states, actions and rewards obtained; w(τ i ) is a weight function that depends on the trajectory τ i ; is in state s t Next take action a t The probability of strategy The logarithmic gradient of is the sum of discounted rewards starting at time t, with the discount factor γ determining the weight of future rewards relative to immediate rewards. As can be seen, by weighting episode quality, the strategy focuses more on episodes that deliver high rewards, effectively improving the quality and speed of learning in a dynamic environment. This also simulates the human brain's increased focus on higher-yield actions.

[0109] At this point, the formula for gradient descent is:

[0110]

[0111] Where, represents the optimization strategy parameter at time step t+1, which is updated after the gradient descent step based on the current parameter; represents the current policy parameter at time step t, which is the parameter value before the update; α represents the learning rate of the inner loop, which is used to control the step size of the parameter adjustment at each update; Represents the objective function J relative to the policy parameter gradient.

[0112] Through the above method, an inner loop framework for adaptive learning of vibration strategies under the same speed environment was established. Next, a general strategy learning algorithm for different speed environments was proposed based on meta-learning (MAML method).

[0113] In meta-learning, although the training and testing tasks are different, all tasks share the same action and state space. The main difference between tasks is that each task corresponds to a Markov decision process (MDP) with a different reward function, environment, or dynamics. The agent trained in meta-learning learns how to adapt to these different MDPs when faced with multiple tasks.

[0114] In this embodiment, a model-independent meta-policy learning method is used, which learns a general policy by combining updated policies from multiple tasks. The main goal of the meta-learning framework is to evaluate the performance of the updated model parameters on the test set, rather than just focusing on its performance during training. The core of this method is that it is consistent with the broad goal of meta-learning: "learning to learn." In this way, the model not only learns solutions to specific tasks, but also acquires the ability to learn across tasks, so that it can quickly adapt and optimize when faced with new tasks. This reflects a deep learning skill that meta-learning aims to cultivate, namely the ability to quickly master new knowledge through experience.

[0115] Specifically, after fast adaptation on all tasks according to the gradient descent formula, all adjusted strategies are used To update the meta parameters

[0116]

[0117] Where, Represents the meta-parameters updated in the outer loop; Represents the current meta parameter, that is, the parameter value before the update; represents the sum of the gradients of the objective function J with respect to the policy parameters of task n; represents the objective function value calculated under task n, β is the learning rate of the outer loop, and N is the total number of tasks.

[0118] After internal updates, an adjusted policy is generated for each task, and the meta-policy is updated by optimizing the objective function of the master task. For tasks with time-varying dynamics, the problem is decomposed into multiple subtasks with fixed time-varying dynamics, each corresponding to a specific MDP model parameter. The MAML approach enables the agent to learn how to effectively handle these different time-varying dynamic tasks, thereby enhancing the flexibility and effectiveness of vibration control.

[0119] S3, online rapid adaptation and model optimization, as follows:

[0120] After obtaining the meta-strategy, due to the differences between the real environment and the simulation training environment, it is necessary to obtain the latest strategy that adapts to the new environment through online data interaction.

[0121] Traditional reinforcement learning algorithms perform gradient updates at each step by extracting a small batch of data from the experience replay buffer. This small batch contains experience samples collected in past policy iterations, but this approach is difficult to adapt to new environments, especially when the environment is very different from the training environment and contains disturbances. Therefore, in this embodiment, a new sample fusion method is proposed in the meta-reinforcement learning algorithm. By fusing the latest samples that perform well in the policy experience and training process, and using the fused and corrected data for gradient descent training, the network is optimized. The specific steps can be divided into two steps: First, the expanded experience tuple in the meta-policy learning stage is In which w(τ i ) represents the specific value of and In the real environment, by comparing the speed and forward ratio of the current environment with the samples in the experience sample pool, we can get the most similar sample set to the current environment during training. The labels of the current speed and forward ratio are defined as v current =(ω current ,μ current ), then the weight ζ of the experience sample can be expressed as:

[0122]

[0123] Where, and ||v current || is the Euclidean norm of the corresponding vector. The closer the ζ value is to 1, the more the sample matches the speed and forward ratio of the current environment. In order to select high-quality samples that are more suitable for the current environment, only samples with ζ≥ζ are selected during the screening process. max Sample.

[0124] For the filtered sample set, use the episode score w(τ i ) as the weighting factor, and implement weighted random sampling. Sample The probability of selection p i Its episode score w(τ i ) is proportional to:

[0125]

[0126] Where p i Represents the sample τ i The probability of being selected; w(τ i ) is the sample τ i The weight indicates the quality or importance of the sample. The higher the weight, the more important the sample is or the more it matches the current environmental conditions. Indicates that in the screening process, all the conditions ζ≥ζ are met maxThe sample τ i The weighted sum of the prioritized data is defined as batch C. The latest m interaction policy experiences q are then added to the batch to obtain the final dataset M for online fast adaptation at the current time, i.e., M = (C) ∪ {q}. M is then used to iterate and learn the meta-strategy.

[0127] In summary, the meta-reinforcement learning algorithm designed in this embodiment divides the helicopter vibration control problem into multiple tasks based on rotational speed. Each task is independently learned and its sample quality is evaluated. These evaluated scores serve as sample weights, helping to integrate learning results in the outer loop of meta-learning to form a broadly adaptable meta-policy. Through brain-inspired intelligent simulation, the algorithm performs performance evaluation and policy optimization in a dynamic environment. It then adjusts policy updates based on performance weighting, ultimately achieving a meta-reinforcement learning algorithm that can rapidly adapt to new environments.

[0128] Simulation experiment

[0129] Simulation setup: To verify the effectiveness of the proposed dynamic helicopter vibration reduction method with intelligent parameter self-adjustment, the helicopter vibration reduction system was first integrated and designed in Matlab 2019b and Python 3.6, and a simulation experiment was conducted. The specific process is as follows:

[0130] (1) Parameter settings

[0131] as well as The settings are shown in Table 1.

[0132] Table 1 Transfer function parameter settings

[0133]

[0134] The dynamic relationship functions F1, F2, F3, F4, and F5 of the speed Ω and forward ratio μ are set as follows:

[0135] F1=-14.5+35.3μ+241.8μ 2 -0.12μΩ

[0136] F2=-14.2+34.2μ+246.8μ 2 -0.11μΩ

[0137] F3=-58.5-12.0μ+0.105Ω+1733μ 2 -0.5μΩ

[0138] F4=-3.7-12.3μ+198.4μ 2 -0.041μΩ

[0139] F5=-14.0+28.2μ+0.025Ω+256.8μ 2 -0.1μΩ

[0140] (2) Meta-strategy reinforcement learning parameter settings, as shown in Table 2.

[0141] Table 2 Meta-strategy reinforcement learning parameters

[0142] parameter Numerical parameter Numerical Optimizer Adam Nonlinear functions ReLU The number of hidden units in each layer 256 Sample pool 10000 Number of hidden layers 2 Entropy target -dim(A) Outer loop learning rate β 1e-4 Inner loop learning rate α 1e-3 <![CDATA[Minimum policy fitness f min > 0.5 Discount factor γ 0.998 Regularization constant ρ 0.5 <![CDATA[Evaluate the running time step P e > 20 <![CDATA[Sample threshold ζ max > 0.6 Small sample batch m 128 Outer loop iteration number P 1000 Inner loop iteration number K 1 or 2

[0143] like Figure 2 As shown in the figure, the performance of the meta-reinforcement learning algorithm designed in this embodiment is better than the basic meta-reinforcement learning algorithm under different inner loop times (K). When K=2, it shows higher average returns and faster convergence speed than when K=1, indicating that more inner loop times contribute to more comprehensive strategy learning and adaptation to environmental changes. The algorithm designed in this patent shows higher final returns and smaller fluctuations when K=2, which shows that the algorithm has higher efficiency and stability when processing complex tasks, and can achieve better performance in diversified tasks. Figure 3 As shown, in this embodiment, a dynamic forward ratio change method is designed. The vibration control effect under the conditions of 720rpm and dynamic forward ratio change is as follows: Figures 4 to 7 As shown. Among them, Figure 4 The control input table corresponding to the trailing edge flap deflection input is shown. Figure 5 and Figure 6 The effects of pitch moment and pitch angular velocity control are demonstrated respectively. The system exhibits a faster vibration convergence speed at a rotation speed of 720 rpm and maintains a smaller vibration amplitude after stabilization. Figure 7 This further demonstrates the excellent performance of pitch moment control under conditions of varying forward ratios, with the system effectively reducing vibration and maintaining a stable state. It also demonstrates a fast and stable response.

[0144] like Figure 8 As shown, this embodiment designs another different dynamic advance ratio change method. The corresponding trailing edge flap input is shown in Figure 9 At 1032rpm and Figure 8 For the vibration control effect under the condition of medium dynamic forward ratio, please refer to Figures 10 to 12 , Figure 10 The rapid convergence of the vertical force is shown, and the system quickly regains stability after the initial disturbance, indicating that the control strategy can effectively cope with the advance ratio changes in a highly dynamic environment. Figure 11 The rolling moment control situation is demonstrated. The curve shows that the system reaches a stable state in a relatively short time and the vibration amplitude is significantly reduced, which reflects the advantage of the algorithm in lateral stability control. Figure 12This demonstrates the stable performance of pitch moment control under the same conditions, and the system maintains low vibration levels even when processing dynamic inputs. Overall, the control method of this embodiment has excellent vibration suppression capabilities and system stability under high speed and complex operating conditions, and vibration control in different directions can quickly achieve and maintain a stable state.

[0145] Therefore, the present invention adopts the above-mentioned multi-dynamic vibration reduction method of helicopter based on meta-reinforcement learning, which can realize vibration reduction control with wide application, autonomous adjustment and optimization effect, solve the problems of insufficient adaptability of vibration reduction control methods, difficulty in parameter adjustment, poor vibration control effect in dynamics, etc., and ensure stability and effectiveness in multi-dynamic flight states.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, characterized in that: The following steps are involved: S1. Based on the Markov decision process, a multi-dynamic helicopter vibration response model is established, in which different speeds are defined as different tasks and kept constant in the same Markov decision process. S2. Based on the policy perturbation mechanism, a meta-strategy learning algorithm for vibration control in dynamic environments is obtained: Based on the meta-strategy, we assign an initial strategy to each task, evaluate the strategy performance using strategy fitness, and update the strategy through a strategy perturbation mechanism, including: For the same speed environment, establish an inner loop framework for strategy adaptive learning: Based on the calculated strategy fitness, a random strategy is obtained through random perturbation; The random strategy interacts with the changing environment to obtain sample episodes, and the sample is expanded using episode quality, speed and forward ratio; among them, the sample episodes Defined as the time from the beginning of the episode to the Sample appearance reward function When the sample episode End reset environment and start next sample episode ; Establish a target gradient function based on episode quality and optimize the strategy through gradient descent method; episode quality is defined as: Where, For sample episodes Rewards, is the regularization constant, Indicates sample episodes The specific number of samples included in For different speed environments, a general strategy learning algorithm is established, and the meta-parameters are updated using the optimized strategy parameters to obtain the optimal meta-strategy. S3. Obtain a rapidly adapted data set through sample fusion method and perform model optimization.

2. A helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning according to claim 1, characterized in that: In step S1, the helicopter vibration problem is described using a Markov decision process. Different speeds are defined as different tasks and are kept constant in the same Markov decision process. The specific expression is as follows: Where, represents a Markov decision process, represents the state space, represents the action space, Indicates state transition, Indicates status Take action Expected rewards, Indicates the rotation speed; Among them, the status is defined as: Where, 、 Respectively and The derivative of is the number of blades, 、 、 Respectively Vertical force, rolling moment, pitching moment, subscript 、 Respectively represent The cosine and sine components of the moment, 、 、 Respectively 、 and Harmonic quantity of trailing edge flap deflection angle; The reward function for helicopter vibration control is defined as: Where, 、 and are all positive definite matrices, Represents the differential symbol.

3. The helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning according to claim 1 is characterized in that: In step S2, the initial strategy for each task is assigned based on the meta-strategy, the performance of the strategy is evaluated using the strategy fitness, and the strategy is updated through the strategy perturbation mechanism; Among them, the strategy fitness is defined as: Where, is the strategy fitness, is the time step, For The number of poorly performing samples.

4. A helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning according to claim 3, characterized in that: Updating the policy through the policy perturbation mechanism involves establishing an inner loop framework for policy adaptive learning for the same speed environment, as follows: First, based on the calculated strategy fitness, a random strategy is obtained through random perturbation: Where, for Time-optimal strategy Parameters, Initialization strategy Parameters, is the parameter factor, is a hyperparameter, is the strategy fitness; Second, we interact the random policy with the changing environment to obtain sample episodes, and then expand the sample using episode quality, speed, and advance ratio: Where, For the expanded sample, 、 、 、 、 Respectively represent the samples in the sample episode The state, action, reward, Moment status and episode quality, is the regularization constant, Indicates sample episodes The number of samples included in As a reward for the sample episode, 、 are the speed and forward ratio respectively; Then, we build a target gradient function based on the episode quality and optimize the strategy through gradient descent: Where, is the objective function Relative to the strategy parameters The gradient, Is in state Take action The probability of strategy The logarithmic gradient of represents the discount factor, Indicates from The total amount of discounted rewards starting at time t, express The optimization strategy parameters at the moment, Represents the learning rate of the inner loop.

5. The method for helicopter multi-dynamic vibration reduction based on meta-reinforcement learning according to claim 4, characterized in that: A general strategy learning algorithm is established for different speed environments to obtain the optimal meta-strategy, including: Utilize optimized strategy parameters Update meta parameters : Where, represents the updated meta-parameters in the outer loop, Indicates the current meta parameters, Represents the objective function Relative to the task The gradient of the policy parameters and, Indicates a task The objective function value of is the learning rate of the outer loop, is the total number of tasks.

6. A helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning according to claim 5, characterized in that: In step S3, it includes obtaining a new sample set using the optimal meta-strategy; Calculate the similarity between the current sample and the experience sample to obtain a high-quality sample set; obtain a sample subset based on the sampling probability of each sample in the high-quality sample set; fuse the sample subset with the new sample set to update the meta-strategy; Among them, the similarity between the current sample and the experience sample for: Where, is the speed and forward ratio label of the current sample, are the speed and forward ratio labels of the empirical samples, represents the Euclidean norm; The sampling probability is obtained according to the quality of the episodes corresponding to the sample: Where, Indicates the The probability that a sample is selected, subscript Indicates high-quality samples. Indicates the quality of the episode.

Images