Helicopter multi-dynamic vibration reduction method based on meta reinforcement learning

Abstract

The invention discloses a multi-dynamic vibration reduction method of a helicopter based on meta reinforcement learning, and relates to the technical field of active vibration reduction control of helicopters. The invention provides a helicopter multi-dynamic vibration reduction method based on meta reinforcement learning, which comprises the following steps: defining different rotating speeds as different task levels based on a Markov decision process, and establishing a multi-dynamic helicopter vibration response model; a random disturbance strategy is obtained based on a strategy disturbance mechanism, the performance of the strategy is evaluated through strategy fitness, the strategy is updated, and a meta-strategy learning algorithm of vibration control in the dynamic environment is obtained; and a fast adaptive data set is obtained through a sample fusion method, and model optimization is carried out. Therefore, by the adoption of the method, autonomous intelligent optimization of the helicopter vibration control strategy can be achieved, and stability and effectiveness in the multi-dynamic flight state are ensured.

Description

Technical Field

[0001] The invention relates to the technical field of active vibration reduction control of helicopters, and in particular to a multi-dynamic vibration reduction method of helicopters based on meta-reinforcement learning. Background Art

[0002] In the multi-mission flight of helicopters, the change of rotation speed is a common operational requirement. Different tasks such as fast forward, precise hovering, and low-speed cruising have different requirements for the helicopter's rotation speed. This variable rotation speed requirement not only affects the flight performance of the helicopter, but also significantly affects its vibration characteristics. In a multi-speed environment, the vibration response of the helicopter will change significantly due to the change of rotation speed, which may cause fatigue damage to the fuselage and onboard equipment. Therefore, designing a vibration control strategy that can adapt to different rotation speed conditions is the key to ensuring the safe and efficient operation of the helicopter in a multi-mission environment.

[0003] The vibration control strategy under variable speed conditions helps to improve the mission flexibility and flight stability of the helicopter. Under different mission requirements, the helicopter vibration control strategy needs to quickly adapt to speed changes to ensure flight stability and improve pilot comfort. In addition, through the design of a universal vibration control strategy, the control system can identify and adapt to different vibration modes without the need for complex parameter adjustments before each mission. This not only saves flight preparation time, but also reduces potential errors caused by human intervention, thereby extending the service life of the helicopter. Therefore, researching and developing vibration control strategies that adapt to environments where speed and forward ratio change simultaneously is of great significance for optimizing the performance of helicopters and being able to quickly adapt to various flight conditions in practical applications.

[0004] In addition, the speed and lead ratio of the helicopter belong to two different change modes and need to be analyzed separately. Moreover, the speed and lead ratio changes greatly affect the vibration response relationship of active control, especially the time-varying characteristics of the unsteady aerodynamic force introduced by the trailing edge flap and the nonlinear characteristics of the material. These factors are often difficult to accurately describe through mathematical models. At present, although the linear parameter time-varying method and the traditional high-order harmonic method can adapt to the changes in the flight environment to a certain extent, they are mainly the continuous change of the lead ratio rather than the speed change. The H∞ method can ensure good control performance in the worst case by minimizing the H∞ norm of the system to a certain extent. However, for the problem of helicopter vibration control, the existing theoretical methods have some shortcomings when designing vibration reduction controllers for helicopters with multiple speeds: (1) The existing algorithms cannot be effectively applied in the mode where both the speed and the lead ratio change. In this multi-dynamic environment, the vibration response of the helicopter becomes more complex and dynamic. The existing control methods are not adaptable enough to deal with such variable flight conditions. (2) In the flight state where the helicopter speed and the lead ratio change simultaneously, the vibration characteristics change with the change of flight conditions, which requires the controller to frequently adjust parameters under different states. The controller parameter tuning process of traditional methods is usually complicated and time-consuming, and it is difficult to adapt to changes in flight in real time. (3) Existing control methods, such as H∞ control, often sacrifice certain vibration reduction performance in order to enhance the robustness of the system. Although the system remains stable under the worst case, the vibration reduction performance may not reach the expected optimal level under variable speeds and complex flight environments.

[0005] Therefore, it is urgent to design an intelligent vibration control method for helicopters in multi-dynamic flight environments, which can cope with the complex vibration responses caused by the simultaneous changes in speed and forward ratio, obtain more flexible control capabilities through comprehensive learning iteration in multiple environments, and realize vibration reduction control with wide application, autonomous adjustment, and optimization effect, solve the problems of insufficient adaptability of vibration reduction control methods, difficulty in parameter adjustment, and poor vibration control effect in dynamics, and make up for the shortcomings of existing methods. Summary of the invention

[0006] The purpose of the present invention is to provide a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, which can autonomously adjust parameters, maintain efficient vibration suppression performance, is suitable for various environments, and has a wide range of applications.

[0007] To achieve the above object, the present invention provides a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, comprising the following steps:

[0008] S1. Establish a multi-dynamic helicopter vibration response model based on Markov decision process;

[0009] S2, based on the strategic perturbation mechanism, obtain the meta-strategy learning algorithm for vibration control in dynamic environment;

[0010] S3. Obtain a fast-adapting data set through sample fusion method and perform model optimization.

[0011] Preferably, step S1 includes using a Markov decision process to describe the helicopter vibration problem, defining different rotation speeds as different tasks, and keeping them constant in the same Markov decision process. The specific expression is as follows:

[0012] MDP(S,A,P,R,Ω)

[0013] In the formula, MDP represents Markov decision process, S represents state space, A represents action space, P represents state transition, and R represents state s t Take action a t The expected reward obtained, Ω represents the rotation speed;

[0014] Among them, the state s t is defined as:

[0015]

[0016] Y t =[F zct F zst M xct M xst M yct M yst ] T

[0017] U t =[η N-1,ct η N-1,st η N,ct η N,st η N+1,ct η N+1,st ]

[0018] In the formula, Respectively represent Y t and U t The derivative of, N is the number of blades, F z 、M x 、M y They represent the N-order vertical force, rolling moment, and pitching moment, respectively. The subscripts ct and st represent the cosine and sine components at time t, respectively. N-1 ,,η N ,,η N+1 , represent the N-1, N and N+1 order trailing edge flap deflection angle harmonics respectively;

[0019] The reward function for helicopter vibration control is defined as:

[0020]

[0021] In the formula, Q, Q d and R d are all positive definite matrices, and d represents the differential sign.

[0022] Preferably, it includes assigning an initial strategy to each task based on a meta-strategy, evaluating the performance of the strategy using strategy fitness, and updating the strategy through a strategy perturbation mechanism;

[0023] Among them, the strategy fitness is defined as:

[0024]

[0025] In the formula, f is the strategy fitness, P e is the time step, P n For P e The number of poorly performing samples.

[0026] Preferably, updating the strategy through the strategy perturbation mechanism includes establishing an inner loop framework of strategy adaptive learning for the same speed environment, as follows:

[0027] First, based on the calculated strategy fitness, a random strategy is obtained through random perturbation:

[0028]

[0029] α e =λ 1-f

[0030] In the formula, is the optimal strategy at time t The parameter θ r Initialization strategy The parameter α e is the parameter factor, λ is the hyperparameter;

[0031] Second, we interact the random policy with the changing environment to obtain sample episodes, and expand the sample using episode quality, rotation speed, and advance ratio:

[0032]

[0033] In the formula, For the expanded sample, w(τ i ) represent the samples in episode τ i The state, action, reward, state at time t+1 and episode quality in , ρ is the regularization constant, num(τ i ) represents the episode τ iThe number of samples contained in r(τ i ) is the reward for the sample episode, are the speed and forward ratio respectively;

[0034] Then, we build a target gradient function based on the episode quality and optimize the strategy by gradient descent:

[0035]

[0036] In the formula, is the objective function J relative to the policy parameters The gradient of is in state s t Take action a t The probability of strategy The logarithmic gradient of , γ represents the discount factor, represents the total discounted reward starting from time t, represents the optimization strategy parameter at time t+1, and α represents the learning rate of the inner loop.

[0037] Preferably, a general strategy learning algorithm is established for different speed environments to obtain the optimal meta-strategy, including:

[0038] Using optimized strategy parameters Update meta parameters

[0039]

[0040] In the formula, represents the updated meta parameters in the outer loop, Represents the current meta parameters, represents the gradient of the objective function J with respect to the policy parameters of task n, represents the objective function value of task n, β is the learning rate of the outer loop, and N is the total number of tasks.

[0041] Preferably, the method includes obtaining a new sample set by using an optimal meta-strategy; calculating the similarity between the current sample and the experience sample to obtain a high-quality sample set; obtaining a sample subset according to the sampling probability of each sample in the high-quality sample set; fusing the sample subset with the new sample set to update the meta-strategy;

[0042] Among them, the similarity ζ between the current sample and the experience sample is:

[0043]

[0044] In the formula, v current is the speed and forward ratio label of the current sample, are the speed and advance ratio labels of the empirical samples, || || represents the Euclidean norm;

[0045] The sampling probability is obtained according to the quality of the episodes corresponding to the sample:

[0046]

[0047] In the formula, p i It represents the probability of the i-th sample being selected, and the subscript j represents the high-quality sample.

[0048] Therefore, the present invention adopts the above-mentioned helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, which has the following technical effects:

[0049] (1) By designing an inner-outer-loop architecture, the complex vibration response of the helicopter under different changes in speed and forward ratio is described by defining discrete changes in speed as different tasks and continuously changing forward ratio as the dynamic environment in a specific mission.

[0050] (2) Under changing flight conditions, the adaptive adjustment mechanism greatly reduces the complexity of manual adjustments, enabling the system to automatically optimize control parameters according to real-time flight status, quickly respond to environmental changes, and improve the convenience and efficiency of operation.

[0051] (3) Through continuous learning and optimization, the algorithm can maintain efficient vibration suppression performance in various flight environments, ensuring the safety and durability of the equipment. It can not only maintain the stability of the system, but also achieve excellent vibration reduction effects under varying speed and forward ratio conditions.

[0052] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 In an embodiment of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning;

[0054] Figure 2 is a graph of an average reward function when training in a multi-dynamic environment in an embodiment of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning;

[0055] Figure 3 is a graph of reward function of forward ratio change at 720 rpm in an embodiment of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning;

[0056] Figure 4 The embodiment of the multi-dynamic vibration reduction method of a helicopter based on meta-reinforcement learning is the input of the trailing edge flap angle when the speed is 720 rpm and the forward ratio changes. Figure 4(a) is the cosine and sine components of the trailing edge flap deflection angle, Figure 4 (b) is the waveform of the trailing edge flap deflection angle change;

[0057] Figure 5 is a vertical force control diagram of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning at a speed of 720 rpm when the forward ratio changes, wherein: Figure 5 (a) is the cosine and sine components of the vertical force, Figure 5 (b) is the waveform of vertical force variation;

[0058] Figure 6 is a rolling moment control diagram of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning at a speed of 720 rpm when the forward ratio changes, wherein: Figure 6 (a) is the cosine and sine components of the rolling moment, Figure 6 (b) is the rolling moment change waveform;

[0059] Figure 7 The invention is a pitch moment control diagram of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning when the speed is 720 rpm and the forward ratio changes, wherein: Figure 7 (a) is the cosine and sine components of the pitch moment, Figure 7 (b) is the pitch moment variation waveform;

[0060] Figure 8 is a graph of reward function of forward ratio change at 960 rpm in an embodiment of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning;

[0061] Fig. 9 is an embodiment of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, wherein, at a speed of 1032 rpm, the trailing edge flap deflection angle input under the condition of a change in the forward ratio, Fig. 9 (a) is the cosine and sine components of the trailing edge flap deflection angle, Fig. 9 (b) is the waveform of the trailing edge flap deflection angle change;

[0062] Fig.10 is a vertical force control diagram of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning when the speed is 1032 rpm and the forward ratio changes, wherein: Fig.10 (a) is the cosine and sine components of the vertical force control. Fig.10 (b) is the vertical force control variation waveform;

[0063] Fig.11 is a rolling moment control diagram under the condition of a change in the forward ratio at a speed of 1032 rpm in an embodiment of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, wherein: Fig.11 (a) is the cosine and sine components of the rolling moment control. Fig.11 (b) is the rolling moment control change waveform;

[0064] Fig.12 is a pitch moment control diagram under the condition of change of forward ratio at 1032 rpm in an embodiment of a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, wherein: Fig.12 (a) is the cosine and sine components of the pitch moment control. Fig.12 (b) is the pitch moment control change waveform. DETAILED DESCRIPTION

[0065] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0066] like Figure 1 As shown, the present invention provides a helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, comprising the following steps:

[0067] S1. In order to establish a general helicopter vibration controller framework that can be applied to multiple speeds, the helicopter vibration problem is transformed into a Markov process, and different speeds are defined as different tasks, thereby realizing the description of the helicopter vibration problem at multiple speeds and laying the foundation for the subsequent design of a meta-learning controller with stronger generalization ability.

[0068] Due to the factors such as coordinate system conversion and hub filtering in the transmission process of helicopter vibration, its vibration presents typical harmonic order characteristics. Therefore, the transmission process of helicopter vibration is specifically expressed as follows:

[0069] First, define the speed Ω as 1 / (rmin -1 ), for the N blades of a helicopter, they are usually expressed in a rotating coordinate system by N-1 / (rmin -1 )、N / (rmin -1 ) and N+1 / (rmin -1 ) order trailing edge flap harmonics are reasonably coordinated to generate effective additional aerodynamic force and moment, thereby reducing the hub vibration load in the non-rotating coordinate system.

[0070] When the helicopter speed changes, N / (rmin -1 ) is converted into a vertical force in a non-rotating system, N / (rmin -1 ) rolling moment M x , pitch moment M y They come from the rotating system N-1 / (r·min -1 ) and N+1 / (r·min-1 )’s deformation moment.

[0071] When N = 4, the transfer function G(s) from the trailing edge flap input to the vibration response of the multi-output helicopter can be expressed as:

[0072]

[0073] Among them, Y s =[F z M x M y ] T , represents the vertical force, rolling moment and pitching moment at the helicopter hub, U s =[η3 η4 η5] T represents the trailing edge flap deflection angle 3 / (rmin -1 ), 4 / (rmin -1 ) and 5 / (rmin -1 ), F1, F2, F3, F4, F5 represent the dynamic relationship function according to the change of speed and forward ratio, as well as They represent the corresponding transfer functions respectively.

[0074] In order to solve the helicopter vibration control problem using reinforcement learning methods, it is necessary to express it as a Markov decision process (MDP). In the context of reinforcement learning, the interaction between the agent and its environment is described using a Markov decision process, usually represented by (S, A, P, R, γ), where S is the state space, A is the action space, P is the state transition, and R is the state in state s. t Take action a t The expected reward obtained, γ∈[0,1) is the discount factor.

[0075] In order to construct the MDP of the helicopter vibration control problem, this embodiment converts the state s t Defined as:

[0076]

[0077] Among them, U t =[η 3ct η 3st η 4ct η 4st η 5ct η 5st ] represents the trailing edge flap deflection angle 3 / (rmin -1 ), 4 / (rmin -1 ) and 5 / (rmin -1 )'s cosine and sine components; Y t =[Fzct F zst M xct M xst M yct M yst ] T N / (r·min) representing vertical force, rolling moment and pitching moment -1 )-order cosine and sine components; d(·) represents the differential of (·), such as dY t Y t The differential of represents the rate of change of the vibration output.

[0078] against Figure 1 The dynamic vibration model of the helicopter rotor with trailing edge flaps established in the figure, the harmonic components calculated by the controller will act on the rotor system through the harmonic synthesis module to achieve vibration control. t Defined as U t , can be directly used as a controller to adjust the harmonic input of the trailing edge flap.

[0079] At this point, the Markov decision can be described as:

[0080] s t+1 ~P(s t+1 ∣s t ,a t )

[0081] In the formula, P(s t+1 ∣s t ,a t ) means in state s t Take action a t Then transfer to state s t+1 For the vibration control problem of the helicopter, the reward function r(s t ,a t ) is defined as:

[0082]

[0083] In the formula, Q, Q d and R d are all positive definite matrices; the reward function takes into account the vibration output, the rate of change of the vibration output, and the rate of change of the trailing edge flap, and calculates the value of the reward function through Q, Q d and R d Adjust the weight.

[0084] After the above process, a traditional Markov problem has been constructed. When the rotor speed Ω changes, not only the vibration model of the helicopter with trailing edge flaps is changed, but also the basic vibration load D that needs to be reduced is changed. t, which increases the complexity of classifying the vibration control problem into the traditional MDP framework (S, A, P, R). Therefore, a new type of MDP (S, A, P, R, Ω) is proposed, which introduces an additional parameter Ω to describe the dynamic changes.

[0085] For the helicopter vibration model with different rotor speeds Ω, the vibration control task is further subdivided into multiple specific tasks, each with fixed time-varying dynamics, forming a series of specific MDPMs i :(S,A,P,R,Ω i ), i = 1,...,n. Each MDP assumes that Ω remains constant. The meta-reinforcement learning method with this setting aims to learn a general strategy that can effectively operate between these MDPs to deal with the vibration control problem in a dynamic environment and achieve autonomous adjustment and adaptation in a new environment.

[0086] S2. Based on the established multi-dynamic helicopter vibration response model, a meta-strategy learning framework in a dynamic environment is constructed as follows:

[0087] For each individual mission, the forward ratio μ is in real-time change, and the working environment of the helicopter is actually dynamic. Defined as:

[0088]

[0089] In the formula, [C1,...,C t-1 ,C t ,…] represents a set of different dynamic environments; since the disturbance changes are continuous, the dynamic environment The definition only indicates that the environment is changing in real time, and does not mean that the disturbance change can be clearly divided into multiple discrete stages, because this is inconsistent with the actual situation.

[0090] Furthermore, during helicopter operations, it should not be assumed that the controller itself is fully aware of the dynamic situation, i.e. the changing environment The specific situation is unknown to the meta-reinforcement learning controller. On the basis of being agnostic to the environment, in order to reduce the number of algorithm iterations and improve the efficiency of learning, the meta-reinforcement learning controller should have a performance evaluation mechanism. Only when the environmental changes affect the learned optimal strategy to a certain extent, the learned optimal strategy is adjusted through fine-tuning. That is, when the environmental changes do not affect the learned optimal strategy, the learned optimal strategy still has strong applicability after the environmental changes.

[0091] In order to adapt to the vibration control problem of the helicopter, the strategy fitness f is defined as:

[0092]

[0093] In the formula, the strategy fitness f is used as an indicator to evaluate the strategy, P e is the running time step of the evaluation, P n is defined as e The number of samples with poor performance in the sample type.

[0094] Specifically, at time t, calculate from (tP e ,…,t) of the total P e From the definition of the reward function, we can see that the reward for the helicopter vibration control problem is composed of the vibration control level, the control amount, and the change in the control amount. When r(a t ,s t )≤r ε When , it means that the strategy learned in the previous environment performs poorly in the current environment, and this sample is counted in P n And the fitness is calculated through the definition of strategy fitness.

[0095] When the strategy fitness f≤f min , it means that in the past P e There are enough samples in the sample that perform poorly, and it is necessary to make incremental adjustments to the strategy and ensure that the strategy is adaptable to the new environment.

[0096] When the environment changes, the agent tends to take the previous optimal action, which may lead to local optimality. In addition, in a new environment, using a random initialization strategy for training, the agent needs more time to learn the optimal strategy. On the other hand, when using importance sampling, the difference between the strategy distribution used for sampling and the strategy distribution used for updating increases, and the variance tends to infinity, which cannot effectively improve the distribution of the learned optimal strategy. To address the above problems, the meta-reinforcement learning control algorithm designs a strategy perturbation mechanism, that is, learning the parameters of the optimal strategy according to environmental changes, which not only reduces the training time, but also prevents the agent from falling into the local optimum.

[0097] Specifically, based on the calculated strategy fitness f, a new strategy is obtained through random perturbation:

[0098]

[0099] In the formula, is the previously learned optimal strategy The parameter θ r is a random initialization strategy The parameter α e is the parameter factor calculated based on the strategy fitness f:

[0100] α e=λ 1-f

[0101] Where λ is a hyperparameter that determines the degree of policy perturbation; the larger λ is, the smaller the degree of policy perturbation is.

[0102] Through the above method, we can get a new strategy with randomness Through new strategies and changing environment By interacting, we can get a collection of new sample episodes. Since the helicopter vibration control problem is different from other problems with clear end conditions, such as obstacle avoidance and target search, in order to simplify the problem, a sample episode τ i Defined as the time from the beginning of the episode to the time when P e Samples appear r t ≤r ε When τ i End resets the environment and starts the next episode τ i+1 .

[0103] This definition can measure the adaptability of the episode to the new environment after the current change. If an episode reaches the termination condition after interacting with the current environment and obtaining multiple samples, it means that the quality of this episode is very high. The episode quality is defined as:

[0104]

[0105] Where ρ is the regularization constant, num(τ i ) represents the episode τ i The specific number of samples contained in ; From this definition, it can be seen that the quality of the episode is determined by the average value of the reward and of the entire episode, so for the episode τ i For each sample in , the experience tuple of the sample can be expanded by combining the current speed and forward ratio information. Expand to in, and w(τ i ) represent the state and action in the episode, the corresponding reward, the next moment state and the corresponding episode quality, respectively. represents the set of speed and forward ratio. Since the episode quality remains constant for all samples in the entire episode, this expanded It does not affect the complexity of the algorithm. On the contrary, this extension will better record the performance of samples from different episodes when the environment changes, and provide high-quality sample selection for subsequent strategy optimization or online rapid adaptation.

[0106] According to the definition of episode quality, the objective gradient function based on episode quality weighting is rewritten as:

[0107]

[0108] In the formula, Represents the objective function J relative to the policy parameter The gradient of Represents the trajectory τ i It is parameterized as Strategy Generated,trajectory τ i Contains a series of states, actions, and rewards; w(τ i ) is a weight function that depends on the trajectory τ i ; is in state s t Take action a t The probability of strategy The logarithmic gradient of is the sum of discounted rewards starting from time t, and the discount factor γ determines the weight of future rewards relative to immediate rewards. It can be seen that by weighting the episode quality, the strategy focuses more on episodes that bring high returns, effectively improving the quality and speed of learning in a dynamic environment. This also simulates the human brain's stronger focus on higher-yield behaviors.

[0109] At this point, the formula for gradient descent is:

[0110]

[0111] In the formula, represents the optimization strategy parameter at time step t+1, which is updated after the gradient descent step based on the current parameter; represents the current policy parameter at time step t, which is the parameter value before the update; α represents the learning rate of the inner loop, which is used to control the step size of parameter adjustment at each update; Represents the objective function J relative to the policy parameter gradient.

[0112] Through the above method, an inner loop framework for adaptive learning of vibration strategies under the same speed environment was established. Next, a general strategy learning algorithm for different speed environments was given based on meta-learning (MAML method).

[0113] In meta-learning, although the training and testing tasks are different, all tasks share the same action and state space. The main difference between tasks is that each task corresponds to a Markov decision process (MDP) with a different reward function, environment, or dynamics, and the agent trained in meta-learning learns how to adapt to these different MDPs when faced with multiple tasks.

[0114] In this embodiment, a model-independent meta-strategy learning method is adopted, which learns a general strategy by combining updated strategies in multiple tasks. The main goal of the meta-learning framework is to evaluate the performance of the updated model parameters on the test set, rather than just focusing on its performance during training. The core of this method is that it is consistent with the broad goal of meta-learning: "learning to learn". In this way, the model not only learns solutions to specific tasks, but also acquires the ability to learn across tasks, so that it can quickly adapt and optimize when faced with new tasks. This reflects a deep learning skill that meta-learning aims to cultivate, namely the ability to quickly master new knowledge through experience.

[0115] Specifically, after fast adaptation on all tasks according to the gradient descent formula, all adjusted strategies are used To update the meta parameters

[0116]

[0117] In the formula, Represents the meta parameters updated in the outer loop; Represents the current meta parameter, that is, the parameter value before the update; represents the sum of the gradients of the objective function J with respect to the policy parameters of task n; represents the objective function value calculated under task n, β is the learning rate of the outer loop, and N is the total number of tasks.

[0118] After internal updates, an adjusted policy is generated for each task, and the meta-policy is updated by optimizing the objective function of the main task. For tasks with time-varying dynamics, the problem is decomposed into multiple subtasks with fixed time-varying dynamics, each of which corresponds to a specific MDP model parameter. The MAML method enables the agent to learn how to effectively handle these different time-varying dynamic tasks, thereby enhancing the flexibility and effectiveness of vibration control.

[0119] S3, online rapid adaptation and model optimization, as follows:

[0120] After obtaining the meta-strategy, due to the differences between the real environment and the simulation training environment, it is necessary to obtain the latest strategy that adapts to the new environment through online data interaction.

[0121] Traditional reinforcement learning algorithms perform gradient updates at each step by extracting a small batch of data from the experience replay buffer. This small batch contains experience samples collected in past policy iterations, but this approach is difficult to adapt to new environments, especially when the environment is very different from the training environment and contains disturbances. Therefore, in this embodiment, a new sample fusion method is proposed in the meta-reinforcement learning algorithm, which fuses the latest samples that perform well in the policy experience and training process, and uses the fused and corrected data for gradient descent training to optimize the network. The specific steps can be divided into two steps: First, the expanded experience tuple in the meta-policy learning stage is In which w(τ i ) represents the specific value of and In the real environment, through the similarity of the speed and forward ratio of the current environment and the samples in the experience sample pool, we can get the collection of samples that are most similar to the current environment during the training process, and define the labels of the current speed and forward ratio as v current =(ω current ,μ current ), then the weight ζ of the experience sample can be expressed as:

[0122]

[0123] In the formula, and ||v current || is the Euclidean norm of the corresponding vector. The closer the ζ value is to 1, the more the sample matches the speed and forward ratio of the current environment. In order to select high-quality samples that are more suitable for the current environment, only ζ ≥ ζ is selected during the screening process. max of the sample.

[0124] For the selected sample set, use the episode score w(τ i ) as the weighting factor to implement weighted random sampling. Sample The probability of selection p i With its episode score w(τ i ) is proportional to:

[0125]

[0126] In the formula, p i Represents the sample τ i The probability of being selected; w(τ i ) is the sample τ i The weight indicates the quality or importance of the sample. The higher the weight, the more important the sample is or the more it matches the current environmental conditions. Indicates that in the screening process, all maxThe sample τ i The data batch that records the priority sorting is defined as batch C, and then the latest m interaction strategy experience q is added to it to obtain the final data set M for online fast adaptation at the current time, that is, M = (C) ∪ {q}, and M is used to iterate and learn the meta-strategy.

[0127] In summary, the meta-reinforcement learning algorithm designed in this embodiment divides the helicopter vibration control problem into multiple tasks according to the rotation speed, and each task learns independently and evaluates the sample quality. These evaluated scores are used as sample weights to help integrate the learning results in the outer loop of meta-learning to form a meta-strategy with broad adaptability. Through brain-like intelligent simulation, the algorithm performs performance evaluation and strategy optimization in a dynamic environment, and adjusts the strategy update through performance weighting, ultimately realizing a meta-reinforcement learning algorithm that can quickly adapt to new environments.

[0128] Simulation experiment

[0129] Simulation setting: In order to verify the effectiveness of the dynamic helicopter vibration reduction method with intelligent parameter self-adjustment proposed in this invention, firstly, the helicopter vibration reduction system was integrated and designed in Matlab2019b and python 3.6, and a simulation experiment was carried out. The specific process is as follows:

[0130] (1) Parameter setting

[0131] as well as The settings are shown in Table 1.

[0132] Table 1 Transfer function parameter settings

[0133]

[0134] The dynamic relationship functions F1, F2, F3, F4, and F5 of the speed Ω and the forward ratio μ are set as follows:

[0135] F1=-14.5+35.3μ+241.8μ 2 -0.12μΩ

[0136] F2=-14.2+34.2μ+246.8μ 2 -0.11μΩ

[0137] F3=-58.5-12.0μ+0.105Ω+1733μ 2 -0.5μΩ

[0138] F4=-3.7-12.3μ+198.4μ 2 -0.041μΩ

[0139] F5=-14.0+28.2μ+0.025Ω+256.8μ 2 -0.1μΩ

[0140] (2) Meta-strategy reinforcement learning parameter settings, as shown in Table 2.

[0141] Table 2 Meta-strategy reinforcement learning parameters

[0142] parameter Numeric parameter Numeric Optimizer Adam Nonlinear functions ReLU The number of hidden units in each layer 256 Sample pool 10000 Number of hidden layers 2 Entropy Target -dim(A) Outer loop learning rate β 1e-4 Inner loop learning rate α 1e-3 <![CDATA[Minimum policy fitness f min > 0.5 Discount factor γ 0.998 Regularization constant ρ 0.5 <![CDATA[Evaluate the running time step P e > 20 <![CDATA[Sample threshold ζ max > 0.6 Small sample batch m 128 Outer loop iteration number P 1000 Inner loop iteration number K 1 or 2

[0143] like Figure 2 As shown, the performance of the meta-reinforcement learning algorithm designed in this embodiment is better than the basic meta-reinforcement learning algorithm at different inner loop times (K). When K=2, it shows higher average returns and faster convergence speed than when K=1, indicating that more inner loop times are conducive to more comprehensive strategy learning and adaptation to environmental changes. The algorithm designed in this patent shows higher final returns and less fluctuations when K=2, which shows that the algorithm has higher efficiency and stability when processing complex tasks, and can achieve better performance in diversified tasks. Figure 3 As shown, in this embodiment, a dynamic forward ratio change method is designed. The vibration control effect under the conditions of 720rpm and dynamic forward ratio change is as follows: Figures 4 to 7 As shown. Among them, Figure 4 The control input table corresponding to the trailing edge flap deflection angle input is shown. Figure 5 and Figure 6 The effects of pitch moment and pitch angular velocity control are demonstrated respectively. The system exhibits a faster vibration convergence speed at a rotation speed of 720rpm and maintains a smaller vibration amplitude after stabilization. Figure 7 This further demonstrates the excellent performance of pitch moment control under forward ratio changes, with the system able to effectively reduce vibration and maintain a stable state. Fast and stable response.

[0144] like Figure 8 As shown, this embodiment designs another different dynamic advance ratio change method, and its corresponding trailing edge flap input is shown in Fig. 9 . At 1032rpm and Figure 8 For vibration control effects under medium dynamic forward ratio conditions, please refer to Figures 10 to 12 , Fig.10 The rapid convergence of the vertical force is shown, and the system quickly regains stability after the initial disturbance, indicating that the control strategy can effectively cope with the advance ratio changes in a highly dynamic environment. Fig.11 The rolling moment control situation is demonstrated, and the curve shows that the system reaches a stable state in a relatively short time, and the vibration amplitude is significantly reduced, which reflects the advantage of the algorithm in lateral stability control. Fig.12The stability of the pitch moment control under the same conditions is demonstrated, and the system can still maintain a low vibration level when processing dynamic inputs. Overall, the control method of this embodiment has excellent vibration suppression capability and system stability under high speed and complex operating conditions, and vibration control in different directions can quickly reach and maintain a stable state.

[0145] Therefore, the present invention adopts the above-mentioned helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, which can achieve vibration reduction control with wide application, autonomous adjustment and optimization effect, solve the problems of insufficient adaptability of vibration reduction control methods, difficulty in parameter adjustment, poor vibration control effect in dynamics, etc., and ensure stability and effectiveness in multi-dynamic flight states.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning, characterized in that: The following steps are involved: S1. Establish a multi-dynamic helicopter vibration response model based on Markov decision process; S2, based on the strategic perturbation mechanism, obtain the meta-strategy learning algorithm for vibration control in dynamic environment; S3. Obtain a fast-adapting data set through sample fusion method and perform model optimization.

2. A helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning according to claim 1, characterized in that: In step S1, the helicopter vibration problem is described by using the Markov decision process, and different rotation speeds are defined as different tasks, and are kept constant in the same Markov decision process. The specific expression is as follows: MDP(S,A,P,R,Ω) In the formula, MDP represents Markov decision process, S represents state space, A represents action space, P represents state transition, and R represents state s t Take action a t The expected reward obtained, Ω represents the rotation speed; Among them, the state s t is defined as: In the formula, Respectively represent Y t and U t The derivative of, N is the number of blades, F z 、M x 、M y They represent the N-order vertical force, rolling moment, and pitching moment, respectively. The subscripts ct and st represent the cosine and sine components at time t, respectively. N-1, , η N ,,η N+1 , represent the N-1, N and N+1 order trailing edge flap deflection angle harmonics respectively; The reward function for helicopter vibration control is defined as: In the formula, Q, Q d and R d are all positive definite matrices, and d represents the differential sign.

3. The helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning according to claim 1 is characterized in that: In step S2, it includes assigning an initial strategy to each task based on the meta-strategy, evaluating the performance of the strategy using the strategy fitness, and updating the strategy through the strategy perturbation mechanism; Among them, the strategy fitness is defined as: In the formula, f is the strategy fitness, P e is the time step, P n For P e The number of poorly performing samples.

4. The helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning according to claim 3 is characterized in that: Updating the strategy through the strategy perturbation mechanism includes establishing an inner loop framework for strategy adaptive learning for the same speed environment, as follows: First, based on the calculated strategy fitness, a random strategy is obtained through random perturbation: a e =λ 1-f In the formula, is the optimal strategy at time t The parameter θ r Initialization strategy The parameter α e is the parameter factor, λ is the hyperparameter; Second, we interact the random policy with the changing environment to obtain sample episodes, and expand the sample using episode quality, rotation speed, and advance ratio: In the formula, For the expanded sample, w(τ i ) represent the samples in episode τ i The state, action, reward, state at time t+1 and episode quality in , ρ is the regularization constant, num(τ i ) represents the episode τ i The number of samples contained in r(τ i ) is the reward for the sample episode, are the speed and forward ratio respectively; Then, we build a target gradient function based on the episode quality and optimize the strategy by gradient descent: In the formula, is the objective function J relative to the policy parameters The gradient of is in state s t Take action a t The probability of strategy The logarithmic gradient of , γ represents the discount factor, represents the total discounted reward starting from time t, represents the optimization strategy parameter at time t+1, and α represents the learning rate of the inner loop.

5. The helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning according to claim 4 is characterized in that: For different speed environments, a general strategy learning algorithm is established to obtain the optimal meta-strategy, including: Using optimized strategy parameters Update meta parameters In the formula, represents the updated meta-parameters in the outer loop, Represents the current meta parameters, represents the gradient of the objective function J with respect to the policy parameters of task n, represents the objective function value of task n, β is the learning rate of the outer loop, and N is the total number of tasks.

6. A helicopter multi-dynamic vibration reduction method based on meta-reinforcement learning according to claim 5, characterized in that: In step S3, it includes obtaining a new sample set using the optimal meta-strategy; Calculate the similarity between the current sample and the experience sample to obtain a high-quality sample set; obtain a sample subset based on the sampling probability of each sample in the high-quality sample set; merge the sample subset with the new sample set to update the meta-strategy; Among them, the similarity ζ between the current sample and the experience sample is: In the formula, v current is the speed and forward ratio label of the current sample, are the speed and advance ratio labels of the empirical samples, |||| represents the Euclidean norm; The sampling probability is obtained according to the quality of the episodes corresponding to the sample: In the formula, p i It represents the probability of the i-th sample being selected, and the subscript j represents the high-quality sample.

Images