A method for airfoil optimization design based on deep reinforcement learning

Abstract

The present invention proposes an airfoil optimization design method based on deep reinforcement learning. This method is different from supervised learning. Instead, it has an autonomous learning strategy and the characteristics of maximum long-term reward. It is a more intelligent optimization method with transferability and universality of strategy. If the design conditions are changed within a certain range, such as the incoming flow Mach number, Reynolds number, etc., the original optimized strategy can still provide the initial optimization direction, and the optimization target can be significantly improved in a very short number of steps.

Description

Technical Field

[0001] The present invention belongs to the field of aircraft design and proposes an airfoil optimization design method based on deep reinforcement learning. Background Art

[0002] As a key component of an aircraft, the airfoil not only enhances the lift required for flight but also ensures the aircraft's stability and maneuverability. With the understanding and exploration of the aerodynamic performance of airfoils, multiple airfoil libraries have been established. Currently, there is no single, fully qualified super-airfoil that satisfies all design requirements. It is necessary to initially select a baseline airfoil from the airfoil library based on factors such as the design application and operating conditions. Based on this baseline airfoil, the baseline airfoil is continuously improved in accordance with design requirements. The initial approach to improving the baseline airfoil was to conduct repeated experiments in a wind tunnel, which consumes enormous material and human resources.

[0003] Since the beginning of the 21st century, advances in science and technology have injected new vitality into computational fluid dynamics (CFD), which has quickly become a primary means of analyzing and solving fluid problems. Using CFD significantly shortens optimization cycles, saving significant manpower and resources. This allows designers to perform repeated calculations until the desired airfoil shape is achieved, exemplified by the widespread application of evolutionary and genetic algorithms in aerodynamic optimization. However, these algorithms have a low efficiency in utilizing the vast amounts of computational data generated during the optimization process.

[0004] The application of machine learning in fluid dynamics has rapidly developed in recent years. Currently, the most common application of machine learning in optimization design is rapid performance prediction methods based on response surfaces. These methods leverage existing data to construct a mapping between inputs and outputs, accelerating the optimization process and partially replacing the role played by CFD techniques in the optimization process. This is a form of supervised learning. Unlike supervised learning, reinforcement learning constructs a mapping from environmental state parameters to action parameters. Its goal is to iteratively update the strategy for interacting with the environment to maximize cumulative rewards. Unlike supervised learning, which requires predicting the actual output, this approach targets rewards associated with specific actions. Airfoil design is not about optimizing a single performance objective, but rather optimizing overall performance through repeated trade-offs. It is a complex system engineering project, often requiring manual refinement based on the engineer's own experience and understanding. The training process of reinforcement learning is very similar to the process by which engineers accumulate experience. Similar to traditional trial-and-error methods, the agent responsible for taking actions in reinforcement learning continuously takes different "actions" during training and observes the benefits of the design results over time (or after a series of actions). This strategy updates its action strategy. As the benefits increase, it can be considered that the agent has, to a certain extent, acquired the same design experience as the engineer. In the area of ​​airfoil optimization, Li Runze used reinforcement learning to optimize airfoil pressure distribution to reduce transonic airfoil drag. Viquerat conducted both constrained and unconstrained optimization experiments. While reinforcement learning for airfoil optimization is still in its infancy, there's a risk that the agent might lose its way and become trapped in a local optimum. Overall, applying reinforcement learning to airfoil optimization in aircraft aerodynamic optimization design can effectively improve optimization efficiency and has a wide range of applications. Summary of the Invention

[0005] Since the current CFD-based method has low optimization efficiency, in order to improve the airfoil optimization efficiency, the present invention proposes an airfoil optimization design method based on deep reinforcement learning. This method is different from supervised learning. Instead, it has an autonomous learning strategy and the characteristics of maximizing long-term rewards. It is a more intelligent optimization method and has transferability, or the universality of the strategy.

[0006] The technical solution of the present invention is:

[0007] The airfoil optimization design method based on deep reinforcement learning comprises the following steps:

[0008] Step 1: Use the free-form surface deformation method to perform geometric parameterization of the airfoil: establish a free-form surface deformation control frame around the reference airfoil, establish a mapping relationship between the control frame and the airfoil, and obtain a new airfoil by changing the position of the control frame points;

[0009] Step 2: Establish an optimization design model and determine the single design objective and constraints based on flight requirements. The design objectives and constraints are various aerodynamic parameters of the airfoil, such as lift coefficient, drag coefficient, and airfoil thickness. Express the design objectives and constraints in mathematical expressions. A general single-objective optimization problem can be expressed in the following mathematical form:

[0010] Minimize:f(x)

[0011] subject to:g w (x)≥0,w=1,2,···,W

[0012] h r (x)=0,r=1,2,···,R

[0013] Where x is the design variable, f(x) is the objective function, g w (x) is an inequality constraint, there are W of them; h r (x) is an equality constraint, there are R in total.

[0014] Step 3: Establish a reward function based on the optimization objective and constraints, where the total reward value is obtained by the linear sum of the reward values ​​of each aerodynamic parameter. The reward value increases when the objective is achieved, does not increase when the constraint is satisfied, and decreases when the constraint is not satisfied. At the same time, the objective reward value and the constraint reward value are multiplied by different coefficients to balance the magnitude difference between the objective and the constraint.

[0015] Step 4: Establish an intelligent agent, including a policy model π and a value function model. The policy function model can output action strategies, while the value function model can output advantage estimates and value functions. Both the policy model and the value function model use an artificial neural network model with two hidden layers and 64 hidden layer nodes. Initialize the policy model parameters θ0 and the value function model parameters The aerodynamic parameters of the airfoil related to the design objectives and constraints are taken as the state, including the lift coefficient, drag coefficient, maximum thickness and moment coefficient of the airfoil. The aerodynamic parameters of the reference wing are taken as the initial state s0.

[0016] Step 5: The current agent is given action a by the policy model based on the state and reward value, which is the new design variable.

[0017] Step 6: Apply the action to the airfoil to obtain a new airfoil.

[0018] Step 7: Create a structured grid for the new airfoil and use the open source solver CFL3D to perform numerical simulation of the flow around the airfoil. The main governing equation is the Reynolds-averaged NS equation, and the turbulence model is the k-ωSST model. The lift coefficient, drag coefficient, maximum thickness, and moment coefficient of the airfoil are calculated to confirm that the aerodynamic parameters of the new airfoil are in the new state.

[0019] Step 8: The reward value is calculated based on the calculated aerodynamic parameters according to the reward function in step 3.

[0020] Step 9: Repeat steps 5-8 for a total of e-1 times using the current policy model to obtain the trajectory and reward value {r e}, trajectory τ={s0,a0,···,s e-1 ,a e-1 ,s e}, where s0 and a0 are the initial state and action, s e is the e-step state, s e-1 is the e-1 step state, a e-1 This is the e-1 step action.

[0021] Step 10: Repeat steps 5-9 for n-1 times based on the current policy model to obtain n trajectories and reward values.

[0022] Step 11: Based on the obtained n trajectory parameters and reward values, calculate the advantage estimate based on the current agent's value function model, that is, the difference between the expected reward of each action a and the mean expected reward of all possible actions in this state.

[0023] Step 12: Construct a loss function based on the advantage estimate, trajectory, and reward value, and optimize the policy model parameters θ and the value function model parameters based on the stochastic gradient descent algorithm. The optimization goal is to minimize the loss function, and use the optimized parameters to update the policy model and value function model to obtain a new policy model and value function model.

[0024] Step 13: Repeat steps 5-12 until the loss function stops decreasing and the training is completed.

[0025] Beneficial effects

[0026] 1. The airfoil optimization design method based on deep reinforcement learning proposed in this invention has the characteristics of autonomous learning strategy and maximum long-term reward. Compared with algorithms such as genetic algorithms that only use large amounts of computational data as evaluation criteria for optimization objectives and constraints, deep reinforcement learning attempts to learn the experience gained during the optimization process, which can improve data utilization efficiency, reduce computational complexity, and improve optimization efficiency.

[0027] 2. The airfoil optimization design method based on deep reinforcement learning proposed in this paper has a certain degree of universality or transferability in the strategy model obtained through deep reinforcement learning optimization: if the design conditions, such as the incoming flow Mach number and Reynolds number, are changed within a certain range, the original optimized strategy can still provide the initial optimization direction, and the optimization target can be significantly improved in a very short number of steps. This is a feature and advantage that traditional optimization methods do not have.

[0028] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0030] Figure 1 This is an overall flow chart of the airfoil optimization method of the present invention.

[0031] Figure 2 It is the control frame of the free surface deformation method.

[0032] Figure 3 Schematic diagram of airfoil design space.

[0033] Figure 4 Schematic diagram of a fully connected network.

[0034] Figure 5 Optimizing processes for reinforcement learning.

[0035] Figure 6 The full and partial images of the airfoil mesh.

[0036] Figure 7 This is the drag convergence curve of the airfoil optimization of the proximal point optimization strategy (PPO) in this paper.

[0037] Figure 8 Drag convergence curve of airfoil optimization using the non-dominated sorting genetic algorithm (NSGA-II) for comparison strategy.

[0038] Figure 9 Drag convergence curve for airfoil optimization using the pre-trained PPO strategy. DETAILED DESCRIPTION

[0039] The following describes in detail embodiments of the present invention. The embodiments are exemplary and intended to explain the present invention, but are not to be construed as limiting the present invention.

[0040] Step 1: This example uses RAE2822 as the reference airfoil and uses the free form deformation method (FFD) to parameterize the reference airfoil. The free form deformation method control box is as follows: Figure 2 As shown, the design variables consist of 14 points that control the upper and lower wing surfaces. The perturbation range of each design point in the y direction is ±0.02. The design space of the airfoil is as follows: Figure 3 shown.

[0041] Step 2: The design goal is to minimize the drag of the airfoil in the design state while satisfying three constraints: (1) the lift coefficient does not decrease, (2) the absolute value of the moment coefficient of the airfoil does not increase, and (3) the maximum thickness of the airfoil does not decrease. The mathematical expression of the optimization design problem is as follows:

[0042] Minimize:C d

[0043] Subject to:C l ≥C l0

[0044] |C m |≤|C m0 |

[0045] t≥t0

[0046] Among them C d is the airfoil drag coefficient, C l is the airfoil lift coefficient, C l0 is the lift coefficient of the reference airfoil, C m is the moment coefficient of the airfoil, C m0 is the moment coefficient of the reference airfoil, t is the thickness of the airfoil, and t0 is the thickness of the reference airfoil.

[0047] Step 3: Construct the reward function based on the design objectives and constraints in step 2 as follows:

[0048] If C d -C d0 <0 reward cd =100×(C d -C d0 ).

[0049] If C l -C l0 <0 reward cl =10×(C l -C l0 ), otherwise let reward cl =0.

[0050] If C m -C m0 <0, then rewardcm =100×(C m -C m0 ), otherwise let reward cm =0.

[0051] If t-t0<0, let reward t =50×(t-t0), otherwise let reward t =0.

[0052] Final reward value reward = reward cd +reward cl +reward cm +100reward t .

[0053] Among them, reward cd is the drag coefficient reward value, reward cl is the lift coefficient reward value, reward cm is the torque coefficient reward value, reward t is the airfoil thickness reward value. 10, 50, and 100 are used to balance the magnitude difference between the target and the constraint.

[0054] Step 4: Establish an intelligent agent, which includes a policy model π and a value function model. The policy function model outputs the action policy, while the value function model outputs the advantage estimate and the value function. Both models use an artificial neural network model with two hidden layers, written in PyTorch, with 64 hidden layer nodes.

[0055] Artificial neural network is a feedforward neural network that connects multiple neuron nodes together. The output of each layer is used as the input of the next layer. Through weight calculation, a grouped hierarchical computing structure is formed. The fully connected network diagram is as follows: Figure 4 Initialize the policy model parameters θ0 and the value function model parameters The lift coefficient, drag coefficient, maximum thickness and moment coefficient of the airfoil are taken as states. Here, the lift coefficient, drag coefficient, maximum thickness and moment coefficient of the reference wing are taken as the initial state s0. The relationship between the various parts in the reinforcement learning optimization process is as follows: Figure 5 shown.

[0056] Step 5: The current agent is given an action based on the state and reward value by the policy model, which is the new airfoil design variable.

[0057] Step 6: Apply actions to the airfoil to obtain a new airfoil.

[0058] Step 7: Create a C-type structured grid for the new 2D airfoil. Figure 6The mesh of the airfoil is shown in full and in part. The calculation conditions are Ma=0.734, Re=6.5×10 6 ,α=2.79°, where Ma is the incoming flow Mach number, Re is the incoming flow Reynolds number, and α is the incoming flow angle of attack. Without considering the effects of body force and external heating, the open source solver CFL3D is used to perform numerical simulation of the flow around the airfoil. The main governing equation is the Reynolds-averaged NS equation, and the turbulence model is the k-ωSST turbulence model. The drag coefficient c of the new airfoil is calculated. d and moment coefficient c m , lift coefficient c l and the airfoil thickness t. Confirm that the aerodynamic parameters of the new airfoil are in the new state.

[0059] Step 8: c obtained by calculation d 、c m 、c l The reward value is calculated based on the reward function in step 3.

[0060] Step 9: Repeat steps 5-8 for a total of e-1 times using the current policy model to obtain the trajectory and reward value {r e}, trajectory τ={s0,a0,···,s e-1 ,a e-1 ,s e}, where s0 and a0 are the initial state and action, s e is the e-step state, s e-1 is the e-1 step state, a e-1 This is the e-1 step action.

[0061] Step 10: Repeat steps 5-9 based on the current policy model to obtain a total of n trajectories and reward values.

[0062] Step 11: Based on the obtained trajectory parameters, calculate the advantage estimate based on the current value function model. The advantage estimate is the interpolation of the expected reward of each action a and the mean of the expected rewards of all actions in the state. If the advantage estimate is greater than 0, it means that the action a is better than the average, otherwise it is worse than the average.

[0063] Step 12: Construct a loss function based on the advantage estimate, trajectory, and reward value, and optimize the policy model parameters θ and the value function model parameters based on the stochastic gradient descent algorithm. The optimization goal is to minimize the loss function, and use the optimized parameters to update the policy model and value function model to obtain a new policy model and value function model.

[0064] Step 13: Repeat steps 5-12 until the loss function stops decreasing and the training is completed.

[0065] Proximal Policy Optimization (PPO) is a deep reinforcement learning algorithm. Figure 7 The optimized drag convergence curve of the proximal optimization strategy in this paper is given, from which it can be seen that the airfoil drag has been significantly improved. Compared with the initial airfoil RAE2822, the drag is reduced by about 50 frames, which is about 30% improvement. In order to verify the optimization effect of PPO in single-objective constrained aerodynamic optimization design, the non-dominated sorting genetic algorithm (NSGA-II) is used as a comparison algorithm. The population size of NSGA-II is 100, the evolutionary number is 100, and the optimization convergence curve is shown as follows: Figure 8 As shown in the figure, PPO's optimization effect still lags behind NSGA-II, so pre-training is used to improve PPO's optimization effect. Pre-training is based on a model that has been trained on similar problems. This provides a better starting point for optimization and avoids falling into local optimality, resulting in better optimization results.

[0066] The network is pre-trained, and the optimization convergence curve of the pre-trained PPO is as follows: Figure 9 As shown in Table 1, the comparison of the optimization results obtained by the two methods and the required amount of calculation is shown. It can be observed that the convergence trend of the optimization is significantly more stable than that of the genetic algorithm, and the convergence trends of the two are the same. When the drag optimization effect is similar, the computational amount of the pre-trained PPO strategy in this paper is 0.05 times that of the genetic algorithm, and the computational amount is significantly reduced. As shown in Table 2, when the fixed computational amount is 500, the drag optimization effect of the PPO strategy in this paper and the pre-trained PPO strategy is also much better than that of the genetic algorithm. It can be seen that the airfoil optimization design method based on deep reinforcement learning of the present invention can significantly reduce the computational amount compared to the genetic algorithm, and it is not easy to fall into the local optimum after pre-training.

[0067] Table 1

[0068]

[0069] Table 2

[0070]

[0071] Next, we change the design conditions to illustrate that the strategy obtained by reinforcement learning has a certain degree of universality: keep the basic airfoil unchanged, and change the design state from Ma=0.734, Re=6.5×10 6 ,α=2.79°becomes 1#:Ma=0.72,Re=1×10 7 ,α=2.79°, 2#: Ma=0.76, Re=8×10 6 ,α=2.79° and 3#: Ma=0.6, Re=6.5×10 6,α=2.79°. To verify the effectiveness of the strategy, the existing strategy was directly used to optimize the three states for 5, 10, 15, and 20 steps, observing the changes in resistance. The results are shown in Table 3. It can be seen that in the three design states, after 20 optimization steps, the resistance of design 1# decreased by 7.4%; the resistance of design 2# decreased by 3.7%; and the resistance of design 3# decreased by 6.8%. The optimization results for all three design states met the required constraints. In summary, reinforcement learning has learned a strategy with a certain degree of universality, which can provide direction and preliminary optimization results for optimization.

[0072] Table 3

[0073]

[0074] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention.

Claims

1. A method for airfoil optimization design based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Use the free-form surface deformation method to perform geometric parameterization of the airfoil: establish a free-form surface deformation control frame around the reference airfoil, establish a mapping relationship between the control frame and the airfoil, and obtain a new airfoil by changing the position of the control frame points; Step 2: Establish an optimization design model and confirm the single design objectives and constraints according to flight requirements; Step 3: Establish a reward function based on the optimization objectives and constraints; Step 4: Establish an agent, including a policy model π and a value function model. The policy function model outputs action strategies, and the value function model outputs advantage estimates and value functions; initialize policy model parameters θ0 and value function model parameters The aerodynamic parameters of the airfoil design objectives and constraints are taken as states, among which the aerodynamic parameters corresponding to the reference wing are taken as the initial state s0; Step 5: The current agent takes actions according to the state and reward value through the policy model to obtain new airfoil design variables; Step 6: Apply actions to the airfoil to obtain a new airfoil; Step 7: Establish a structured grid for the obtained new airfoil, perform numerical simulation of the flow around the new airfoil, and calculate the aerodynamic parameters of the new airfoil as the new state; Step 8: Using the aerodynamic parameters calculated in step 7, the reward value is calculated according to the reward function in step 3; Step 9: Repeat steps 5-8 for a total of e-1 times using the current policy model to obtain the trajectory and reward value {r e }, trajectory τ = {s0, a0, ..., s e-1 , a e-1 ,s e }, where s0 and a0 are the initial state and action, s e is the e-step state, s e-1 is the e-1 step state, a e-1 It is e-1 step action; Step 10: Repeat steps 5-9 for a total of n-1 times based on the current policy model to obtain n trajectories and reward values; Step 11: Calculate the advantage estimate based on the value function model of the current agent according to the obtained n trajectory parameters and reward values; Step 12: Construct a loss function based on advantage estimates, trajectories, and reward values ​​to optimize policy model parameters θ and value function model parameters The optimization goal is to minimize the loss function, and use the optimized parameters to update the policy model and value function model to obtain a new policy model and value function model; Step 13: Repeat steps 5-12 until the loss function stops decreasing and the training is completed.

2. The airfoil optimization design method based on deep reinforcement learning according to claim 1, characterized in that: In step 2, the design objectives and constraints are the aerodynamic parameters of the airfoil.

3. The airfoil optimization design method based on deep reinforcement learning according to claim 2 is characterized in that: In step 2, the design goal is to minimize the drag of the airfoil in the design state while satisfying three constraints: (1) the lift coefficient does not decrease, (2) the absolute value of the moment coefficient of the airfoil does not increase, and (3) the maximum thickness of the airfoil does not decrease; The mathematical expression of the optimization design problem is as follows: Minimize:C d Subject to: |C m |≤|C m0 | t≥t0 Among them C d is the airfoil drag coefficient, C l is the airfoil lift coefficient, is the lift coefficient of the reference airfoil, C m is the moment coefficient of the airfoil, C m0 is the moment coefficient of the reference airfoil, t is the thickness of the airfoil, and t0 is the thickness of the reference airfoil.

4. The airfoil optimization design method based on deep reinforcement learning according to claim 3 is characterized in that: In step 3, the total reward value is obtained by the linear sum of the reward values ​​of each aerodynamic parameter, where the reward value is increased if the target is achieved, the reward value is not increased if the constraint is met, and the reward value is reduced if the constraint is not met. At the same time, the target reward value and the constraint reward value are multiplied by different coefficients to balance the magnitude difference between the target and the constraint.

5. The airfoil optimization design method based on deep reinforcement learning according to claim 4 is characterized in that: The reward function is constructed based on the design objectives and constraints in step 2 as follows: If C d -C d0 <0 reward cd =100×(C d -C d0 ); if Otherwise, reward cl =0; If C m -C m0 <0, then reward cm =100×(C m -C m0 ), otherwise let reward cm =0; If t-t0<0, let reward t =50×(t-t0), otherwise let reward t =0; Final reward value reward = reward cd +reward cl +reward cm +100reward t ; Among them, reward cd is the drag coefficient reward value, reward cl is the lift coefficient reward value, reward cm is the torque coefficient reward value, reward t is the bonus value for airfoil thickness.

6. The airfoil optimization design method based on deep reinforcement learning according to claim 1, characterized in that: In step 4, both the policy model and the value function model use an artificial neural network model with two hidden layers and 64 hidden layer nodes.

7. The airfoil optimization design method based on deep reinforcement learning according to claim 1, characterized in that: In step 7, the open source solver CFL3D is used to perform numerical simulation of the flow around the new airfoil. The main control equation is the Reynolds-averaged NS equation, and the turbulence model is the k-ωSST model. The aerodynamic parameters of the new airfoil are calculated.

Images