A design method for efficient flow control based on transfer learning

Abstract

The present invention provides a flow active control efficiency design method based on transfer learning, which belongs to the field of flow control design. The method comprises: sharing all network parameters of a first airfoil active rudder control model to an initial second airfoil active rudder control model for initialization, and interacting with a flow field solver to obtain relevant response data of the airfoil flow to the rudder action, calculating a corresponding reward value, forming an initial array and storing it in an experience replay pool; randomly sampling a preset sampling data group data, and calculating the evaluation value of each network of the second airfoil active rudder control model, updating the evaluation network, introducing transfer learning to update the strategy network, obtaining the second airfoil optimal active rudder control model and using it for flow active control; the present invention introduces transfer learning in the training of the active rudder control model, thereby solving the problem that the control law design model requires a large amount of simulation or experimental data and the control law design model is strongly correlated with the control object.

Description

Technical Field

[0001] The present invention belongs to the field of flow control design, and in particular relates to a flow active control efficiency enhancement design method based on transfer learning. Background Art

[0002] Unstable separation flow is a flow phenomenon often encountered in many fields such as aerospace, wind engineering, marine engineering, energy and power. The flow structure of these separation flows is very complex, the flow has strong unsteady and nonlinear characteristics, and will cause many adverse effects. Flow control is an effective means to eliminate the above-mentioned adverse effects of separation flow. Depending on whether energy injection is required, flow control can be divided into passive and active control methods. Passive control has the risk of failure or reverse effect under non-design conditions because its control parameters cannot be changed. Therefore, for complex flow control problems, active control is more advantageous, especially active closed-loop control with feedback loops.

[0003] The active control design method based on deep reinforcement learning has strong nonlinearity and is suitable for active closed-loop control design of complex flows. In the prior art, this method has accumulated a large number of successful implementation cases, whether through simulation or experimental verification. However, training a control law design model based on reinforcement learning requires a large amount of simulation or experimental data, which is very time-consuming, and the control law design model is strongly related to the control object. When the geometric boundary of the control object changes or its flow characteristics change, the control law design model needs to be retrained. Therefore, how to improve the efficiency of active control law design based on reinforcement learning is the key to solving the problem that this control law design method is difficult to apply to practical engineering problems. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a flow active control efficiency enhancement design method based on transfer learning, which solves the problem that the control law design model requires a large amount of simulation or experimental data and the control law design model is strongly correlated with the control object.

[0005] In order to achieve the above objectives, the technical solution adopted by the present invention is: a flow active control efficiency enhancement design method based on transfer learning, comprising the following steps:

[0006] S1, sharing all network parameters of the optimal active rudder control model of the first airfoil to the initial second airfoil active rudder control model for initialization, and interacting the initialized second airfoil active rudder control model with the flow field solver to obtain relevant response data of the airfoil flow to the rudder action guided by the second airfoil active rudder control model, and calculating the corresponding reward value, forming an initial array and storing it in the experience replay pool;

[0007] S2, in response to the number of data arrays in the experience playback pool exceeding the preset sampling data amount, randomly sampling the preset sampling data amount group data, and calculating the evaluation values ​​of the first evaluation network, the second evaluation network, the target network of the first evaluation network and the target network of the second evaluation network of the second airfoil active control surface control model, and updating the parameters of the first evaluation network and the second evaluation network by using the calculation method of the timing difference error and the update strategy;

[0008] S3. After updating the first evaluation network and the second evaluation network for a preset number of times, introduce transfer learning to update the strategy network, freeze all parameters of the hidden layers except the last hidden layer in the strategy network, update the weights and biases of each neuron on the last hidden layer in the strategy network, and obtain the optimal active rudder control model for the second airfoil based on the updated first evaluation network, the second evaluation network and the strategy network. Use the optimal active rudder control model for the second airfoil for flow active control, and complete the flow active control efficiency enhancement design.

[0009] The beneficial effects of the present invention are as follows: the present invention adopts transfer learning to improve the training efficiency of the active control design model based on deep reinforcement learning, and realizes the rapid training of the optimal active control design model of the variable airfoil; the present invention completes the migration from the first optimal active rudder surface control model to the second airfoil optimal active rudder surface control model. The present invention uses the transfer learning method without changing the weight combination between the control objectives in the reward function, and does not need to adjust the changing law of the learning rate. The number of rounds required for model training is reduced to fifty percent of the number of rounds required for model training without introducing transfer learning, thereby reducing the time required for weight optimization and flow field calculation.

[0010] Furthermore, the S1 comprises the following steps:

[0011] S101, selecting a trailing edge control surface with a preset chord-wise length as a control device, and randomly initializing a strategy network, a target network of the strategy network, a first evaluation network, a second evaluation network, a target network of the first evaluation network, and a target network of the second evaluation network of a first airfoil active control surface control model;

[0012] S102, optimizing the active rudder control model of the first airfoil using a double-delayed deep deterministic policy gradient algorithm to obtain an optimal active rudder control model for the first airfoil, and sharing all parameters of each network in the optimal active rudder control model for the first airfoil to an initial active rudder control model of the second airfoil for initialization;

[0013] S103, setting a reward function, and setting a weight combination of each target in the reward function according to the optimal active rudder control model of the first airfoil;

[0014] S104, interacting the trailing edge rudder control guided by the active rudder control model of the second airfoil with the flow field solver, and inputting the flow field information at the current moment into the strategy network of the active rudder control model of the second airfoil, obtaining the rudder deflection at the next moment, and realizing the change of the airfoil shape according to the dynamic grid, to obtain the response data of the two-dimensional airfoil;

[0015] S105. Calculate the reward value of the current rudder action according to the response data of the two-dimensional airfoil, form an initial array with the flow field information at the current moment, the rudder deflection amount at the next moment, the response data of the two-dimensional airfoil, and the reward value of the current rudder action, and store them in the experience replay pool.

[0016] Furthermore, the expression of the reward function is as follows:

[0017] ;

[0018] ;

[0019] ;

[0020] ;

[0021] in, represents the reward function, represents the pulsation amplitude of the airfoil lift coefficient response, Indicates the change in the lift coefficient response of the airfoil at adjacent moments, represents the deflection amplitude of the trailing edge rudder surface, , , Both represent weight parameters, is the lift coefficient, express to The average value of the lift coefficient of the airfoil at time represents the change in lift coefficient, express to The average value of the active rudder surface deflection amplitude at each moment.

[0022] The beneficial effect of the above further scheme is: the present invention sets a reward function, which improves the migration of the optimal active rudder control model of the second airfoil, and realizes that there is no need to change the relevant settings of the reward function and the learning rate when the airfoil model is migrated to other airfoils.

[0023] Furthermore, the S2 comprises the following steps:

[0024] S201, randomly sampling a preset number of arrays from the experience replay pool according to a preset sampling data volume, bringing the response data of the two-dimensional airfoil in the array into the target network of the strategy network to obtain a target control law, and bringing the response data of the two-dimensional airfoil in the array and the target control law into the target network of the first evaluation network and the target network of the second evaluation network to obtain a first target evaluation value and a second target evaluation value respectively;

[0025] S202, selecting the smallest amount between the first target evaluation value and the second target evaluation value and multiplying it by a discount coefficient as an update target of the evaluation network;

[0026] S203, inputting the flow field information at the current moment, the deflection amount of the control surface at the next moment, and the response data of the two-dimensional airfoil in the array into the first evaluation network and the second evaluation network, and obtaining the first evaluation value at the current moment and the second evaluation value at the current moment respectively;

[0027] S204, calculating the difference between the update target of the evaluation network and the first evaluation value and the second evaluation value at the current moment, and minimizing the difference through gradient optimization, obtaining the parameter combination of the first evaluation network and the parameter combination of the second evaluation network in response to the difference convergence, and updating the first evaluation network and the second evaluation network.

[0028] Furthermore, the expression of the update target of the evaluation network is as follows:

[0029] ;

[0030] in, represents the update target of the evaluation network, Indicates k The reward function for the group rudder action, represents the discount factor, Indicates i Target evaluation value, Indicates k +1 set of flow field information in the sampling data, Indicates the k +1 The flow field information in the sampled data set is the action value output by the policy network when it is input. and All represent target network parameters. Representation parameters A normal distribution with truncated measures, represents a normal distribution, represents the variance of the normal distribution, Represents the amplitude of the truncated normal distribution.

[0031] Furthermore, the expressions of the parameter combination of the first evaluation network and the parameter combination of the second evaluation network are as follows:

[0032] ;

[0033] in, i =1, the above expression represents the parameter combination of the first evaluation network. i =2, the above expression represents the parameter combination of the second evaluation network. Indicates i Evaluate the weights and bias parameters of each neuron in the network, Indicates the preset sampling data volume, Indicates i Evaluation network update target, Indicates the k The flow field information and rudder deflection angle in the group of sampling data are input as i The evaluation value of the current action output by the evaluation network, Indicates k The flow field information in the group sampling data, Indicates k The deflection angle of the rudder surface in the group of sampled data.

[0034] The beneficial effects of the above further scheme are as follows: the present invention utilizes the target evaluation value multiplied by the discount coefficient as the update target of the evaluation network, thereby avoiding the overestimation problem of the evaluation network; and utilizes the calculation method of the timing difference error and the update strategy to update the evaluation network, so that the evaluation network gradually reaches the optimal value and reduces the time of model training.

[0035] Furthermore, the S3 comprises the following steps:

[0036] S301, after updating the first evaluation network and the second evaluation network for a preset number of times, introduce the policy network update strategy of transfer learning, adjust the update frequency of the policy network, freeze all parameters of the hidden layers except the last hidden layer in the policy network, change the weights and biases of each neuron on the last hidden layer of the policy network through gradient optimization, and update the parameters of the target network of the policy network, the target network of the first evaluation network, and the target network of the second evaluation network by soft update;

[0037] S302, the second airfoil active control surface control model interacts with the flow field solver according to the parameters of the target network of the updated strategy network, the target network of the first evaluation network, and the target network of the second evaluation network to obtain an updated array and put it into the experience playback pool;

[0038] S303, repeating steps S201-S302, in response to the convergence of the reward function, obtaining the optimal active rudder surface control model for the second airfoil, and using the optimal active rudder surface control model for the second airfoil for flow active control to complete the flow active control efficiency enhancement design.

[0039] Furthermore, the expression of the gradient optimization is as follows:

[0040] ;

[0041] in, represents the weights and biases of the last hidden layer neurons in the policy network, Indicates the k The flow field information and rudder deflection angle in the group of sampling data are input as i The evaluation value of the current action output by the evaluation network, Indicates the preset sampling data volume, represents the flow field information in the kth group of sampling data, Indicates k The deflection angle of the rudder surface in the group sampling data, Indicates the k The flow field information in the group sampling data is the action value output by the policy network when it is input. represents the vector differential operator, represents the objective function for updating the policy network parameters;

[0042] The expression of the soft update method is as follows:

[0043] ;

[0044] in, Indicates i Evaluate the weights and bias parameters of each neuron in the network, is the soft update coefficient, and its value range is , represents the target network parameters for the evaluation network, Represents the target network parameters of the policy network.

[0045] The beneficial effects of the above further scheme are as follows: the present invention introduces a strategy network update strategy for transfer learning, and retains the training experience of the optimal active rudder control model of the first airfoil by freezing parameters, thereby achieving the purpose of obtaining better control effect within the same training time, and obtaining a better flow active control model within the same time, thereby achieving efficiency improvement; by finding the action that can obtain the maximum evaluation value at the next moment, the experience of the original standard airfoil trailing edge rudder reinforcement learning control design model is retained to the greatest extent, so that the active control design model for the new airfoil converges to a model that achieves better control effect within a certain training time, thereby improving the training efficiency of the active rudder control model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The figure is a flow chart of the method of the present invention.

[0047] Figure 2 This is a flow chart of the active flow control efficiency enhancement design method in this embodiment.

[0048] Figure 3 This is a schematic diagram of introducing transfer learning in this embodiment to retain existing model training experience.

[0049] Figure 4 This is a comparison chart of the training history of the active rudder control model of the RAE2822 airfoil with bulges with and without the introduction of transfer learning in this embodiment.

[0050] Figure 5 This is a comparison chart of the control effects of the rudder control law guided by the optimal active rudder control model for the RAE2822 airfoil with bulge obtained when transfer learning is introduced and not introduced in this embodiment. DETAILED DESCRIPTION

[0051] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0052] Before describing this embodiment, the following terms are explained:

[0053] TD3 algorithm: double-delayed deep deterministic policy gradient algorithm;

[0054] TD-error: timing difference error.

[0055] Example

[0056] like Figure 1 As shown, the present invention provides a flow active control efficiency enhancement design method based on transfer learning, and its implementation method is as follows:

[0057] S1. Share all network parameters of the optimal active rudder control model of the first airfoil with the initial active rudder control model of the second airfoil for initialization, and interact the initialized active rudder control model of the second airfoil with the flow field solver to obtain the relevant response data of the airfoil flow to the rudder action guided by the active rudder control model of the second airfoil, and calculate the corresponding reward value to form an initial array and store it in the experience replay pool. The specific steps are as follows:

[0058] S101, selecting a trailing edge control surface with a preset chord-wise length as a control device, and randomly initializing a strategy network, a target network of the strategy network, a first evaluation network, a second evaluation network, a target network of the first evaluation network, and a target network of the second evaluation network of a first airfoil active control surface control model;

[0059] S102, optimizing the active rudder control model of the first airfoil using a double-delayed deep deterministic policy gradient algorithm to obtain an optimal active rudder control model for the first airfoil, and sharing all parameters of each network in the optimal active rudder control model for the first airfoil to an initial active rudder control model of the second airfoil for initialization;

[0060] S103, setting a reward function, and setting a weight combination of each target in the reward function according to the optimal active rudder control model of the first airfoil;

[0061] S104, interacting the trailing edge rudder control guided by the active rudder control model of the second airfoil with the flow field solver, and inputting the flow field information at the current moment into the strategy network of the active rudder control model of the second airfoil, obtaining the rudder deflection at the next moment, and realizing the change of the airfoil shape according to the dynamic grid, to obtain the response data of the two-dimensional airfoil;

[0062] S105. Calculate the reward value of the current rudder action according to the response data of the two-dimensional airfoil, form an initial array with the flow field information at the current moment, the rudder deflection amount at the next moment, the response data of the two-dimensional airfoil, and the reward value of the current rudder action, and store them in the experience replay pool.

[0063] In this embodiment, transfer learning that can retain certain model training experience is introduced to improve the training efficiency of the reinforcement learning active-passive collaborative control design model, thereby promoting the application of active-passive collaborative control in engineering practice.

[0064] In this embodiment, the existing NACA0012 airfoil optimal active rudder surface control model is used as the first airfoil active rudder surface control model, and the bulged RAE2822 airfoil active rudder surface control model is used as the second airfoil active rudder surface control model.

[0065] In this embodiment, Figure 2 As shown in the figure, data preparation is first performed: all network parameters of the existing NACA0012 airfoil optimal active rudder control model are shared with the initial RAE2822 airfoil active rudder control model with bulge for initialization; then, the initialized RAE2822 airfoil active rudder control model with bulge is interacted with the flow field solver to obtain the rudder action guided by the current control model for the airfoil spoiler. Related response data (including feedback signals for closed-loop control, such as lift coefficient etc.), and calculate the corresponding reward value , forming an array Stored in the experience replay pool;

[0066] In this embodiment, the preferred control model of the initialization model is the NACA0012 airfoil active rudder control model. In the initial model parameter acquisition, a chord length of 0.15 is selected for the transonic buffeting phenomenon of the NACA0012 airfoil. ( The trailing edge rudder of the NACA0012 airfoil (the chord length of the airfoil) is used as the control device, and the deep reinforcement learning TD3 algorithm is used as the optimization algorithm of the NACA0012 airfoil active rudder control model; six networks of the NACA0012 airfoil active rudder control model are randomly initialized: the strategy network, the target network of the strategy network, the first evaluation network, the second evaluation network, the target network of the first evaluation network, and the target network of the second evaluation network; the NACA0012 airfoil active rudder control model is updated according to the traditional TD3 algorithm until it converges to the optimal. All parameters of each network in the obtained NACA0012 airfoil optimal active rudder control model are shared with the initial RAE2822 airfoil active rudder control model with bulge; if there is a suitable control model that can be used as the initialization model, that is, the first airfoil active rudder control model, the above steps can be omitted, and all parameters of the control model network with the suitable initialization model can be directly shared with the initial RAE2822 airfoil active rudder control model with bulge.

[0067] In this embodiment, with respect to the reward function setting, in order to improve the training efficiency of the active rudder control model of the RAE2822 airfoil with bulge, the weight combination between the objectives in the reward function needs to be fully optimized in the training of the active rudder control model of the NACA0012 airfoil, so that the relevant settings of the reward function and the learning rate do not need to be changed when the NACA0012 airfoil active rudder control model is migrated to other airfoils (such as the RAE2822 airfoil with bulge); the expression of the reward function is as follows:

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] in, represents the reward function, represents the pulsation amplitude of the airfoil lift coefficient response, Indicates the change in the lift coefficient response of the airfoil at adjacent moments, represents the deflection amplitude of the trailing edge rudder surface, , , Both represent weight parameters, is the lift coefficient, express to The average value of the lift coefficient of the airfoil at time represents the change in lift coefficient, express to The average value of the active rudder surface deflection amplitude at each moment;

[0073] The trailing edge rudder control under the guidance of the active rudder control model of the RAE2822 airfoil with bulge interacts with the flow field solver and converts the current flow field response information into Input the strategy network of the active rudder control model of the RAE2822 airfoil with bulge to obtain the rudder deflection at the next moment ( ), the airfoil shape is changed by moving the mesh, and then the deflection of the rudder surface is obtained Response data of two-dimensional airfoil under the action of , and based on the response data of the two-dimensional airfoil Calculate the reward value for this action , the obtained relevant data is formed into an array Stored in the experience replay pool.

[0074] S2, in response to the number of data arrays in the experience playback pool exceeding the preset sampling data volume, randomly sampling the preset sampling data volume group data, and calculating the evaluation values ​​of the first evaluation network, the second evaluation network, the target network of the first evaluation network, and the target network of the second evaluation network of the second airfoil active control surface control model, and updating the parameters of the first evaluation network and the second evaluation network by using the calculation method of the timing difference error and the update strategy, the specific steps are as follows:

[0075] S201, randomly sampling a preset number of arrays from the experience replay pool according to a preset sampling data volume, bringing the response data of the two-dimensional airfoil in the array into the target network of the strategy network to obtain a target control law, and bringing the response data of the two-dimensional airfoil in the array and the target control law into the target network of the first evaluation network and the target network of the second evaluation network to obtain a first target evaluation value and a second target evaluation value respectively;

[0076] S202, selecting the smallest amount between the first target evaluation value and the second target evaluation value and multiplying it by a discount coefficient as an update target of the evaluation network;

[0077] S203, inputting the flow field information at the current moment, the deflection amount of the control surface at the next moment, and the response data of the two-dimensional airfoil in the array into the first evaluation network and the second evaluation network, and obtaining the first evaluation value at the current moment and the second evaluation value at the current moment respectively;

[0078] S204, calculating the difference between the update target of the evaluation network and the first evaluation value and the second evaluation value at the current moment, and minimizing the difference through gradient optimization, obtaining the parameter combination of the first evaluation network and the parameter combination of the second evaluation network in response to the difference convergence, and updating the first evaluation network and the second evaluation network.

[0079] In this embodiment, the evaluation network is updated. When the number of arrays in the experience replay pool exceeds the preset sampling data volume, When, and randomly sample The group data is used to calculate the two evaluation networks and the corresponding target network Q The relevant parameters of the two evaluation networks are updated using the TD-error calculation method and update strategy.

[0080] In this embodiment, a preset sampling data volume is selected for sampling and calculating the update target of the evaluation network. , randomly sampled from the experience replay pool Group data, Bring it into the target network of the strategy network and get the target control law , and then the response data of the two-dimensional airfoil in the array and the target control law The target network of the first evaluation network and the target network of the second evaluation network are respectively used to obtain the first target evaluation value and the second target evaluation value. , select the minimum of the first target evaluation value and the second target evaluation value and multiply it by the discount coefficient as the update target of the evaluation network, which is used to avoid the problem of overestimation of a certain action by the evaluation network;

[0081] The expression of the update target of the evaluation network is as follows:

[0082] ;

[0083] in, represents the update target of the evaluation network, Indicates k The reward function for the group rudder action, represents the discount factor, Indicates i Target evaluation value, Indicates k +1 set of flow field information in the sampling data, Indicates the k+1 The flow field information in the sampled data set is the action value output by the policy network when it is input. and All represent target network parameters. Representation parameters A normal distribution with truncated measures, represents a normal distribution, represents the variance of the normal distribution, represents the amplitude of the truncated normal distribution;

[0084] To update the evaluation network using the calculation method and update strategy of TD-error, As the input of the first evaluation network and the second evaluation network, the evaluation values ​​of the first evaluation network and the second evaluation network at the current moment are obtained. , and calculate the difference between the update target of the evaluation network and the first evaluation value and the second evaluation value at the current moment, and minimize the difference through gradient optimization. When the difference converges, obtain the parameter combination of the first evaluation network and the parameter combination of the second evaluation network, and update the first evaluation network and the second evaluation network;

[0085] The expressions of the parameter combination of the first evaluation network and the parameter combination of the second evaluation network are as follows:

[0086] ;

[0087] in, i =1, the above expression represents the parameter combination of the first evaluation network. i =2, the above expression represents the parameter combination of the second evaluation network. Indicates i Evaluate the weights and bias parameters of each neuron in the network, Indicates the preset sampling data volume, Indicates i Evaluation network update target, Indicates the k The flow field information and rudder deflection angle in the group of sampling data are input as i The evaluation value of the current action output by the evaluation network, Indicates k The flow field information in the group sampling data, Indicates k The deflection angle of the rudder surface in the group of sampled data.

[0088] S3. After updating the first evaluation network and the second evaluation network for a preset number of times, introduce transfer learning to update the strategy network, freeze all parameters of the hidden layers except the last hidden layer in the strategy network, update the weights and biases of each neuron on the last hidden layer in the strategy network, and obtain the optimal active rudder control model of the second airfoil according to the updated first evaluation network, the second evaluation network and the strategy network, and use the optimal active rudder control model of the second airfoil for flow active control to complete the flow active control efficiency design. The specific steps are as follows:

[0089] S301, after updating the first evaluation network and the second evaluation network for a preset number of times, introduce the policy network update strategy of transfer learning, adjust the update frequency of the policy network, freeze all parameters of the hidden layers except the last hidden layer in the policy network, change the weights and biases of each neuron on the last hidden layer of the policy network through gradient optimization, and update the parameters of the target network of the policy network, the target network of the first evaluation network, and the target network of the second evaluation network by soft update;

[0090] S302, the second airfoil active control surface control model interacts with the flow field solver according to the parameters of the target network of the updated strategy network, the target network of the first evaluation network, and the target network of the second evaluation network to obtain an updated array and put it into the experience playback pool;

[0091] S303, repeating steps S201-S302, in response to the convergence of the reward function, obtaining the optimal active rudder surface control model for the second airfoil, and using the optimal active rudder surface control model for the second airfoil for flow active control to complete the flow active control efficiency enhancement design.

[0092] In this embodiment, finally for the update of the policy network, the network update strategy adopted by the present invention is the asynchronous gradient update strategy of the policy network and the evaluation network. The number of updates is preset. In this embodiment, a delay of 2 is taken as an example. After the evaluation network is updated twice, the policy network is updated. At the same time, transfer learning is introduced to freeze all the parameters of the hidden layers except the last hidden layer in the policy network. The weights and biases of each neuron on the last hidden layer of the policy network are changed through gradient optimization to retain the relevant experience of the previous control design model and ensure the controllability of the network update direction. Then, without changing the reward function weight combination, learning rate and some network parameters, the migration from the optimal active rudder control model of NACA0012 to the optimal active rudder control model of RAE2822 airfoil with bulge is realized, thereby completing the flow active control efficiency enhancement design.

[0093] In this embodiment, for the asynchronous gradient update strategy, in order to prevent the policy network from falling into a local optimum or failing to find the optimal network parameter combination due to the continuous update of the evaluation network, the update frequency of the policy network is adjusted to 0.5 times the update frequency of the evaluation network;

[0094] For the policy network update strategy, transfer learning is introduced. The purpose of the policy network update is to find the rudder action that can obtain the maximum evaluation value at the next moment. , it is believed that the original NACA0012 airfoil optimal active rudder control model is a suboptimal model close to the optimal active rudder control model of the RAE2822 airfoil with bulge. In order to retain the training experience of the NACA0012 airfoil optimal active rudder control model to the greatest extent (the policy network is directly related to the action of the control device, and the weight optimization between the objectives in the reward function and the learning rate selection have a greater impact on the convergence of the policy network), it is chosen to update the weights and biases of each neuron in the policy network as little as possible. Therefore, only the parameters of each neuron in the last hidden layer of the policy network are changed through gradient optimization;

[0095] The expression of the gradient optimization is as follows:

[0096] ;

[0097] in, represents the weights and biases of the last hidden layer neurons in the policy network, Indicates the k The flow field information and rudder deflection angle in the group of sampling data are input as i The evaluation value of the current action output by the evaluation network, Indicates the preset sampling data volume, Indicates k The flow field information in the group sampling data, Indicates k The deflection angle of the rudder surface in the group sampling data, Indicates the k The flow field information in the group sampling data is the action value output by the policy network when it is input. represents the vector differential operator, represents the objective function for updating the policy network parameters;

[0098] The gradient update formula of the above strategy network related parameters is obtained by the chain rule of gradient operation; then the target network parameters are updated by soft update;

[0099] The expression of the soft update method is as follows:

[0100] ;

[0101] in, Indicates i Evaluate the weights and bias parameters of each neuron in the network, is the soft update coefficient, and its value range is , represents the target network parameters for the evaluation network, represents the target network parameters of the policy network;

[0102] The updated array is obtained by using the updated RAE2822 airfoil active control model with bulge to interact with the flow field controller. , and continue to put it into the experience playback pool. Repeat the update of the evaluation network and strategy network of the active rudder control model of the RAE2822 airfoil with bulge, and then update the active rudder control model of the RAE2822 airfoil with bulge to obtain the updated array All steps are repeated until the reward function converges, and the optimal active rudder control model of the RAE2822 airfoil with bulge is obtained and used for flow active control, completing the efficiency enhancement design of flow active control.

[0103] In this embodiment, Figure 3 As shown in the figure, based on the trained optimal active rudder control model of NACA0012 airfoil for transonic buffeting problem, transfer learning is introduced. By freezing the weights and biases of some neurons in the strategy network of the reinforcement learning model, the weight coefficients and learning rates between each item in the reward function are kept unchanged, and the optimal active rudder control model of NACA0012 airfoil is further trained to make it the optimal active rudder control design model of RAE2822 airfoil with bulge, which reduces the time consumed in optimizing the weight of the reward function and improves the efficiency of active control design based on reinforcement learning under variable shape or variable working conditions.

[0104] In this embodiment, Figure 4 As shown in the figure, taking the optimal NACA0012 airfoil active rudder control model as the initialization, the different training processes of the RAE2822 airfoil active rudder control model with bulge when and without introducing transfer learning are compared, and the reward value curve is plotted; it is found that the number of rounds required for the model with introduced transfer learning to reach the maximum reward value is about half of that without introducing transfer learning, which proves that introducing transfer learning can improve the training efficiency of the reinforcement learning model.

[0105] In this embodiment, Figure 5As shown, taking the optimal active rudder control model of NACA0012 airfoil as the initial, the different control effects of the active rudder control model of RAE2822 airfoil with bulge when introducing and not introducing transfer learning are compared, and the lift coefficient and rudder deflection angle curves are plotted; the optimal reward function weight combination of the optimal active rudder control model of NACA0012 airfoil is different from the optimal reward function weight combination of the active control design model of RAE2822 airfoil with bulge. If transfer learning is not introduced, the control law that satisfies the control constraint of no deflection of the trailing edge rudder at the end of control cannot be obtained, and the introduction of transfer learning can obtain the control law that satisfies the three control objectives without changing the reward function weight combination; the optimal active rudder control model of NACA0012 airfoil fails to achieve the complete suppression of the lift coefficient, nor the goal of the rudder control returning to zero, but the active rudder control model of RAE2822 airfoil with bulge not only completely suppresses the pulsation phenomenon of the lift response, but also makes the rudder return to the initial uncontrolled state after the control ends. The necessity of transfer learning in the method of the present invention is proved.

Claims

1. A flow active control efficiency enhancement design method based on transfer learning, characterized in that: The following steps are involved: S1. Share all network parameters of the optimal active rudder control model of the first airfoil with the initial second airfoil active rudder control model for initialization, and interact the initialized second airfoil active rudder control model with the flow field solver to obtain the relevant response data of the airfoil flow to the rudder action guided by the second airfoil active rudder control model, and calculate the corresponding reward value to form an initial array and store it in the experience replay pool, specifically: A trailing edge control surface with a preset chord-wise length is selected as a control device, and a strategy network, a target network of the strategy network, a first evaluation network, a second evaluation network, a target network of the first evaluation network, and a target network of the second evaluation network of a control model of an active control surface of a first airfoil are randomly initialized; The active rudder control model of the first airfoil is optimized by using a double-delayed deep deterministic policy gradient algorithm to obtain the optimal active rudder control model of the first airfoil, and all parameters of each network in the optimal active rudder control model of the first airfoil are shared with the initial active rudder control model of the second airfoil for initialization; A reward function is set, and a weight combination of each objective in the reward function is set according to the optimal active rudder control model of the first airfoil; The trailing edge rudder control guided by the active rudder control model of the second airfoil interacts with the flow field solver, and the flow field information at the current moment is input into the strategy network of the active rudder control model of the second airfoil to obtain the rudder deflection at the next moment, and the change of the airfoil shape is realized according to the dynamic grid to obtain the response data of the two-dimensional airfoil; The reward value of the current rudder action is calculated based on the response data of the two-dimensional airfoil, and the flow field information at the current moment, the deflection amount of the rudder at the next moment, the response data of the two-dimensional airfoil, and the reward value of the current rudder action are combined into an initial array and stored in the experience playback pool; S2, in response to the number of data arrays in the experience playback pool exceeding the preset sampling data amount, randomly sampling the preset sampling data amount group data, and calculating the evaluation values ​​of the first evaluation network, the second evaluation network, the target network of the first evaluation network and the target network of the second evaluation network of the second airfoil active control surface control model, and updating the parameters of the first evaluation network and the second evaluation network by using the calculation method of the timing difference error and the update strategy; S3. After updating the first evaluation network and the second evaluation network for a preset number of times, introduce transfer learning to update the strategy network, freeze all parameters of the hidden layers except the last hidden layer in the strategy network, update the weights and biases of each neuron on the last hidden layer in the strategy network, and obtain the optimal active rudder control model for the second airfoil based on the updated first evaluation network, the second evaluation network and the strategy network. The optimal active rudder control model for the second airfoil is used for flow active control to complete the flow active control efficiency enhancement design.

2. The flow active control efficiency enhancement design method based on transfer learning according to claim 1 is characterized in that: The expression of the reward function is as follows: in, represents the reward function, represents the pulsation amplitude of the airfoil lift coefficient response, Indicates the change in the lift coefficient response of the airfoil at adjacent moments, represents the deflection amplitude of the trailing edge rudder surface, , , Both represent weight parameters, is the lift coefficient, express to The average value of the lift coefficient of the airfoil at time represents the change in lift coefficient, express to The average value of the active rudder surface deflection amplitude at each moment.

3. The flow active control efficiency enhancement design method based on transfer learning according to claim 1 is characterized in that: The S2 comprises the following steps: S201, randomly sampling a preset number of arrays from the experience replay pool according to a preset sampling data volume, bringing the response data of the two-dimensional airfoil in the array into the target network of the strategy network to obtain a target control law, and bringing the response data of the two-dimensional airfoil in the array and the target control law into the target network of the first evaluation network and the target network of the second evaluation network to obtain a first target evaluation value and a second target evaluation value respectively; S202, selecting the smallest amount between the first target evaluation value and the second target evaluation value and multiplying it by a discount coefficient as an update target of the evaluation network; S203, inputting the flow field information at the current moment, the deflection amount of the control surface at the next moment, and the response data of the two-dimensional airfoil in the array into the first evaluation network and the second evaluation network, and obtaining the first evaluation value at the current moment and the second evaluation value at the current moment respectively; S204, calculating the difference between the update target of the evaluation network and the first evaluation value and the second evaluation value at the current moment, and minimizing the difference through gradient optimization, obtaining the parameter combination of the first evaluation network and the parameter combination of the second evaluation network in response to the difference convergence, and updating the first evaluation network and the second evaluation network.

4. The flow active control efficiency enhancement design method based on transfer learning according to claim 3 is characterized in that: The expression of the update target of the evaluation network is as follows: in, Indicates i The update target of the evaluation network, Indicates k The reward function for the rudder action in the group sampling data, represents the discount factor, Indicates i Target evaluation value, Indicates k +1 set of flow field information in the sampling data, Indicates the k +1 The flow field information in the sampled data set is the action value output by the policy network when it is input. represents the target network parameters for the evaluation network, represents the target network parameters, Representation parameters A normal distribution with truncated measures, represents a normal distribution, represents the variance of the normal distribution, Represents the amplitude of the truncated normal distribution.

5. The flow active control efficiency enhancement design method based on transfer learning according to claim 3 is characterized in that: The expressions of the parameter combination of the first evaluation network and the parameter combination of the second evaluation network are as follows: in, i =1, the above expression represents the parameter combination of the first evaluation network. i =2, the above expression represents the parameter combination of the second evaluation network. Indicates i Evaluate the weights and bias parameters of each neuron in the network, Indicates the preset sampling data volume, Indicates i The update target of the evaluation network, Indicates the k The flow field information and rudder deflection angle in the group of sampling data are input as i The evaluation value of the current action output by the evaluation network, Indicates k The flow field information in the group sampling data, Indicates k The deflection angle of the rudder surface in the group of sampled data.

6. The flow active control efficiency enhancement design method based on transfer learning according to claim 3 is characterized in that: The S3 comprises the following steps: S301, after updating the first evaluation network and the second evaluation network for a preset number of times, introduce the policy network update strategy of transfer learning, adjust the update frequency of the policy network, freeze all parameters of the hidden layers except the last hidden layer in the policy network, change the weights and biases of each neuron on the last hidden layer of the policy network through gradient optimization, and update the parameters of the target network of the policy network, the target network of the first evaluation network, and the target network of the second evaluation network by soft update; S302, the second airfoil active control surface control model interacts with the flow field solver according to the parameters of the target network of the updated strategy network, the target network of the first evaluation network, and the target network of the second evaluation network to obtain an updated array and put it into the experience playback pool; S303, repeating steps S201-S302, in response to the convergence of the reward function, obtaining the optimal active rudder surface control model for the second airfoil, and using the optimal active rudder surface control model for the second airfoil for flow active control to complete the flow active control efficiency enhancement design.

7. The flow active control efficiency enhancement design method based on transfer learning according to claim 6 is characterized in that: The expression of the gradient optimization is as follows: in, represents the weights and biases of the last hidden layer neurons in the policy network, Indicates the k The flow field information and rudder deflection angle in the group of sampling data are input as i The evaluation value of the current action output by the evaluation network, Indicates the preset sampling data volume, Indicates k The flow field information in the group sampling data, Indicates k The deflection angle of the rudder surface in the group sampling data, Indicates the k The flow field information in the group sampling data is the action value output by the policy network when it is input. represents the vector differential operator, represents the objective function for updating the policy network parameters; The expression of the soft update method is as follows: in, Indicates i Evaluate the weights and bias parameters of each neuron in the network, is the soft update coefficient, and its value range is , represents the target network parameters for the evaluation network, Represents the target network parameters of the policy network.

Images