An Active Flow Control Method Based on Agent Model and Transfer Learning

Abstract

The present invention belongs to the field of active flow control technology, and discloses an active flow control method based on agent model and transfer learning, including the following steps: S1, data sampling based on model-free reinforcement learning; S2, training the neural network based on the interaction data between the intelligent agent module and the CFD environment module in step S1; S3, active flow control training based on the agent model; S4, based on the idea of ​​transfer learning, using the agent model reinforcement learning framework established in step S3 to perform active flow control training in complex environments. The present invention adopts the above-mentioned active flow control method based on agent model and transfer learning, trains the agent model based on model-free reinforcement learning interaction experience, reduces the time and computing power of pre-simulating data in advance, improves the interaction speed, and quickly implements the active flow control strategy in complex environments.

Description

Technical Field

[0001] The present invention relates to the technical field of active flow control, and in particular to an active flow control method based on an agent model and transfer learning. Background Art

[0002] Drag reduction is an important topic in fluid mechanics and many industrial fields. Technologies to achieve drag reduction include passive and active flow control technologies. Active flow control (AFC) technology can further reduce drag during transportation based on passive flow control.

[0003] In order to achieve active flow control in complex environments, deep reinforcement learning (DRL) shows great potential due to its beneficial control effects in complex environments. Among them, the environmental module of DRL is mainly introduced in the form of computational fluid dynamics (CFD). Active flow control technology based on DRL mainly uses the interaction between the agent and the environment to generate a large amount of data. The agent learns the size of the reward obtained by performing different actions in different states, thereby achieving the control effect. At present, for the research on active flow control, most of the focus is on the application of deep reinforcement learning in different fields. Few studies have involved the agent model as a means to accelerate the DRL process, and the idea of ​​​​the agent model transfer learning method to enhance the control of more complex environments has not yet been proposed. In order to implement active flow control technology in a wider range of fluid flow environments, it is necessary to accelerate the development of strategies and enhance the control effect of strategies. At the same time, for more complex flow fields (high Reynolds number), how to use prior knowledge to accelerate and improve performance has not yet been proposed.

[0004] In terms of active flow control technology based on deep learning, although the existing technology can be applied to complex active flow control technology, using OpenFOAM as a control method for environment generation, the strategy generation speed is slow, and the bottleneck of strategy generation lies in the CFD calculation speed. There are also methods for improving the training of active flow control strategies. Although the flow field reduction model of the masked deep neural network is used as the environment to interact with the deep reinforcement learning algorithm to achieve rapid solution of the active flow control strategy. However, on the one hand, the construction of the reduced-order model requires pre-simulation of a large amount of data, which is time-consuming; on the other hand, for more complex environments, the accuracy of the reduced-order model has errors, which also has a greater impact on the effect of the generated strategy.

[0005] In summary, existing active flow control technologies based on reinforcement learning do not yet involve means for accelerating the generation and optimizing the effects of flow control strategies in complex flow environments. Summary of the invention

[0006] The purpose of the present invention is to provide an active flow control method based on agent model and transfer learning, which trains the agent model based on model-free reinforcement learning interactive experience, reduces the time and computing power of pre-simulating data in advance, improves the interaction speed, and quickly realizes the active flow control strategy in complex environments.

[0007] To achieve the above object, the present invention provides an active flow control method based on agent model and transfer learning, comprising the following steps:

[0008] S1. Data sampling based on model-free reinforcement learning;

[0009] S2, training the neural network based on the interaction data between the intelligent agent module and the CFD environment module in step S1;

[0010] S3, active flow control training based on agent model;

[0011] S4. Based on the idea of ​​transfer learning, the agent model reinforcement learning framework established in step S3 is used to conduct active flow control training in complex environments.

[0012] Preferably, in step S1, model-free reinforcement learning is composed of a CFD environment module and an agent module; wherein the CFD environment module is composed of ANSYS Fluent; the agent module is composed of a proximal strategy optimization algorithm, and the proximal strategy optimization algorithm is used to iteratively optimize the neural network.

[0013] Preferably, in step S1, data sampling is performed based on model-free reinforcement learning, and the specific process is as follows:

[0014] S11. Connect the agent module and the environment module through pyfluent to realize the agent's change of boundary conditions in ANSYSFluent to obtain the change of flow resistance under the applied action;

[0015] S12. In each step of interaction, the agent observes the current state s t , choose action a according to its strategy t , and then perform this action in the environment;

[0016] S13. After executing the action, the agent will observe a new state s t+1 and a reward r t , these data are based on (s t , a t , r t ,s t+1 ) is recorded in the form of an experience tuple or trajectory, which is used to update the action network and the critic network.

[0017] Preferably, in step S2, the neural network is trained based on the interaction data between the intelligent agent module and the CFD environment module in step S1, and the specific process is as follows:

[0018] S21, with the help of fully connected neural network, the current state s t and the action a taken in this state t As the input of the neural network; with the current state s t Take a t The flow field obtained after t+1 and the reward r t As output, a database is constructed;

[0019] S22, initialize the weights and biases of the fully connected neural network, and set s t and a t Passed as input to the neural network, the predicted s is calculated t+1 ′ and r t ′;

[0020] S23. Use loss function to measure the predicted value s t+1 ′ and r t ′ and the true value s t+1 and r t The gap between them is used to update the weights of the neural network using the gradient calculated by the loss function;

[0021] S24. Repeat steps S21-S23 until the network converges or reaches a predetermined number of training rounds, and obtain proxy models under different Reynolds numbers.

[0022] Preferably, the proxy model takes the form of a fully connected neural network that can process input vectors of any size and map them to another vector space to predict the output.

[0023] Preferably, in step S3, an active flow control method based on an agent model is established, and the specific implementation process is as follows:

[0024] S31, take the flow field without control as the initial state s t , contains all physical parameters related to the flow field;

[0025] S32, initialize the neural network of the intelligent agent module, and design the receiving flow field state s t , with the received state s t As input, the agent generates an action a through its internal decision-making mechanism t ;

[0026] S33, the current state s t and the action a generated by the agent tCombined into an input array, passed to the proxy model; the proxy model uses a neural network to simulate the response of the real flow field and quickly predict the execution action a t The new flow state s t+1 And the corresponding reward r t ; New flow field state s t+1 Become the input state of the next time step, thus forming a closed-loop system;

[0027] S34, the agent is rewarded according to r t By updating its internal strategy, through continuous iteration, the agent learns the optimal flow control strategy and realizes active flow control.

[0028] Preferably, reward r t It is used to evaluate the contribution of the actions taken by the agent to the flow control goal and reflects the goal of flow control.

[0029] Preferably, in step S4, based on transfer learning, the agent model reinforcement learning framework established in step S3 is utilized, the strategy of active flow control of the flow field in a simple environment is used as the initial strategy, training is performed in a complex environment, and the technology of the original strategy is fine-tuned. The intelligent agent continuously optimizes the control strategy to adapt it to more complex flow field conditions, and finds new strategies for achieving flow control in complex environments, thereby achieving active flow control in complex environments.

[0030] Therefore, the present invention adopts the above-mentioned active flow control method based on agent model and transfer learning, and trains the agent model based on model-free reinforcement learning interactive experience, which significantly reduces the time and computing power of pre-simulating data in advance, improves the interaction speed, and solves the shortcomings of slow interaction speed of existing active flow control, and solves the shortcomings of insufficient control strategy caused by limited accuracy of agent models in complex environments, and quickly realizes active flow control strategies in complex environments.

[0031] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of model-free reinforcement learning of the present invention;

[0033] Figure 2 It is a schematic diagram of the flow field structure of the jet around the cylindrical surface;

[0034] Figure 3 It is a schematic diagram of active flow control based on the agent model of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0036] The present invention provides an active flow control method based on an agent model and transfer learning, comprising the following steps:

[0037] S1. Data sampling based on model-free reinforcement learning;

[0038] S2, training the neural network based on the interaction data between the intelligent agent module and the CFD environment module in step S1;

[0039] S3, active flow control training based on agent model;

[0040] S4. Based on the idea of ​​transfer learning, the agent model reinforcement learning framework established in step S3 is used to conduct active flow control training in complex environments.

[0041] Example

[0042] S1. Data sampling based on model-free reinforcement learning.

[0043] like Figure 1 As shown in Figure 1, model-free reinforcement learning consists of a CFD environment module and an agent module. The CFD environment module of model-free reinforcement learning consists of ANSYS Fluent. Taking the jet around a cylinder as an example, through ANSYS Fluent simulation, the velocity between the jet inlet and outlet on the upper and lower surfaces of the cylinder is adjusted to change the flow pattern and reduce the flow resistance. Figure 2 As shown in the figure, the velocity and pressure of the flow field are solved by solving the incompressible Navier-Stokes equations and the continuity equation. The intelligent agent module of model-free reinforcement learning consists of the proximal policy optimization (PPO) algorithm, which performs iterative optimization of the neural network. The velocity probe value in the flow field is used as s t , sensing the flow changes in the flow field; the output action of the action network is a t , which is the boundary condition of the jet inlet and outlet. The specific implementation process of data sampling is as follows:

[0044] S11. Connect the agent module and the environment module through pyfluent to realize the agent's change of boundary conditions in ANSYSFluent to obtain the change of flow resistance under the applied action.

[0045] S12. In each step of interaction, the agent observes the current state s t , choose action a according to its strategy t , and then execute this action in the environment.

[0046] S13. After executing the action, the agent will observe a new state s t+1 and a reward r t , these data are based on (st , a t , r t ,s t+1 ) is recorded in the form of an experience tuple or trajectory, which is used to update the action network and the critic network.

[0047] S2. Based on the interaction data between the intelligent agent module and the CFD environment module in step S1, the neural network is trained.

[0048] In the interaction between the agent module and the CFD environment module in step S1, a large amount of data will be generated, which consists of (s t , a t , r t ,s t+1 ), these experience tuples will be used to train the proxy model so that the agent can learn and make decisions more efficiently. The proxy model is usually in the form of a fully connected neural network (FCN), which is able to process input vectors of any size and map them to another vector space to predict the output. The role of the proxy model is to predict the next state s without actually running the CFD simulation. t+1 and reward r t , which can significantly reduce the computing resources and time required for actual simulation. The specific process of training a neural network is as follows:

[0049] S21, with the help of FCN, the current state s t and the action a taken in this state t As the input of the neural network; in state s t Next take a t Then we get the flow field s t+1 and the reward r t As output, a database is constructed.

[0050] S22, initialize the weights and biases of FCN, and set s t and a t Passed as input to the neural network, the predicted s is calculated t+1 ′ and r t ′.

[0051] S23. Use an appropriate loss function (such as mean square error) to measure the predicted value s t+1 ′ and r t ′ and the true value s t+1 and r t The weights of the neural network are updated according to the gradient calculated by the loss function.

[0052] S24. Repeat steps S21-S23 until the network converges or reaches a predetermined number of training rounds, and obtain proxy models under different Reynolds numbers.

[0053] S3. Based on the agent model of step S2 and the intelligent agent module, an active flow control method based on the agent model is established.

[0054] like Figure 3 As shown, the agent model trained at low Reynolds number in step S2 is used as the environment module of reinforcement learning, and combined with the intelligent agent module in step S1, an active flow control method based on the agent model is established. The specific implementation process is as follows:

[0055] S31, take the flow field without control as the initial state s t , which contains all physical parameters related to the flow field (such as velocity, pressure distribution, etc.).

[0056] S32, initialize the neural network of the intelligent agent module, and design the receiving flow field state s t , with the received state s t As input, the agent generates an action a through its internal decision-making mechanism (such as a deep neural network) t .

[0057] S33, the current state s t and the action a generated by the agent t Combined into an input array, passed to the proxy model; the proxy model uses a neural network to simulate the response of the real flow field and quickly predict the execution action a t The new flow state s t+1 And the corresponding reward r t ; New flow field state s t+1 becomes the input state of the next time step, thus forming a closed-loop system.

[0058] S34, Reward t It is used to evaluate the contribution of the actions taken by the agent to the flow control target, reflecting the flow control target (such as reducing drag, increasing lift, etc.). The agent is rewarded according to r t Update its internal strategy to make better action choices in the future. By continuously iterating this process, the agent can gradually learn the optimal flow control strategy, thereby achieving efficient and accurate active flow control.

[0059] S4. Based on the idea of ​​transfer learning, the agent model reinforcement learning framework established in step S3 is used to conduct active flow control training in complex environments.

[0060] Based on transfer learning, the agent model reinforcement learning framework established in step S3 is used to take the strategy of active flow control in the flow field under a simple environment (low Reynolds number) as the initial strategy, and training is performed in a complex environment (high Reynolds number). The original strategy is fine-tuned technically, and the intelligent agent continuously optimizes the control strategy to adapt it to more complex flow field conditions, finds new strategies that can achieve efficient flow control under high Reynolds numbers, and quickly realizes active flow control in complex environments.

[0061] Therefore, the present invention adopts the above-mentioned active flow control method based on agent model and transfer learning, and trains the agent model based on model-free reinforcement learning interactive experience, which significantly reduces the time and computing power of pre-simulating data in advance, improves the interaction speed, and solves the shortcomings of slow interaction speed of existing active flow control, and solves the shortcomings of insufficient control strategy caused by limited accuracy of agent models in complex environments, and quickly realizes active flow control strategies in complex environments.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An active flow control method based on agent model and transfer learning, characterized in that: The following steps are involved: S1. Data sampling based on model-free reinforcement learning. The specific process is as follows: S11. Connect the agent module and the environment module through pyfluent to realize the agent's change of boundary conditions in ANSYS Fluent to obtain the change of flow resistance under the applied action; S12. In each step of interaction, the agent observes the current state s t , choose action a according to its strategy t , and then perform this action in the environment; S13. After executing the action, the agent will observe a new state s t+1 and a reward r t , these data are based on (s t , a t , r t ,s t+1 ) is recorded in the form of an experience tuple or trajectory, which is used to update the action network and the critic network; S2. Based on the interaction data between the agent module and the CFD environment module in step S1, the neural network is trained. The specific process is as follows: S21, with the help of fully connected neural network, the current state s t and the action a taken in this state t As the input of the neural network; with the current state s t Take a t The flow field obtained after t+1 and the reward r t As output, a database is constructed; S22, initialize the weights and biases of the fully connected neural network, and set s t and a t Passed as input to the neural network, the predicted s is calculated t+1 ′ and r t ′; S23. Use loss function to measure the predicted value s t+1 ′ and r t ′ and the true value s t+1 and r t The gap between them is used to update the weights of the neural network using the gradient calculated by the loss function; S24, repeating steps S21-S23 until the network converges or reaches a predetermined number of training rounds, and obtaining proxy models under different Reynolds numbers; S3. Active flow control training based on agent model: Establish an active flow control method based on agent model. The specific process is as follows: S31, take the flow field without control as the initial state s t , contains all physical parameters related to the flow field; S32, initialize the neural network of the intelligent agent module, and design the receiving flow field state s t , with the received state s t As input, the agent generates an action a through its internal decision-making mechanism t ; S33, the current state s t and the action a generated by the agent t Combined into an input array, passed to the proxy model; the proxy model uses a neural network to simulate the response of the real flow field and quickly predict the execution action a t The new flow state s after t+1 And the corresponding reward r t ; New flow field state s t+1 Become the input state of the next time step, thus forming a closed-loop system; S34, the agent based on the reward r t Update its internal strategy. Through continuous iteration, the agent learns the optimal flow control strategy and realizes active flow control. S4. Based on the idea of ​​transfer learning, the agent model reinforcement learning framework established in step S3 is used to conduct active flow control training in complex environments.

2. The active flow control method based on agent model and transfer learning according to claim 1, characterized in that: In step S1, model-free reinforcement learning is composed of a CFD environment module and an agent module; wherein the CFD environment module is composed of ANSYS Fluent; the agent module is composed of a proximal strategy optimization algorithm, and the proximal strategy optimization algorithm is used to iteratively optimize the neural network.

3. The active flow control method based on agent model and transfer learning according to claim 1, characterized in that: The proxy model takes the form of a fully connected neural network that is able to process input vectors of arbitrary size and map them into another vector space to predict the output.

4. The active flow control method based on agent model and transfer learning according to claim 1, characterized in that: Reward t It is used to evaluate the contribution of the actions taken by the agent to the flow control goal and reflects the goal of flow control.

5. The active flow control method based on agent model and transfer learning according to claim 1, characterized in that: In step S4, based on transfer learning, the agent model reinforcement learning framework established in step S3 is used to take the strategy of active flow control of the flow field in a simple environment as the initial strategy, and training is performed in a complex environment. The original strategy is fine-tuned technically, and the intelligent agent continuously optimizes the control strategy to adapt it to more complex flow field conditions, and finds new strategies to achieve flow control in complex environments, thereby achieving active flow control in complex environments.

Images