Active flow control data acquisition method, active flow control method and device based on expert demonstration

Abstract

This invention provides a method for acquiring active flow control data based on expert demonstrations, as well as an active flow control method and apparatus. The invention establishes a multi-dimensional surrogate model based on a deep neural network, combining it with a Gaussian mixture model to characterize the uncertainty of the surrogate model. While ensuring model accuracy, a particle swarm optimization heuristic algorithm is used to dynamically iteratively find the optimal static active flow closed-loop control parameters to provide deep learning expert demonstration knowledge. This invention establishes a decision-making agent that performs numerical simulations of the flow control process and evaluates the flow control strategy in rounds. Simulation data is stored in an experience pool pre-mixed with expert demonstration knowledge. Deep reinforcement learning randomly reads past experiences from the experience pool to adjust the strategy. The reinforcement learning method based on expert demonstration technology can quickly explore robust and efficient active flow control strategies to suppress vortex-induced vibrations in cylinders, reducing the amount of sample data required for training the active flow control device.

Description

Technical Field

[0001] This invention relates to a method for acquiring active flow control data based on expert demonstrations, an active flow control method, and an apparatus. It establishes a surrogate model using deep neural network technology, dynamically updates and improves the surrogate model using a Gaussian mixture model, and searches for expert demonstration knowledge using a particle swarm optimization heuristic algorithm, pre-mixing it into an experience pool established by a soft actor-critic algorithm. This provides high-quality data for dynamically adjusting the parameters of the artificial neural network, reduces the data requirements for numerical simulation of active flow control, and improves training efficiency, belonging to the field of active flow control. Background Technology

[0002] Due to the high dimensionality and strong nonlinearity of fluid mechanics, designing active flow control strategies using traditional control methods is often a very tedious task. Reinforcement learning based on deep neural networks, through iterative interaction with the simulation environment, establishes an end-to-end control strategy directly from the system state to the control output through trial and error. However, reinforcement learning requires a large number of interactive errors to correct the control strategy, which is a significant burden on time-consuming and resource-intensive computational fluid dynamics numerical simulation techniques. Therefore, to improve the practical application value of reinforcement learning in active flow control engineering problems, it is necessary to provide a more efficient application framework for deep reinforcement learning algorithms that improves data utilization and reduces sample requirements. Summary of the Invention

[0003] The purpose of this invention is to solve the above-mentioned technical problems and provide an active flow control data acquisition method, an active flow control method, and an apparatus based on expert demonstrations.

[0004] The technical solution adopted in this invention is as follows:

[0005] A method for acquiring active flow control data based on expert demonstrations, comprising:

[0006] A surrogate model is established to characterize the relationship between static control parameters and the amplitude of vortex-induced vibration of a cylinder; the surrogate model is established using deep neural network technology.

[0007] Y A =f NN (g1,g2)

[0008] Among them, Y A f is the stable vibration amplitude of the cylinder's center of mass. NN The mapping is established by a deep neural network; the static control parameters (g1, g2) are the control laws of the closed-loop control, which are static parameters under the specified strategy conditions and do not change with the flow state.

[0009] The surrogate model is trained based on the stable vibration amplitude data of the cylinder's center of mass obtained from spatial sampling and numerical simulation of static control parameters. Then, the expert demonstration trajectory is generated based on the prediction results of the trained surrogate model to obtain active flow control data.

[0010] The methods for static control parameter space sampling include the following dynamic sampling methods:

[0011] Based on the Gaussian mixture model to characterize the uncertainty of the surrogate model, a heuristic algorithm is used to solve the multi-objective optimization problem by dynamically sampling in the static control parameter space; wherein the Gaussian mixture model in the static control parameter space uses multiple Gaussian probability distributions to accurately quantify the distribution of static control parameters, as shown below:

[0012]

[0013]

[0014] Where k represents the probability density function of the k-th Gaussian probability distribution, K indicates that the model contains K Gaussian probability distributions, N represents the probability density function of a Gaussian probability distribution, x represents the sample point, i.e., the static control parameters (g1, g2), and Y... p τ represents the probability of a sample point under this Gaussian mixture model distribution, where τ is the parameter of the Gaussian mixture model, including the weights α of each Gaussian probability density function. k Mean μ k and variance The parameter update formula for the Gaussian mixture model is shown below:

[0015]

[0016]

[0017] q ik =p(z) k |x i ;τ n-1 )

[0018] Where n represents the number of iterations, i represents the i-th actual sample point collected, M represents the number of actual sample points collected in the dataset, and z k This indicates that it belongs to the k-th Gaussian probability distribution; p is the probability function, and B is the lower bound function of the log-likelihood function of parameter τ, which is also the objective function of optimization;

[0019] The multi-objective optimization problem is:

[0020]

[0021] Among them, Y AY represents the stable vibration amplitude corresponding to sample point x predicted based on the surrogate model. p The value represents the probability of a sample point under this Gaussian mixture model distribution, where α is the weight and x is the weight. next These are the static control parameters for the next evaluation.

[0022] Furthermore, the process of generating expert demonstration trajectories based on the prediction results of the trained surrogate model to obtain active flow control data specifically involves: solving an optimization problem based on the surrogate model, finding the optimal static control parameters, and performing flow control simulation using computational fluid dynamics techniques. The control trajectory is then stored in the experience pool of deep reinforcement learning. The optimization problem based on the surrogate model is as follows:

[0023]

[0024] Where, x optimal The optimal static control parameters are predicted by the surrogate model; for each expert demonstration knowledge trajectory, the instantaneous system state s of the cylinder and its surrounding flow field at time t is formatted according to the Markov overchain model required by the deep reinforcement learning algorithm. t Control action a t The instantaneous system state s of the cylinder and its surrounding flow field at time t+1 t+1 and control action a t The corresponding reward and punishment function r t and the final state variable done t The instantaneous system state s of the cylinder and its surrounding flow field t Includes information on the cylinder's motion state, velocity and pressure in the flow field; reward / penalty function r t It is expressed as follows:

[0025] r t =κd t +χv t

[0026] Where, d t v is the displacement of the center of mass of the cylinder. t Let be the velocity of the cylinder's center of mass, and κ and χ be the corresponding weights.

[0027] Furthermore, the relationship between the static control parameters (g1, g2) and the control action is as follows:

[0028] a t =g1·d t +g2·v t

[0029] The subscript t represents the time step number, a t Let d be the control action at time t. t Let v be the displacement of the center of mass of the vortex-induced vibrating cylinder at time t. tLet be the velocity of the center of mass of the vortex-induced vibrating cylinder at time t. Control action a. t The action of the blowing and suction device applied to the surface of the cylinder is used to change the flow field in the vicinity.

[0030] Furthermore, the artificial neural network structure of the surrogate model includes three fully connected hidden layers, each containing 16, 32, and 16 neurons respectively. The surrogate model takes static control parameters as input and outputs the amplitude of cylindrical vortex-induced vibration under active flow control.

[0031] Furthermore, the surrogate model is trained based on the Adaptive Moment Estimation algorithm. In the initial training stage, 10 sets of sample points are randomly selected in the static control parameter space based on the Latin hypercube method as the basic data for training the surrogate model. Then, sample points are sampled 10 times using the dynamic sampling method as the training data for the surrogate model to continuously improve the surrogate model.

[0032] Furthermore, the parameters of the Gaussian mixture model are initialized using the K-means algorithm and iteratively learned using the expectation-maximization algorithm, with a maximum number of iterations of 2000.

[0033] Furthermore, the particle swarm optimization heuristic algorithm is used to solve the multi-objective optimization problem. Dynamic sampling is performed in the static control parameter space, with 40 populations set for each generation of optimization, and a total of 50 generations are iterated.

[0034] An active flow control device based on expert demonstration is disclosed. This device is used to execute actions to control a blow-suction device arranged on the surface of a cylinder based on the instantaneous system state output of the flow field around the cylinder. The active flow control device is composed of an artificial neural network, trained using a method guided by expert demonstration knowledge. This method learns an active flow control strategy that suppresses vortex-induced vibrations of the cylinder by outputting actions to control the blow-suction device arranged on the surface of the cylinder based on the flow field information around the cylinder, thereby improving data utilization efficiency during the learning process. Specifically:

[0035] A decision-making agent incorporating an active flow control device and a vortex-induced vibration numerical simulation environment are constructed. The vortex-induced vibration numerical simulation environment is used to simulate the active flow control process of the decision-making agent: by changing the nozzle flow rate of the surface blowing and suction device according to the control action, the changes in the flow field are deduced to obtain the instantaneous system state s of the cylinder and its surrounding flow field at time t+1. t+1The control trajectory is stored in the experience pool of the deep reinforcement learning algorithm; the experience pool also contains active flow control data acquired by the aforementioned active flow control data acquisition method based on expert demonstration; the decision-making agent also includes an evaluator, Critic, wherein the active flow control device acts as the decision-maker to determine the instantaneous system state s of the cylinder and its surrounding flow field at time t. t Give control action a t The evaluator is used to assess the effectiveness of the current action based on the instantaneous system state of the cylinder and the surrounding flow field. The evaluator is composed of an artificial neural network.

[0036] The training process of the active flow control device begins in rounds, with each new round starting from an uncontrolled steady flow. A fluid-structure interaction (FSI) example of a cylinder in unsteady flow at Reynolds number 100 is calculated using the finite volume method and dynamic mesh technique in the open-source computational fluid dynamics software OpenFOAM, with the periodic steady state of the uncontrolled flow serving as the starting state for each round.

[0037] Training begins by reading past data containing expert demonstration information from the experience pool and training the decision agent using a soft decision-maker-evaluator algorithm. The Bellman backtracking equation is used as the objective function for training the evaluator.

[0038]

[0039] Where π represents the current decision-maker's control strategy, ρ π Let θ represent the transition probability distribution corresponding to the current control strategy, and let θ represent the parameters of the evaluator's artificial neural network. It is an expectation, Q θ This represents the state-action evaluation value given by the current evaluator's artificial neural network. This represents a more precise assessment value:

[0040]

[0041] Where H is the entropy corresponding to policy π in the current observation state. Let ω be the weight of the action entropy in the evaluation function, and ω be the influence factor of future evaluation items on the current evaluation; ·|S t This represents all possible actions that can be taken in the current state; r t This is the reward / punishment function; the objective function for training the artificial neural network of the decision-maker (Actor) is as follows:

[0042]

[0043] Where φ represents the parameters of the decision-maker's artificial neural network;

[0044] If the flow state reaches the end of the round, then the current round ends; otherwise, continue training and deducing the control process of the current flow.

[0045] If the control strategy represented by the decision agent has converged, the training process ends; otherwise, a new training round begins.

[0046] Furthermore, the artificial neural network structure of the active flow control device comprises three fully connected hidden layers, each with 200 neural units. The activation function is the ReLU function. The input is the flow field state s, and the output is the average value μ(s) and standard deviation σ(s) of the action. The final action can be calculated as follows:

[0047] a=μ(s)+σ(s)⊙z

[0048] in,

[0049] The evaluator's artificial neural network structure consists of two fully connected hidden layers, each with 200 neurons. The activation function is ReLU. The inputs are the flow field state s and the action to be performed, and the output is the quality Q of the current state. θ (S t ,a t ).

[0050] Furthermore, the numerical simulation environment was built on the open-source software OpenFOAM platform, using the laminar model and the pimpleFoam incompressible transient solver; the dynamic mesh solver used was a six-degree-of-freedom motion model solver, and the maximum allowable deformation range of the mesh was 7.

[0051] Furthermore, the active flow control system is equipped with two blowing and blowing devices, symmetrically installed at the poles of the cylinder in the transverse flow direction. The sum of the blowing and blowing control flow rates of the two devices at the same time is zero. A monitoring probe is installed in the flow field to transmit the corresponding velocity field and pressure field information in real time.

[0052] A control method based on the above-mentioned expert-demonstration-based active flow control device specifically includes:

[0053] Monitoring probes are arranged around the cylinder to collect the instantaneous system state s of the cylinder and the surrounding flow field in real time. t The data is then transmitted to a trained active flow control device, which uses the instantaneous system state s of the cylinder and its surrounding flow field as input. t Output the action 'a' at each time step. t By changing the flow rate at the nozzle of the blowing and suction device, active flow control can be achieved.

[0054] The beneficial effects of this invention are as follows:

[0055] 1. The active flow control device of the present invention overcomes the shortcomings of traditional methods in modeling high-dimensional fluid dynamics problems by directly establishing an end-to-end control strategy to achieve the goal of suppressing eddy-induced vibration.

[0056] 2. The data acquisition method of this invention provides expert demonstration knowledge for general deep reinforcement learning methods, which can overcome the shortcomings of large sample requirements and low data utilization efficiency in deep reinforcement learning, and accelerate the training and convergence speed of decision-making agents.

[0057] 3. The active flow control device trained by the deep reinforcement learning algorithm guides the suction and blowing device on the cylindrical surface to eject vortices in the same direction as the original periodic shedding vortices. The strong friction and collision between the vortices weaken the vortex energy and intensity, and the energy transfer between the fluid system and the solid system is attenuated, effectively suppressing the vortex-induced vibration phenomenon.

[0058] 4. The active flow control method of the present invention can be applied to the active flow control of various cylindrical objects, such as submarine vertical oil pipelines, cylindrical buildings and wings. Attached Figure Description

[0059] Figure 1 The flowchart illustrates the active flow control data acquisition method and active flow control reinforcement learning method based on expert demonstration of the present invention.

[0060] Figure 2 This is a schematic diagram of an active flow control model for vortex-induced vibration problems.

[0061] Figure 3 Real-time prediction graph during the construction of deep neural network agent models;

[0062] Figure 4 The final prediction graph for the deep neural network surrogate model;

[0063] Figure 5 A comparison chart showing direct deep reinforcement learning and expert demonstrations of deep reinforcement learning;

[0064] Figure 6 A comparison of the first mode of principal component analysis for actively controlled and uncontrolled flow fields; Detailed Implementation

[0065] The present invention will now be described in detail with reference to the accompanying drawings.

[0066] Figure 1 The right side is a flowchart of an active flow control data acquisition method for expert demonstration provided by this invention. The method includes the following steps:

[0067] Step S1: Establish a surrogate model representing the relationship between static control parameters and the amplitude of vortex-induced vibration of the cylinder. This surrogate model is built using deep neural network technology to predict the amplitude of vortex-induced vibration of the cylinder corresponding to the static control parameters. The static control parameters (g1, g2) are the control law of the closed-loop control, which are static parameters under the specified strategy conditions and do not change with the flow state, as shown below:

[0068] a t =g1·d t +g2·v t

[0069] The subscript t represents the time step number, a t Let d be the control action at time t. t Let v be the displacement of the center of mass of the vortex-induced vibrating cylinder at time t. t Let be the velocity of the center of mass of the vortex-induced vibrating cylinder at time t. Control action a. t A suction / blowing device applied to the surface of a cylinder alters the surrounding flow field. After a certain period of time, the intensity of eddies in the flow field near the cylinder either increases or decreases, affecting the coupling between the fluid and the solid, which is directly reflected in the change in the vibration amplitude of the cylinder's center of mass. Different static control parameters correspond to different vibration amplitudes in the final stable flow state. This mapping relationship is reflected by a surrogate model constructed using conventional deep neural network techniques.

[0070] Y A =f NN (g1,g2)

[0071] Among them, Y A f is the stable vibration amplitude of the cylinder's center of mass. NN This is a mapping established by a deep neural network. As one implementation, the artificial neural network structure of the surrogate model includes three fully connected hidden layers, each containing 16, 32, and 16 neurons respectively.

[0072] Step S2: The surrogate model takes static control parameters as input and outputs the vortex-induced vibration amplitude of the cylinder under active flow control. It is trained based on the stable vibration amplitude data of the cylinder's center of mass obtained from sampling in the static control parameter space and numerical simulation. Then, based on the prediction results of the trained surrogate model, an expert demonstration trajectory is generated to obtain active flow control data. Specifically, the process of establishing and training the surrogate model is a dynamic process. To minimize the cost of numerical simulation, this invention introduces a dynamic sampling method: a Gaussian mixture model is used to characterize the uncertainty of the surrogate model, a heuristic algorithm is used to solve the multi-objective optimization problem, and dynamic sampling is performed in the static control parameter space. In the static control parameter state space, the Gaussian mixture model uses multiple Gaussian probability distributions to accurately quantify the distribution of the static control parameters, as shown below:

[0073]

[0074]

[0075] Where k represents the probability density function of the k-th Gaussian probability distribution, K indicates that the model contains K Gaussian probability distributions, N represents the probability density function of a Gaussian probability distribution, x represents the sample point, i.e., the static control parameters (g1, g2), and Y... p τ represents the probability of a sample point under this Gaussian mixture model distribution, where τ is the parameter of the Gaussian mixture model, including the weights α of each Gaussian probability density function. k Mean μ k and variance The parameters of the Gaussian mixture model are initialized using the K-means algorithm and updated using the Iterative Expectation-maximization Algorithm, with a maximum number of iterations that can be set to 2000. The update formula is shown below:

[0076]

[0077]

[0078] q ik =p(z) k |x i ;τ n-1 )

[0079] Where n represents the number of iterations of the maximum expectation iteration algorithm, i represents the i-th actual sample point that has been collected, M represents the number of actual sample points collected in the dataset, and z k Let p represent the probability function, indicating that the model belongs to the k-th Gaussian probability distribution. B represents the lower bound of the log-likelihood function of parameter τ, which is also the objective function for optimization. The Gaussian mixture model can provide uncertainty information about the current surrogate model. Based on this uncertainty information, the control effect of static control parameters in unknown regions can be tested, thereby improving the prediction accuracy of the surrogate model. Simultaneously, to ensure the optimality of the expert demonstration knowledge, the sampling rules in the process of establishing and improving the surrogate model are defined as a multi-objective optimization problem:

[0080]

[0081] Among them, Y A Let Y be the stable vibration amplitude of the cylinder's center of mass corresponding to the sample point x, i.e., the static control parameters (g1, g2) (predicted by the surrogate model). p The value represents the probability of a sample point under this Gaussian mixture model distribution, where α is the weight of each objective in the multi-objective problem, and x represents the probability of the sample point under this distribution. nextThese are the static control parameters for the next evaluation. The optimization problem is solved using a particle swarm optimization heuristic, with 40 populations per generation, for a total of 50 iterations.

[0082] Dynamic sampling methods can reduce the cost of numerical simulation, but the calculation is relatively complex. As an implementation method, in the initial stage of training, the surrogate model randomly selects one or more sets (e.g., 10 sets) of sample points in the static control parameter space based on the Latin hypercube method as the basic data for training the surrogate model. Then, the surrogate model is continuously improved by sampling sample points multiple times (e.g., 10 times) using dynamic sampling methods as training data.

[0083] Step S3: The surrogate model can be trained based on the Adaptive Moment Estimation algorithm. The training objective is to minimize the difference between the predicted value and the numerically simulated stable vibration amplitude data of the cylinder's center of mass. If the surrogate model meets the accuracy requirements, the dynamic modeling process stops; otherwise, return to step S1.

[0084] Step S4: Generate expert demonstration trajectory (the relationship curve between the control action and the instantaneous system state of the cylinder and the surrounding flow field over time under a set of static control parameters) based on the surrogate model prediction results, that is, obtain active flow control data, and premix the control trajectory into the experience pool of the deep reinforcement learning algorithm.

[0085] As a preferred approach, the optimization problem based on the surrogate model is solved to find the optimal static control parameters. Computational fluid dynamics (CFD) is then used to simulate and model the flow control. The control trajectory is stored in the experience pool of deep reinforcement learning, providing expert demonstration knowledge for learning and further reducing the algorithm's requirement for sample data. The optimization objective based on the surrogate model is as follows:

[0086]

[0087] Where, x optimal The optimal static control parameters are predicted by the surrogate model. This optimization problem is also solved using a particle swarm optimization heuristic. For each expert demonstration knowledge trajectory, the instantaneous system state s of the cylinder and its surrounding flow field at time t is formatted according to the Markov chain model required by the deep reinforcement learning algorithm. t Control action a t The instantaneous system state s of the cylinder and its surrounding flow field at time t+1 t+1 and control action a t The corresponding reward and punishment function r t and the final state variable done t The instantaneous system state s of the cylinder and the surrounding flow field. t Includes cylindrical motion state information (i.e., d) t and vt The velocity and pressure information of the flow field are acquired by the monitoring probe. The reward / penalty function r... t Closely related to the control objectives, as follows:

[0088] r t =κd t +χv t

[0089] Where, d t v is the displacement of the center of mass of the cylinder. t Let be the velocity of the cylinder's center of mass, and κ and χ be the corresponding weights. The trajectory is formatted so that deep reinforcement learning algorithms can imitate and learn from it.

[0090] Based on the above-described method for acquiring active flow control data based on expert demonstration, this invention also provides an active flow control device based on expert demonstration. The active flow control device is used to execute actions to control a blowing and suction device arranged on the surface of the cylinder based on the instantaneous system state output of the flow field around the cylinder. The active flow control device is composed of an artificial neural network, such as... Figure 1 As shown, it is obtained through training using the following method:

[0091] A decision-making agent incorporating an active flow control device and a vortex-induced vibration numerical simulation environment are constructed. The vortex-induced vibration numerical simulation environment is used to simulate the active flow control process of the decision-making agent: by changing the nozzle flow rate of the surface blowing and suction device according to the control action, the changes in the flow field are deduced to obtain the instantaneous system state s of the cylinder and its surrounding flow field at time t+1. t+1 The control trajectory is stored in the experience pool of the deep reinforcement learning algorithm; the experience pool also contains active flow control data acquired by the aforementioned active flow control data acquisition method based on expert demonstration; the decision-making agent also includes an evaluator, Critic, wherein the active flow control device acts as the decision-maker to determine the instantaneous system state s of the cylinder and the surrounding flow field at time t. t Give control action a t The evaluator is used to output a rating of the effectiveness of the current action under the current system state based on the flow field information of the cylinder and its surroundings. The evaluator is composed of an artificial neural network.

[0092] As one implementation scheme, the artificial neural network structure of the active flow control device in this embodiment includes three fully connected hidden layers, each with 200 neurons. The activation function is the ReLU function. The input is the flow field state s, and the output is the average value μ(s) and standard deviation σ(s) of the action. The final action can be calculated as follows:

[0093] a=μ(s)+σ(s)⊙z

[0094] in,

[0095] The evaluator's artificial neural network structure consists of two fully connected hidden layers, each with 200 neurons. The activation function is ReLU. The inputs are the flow field state s and the action to be performed, and the output is the quality Q of the current state. θ (S t ,a t ).

[0096] The numerical simulation environment is built on the open-source software OpenFOAM platform, using the laminar model and the pimpleFoam incompressible transient solver; the dynamic mesh solver used is a six-degree-of-freedom motion model solver, and the maximum allowable deformation range of the mesh is 7.

[0097] The blowing and blowing device comprises two devices, symmetrically installed at the poles of the cylinder in the transverse flow direction. The sum of the blowing and blowing control flow rates of the two devices at the same time is zero. A monitoring probe is installed in the flow field, which can transmit the corresponding velocity field and pressure field information in real time.

[0098] The training process of deep reinforcement learning for the active flow control device begins in rounds, with each new round starting from an uncontrolled steady flow. A fluid-structure interaction example of a cylinder in unsteady flow at Reynolds number 100 is calculated using the finite volume method and dynamic mesh technique in the open-source computational fluid dynamics software OpenFOAM, with the periodic steady state of the uncontrolled flow serving as the starting state for each round.

[0099] Training begins by reading past experience data containing expert demonstration information from the experience pool and training the decision agent using the Soft Actor-Critic algorithm. The decision agent consists of an Actor and an Critic, both represented by deep neural networks. The Bellman backtracking equation is used as the objective function for training the Critic.

[0100]

[0101] Where π represents the current decision-maker's control strategy, ρ π Let θ represent the transition probability distribution corresponding to the current control strategy, and let θ represent the parameters of the evaluator's artificial neural network. It is an expectation, Q θ This represents the state-action evaluation value given by the current evaluator's artificial neural network. This represents a more precise assessment value:

[0102]

[0103] Where H is the entropy corresponding to policy π in the current observation state. Let be the weight of the action entropy in the evaluation function, and ω be the influence factor of future evaluation items on the current evaluation. ·|S t This represents all possible actions that can be taken in the current state. The objective function for training the artificial neural network of the decision-maker (Actor) is as follows:

[0104]

[0105] Where φ represents the parameters of the decision-maker's artificial neural network.

[0106] A computational fluid dynamics numerical simulation platform simulates the active flow control process of a decision-making agent, storing the control trajectory in the experience pool of a deep reinforcement learning algorithm. The decision-making agent discretizes the continuous control process into a Markov decision process, observing the instantaneous system state s of the cylinder and its surrounding flow field at time t. t The decision-maker, the Actor, bases its decisions on the instantaneous state s at time t. t Give control action a t The numerical simulation platform changes the nozzle flow rate of the surface blowing and suction device according to the control action, and infers the change of the flow field to obtain the instantaneous system state s of the cylinder and the surrounding flow field at time t+1. t+1 And the reward / penalty function r corresponding to the control action. t and the final state variable done t .

[0107] If the flow state reaches the preset round end state, the current round ends, and the final state variable `done` is executed. t Set to 1; otherwise, return to continue training and deducing the current flow control process, terminating the state variable "done". t Set to 0.

[0108] If the control strategy represented by the decision agent has converged, the deep reinforcement learning training process ends; otherwise, return to start a new training round.

[0109] Figure 2 This is a schematic diagram of an active flow control model for vortex-induced vibration. In a flow field with a downstream length of 29D and a transverse width of 16D, an elastically mounted cylinder is placed. The cylinder's center is 8D from both the upper and lower symmetric boundaries, 8D from the front inlet boundary, and 21D from the rear outlet boundary. The inflow rate is 1D / s, the fluid density ρ is taken as 1 kg / m³, and the kinematic viscosity coefficient μ is taken as 0.001 kg / (m*s). Under the influence of the inflow, the cylinder exhibits vortex-induced vibration, and the motion of its center of mass is controlled by a second-order equation of motion:

[0110]

[0111] in, It is the acceleration of the cylinder's center of mass in the transverse direction. ξ is the velocity of the cylinder's center of mass in the transverse direction, y is the displacement of the cylinder's center of mass in the transverse direction, m is the mass of the cylinder, ξ is the damping coefficient of the damper in the cylindrical structure model, k is the elastic coefficient of the spring in the cylindrical structure model, and F fluids This refers to the flow field forces acting on the cylinder. The numerical simulation of the entire flow field relies on the open-source software OpenFOAM platform. The simple steady-state algorithm is used to solve the problem within each time step, and the Piso algorithm is used with a time step of 0.005 seconds. A six-degree-of-freedom model and dynamic mesh technology are used to simulate the eddy-induced vibration process.

[0112] Figure 3 This is a real-time prediction graph used during the construction of a deep neural network surrogate model. Initially, a basic surrogate model is built based on sample points sampled using the Latin hypercube method (10 times). Each subgraph (ak) contains two parts: a surface and contour lines. The surface represents the prediction results of the surrogate model, and the contour lines represent sample points with the same probability density function represented by the Gaussian mixture model. During 10 dynamic updates, the surrogate model's prediction results for static control parameters gradually stabilize, resulting in... Figure 4 .

[0113] Figure 4 The final prediction contour map for the deep neural network surrogate model is shown, with the horizontal and vertical axes representing two static control parameters. Different regions indicate the amplitude range of vortex-induced stable vibration under those static control parameters. The lower the vibration amplitude, the better the control effect. Near the upper triangular region, 10 sets of static parameters (represented by black circles on white backgrounds) are selected as source data for expert knowledge demonstration.

[0114] Figure 5 This is a comparison chart of direct deep reinforcement learning (excluding expert demonstration information) and the expert demonstration deep reinforcement learning method of this invention. Sub-figure (a) shows the change in total reward in each round during training. To reduce random error, both direct deep reinforcement learning and expert demonstration deep reinforcement learning were repeated five times. The experimental results show that expert demonstration deep reinforcement learning converges faster and reduces the amount of data required. Sub-figures (b) and (c) are comparison charts of the control policies obtained by the two methods, respectively. It can be seen that they are basically overlapping, indicating that the expert demonstration method does not change the convergence solution of the original deep reinforcement learning.

[0115] Figure 6Comparison of the first mode of principal component analysis in actively controlled and uncontrolled flow fields. Subfigure (a) shows the pseudoenergy distribution of the first mode in the uncontrolled flow, and subfigure (b) shows the pseudoenergy distribution of the first mode in the actively controlled flow. In both figures, the modal pseudoenergy is symmetrical along the flow centerline and contains the most significant flow energy. The distribution of the first mode pseudoenergy in the actively controlled flow is significantly narrower in the transverse direction than in the uncontrolled flow, which is consistent with the controlled vortex-induced vibration phenomenon and the suppression of vortex energy.

[0116] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for acquiring active flow control data based on expert demonstrations, characterized in that, include: A surrogate model is established to characterize the relationship between static control parameters and the amplitude of vortex-induced vibration of a cylinder; the surrogate model is established using deep neural network technology. ； in, The amplitude of stable vibration at the center of mass of the cylinder. The mapping is established by a deep neural network; static control parameters. The control law for closed-loop control is a static parameter under the specified strategy conditions, which does not change with the flow state. The surrogate model is trained based on the stable vibration amplitude data of the cylinder's center of mass obtained by spatial sampling and numerical simulation of static control parameters. Then, the expert demonstration trajectory is generated based on the prediction results of the trained surrogate model to obtain active flow control data. The methods for static control parameter space sampling include the following dynamic sampling methods: Based on the Gaussian mixture model to characterize the uncertainty of the surrogate model, a heuristic algorithm is used to solve the multi-objective optimization problem by dynamically sampling in the static control parameter space; wherein the Gaussian mixture model in the static control parameter space uses multiple Gaussian probability distributions to accurately quantify the distribution of static control parameters, as shown below: ； ； in, Indicates the first The probability density function of a Gaussian probability distribution. This indicates that the model contains a total of A Gaussian probability distribution, Let x represent the probability density function of a Gaussian probability distribution, and let x represent the sample point, i.e., the static control parameter. , This represents the probability of a sample point under this Gaussian mixture model distribution. The parameters of the Gaussian mixture model include the weights of each Gaussian probability density function. mean and variance The parameter update formula for the Gaussian mixture model is shown below: ； ； ； in, Indicates the number of iterations. Indicates the first One actual sample point has been collected. This indicates the actual number of sample points collected in the dataset. Indicates belonging to the first A Gaussian probability distribution; Let be a probability function. Indicates parameters The lower bound function of the log-likelihood function is also the objective function for optimization; The multi-objective optimization problem is: ； in, Sample points predicted based on the surrogate model The corresponding stable vibration amplitude, This represents the probability of a sample point under this Gaussian mixture model distribution. As weight, These are the static control parameters for the next evaluation.

2. The method according to claim 1, characterized in that, The process of generating expert demonstration trajectories based on the prediction results of the trained surrogate model to obtain active flow control data specifically involves: solving an optimization problem based on the surrogate model, finding the optimal static control parameters, and performing flow control simulation using computational fluid dynamics techniques. The control trajectory is then stored in the experience pool of deep reinforcement learning. The optimization problem based on the surrogate model is as follows: ； in, The optimal static control parameters are predicted by the surrogate model; for each expert demonstration knowledge trajectory, the instantaneous system state of the cylinder and its surrounding flow field at time t is formatted according to the Markov overchain model required by the deep reinforcement learning algorithm. Controlling actions , Instantaneous system state of the cylinder and the surrounding flow field at any given moment and control actions Corresponding reward and punishment functions and final state variables Instantaneous system state of the cylinder and its surrounding flow field Includes information on the cylinder's motion state, velocity and pressure in the flow field; reward / penalty function. It is expressed as follows: ； in, This represents the displacement of the cylinder's center of mass. Let the velocity be the center of mass of the cylinder. and For the corresponding weights.

3. The method according to claim 2, characterized in that, Static control parameters The relationship with the control action is as follows: ； Subscript Indicates the time step number. for Controlling actions at all times for The displacement of the center of mass of the cylinder under vortex-induced vibration at time t. for The velocity of the center of mass of the vortex-induced vibration cylinder at any given moment; control action. The action of the blowing and suction device applied to the surface of the cylinder is used to change the flow field in the vicinity.

4. The method as described in claim 1, characterized in that, The artificial neural network structure of the surrogate model contains three fully connected hidden layers, each containing 16, 32, and 16 neurons respectively. The surrogate model takes static control parameters as input and outputs the amplitude of cylindrical vortex-induced vibration under active flow control.

5. The method as described in claim 1, characterized in that, The surrogate model is trained based on the Adaptive Moment Estimation algorithm. In the initial training stage, 10 sets of sample points are randomly selected in the static control parameter space based on the Latin hypercube method as the basic data for training the surrogate model. Then, sample points are sampled 10 times using the dynamic sampling method as the training data for the surrogate model to continuously improve the surrogate model.

6. The method as described in claim 1, characterized in that, The parameters of the Gaussian mixture model are initialized using the K-means algorithm and iteratively learned using the expectation-maximization algorithm, with a maximum number of iterations of 2000.

7. The method as described in claim 1, characterized in that, The multi-objective optimization problem is solved using a particle swarm optimization heuristic algorithm. Dynamic sampling is performed in the static control parameter space, with 40 populations set for each generation of optimization, for a total of 50 generations.

8. An active flow control device based on expert demonstration, characterized in that, The active flow control device is used to execute actions to control the blowing and suction devices arranged on the surface of the cylinder based on the instantaneous system state output of the flow field in and around the cylinder; the active flow control device is composed of an artificial neural network, which is trained and obtained through the following method: A decision-making agent incorporating an active flow control device and a vortex-induced vibration numerical simulation environment are constructed. The vortex-induced vibration numerical simulation environment is used to simulate the active flow control process of the decision-making agent: by changing the nozzle flow rate of the surface blowing and suction device according to the control action, the changes in the flow field are deduced. The instantaneous system state of the cylinder and its surrounding flow field at time t. The control trajectory is stored in the experience pool of the deep reinforcement learning algorithm; the experience pool also includes the active flow control data acquired by the active flow control data acquisition method based on expert demonstration as described in claim 1; the decision-making agent also includes an evaluator, Critic, wherein the active flow control device acts as the decision-maker to determine the instantaneous system state of the cylinder and its surrounding flow field at time t. Give control actions The evaluator is used to assess the effectiveness of the current action based on the instantaneous system state of the cylinder and the surrounding flow field. The evaluator is composed of an artificial neural network. The training process of the active flow control device is started in rounds, with each new round starting from an uncontrolled steady flow. Based on the finite volume method and dynamic mesh technology in the open-source computational fluid dynamics software OpenFOAM, a fluid-structure interaction example of a cylinder in unsteady flow at Reynolds number 100 is calculated, with the periodic steady state of the uncontrolled flow as the starting state of each round. Training begins by reading past data containing expert demonstration information from the experience pool and training the decision agent using a soft decision-maker-evaluator algorithm; the Bellman backtracking equation is used as the objective function for training the evaluator. ； in, This indicates the current decision-maker's control strategy. This represents the transition probability distribution corresponding to the current control strategy. The parameters of the artificial neural network representing the evaluator, It is an expectation. This represents the state-action evaluation value given by the current evaluator's artificial neural network. This represents a more precise assessment value: ； in, The strategy under the current observation state The corresponding entropy, The weight of the action entropy in the evaluation function. This refers to the impact factors of future evaluation items on the current evaluation; This represents all possible actions that can be taken in the current state; This is the reward / punishment function; the objective function for training the artificial neural network of the decision-maker (Actor) is as follows: ； in, The parameters of the artificial neural network representing the decision-maker; If the flow state reaches the end of the round, then the current round ends; otherwise, continue training and deducing the control process of the current flow. If the control strategy represented by the decision agent has converged, the training process ends; otherwise, a new training round begins.

9. The active flow control device as described in claim 8, characterized in that, The artificial neural network structure of the active flow control device includes three fully connected hidden layers, each with 200 neural units, and the activation function is the ReLU function. The input is the flow field state. The average value of the output action and standard deviation The final action can be calculated as ； in, ; The evaluator's artificial neural network structure consists of two fully connected hidden layers, each with 200 neurons, using the ReLU activation function, and the input flow field state. And execute actions, outputting the current state's condition (good or bad). .

10. A control method based on the expert demonstration-based active flow control device as described in claim 8, characterized in that, Specifically: Monitoring probes are arranged around the cylinder to collect the instantaneous system state of the cylinder and the surrounding flow field in real time. The data is then transmitted to a trained active flow control device, which uses the instantaneous system state of the cylinder and its surrounding flow field as input. Output the actions performed at each time step. By changing the flow rate at the nozzle of the blowing and suction device, active flow control can be achieved.

Images