An active control method for suppressing lift fluctuations in a sphere based on reinforcement learning

Abstract

This invention relates to an active control method for suppressing lift fluctuations in a sphere based on reinforcement learning. A three-dimensional flow environment around the sphere is constructed, and a formula for solving the lift fluctuations of the sphere is provided. The feedback signal of the flow environment at the current moment is the lift fluctuation F′. l The control effect (reward value) from the previous moment is input to the controller in the closed-loop active flow control process. The controller outputs the action signal (rotation parameter) of the controlled flow environment. After receiving the action signal (rotation parameter), the flow environment executes the rotation parameter command and then transmits the control effect of the signal to the agent. This process is repeated multiple times to find the rotation control rate with the optimal control effect. In this invention, the agent module utilizes reinforcement learning to output multiple control signals to actuators at different positions. By appropriately setting the action network and evaluation network, the complex goal of suppressing lift fluctuations around a sphere can be achieved. This invention achieves a lift fluctuation suppression effect of approximately 70%.

Description

Technical Field

[0001] This invention belongs to the fields of fluid mechanics, flow control, and machine learning, and relates to an active control method for suppressing lift fluctuations of a sphere based on reinforcement learning, which has wide application value. Background Technology

[0002] Flow around a sphere is a typical three-dimensional flow, and its flow state can vary greatly depending on the Reynolds number. During operation, aircraft and spacecraft often experience lift fluctuations that disrupt their normal function. Active flow control can modulate the flow field structure by artificially applying excitations of a certain magnitude, thereby achieving control objectives. As a typical structure, the study of sphere control will drive research into the control of complex structures such as aircraft and spacecraft. Furthermore, reinforcement learning can be used to achieve even more complex control objectives.

[0003] A prior art search revealed Chinese invention patent number CN201120533048.0, entitled "Lift Suppression Device for Flow Around a Cylindrical Circulation." This invention employs a passive control method, adding a right prism with an equilateral triangular base between the incoming flow and the cylinder. This effectively suppresses the lift force experienced by the cylinder in the fluid under low Reynolds number conditions. Another prior art search revealed Chinese invention patent number CN202110572871.0, entitled "Active Flow Controller and Control Method for Suppressing Eddy-Induced Vibration Based on Deep Reinforcement Learning." This invention uses deep reinforcement learning to establish a decision-making agent. By controlling a symmetrical blowing and suction device in the transverse flow direction of the cylinder, it can achieve the goals of vibration suppression and drag reduction for the cylinder.

[0004] Reinforcement learning is a framework for learning, prediction, and decision-making. It can greatly help automate and optimize manually designed strategies, find the optimal control strategy for active control systems, and is widely used in the field of control. Summary of the Invention

[0005] Technical problems to be solved

[0006] To overcome the shortcomings of existing technologies, this invention proposes an active control method based on reinforcement learning to suppress lift disturbances in spherical flow, aiming to solve the lift disturbance problem in spherical flow. This method, utilizing numerical simulation results and reinforcement learning, achieves closed-loop active flow control, thereby suppressing lift disturbances in spherical flow.

[0007] Technical solution

[0008] An active control method for suppressing lift fluctuations in a sphere based on reinforcement learning, characterized by the following steps:

[0009] Step 1: Construct the flow environment around a three-dimensional sphere to obtain the flow state around the sphere;

[0010] The control objective is to suppress lift fluctuations in the sphere, and high-fidelity three-dimensional flow data around the sphere is the foundation of the control.

[0011] The flow environment was constructed using the Direct Numerical Simulation (DNS) method, with the core solver being the Lattice Boltzmann Method (LBM) for solving the Navier-Stokes equations. The governing equations are as follows:

[0012]

[0013]

[0014] Where u is the velocity vector, p is the flow pressure, and v is the kinematic viscosity of the fluid. It is a vector differential operator;

[0015] The submerged boundary method is used to solve for the surface boundary and structural motion. IB This refers to the forces at the boundary in the submerged boundary method;

[0016] The formula for solving the lift fluctuation of a sphere is as follows.

[0017]

[0018] Where: n is the unit normal vector, Γ is the sphere, and μ is the dynamic viscosity of the fluid. It is vorticity; lift F l Decomposed into average lift F l and lift fluctuation F′ l ;

[0019] Step 2: Build a reinforcement learning framework for implementing active flow control:

[0020] In the active flow control framework based on reinforcement learning, the feedback signal of the surrounding environment at the current moment is the lift fluctuation F′. l The control effect (reward value) from the previous moment is input into the deep reinforcement learning controller. The controller outputs the action signal (rotation parameter) used to control the flow field around the flow environment. After receiving the action signal, the flow environment executes the rotation parameter instruction and then transmits the control effect of the signal to the agent. Through multiple iterations, the rotation control law with the optimal control effect is found.

[0021] The controller employs a deep reinforcement learning agent with two sets of neural networks: an actor and an evaluator. The two networks construct a fitting relationship between the environmental state, the action, and the reward. The actor and the evaluator simultaneously receive the state and reward signals. The evaluator evaluates the control effect, i.e., the reward value, and the actor outputs the action signal, i.e., the rotation parameter.

[0022] Step 3: Setting agent parameters in the reinforcement learning framework:

[0023] The inputs to the actuator and the evaluator are the same: the lift fluctuation F′ of the flow field at the current moment. l And the reward function for the next time step; in each iteration, a DRL runs the policy N times and collects a trajectory, which is a series of state-action-reward combinations;

[0024] Use discount reward R t For behavioral control, for the lift fluctuation F′ under the behavior l Perform the calculation:

[0025] R t =∑ t′>t γ t′-t r t′

[0026] Where r t′ It is the ratio of the probability before and after the control policy update, and γ is the discount factor; this definition gives higher weight to later rewards when calculating rewards;

[0027] Objective function J CRITIC To minimize the difference between the predicted discount reward and the actual discount reward:

[0028]

[0029]

[0030] in It is the reward for time t, V ⊙ (s t () represents the predicted discount reward;

[0031] The output of the action network is the action distribution, calculated using the following formula:

[0032]

[0033] in It represents the expectation, and ε is a hyperparameter;

[0034] Select the Adam optimizer when updating the strategy;

[0035] Step 4: Use reinforcement learning to obtain the optimal control effect.

[0036] Using the flow field environment from step 1 to provide real-time environmental feedback, and employing the reinforcement learning framework from step 2 and the agent parameter settings from step 3, closed-loop active flow control is performed on the lift fluctuations of the flow field; when the reward R... tAt its maximum, the agent considers it to be the optimal control law, and the training will gradually converge to obtain the optimal control effect, ultimately achieving the suppression of lift fluctuations in the flow around the three-dimensional sphere.

[0037] Both the actuator and the evaluator use a multilayer perceptron network (MLP), with two layers in each network. Each layer of the actuator has 50 neurons, and each layer of the evaluator has 80 neurons.

[0038] The hyperparameter ε in step 3 is typically 0.1 or 0.2.

[0039] The deduction coefficient γ in step 3 is usually 1.

[0040] Beneficial effects

[0041] This invention proposes an active control method for suppressing lift fluctuations in a sphere based on reinforcement learning. It constructs a three-dimensional flow environment around a sphere and provides a formula for solving the lift fluctuations of the sphere. The feedback signal of the flow environment at the current moment is the lift fluctuation F′. l The control effect, i.e., the reward value, from the previous moment is input to the controller in the closed-loop active flow control process. The controller outputs the action signal, i.e., the rotation parameter, of the controlled flow field around the environment. After receiving the action signal, i.e., the rotation parameter, the flow field around the environment executes the rotation parameter command and then transmits the control effect of the signal to the agent. Through multiple iterations, the rotation control rate with the optimal control effect is found.

[0042] Beneficial effects:

[0043] 1. Compared with traditional control methods, this invention can quickly find the optimal control law through machine learning, overcoming the problem that it is extremely difficult to obtain the control law for fluids due to their high-dimensional, non-convex, and nonlinear properties.

[0044] 2. The intelligent agent module of this invention utilizes reinforcement learning to receive feedback information from multiple flow field measurement points. Through online calculation, it outputs multiple control signals to actuators at different positions. By appropriately configuring the action network and evaluation network, the complex goal of suppressing lift fluctuations around a sphere can be achieved. This invention achieves a lift fluctuation suppression effect of approximately 70%. Attached Figure Description

[0045] Figure 1 The numerical calculation results of the flow field with a Reynolds number of 3700, implemented using the lattice Boltzmann method in the invention.

[0046] Figure 2 Closed-loop control diagram in this invention

[0047] Figure 3 This invention presents a reinforcement learning-based closed-loop flow control framework.

[0048] Figure 4 For the control effect in this invention Detailed Implementation

[0049] The present invention will now be further described in conjunction with the embodiments and accompanying drawings:

[0050] This invention includes the following steps:

[0051] S1: Constructing a flow environment around a three-dimensional sphere:

[0052] The control objective of this method is to suppress lift fluctuations around the sphere, and high-fidelity three-dimensional flow data around the sphere is the foundation of the control. The flow environment is constructed using the Direct Numerical Simulation (DNS) method, with the core solver being the Lattice Boltzmann Method (LBM) to solve the Navier-Stokes equations. The governing equations are as follows:

[0053]

[0054]

[0055] Where u is the velocity vector, p is the flow pressure, and v is the kinematic viscosity of the fluid. It is a vector differential operator. This method uses the submerged boundary method to solve for the surface boundary and structural motion, F. IB This refers to the forces at the boundary in the submerged boundary method.

[0056] The formula for solving the lift fluctuation of a sphere is as follows.

[0057] F l =∫ Γ (-pn+μω×n)·n y dΓ

[0058] Where n is the unit normal vector, Γ is the sphere, and μ is the dynamic viscosity of the fluid. It's vorticity. Lift F l It can be broken down into average lift F l and lift fluctuation F′ l .

[0059] Reinforcement learning is a method of learning through interaction and feedback with the environment. During the learning process, it requires immediate feedback of control effects to the control agent to update and iterate the control law. Therefore, to efficiently achieve high-fidelity simulation of flow control processes, we use parallel methods with graphics processing units (GPUs) to accelerate computation. Figure 1 The results show the flow field around an uncontrolled three-dimensional sphere.

[0060] S2: Building a reinforcement learning framework for implementing active flow control:

[0061] Closed-loop active flow control has three elements: actuator, sensor, and controller, such as Figure 2 As shown. The exciter acts directly on the flow field, modulating the flow structure, thereby further affecting the structural stress; the sensor is used to observe the flow, structural stress, wall pressure, etc., and provides feedback signals; the controller, based on the sensor feedback signals, assigns an appropriate excitation intensity to the exciter according to the control target.

[0062] In this method, based on Figure 2 The framework of closed-loop active flow control is constructed using a proximal policy optimization method based on reinforcement learning. Figure 3 The reinforcement learning framework is shown. The feedback signal from the current flow environment (lift fluctuation F′) is also shown. l The control effect (reward value) from the previous time step and the current state are fed into the deep reinforcement learning agent. The agent, equivalent to a controller in a closed-loop active flow control process, has two neural networks: an actuator and an evaluator. These two networks construct an approximate relationship between the environmental state, the action, and the reward. Both the actuator and the evaluator use multilayer perceptron networks (MLPs), each with two layers. Each layer of the actuator has 50 neurons, and each layer of the evaluator has 80 neurons. The actuator and the evaluator simultaneously receive the state and reward signals. The evaluator evaluates the control effect (reward value), and the actuator outputs the action signal (rotation parameters). After receiving the action signal (rotation parameters), the flow field executes the corresponding instructions and then feeds the control effect of the signal back into the agent. Through multiple iterations, the agent finds the rotation control law with the optimal control effect.

[0063] S3: Agent parameter settings in reinforcement learning framework:

[0064] The inputs to the actuator and the evaluator are the same: the lift fluctuation F′ of the flow field at the current moment. l And the reward function for the next time step. In each iteration, a DRL runs the policy N times and collects a trajectory, which is a series of state-action-reward combinations. To consider the long-term effects of these actions, we use the control effect of discounted rewards on these actions, i.e., the lift fluctuation F′ under these actions. l Perform the calculation:

[0065] R t =∑ t′>t γ t′-t r t′

[0066] Where: r t′ It is the ratio of the probabilities before and after the control policy update, and γ is the discount factor, usually 1. This definition gives higher weight to later rewards when calculating rewards.

[0067] To update the strategy, it is important to correctly define the objective function for each group. The evaluator's output is a long-run discounted prediction of the effect, and the goal is to use the objective function J. CRITIC To minimize the difference between the predicted discount reward and the actual discount reward:

[0068]

[0069]

[0070] in It is the reward for time t, V ⊙ (s t () represents the predicted discount reward;

[0071] However, as learning progresses, the agent always aims to obtain a larger reward than predicted, i.e., to gain a greater advantage. To achieve this, the policy-based DRL algorithm employs an actioner network, whose output is the action distribution, calculated as follows:

[0072]

[0073] in: It represents the expectation, and ε is a hyperparameter;

[0074] When updating the policy, we choose the Adam optimizer, which outperforms traditional stochastic gradient descent optimizers in terms of fast convergence. When dealing with continuous control, the actor network does not directly generate actions; instead, it generates a combination of parameters for a probability distribution of the actions. In this invention, we use a Gaussian distribution from which actions are sampled within a predefined range.

[0075] S4: Using reinforcement learning to achieve optimal control performance:

[0076] Using the flow field environment from step 1 to provide real-time environmental feedback, and employing the reinforcement learning framework from step 2 and the agent parameter settings from step 3, closed-loop active flow control is performed to address lift fluctuations in the flow field. When the reward R... t At its maximum, the agent believes it has found the optimal control law, and training gradually converges. Finally, we observe the optimal control effect found by machine learning, and the results are as follows... Figure 4 As shown, by controlling rotation, the lift fluctuation of the flow around the three-dimensional sphere is ultimately suppressed.

[0077] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the substantive content of the present invention. In the absence of conflict, any modifications or improvements to the inventive concept fall within the protection scope of the present invention.

Claims

1. A method for active control of suppression of lift wave fluctuation of a ball based on reinforcement learning, characterized by The steps are as follows: Step 1: Build a flow environment of flow around a three-dimensional sphere to obtain the flow state of the sphere flow; The control target is to suppress the fluctuation of the sphere lift, and the high-fidelity three-dimensional sphere flow data is the basis for control; The flow environment is built using the direct numerical simulation DNS method of numerical calculation, and the core solver is the lattice Boltzmann method LBM to solve the Navier-Stokes equation, and the control equation is as follows: wherein is the velocity vector, is the flow field pressure, is the kinematic viscosity of the fluid, is the vector differential operator; The immersed boundary method is used to solve the curved boundary and the structure motion, That is the force on the boundary in the immersed boundary method. The formula for solving the fluctuation of the sphere lift is as follows, where is the unit normal vector, is the sphere, is the kinematic viscosity of the fluid, is the vorticity; lift is split into mean lift and lift fluctuations ; Step 2, build a reinforcement learning framework for active flow control: In the active flow control framework based on reinforcement learning, the feedback signal of the current moment of the flow environment, i.e. the lift fluctuation , the control effect of the previous moment, i.e. the reward value, is input into the deep reinforcement learning controller, the controller outputs the action signal, i.e. the rotation parameter, for controlling the flow field of the flow environment; after receiving the action signal, the flow field of the flow environment executes the rotation parameter instruction, and then the control effect of the signal is transmitted into the agent, and multiple iterations are performed to find the rotation control law with the optimal control effect. The controller uses a deep reinforcement learning agent, which has two sets of neural networks, an actor and an evaluator; The two networks build a fitting relationship between the environment state, action and reward, the actor and the evaluator receive the state and reward signals at the same time, the evaluator evaluates the control effect, that is, the reward value, and the actor outputs the action signal, that is, the rotation parameter; Step 3, parameter setting of the agent in the reinforcement learning framework: The inputs to the actuator and the evaluator are the same: the lift fluctuation of the flow field at the current moment. And the reward function for the next time step; in each iteration, a DRL runs the policy N times and collects a trajectory, which is a series of state-action-reward combinations; Using discount rewards Controlling behavior, lift fluctuations under behavior Performing calculations: where: is a discount factor; this definition gives a higher weight to rewards that come later in the computation of the reward. Objective function to minimize the difference between the predicted discounted reward and the actual discounted reward: ； wherein is the reward at time t, is the predicted discounted reward; The output of the actor network is the action distribution, and the calculation formula is as follows: wherein is desired, is a hyperparameter; Adam optimizer is selected when updating the strategy; Step 4 uses reinforcement learning to obtain the optimal control effect: The flow field environment in step 1 provides real-time environment feedback, the reinforcement learning framework in step 2 and the agent parameter setting in step 3 perform closed-loop active flow control on the lift fluctuation of the flow field; when the reward When the reward is maximum, the agent considers the control law to be optimal, the training gradually converges to obtain the optimal control effect, and finally realizes the lift fluctuation suppression of the three-dimensional sphere flow.

2. The method of claim 1, wherein the method is characterized by: The actor and the evaluator of step 3 use a multi-layer perceptron network MLP, the number of network layers is 2, the actor has 50 neurons in each layer, and the evaluator has 80 neurons in each layer.

3. The method of claim 1, wherein: The hyperparameters of step 3 are 0.1 or 0.

2.

4. The method of claim 1, wherein: The discount factor of step 3 is 1.

Images