Intelligent plasma turbulence friction resistance reduction method based on deep reinforcement learning

Abstract

The intelligent plasma turbulence friction resistance reduction control system based on deep reinforcement learning comprises environment building, an upper computer and a controller, and the environment building comprises a high-voltage power supply, a plasma exciter, a state sensor, an evaluation sensor and a signal acquisition board card. The invention further provides an intelligent plasma turbulence friction resistance reduction method based on deep reinforcement learning. According to the invention, a hot line and a hot film are selected as sensors, array dielectric barrier discharge plasma with quick response is used as an exciter, and a control strategy of an intelligent agent is learned and optimized through continuous high-speed interaction between a deep reinforcement learning algorithm deployed in a computer and an FPGA controller and a turbulent boundary layer. And finally, real-time closed-loop control of turbulence friction resistance reduction is realized. Compared with traditional open-loop control, the method has the advantages of being better in adaptability under complex incoming flow, higher in drag reduction energy efficiency and the like.

Description

Technical Field

[0001] The present invention relates to the field of active flow control, and in particular to an intelligent plasma turbulence friction drag reduction method based on deep reinforcement learning. Background Art

[0002] Turbulence is ubiquitous in engineering applications. Typical scenarios include oil transportation in pipelines, air flow on the surface of aircraft, and ship navigation in water. In order to improve transportation efficiency, it is often necessary to regulate turbulent flow to reduce wall friction resistance. Typical turbulent flow control methods can be divided into two categories: passive control and active control. The groove surface is a typical passive control method. Although the structure is simple, its drag reduction effect is weak under non-design conditions and may even cause increased drag. Relatively speaking, the active control method can adjust the excitation intensity according to the changes in external flow conditions, so it has a broader application prospect. Depending on whether there is a sensor feedback signal in the control system, active flow control technology can be divided into open-loop control and closed-loop control. The excitation parameters of open-loop control are pre-set, while closed-loop control can adaptively adjust the output of the actuator through the instantaneous flow field signal collected by the sensor in real time, which is expected to maintain a high drag reduction efficiency. However, the chaotic characteristics of turbulence make it exhibit multi-scale, high-dimensional and highly nonlinear behavior even on simple geometric shapes such as smooth walls. This uncertainty in flow state and the multiple degrees of freedom of control input pose a huge challenge to how to achieve optimal closed-loop control in a turbulent environment. Existing control methods usually rely on reduced-order models formed by researchers simplifying physical problems, which are methods based on physical models. However, for complex flows, it is almost impossible to establish an accurate theoretical model to describe turbulent behavior. Model-free control represented by deep reinforcement learning (DRL) does not rely on any underlying model description from input to output. It can achieve environmental cognition and optimization of control strategies through continuous trial and error, showing the potential to undertake complex control tasks. When DRL is combined with turbulent friction reduction, it is expected to improve benefits and adaptability and promote innovation and development in this field. Summary of the invention

[0003] The present invention proposes an intelligent plasma turbulence friction reduction control system based on deep reinforcement learning, comprising an environmental component, a host computer and a controller, wherein:

[0004] Environmental components include high voltage power supply, plasma actuator, status sensor, evaluation sensor, signal acquisition board;

[0005] A high voltage power supply for driving the plasma exciter to discharge;

[0006] A plasma actuator installed at a position to be controlled of the object to be controlled so as to effectively disturb the turbulent boundary layer;

[0007] A state sensor, which is located upstream of the plasma actuator and is used to sense the state of the flow field and output it to the host computer and controller through a signal acquisition board;

[0008] An evaluation sensor is located at the plasma exciter or downstream of the plasma exciter to evaluate the change of turbulent friction resistance of the flow field after plasma excitation and output the result to the host computer through a signal acquisition board;

[0009] A signal acquisition board collects the real-time output flow field data of the status sensor and the evaluation sensor, and converts the voltage signal into a digital signal and outputs it to the host computer;

[0010] The host computer receives the digital signal output by the signal acquisition board, combines, processes and analyzes the collected data, and performs network learning training through the DDPG algorithm deployed in the host computer;

[0011] The controller receives the flow field status information output by the status sensor, outputs action instructions to the high-voltage power supply through the internally deployed Actor network, and transmits the current action data to the host computer.

[0012] In a specific embodiment of the present invention, the state sensor is a hot wire sensor, and the evaluation sensor is a hot film sensor.

[0013] In one embodiment of the present invention, the hot wire anemometer converts the velocity and shear stress signals collected by the hot wire sensor and the thermal film sensor into voltage signals respectively, and then converts them into digital signals through a signal acquisition card and transmits them to the host computer; when using a hot wire or thermal film sensor, a hot wire anemometer must be used in conjunction.

[0014] In another embodiment of the present invention, the control system includes: a wind tunnel plate 1, a double-wire wall hot wire 2, a plasma actuator 3, a hot film sensor 4, a high-voltage power supply 5, an IGBT high-voltage switch 6, a control computer, an FPGA controller 7 and a CTA wind speed hot wire instrument 8;

[0015] The wind tunnel flat plate is set in the middle of the wind tunnel to generate a flat turbulent boundary layer, and the incoming flow direction is from left to right; from upstream to downstream, double-wire wall hot wire 2, plasma actuator 3 and hot film sensor 4 are arranged in sequence; high-voltage power supply 5, IGBT high-voltage switch 6, control computer and FPGA controller 7 and CTA wind speed hot wire instrument 8 are arranged outside the wind tunnel, and are connected to various devices in the wind tunnel through wires, BNC lines and RS232 serial lines;

[0016] The double-wire wall hot line 2 is distributed along the spanwise array to capture the distribution of high and low speed strips in the turbulent boundary layer along the spanwise direction. The larger the experimental wind speed, the smaller the sensor spacing should be to match the scale of the strip structure. The double-wire wall hot line 2 is used to measure the time series u(t) and v(t) of the streamwise (u) and normal (v) components of the incoming flow velocity, and return the flow field state s t =(u(t),v(t)), “(u(t),v(t))” represents a combination of u(t) and v(t);

[0017] The positive and negative electrodes of the high-voltage power supply 5 are respectively connected to the exposed electrode and the ground electrode of the plasma actuator 3 through high-voltage wires, and an IGBT high-voltage switch 6 for controlling the high-frequency switching of the circuit is also connected in series between the plasma actuator 3 and the positive electrode of the high-voltage power supply 5;

[0018] The hot film sensor 4 is attached to the downstream of the plasma actuator 3 and arranged in a spanwise array. The spanwise measurement point position is consistent with the upstream double-wire wall thermal line 2 to evaluate the change of the surface shear stress τ after convection, corresponding to the benefit r t =1-τ actuation / τ baseline , where τ baseline and τ actuation Corresponding to the shear stress under the base and excitation conditions, respectively;

[0019] The double-wire wall heat line 2 and the thermal film sensor 4 collect flow field data and transmit the signal to the control computer and FPGA controller 7.

[0020] In yet another embodiment of the present invention,

[0021] The double-wire wall surface hot line 2 is distributed along the span direction array, and the array spacing range is 5-20mm; the normal position measured by the double-wire wall surface hot line 2 needs to be controlled in the viscous wall area y + <Within 50, y + Represents the dimensionless normal wall height, the height range of the double-wire wall hot line 2 is 10 <y + <50;

[0022] The plasma actuator 3 is a comb-shaped array type, with an exposed electrode on the left and a ground electrode on the right;

[0023] The high voltage power supply 5 is a high voltage sinusoidal AC power supply, a nanosecond pulse power supply or a pulse-DC power supply;

[0024] The double-wire wall hot wire 2 and the hot film sensor 4 both use a constant temperature CTA wind speed hot wire instrument 8 to collect flow field data.

[0025] In another specific embodiment of the present invention, the distance L between the double-wire wall heating wires 2 is s =10mm;y+ =15;

[0026] The array spacing of the plasma actuator 3 is the span spacing L of the double-wire wall heat wire 2 s Half of them are intertwined;

[0027] The high voltage power supply 5 is a high voltage sinusoidal AC power supply, a nanosecond pulse power supply or a pulse-DC power supply.

[0028] In another embodiment of the present invention, the double-wire wall heating wire 2 comprises a heating wire base 21, a conical tip bracket 22, a welding point 23 and a tungsten wire 24;

[0029] On the upper surface of the hot wire base 21, two conical tip brackets 22 are arranged along a diameter direction, and the whole is conical, and the welding point 23 arranged at the top thereof forms a thin and high cone; the two conical tip brackets 22 are arranged symmetrically about the center of the upper surface of the hot wire base 21;

[0030] On the upper surface of the hot wire base 21, two low cones are arranged along another diameter direction, and the two low cones are symmetrically arranged about the center of the upper surface of the hot wire base 21;

[0031] Two tungsten wires 24 are arranged on each double-wire wall heating wire 2. Each tungsten wire 24 is connected to a high and a low conical tip bracket 22 through a welding point 23. The two tungsten wires 24 are X-shaped along the flow direction and cross each other.

[0032] A method for intelligent plasma turbulence friction drag reduction based on deep reinforcement learning is also provided. The method is based on the above-mentioned intelligent plasma turbulence friction drag reduction control system based on deep reinforcement learning. The method includes two parts: a control loop and a learning loop. The control loop is used to interact with the real scene, and the learning loop is based on the experience accumulated in the control loop for learning. The details are as follows:

[0033] The control cycle process is as follows:

[0034] Step 1: The hot wire sensor monitors the state parameters s of the turbulent boundary layer in real time t =[(u(t),v(t)),(u(t-τ),v(t-τ)),(u(t-2τ),v(t-2τ))], and the state parameter s t Sent to the Actor network in the FPGA controller, where u(t) and v(t) represent the flow velocity and normal velocity at time t, respectively;

[0035] Step 2: The Actor network will change the current state s t As the input of the neural network, make the corresponding action a t As the output of the network;

[0036] Step 3: The deep deterministic policy gradient network DDPG adds a Gaussian noise function ε to the action strategy and then sends the corresponding control instruction to the high-voltage power supply, i.e., action a t =φ(U,f m ,DC)+ε, where φ(.,.,.) represents U, f m , DC, which is a combination of abstract expression parameters, that is, action a is related to U, f m , DC related; U, f m , DC are voltage, modulation frequency and duty cycle respectively;

[0037] Step 4: The high voltage power supply controls the plasma actuator to exert disturbance on the turbulent boundary layer in the specified excitation form, excitation intensity and frequency according to the instruction of the parameter combination transmitted by the Actor network;

[0038] Step 5: The turbulent boundary layer is disturbed, and the flow field state changes further. After the plasma actuator is driven, the hot wire sensor monitors the new state parameter s again. ’ t , and use it as the input of the next round of Actor network;

[0039] Step 6: The action of the hot film sensor on the plasma actuator t Evaluate and get the profit r t ;

[0040] Step 7: FPGA controller will get the data information of this round [s t ,a t ,r t ,s ’ t ]Package and send to the host computer for learning cycle;

[0041] The learning cycle process is as follows:

[0042] The host computer obtains the data information obtained by multiple interactions between the FPGA controller and the flow field [s t ,a t ,r t ,s ’ t ] is stored in the experience pool. When the amount of data in the memory pool reaches the threshold, multiple sets of data information [s t ,a t ,r t ,s ’ t] randomly extracts quantitative data as learning experience, calculates the loss function according to formula (1) and updates the parameters of the Actor network and the Critic network; the DDPG algorithm introduces two target networks based on the current network architecture, and the data in the experience pool is used to update the four network parameters; the learning optimization goal of the Actor network is to make the action-state value function Q output by the Critic network μ (s t ,a t ) to maximize the optimal strategy, that is, to use the gradient descent method to make [-Q μ (s t ,a t )] is minimized; the goal of the Critic network is to minimize the error between the current network and the target Critic network; the loss functions of the two networks are as follows:

[0043]

[0044] Among them, L Actor and L Critic Represent the two loss functions of the Actor network and the Critic network respectively; N represents the number of selected experience groups; θ and μ represent the parameters of the Actor network and the Critic network respectively, corresponding to Q μ (s,a|θ) is the Q value output by the Critic network with the Actor network parameter θ and the parameter μ. s and a represent the current state s respectively. t and the current action a t , r(s,a) represents the corresponding benefit r under the current state and action t , Q μ (s,a) represents the Q value output by the critic network with parameter μ in the current state. γ∈[0,1] is the discount factor. When γ is close to 0, it means that the agent attaches more importance to short-term rewards; when γ is close to 1, the agent attaches more importance to long-term cumulative rewards. Q μ (s ’ ,a ’ ) is the target Critic network according to the next state s in the experience library ’ t Output value, s ’ 、a ’ Represents the next state s respectively ’ t and the next action a ’ t ; After the network parameters θ and μ of the Actor network and Critic network parameters are updated, the current network parameters θ and μ are respectively transferred to the corresponding target network at regular time steps, and the network parameters θ of the two target networks are updated using the sliding average update method ’ and μ’ Through continuous experience replay and learning, the control strategy of the Actor network is optimized, so that the intelligent agent trained by the FPGA controller can make the best action in the face of complex and changeable turbulent boundary layers and reduce friction resistance.

[0045] In one embodiment of the present invention, the algorithm training and learning process is specifically as follows:

[0046] The first step is to establish a learning scenario. Hardware level: open the wind tunnel to establish a turbulent boundary layer, start the double-wire hot wire sensor, hot film sensor and plasma actuator, put the acquisition equipment in real-time monitoring mode, and put the control equipment in a standby state. Software level: define the neural network structure required by the DDPG algorithm and the hyperparameters corresponding to the learning control, use Gaussian random noise to initialize the network parameters, and send them to the four neural networks.

[0047] The second step is to accumulate learning experience; the double-wire hot wire sensor obtains the current turbulent boundary layer flow field state s t =[(u(t),v(t)),(u(t-τ),v(t-τ)),(u(t-2τ),v(t-2τ))], sent to the Actor network, output action a in the current state t =φ(U,f m ,DC)+ε; The plasma actuator is controlled by a high voltage power supply and an IGBT high voltage switch. t The discharge waveform is changed to achieve excitation of different intensities and frequencies, and to regulate the turbulent boundary layer; after the excitation is completed, the double-wire hot wire sensor collects the next state s' t , the hot film sensor evaluates the control yield r t =1-τ actuation / τ baseline ; The FPGA controller converts the data information obtained from this round of interaction into t ,a t ,r t ,s ’ t ]After packaging and sending to the control computer, enter the next round of interaction again;

[0048] The third step is to replay the experience and update the network parameters. The control computer records the learning experience sent by the FPGA controller into the experience pool, and determines in real time whether the data in the experience pool reaches the set threshold. If the experience pool is full, the program automatically randomly samples from the experience pool and calculates the corresponding loss function for network parameter update. Otherwise, it continues to accumulate learning experience.

[0049] Step 4: Learning ends. The control computer determines whether the drag reduction rate has reached convergence or the number of running steps has reached the set number of steps based on the data changes of the thermal film sensor. If the conditions are met, learning ends and the FPGA controller controls the turbulent boundary layer with the currently learned strategy. If the conditions are not met, the aforementioned learning process continues.

[0050] The present invention selects hot wires and hot films as sensors and array dielectric barrier discharge plasma with fast response as actuators. Through the deep reinforcement learning algorithm deployed in the computer and FPGA controller, it continuously interacts with the turbulent boundary layer at high speed, learns and optimizes the control strategy of the intelligent body, and finally realizes the real-time closed-loop control of turbulent friction drag reduction. Compared with the traditional open-loop control, this method has the advantages of better adaptability to complex incoming flows and higher drag reduction efficiency.

[0051] The advantages of the present invention are as follows:

[0052] 1. Better adaptability to external flow: In the traditional open-loop control mode, the operation of the actuator is independent of the flow field state, lacks adjustment ability, and has poor adaptability to complex flow conditions. The control method provided by the present invention, combined with the DDPG-LSTM algorithm, can perceive the state of the flow field in the turbulent boundary layer in real time, and adjust the learning strategy in real time according to the flow field changes, thereby significantly improving adaptability.

[0053] 2. Higher drag reduction efficiency: The control method of the present invention can realize situation-based excitation according to the flow field state, eliminate unnecessary power consumption, especially when no excitation is required, and achieve higher control efficiency on the basis of improving drag reduction capability.

[0054] 3. Data-driven, saving R&D costs: Compared with the model-based control method, the control method of the present invention does not need to rely on high-precision modeling. It only needs to arrange the control system in a real environment for interaction to learn the corresponding control strategy (model). In addition, the method does not require manual intervention during operation, which significantly saves manpower time and reduces construction costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The framework of intelligent plasma turbulence friction reduction control system based on deep reinforcement learning is presented;

[0056] Figure 2 A schematic diagram of an intelligent closed-loop turbulent friction drag reduction scheme based on deep reinforcement learning is shown;

[0057] Figure 3 A schematic diagram of the double-wire wall heat line structure is shown;

[0058] Figure 4 The flowchart of the intelligent closed-loop turbulent friction reduction scheme based on deep reinforcement learning is shown.

[0059] Reference numerals:

[0060] 1 Wind tunnel plate 2 Double wire wall hot wire 21 Hot wire base 22 Conical tip bracket 23 Welding point 24 Tungsten wire 3 Plasma actuator 4 Hot film sensor 5 High voltage power supply 6 IGBT high voltage switch 7 Control computer and FPGA controller 8 CTA wind speed hot wire instrument DETAILED DESCRIPTION

[0061] The present invention will be described in detail below with reference to the accompanying drawings.

[0062] Figure 1 The intelligent plasma turbulence friction reduction control system framework based on deep reinforcement learning proposed by the present invention is shown. The control system framework is divided into three parts: environment construction, host computer and controller.

[0063] The environment construction mainly includes high-voltage power supply, plasma actuator, state sensor (hot wire sensor), evaluation sensor (hot film sensor), hot wire anemometer, signal acquisition board, and the object to be controlled is turbulent boundary layer.

[0064] The high voltage power supply mainly provides driving power for the plasma actuator, and drives the plasma actuator to discharge and generate plasma with specified excitation parameters (voltage, modulation frequency and duty cycle). The high voltage power supply can be a high voltage sinusoidal AC power supply, a nanosecond pulse power supply, a pulse-DC power supply, etc., and only needs to drive the plasma actuator to discharge, without specific restrictions.

[0065] There are various types of plasma actuators (such as surface arc plasma actuators, synthetic jet plasma actuators, dielectric barrier discharge plasma actuators, etc.), and dielectric barrier discharge plasma actuators with simple structure and rapid response are preferred. Plasma actuators suitable for turbulent drag reduction have various structural forms, which can be roughly divided into three types: spanwise jet, normal jet and streamwise jet according to the different forms of disturbance generated in the flow field. The present invention only requires that the plasma actuator can effectively disturb the turbulent boundary layer, and does not limit its structural form. The plasma actuator is installed at the controlled position of the object to be controlled (turbulent boundary layer).

[0066] As an important part of the control system, the state sensor is used to sense the state of the flow field and monitor the parameters such as velocity, temperature, shear stress, etc. that can directly reflect the state of the flow field. It is generally located upstream of the plasma actuator. Hot wire sensors with high frequency response (above 10kHz) and small disturbance to the flow field are preferred to reduce the introduction of additional disturbances as much as possible while improving the accuracy of flow field perception.

[0067] The evaluation sensor is mainly used to evaluate the change of turbulent friction resistance (surface shear stress). The optional wall shear stress measurement technologies include force balance, Preston tube, oil film interferometer, hot film sensor, wall hot wire, etc. The measurement position of the evaluation sensor can be located at the plasma actuator or downstream of the plasma actuator. It is consistent with the requirements of the aforementioned state monitoring sensor. Hot film sensors with higher frequency response are preferred to avoid delaying the control time due to the low frequency response of the sensor, so as to improve the control frequency of the overall closed-loop control system.

[0068] The hot wire anemometer converts the velocity and shear stress signals collected by the state sensor (such as the hot wire sensor) and the evaluation sensor (such as the hot film sensor) into voltage signals, which are then converted into digital signals through the signal acquisition card and transmitted to the host computer. Note that a hot wire anemometer is usually required when using hot wire or hot film sensors; if other sensors are used, a hot wire anemometer is usually not required.

[0069] The solution of the present invention does not make any specific requirements on the signal acquisition board, as long as it can meet the high-speed acquisition of sensor data signals.

[0070] The host computer and controller are responsible for the control hardware and software algorithm parts in the control system framework.

[0071] The host computer can be a mature computer, and the controller can be a computer, a single-chip microcomputer, an FPGA (field programmable gate array) controller, and other control devices that can run stably and at high speed. The present invention does not limit the specific implementation method and equipment of the algorithm. It is preferred that the FPGA controller interacts with the environment. Compared with the single-chip microcomputer controller and the traditional CPU control computer, the FPGA controller can reduce the control delay time to the order of microseconds, which can fully meet the high-speed intelligent closed-loop control requirements of the present invention for turbulent drag reduction control.

[0072] The present invention mainly adopts the algorithm combining the deep deterministic policy gradient (DDPG) [Lillicrap TP, Hunt JJ, PritzelA, et al. Continuous control with deep reinforcement learning [J]. arXiv preprint arXiv: 1509.02971, 2015.] and the long short-term memory (LSTM) network [Hochreiter S. Long Short-term Memory [J]. Neural Computation MIT-Press, 1997.] to explore and optimize the turbulent drag reduction control law. The DDPG algorithm is a powerful and flexible reinforcement learning algorithm. By combining the ideas of deep learning and policy gradient, compared with the deep Q network algorithm, the DDPG algorithm can achieve efficient policy learning and optimization in the continuous action space, and enhance the ability to understand the environment by introducing a noise exploration mechanism.

[0073] The intelligent plasma turbulence friction drag reduction method based on deep reinforcement learning of the present invention is as follows:

[0074] The DDPG algorithm provides the learning experience gained from interacting with the environment to the simulated intelligent agent for training. The intelligent agent continuously updates its control strategy according to the changes in the environment, and finally trains a suitable intelligent agent. Specifically, the DDPG algorithm is based on the actor-critic framework, and has two network structures: actor and critic (the construction method is well known to those skilled in the art). The actor network is based on the current state s t Make corresponding actions t , and the Critic network is about the current state s t The action a taken by the Actor network t The quality of the action is judged (the value of the action value function Q is used for subsequent judgment), that is, the output action-state value function Q μ (s t ,a t )(This function is well known to those skilled in the art). The LSTM network architecture can effectively mine and learn highly correlated features in time series data due to its special memory structure and gate structure. Compared with the fully connected neural network structure commonly used in the DDPG algorithm, the LSTM network has more advantages in dealing with time series prediction and long-term dependency problems, and is expected to be consistent with the convection characteristics and correlation of the strip structure in the turbulent boundary layer. Therefore, all neural networks in the present invention adopt the LSTM architecture.

[0075] The present invention provides an intelligent plasma turbulence friction drag reduction method based on deep reinforcement learning, which includes a control loop and a learning loop; the control loop interacts with the real scene to generate experience data, and the learning loop learns and updates the parameters of the algorithm network based on the experience accumulated by the control loop, and the two are executed in parallel, as follows:

[0076] The control loop process is as follows.

[0077] Step 1: The hot wire sensor monitors the state parameters s of the turbulent boundary layer in real time t =[(u(t),v(t)),(u(t-τ),v(t-τ)),(u(t-2τ),v(t-2τ))](tracing back two time steps τ), where u(t) and v(t) represent the streamwise velocity and normal velocity at time t, respectively, and the state parameter s t Sent to the Actor network in the FPGA controller.

[0078] Step 2: The Actor network will change the current state s t As the input of the neural network, make the corresponding action a t as the output of the network.

[0079] Step 3: To increase the agent’s exploration in the environment, DDPG adds a Gaussian noise function ε to the action strategy and then sends the corresponding control instruction to the high-voltage power supply, i.e., action a t =φ(U,f m ,DC)+ε, where φ(.,.,.) represents U, f m , DC, which is a combination of abstract excitation parameters, that is, action a is related to U, f m , DC related, according to user needs or different high voltage power supplies to define, no specific restrictions. U, f m , DC are voltage, modulation frequency and duty cycle respectively.

[0080] Step 4: The high voltage power supply, according to the instruction of the parameter combination (voltage, frequency, and duty cycle) transmitted by the Actor network, causes the plasma actuator to exert disturbance on the turbulent boundary layer in a specific excitation form, excitation intensity, and frequency control. This method is well known to those skilled in the art and will not be described again.

[0081] Step 5: The turbulent boundary layer is disturbed, and the flow field state changes further. After the plasma actuator is driven, the hot wire sensor monitors the new state parameter s again. ’ t And use it as the input of the next round of Actor network.

[0082] Step 6: The action of the hot film sensor on the plasma actuator t Perform an evaluation (as described later) and obtain the profit r t .

[0083] Step 7: FPGA controller will get the data information of this round [s t ,a t ,r t ,s ’ t ]Package and send to the host computer for learning cycle.

[0084] The learning cycle process is as follows.

[0085] The host computer obtains the data information obtained by multiple interactions between the FPGA controller and the flow field [s t ,a t ,r t ,s ’ t ] is stored in the experience pool. When the amount of data in the memory pool reaches the threshold, multiple sets of data information [s t ,a t ,r t ,s ’ t ] randomly extracts quantitative data as learning experience (the specific number is determined according to the experimental needs and is not limited), and calculates the loss function according to formula (1) and updates the parameters of the Actor network and the Critic network. The DDPG algorithm introduces two target networks (target Actor network and target Critic network) based on the current network architecture to improve the stability of the learning process. The data in the experience pool is used to update the four network parameters. Specifically, the learning optimization goal of the Actor network is to make the action-state value function Q output by the Critic network μ (s t ,a t ) to maximize the optimal strategy, that is, to use the gradient descent method to make [-Q μ (s t ,a t )] is minimized (the "gradient descent method" is well known to those skilled in the art); and the goal of the critic network is to minimize the error between the current network and the target critic network. The loss functions of the two networks are as follows:

[0086]

[0087] Among them, L Actor and L CriticRepresent the two loss functions of the Actor network and the Critic network respectively; N represents the number of selected experience groups; θ and μ represent the parameters of the Actor network and the Critic network respectively, corresponding to Q μ (s,a|θ) is the Q value output by the Critic network with the Actor network parameter θ and the parameter μ. s and a represent the current state s respectively. t and the current action a t , r(s,a) represents the corresponding benefit r under the current state and action t , Q μ (s,a) represents the Q value output by the critic network with parameter μ in the current state. γ∈[0,1] is the discount factor. When γ is close to 0, it means that the agent attaches more importance to short-term rewards; when γ is close to 1, the agent attaches more importance to long-term cumulative rewards. Q μ (s ’ ,a ’ ) is the target Critic network according to the next state s in the experience library ’ t Output value, s ’ 、a ’ Represents the next state s respectively ’ t and the next action a ’ t Further, after the network parameters θ and μ of the Actor network and the Critic network are updated (these two network parameters are known to those skilled in the art), the current network parameters θ and μ are respectively transferred to the corresponding target networks (i.e., the target Actor network and the target Critic network) at regular time steps, and the network parameters θ of the two target networks are updated using a sliding average update method (this method is known to those skilled in the art). ’ and μ ’ Finally, through continuous experience replay and learning, the control strategy of the Actor network is optimized, so that the intelligent agent trained by the FPGA controller can make the best action in the face of complex and changeable turbulent boundary layers and achieve the reduction of friction resistance.

[0088] Specific implementation cases:

[0089] Figure 2 The figure is a schematic diagram of an intelligent closed-loop turbulent friction drag reduction scheme based on deep reinforcement learning. It mainly includes: a wind tunnel plate 1, a double-wire wall hot wire 2, a plasma actuator 3, a hot film sensor 4, a high-voltage power supply 5, an IGBT high-voltage switch 6, a control computer and FPGA controller 7, and a CTA wind speed hot wire instrument 8. It should be noted that the double-wire wall hot wire and the hot film sensor correspond to the state sensor and evaluation sensor of the aforementioned control system, respectively. These devices are only used as examples here and are not strictly limited.

[0090] The implementation case was carried out in the wind tunnel laboratory. The wind tunnel flat plate was set in the middle of the wind tunnel to produce a flat turbulent boundary layer, and the incoming flow direction was from left to right. From upstream to downstream, there were double-wire wall hot wire 2, plasma actuator 3 and hot film sensor 4. High-voltage power supply 5, IGBT high-voltage switch 6, control computer and FPGA controller 7 and CTA wind speed hot wire instrument 8 were arranged on the experimental table outside the wind tunnel, and connected to various devices in the wind tunnel through wires, BNC cables and RS232 serial cables.

[0091] The double-wire wall hot line 2 is distributed in an array along the span direction (z axis) to capture the distribution of high and low speed strips in the turbulent boundary layer along the span direction. The array spacing range is 5-20 mm. The larger the experimental wind speed, the smaller the sensor spacing should be to match the strip structure scale. The preferred L s = 10mm. The structure of the double-wire wall heating wire 2 is as follows Figure 3 As shown, it consists of a hot wire base 21, a conical tip bracket 22, a welding point 23 and a tungsten wire 24. The shape of the hot wire base 21 is determined according to the needs of the installation position of the specific experimental model, and is shown in the figure as a cylindrical shape. On the upper surface of the hot wire base 21, two conical tip brackets 22 are arranged along a diameter direction, which are conical as a whole, and form a thin and tall cone with the welding point 23 arranged at the top thereof. The two conical tip brackets 22 are arranged symmetrically about the center of the upper surface of the hot wire base 21. Similarly, two similar low cones are arranged along another diameter direction, and the two low cones are arranged symmetrically about the center of the upper surface of the hot wire base 21. Two tungsten wires 24 are arranged on each sensor, and each tungsten wire 24 is connected to a high and a low conical tip bracket 22 through a welding point 23. The two tungsten wires 24 are X-shaped along the flow direction and cross each other, and jointly measure the time series u(t) and v(t) of the flow direction (u) and normal (v) components of the incoming flow velocity, and return the flow field state s t =(u(t),v(t)), "(u(t),v(t))" represents a combination of u(t) and v(t), which can be defined by the user without any specific restrictions. Considering that the wall hot line needs to monitor the strip structure distribution in the turbulent boundary layer in real time, the normal position measured by the hot line needs to be controlled in the viscous wall area (y + <50), y + represents the dimensionless normal wall height (well known to those skilled in the art), but near the wall, the hot line inevitably has measurement errors caused by the wall thermal effect. Therefore, the hot line height range is 10 <y + <50, preferably y + =15, the turbulence of the streamwise velocity pulsation reaches its peak here.

[0092] In this case, the structure of the plasma actuator 3 is a common comb-shaped array, with an exposed electrode on the left and a grounded electrode on the right. The array spacing is not limited, and the preferred spacing is L between the double-wire wall heat wires 2. s half, intertwined with each other.

[0093] The plasma actuator 3 is driven by a high-voltage power supply 5, which may be a high-voltage sinusoidal AC power supply, a nanosecond pulse power supply, a pulse-DC power supply, etc. The positive and negative electrodes of the high-voltage power supply 5 are respectively connected to the exposed electrode and the ground electrode of the plasma actuator 3 through high-voltage wires, and an IGBT high-voltage switch 6 for controlling the high-frequency switching of the circuit is also connected in series between the plasma actuator 3 and the positive electrode of the high-voltage power supply 5.

[0094] The hot film sensor 4 is attached downstream of the plasma actuator 3, also at a distance L s Along the spanwise array, the spanwise measurement point position is consistent with the upstream double-wire wall hot line 2 to evaluate the change in surface shear stress τ after convection, corresponding to the gain r t =1-τ actuation / τ baseline , where τ baseline and τ actuation Corresponding to the shear stress under base and excitation conditions, respectively.

[0095] The double-wire wall hot wire 2 and the hot film sensor 4 both use a constant temperature CTA wind speed hot wire instrument 8 to collect flow field data and transmit the signal to the control computer and FPGA controller 7. The above measurement method is a common measurement method in the field of flow measurement and will not be introduced in detail here.

[0096] The following introduces the algorithm training and learning process for a specific implementation case.

[0097] The first step is to establish a learning scenario. Hardware level: Open the wind tunnel to establish a turbulent boundary layer, start the double-wire hot wire sensor, hot film sensor and plasma actuator, put the acquisition equipment in real-time monitoring mode, and put the control equipment in a standby state. Software level: Define the neural network structure required by the DDPG algorithm and the hyperparameters corresponding to the learning control, use Gaussian random noise to initialize the network parameters, and send them to the four neural networks.

[0098] The second step is to accumulate learning experience. The double-wire hot wire sensor obtains the current turbulent boundary layer flow field state s t =[(u(t),v(t)),(u(t-τ),v(t-τ)),(u(t-2τ),v(t-2τ))], sent to the Actor network, output action a in the current state t =φ(U,f m ,DC)+ε. The plasma actuator is controlled by a high voltage power supply and an IGBT high voltage switch.t The discharge waveform is changed to achieve different intensities and frequencies of excitation, and to regulate the turbulent boundary layer. After the excitation is completed, the double-wire hot wire sensor collects the next state s' t , the hot film sensor evaluates the control yield r t =1-τ actuation / τ baseline FPGA controller will get the data information from this round of interaction [s t ,a t ,r t ,s ’ t ]After packaging and sending to the control computer, enter the next round of interaction again.

[0099] The third step is to replay the experience and update the network parameters. The control computer records the learning experience sent by the FPGA controller into the experience pool, and determines in real time whether the data in the experience pool reaches the set threshold. If the experience pool is full, the program automatically randomly samples from the experience pool and calculates the corresponding loss function for network parameter update. Otherwise, it continues to accumulate learning experience.

[0100] Step 4: Learning ends. The control computer determines whether the drag reduction rate has reached convergence or the number of running steps has reached the set number of steps based on the data changes of the hot film sensor. If the conditions are met, learning ends, and the FPGA controller controls the turbulent boundary layer with the currently learned strategy. If the conditions are not met, the above learning process continues.

Claims

1. An intelligent plasma turbulence friction reduction control system based on deep reinforcement learning, characterized in that: The system includes environmental components, host computer and controller. in, Environmental components include high voltage power supply, plasma actuator, status sensor, evaluation sensor, signal acquisition board; A high voltage power supply for driving the plasma exciter to discharge; A plasma actuator installed at a position to be controlled of the object to be controlled so as to effectively disturb the turbulent boundary layer; A state sensor, which is located upstream of the plasma actuator and is used to sense the state of the flow field and output it to the host computer and controller through a signal acquisition board; An evaluation sensor is located at the plasma exciter or downstream of the plasma exciter to evaluate the change of turbulent friction resistance of the flow field after plasma excitation and output the result to the host computer through a signal acquisition board; A signal acquisition board collects the real-time output flow field data of the status sensor and the evaluation sensor, and converts the voltage signal into a digital signal and outputs it to the host computer; The host computer receives the digital signal output by the signal acquisition board, combines, processes and analyzes the collected data, and performs network learning training through the DDPG algorithm deployed in the host computer; The controller receives the flow field status information output by the status sensor, outputs action instructions to the high-voltage power supply through the internally deployed Actor network, and transmits the current action data to the host computer.

2. The intelligent plasma turbulence friction reduction control system based on deep reinforcement learning according to claim 1, characterized in that: The status sensor uses a hot wire sensor, and the evaluation sensor uses a hot film sensor.

3. The intelligent plasma turbulence friction reduction control system based on deep reinforcement learning as claimed in claim 2, characterized in that: The hot wire anemometer converts the velocity and shear stress signals collected by the hot wire sensor and the hot film sensor into voltage signals respectively, and then converts them into digital signals through the signal acquisition card and transmits them to the host computer; when using hot wire or hot film sensors, a hot wire anemometer must be used in conjunction.

4. The intelligent plasma turbulence friction reduction control system based on deep reinforcement learning as claimed in claim 1, characterized in that: The control system includes: a wind tunnel plate 1, a double-wire wall hot wire 2, a plasma actuator 3, a hot film sensor 4, a high-voltage power supply 5, an IGBT high-voltage switch 6, a control computer, an FPGA controller 7 and a CTA wind speed hot wire instrument 8; The wind tunnel flat plate is set in the middle of the wind tunnel to generate a flat turbulent boundary layer, and the incoming flow direction is from left to right; from upstream to downstream, double-wire wall hot wire 2, plasma actuator 3 and hot film sensor 4 are arranged in sequence; high-voltage power supply 5, IGBT high-voltage switch 6, control computer and FPGA controller 7 and CTA wind speed hot wire instrument 8 are arranged outside the wind tunnel, and are connected to various devices in the wind tunnel through wires, BNC lines and RS232 serial lines; The double-wire wall hot line 2 is distributed along the spanwise array to capture the distribution of high and low speed strips in the turbulent boundary layer along the spanwise direction. The larger the experimental wind speed, the smaller the sensor spacing should be to match the scale of the strip structure. The double-wire wall hot line 2 is used to measure the time series u(t) and v(t) of the streamwise (u) and normal (v) components of the incoming flow velocity, and return the flow field state s t =(u(t),v(t)), "(u(t),v(t))" represents a combination of u(t) and v(t); The positive and negative electrodes of the high-voltage power supply 5 are respectively connected to the exposed electrode and the ground electrode of the plasma actuator 3 through high-voltage wires, and an IGBT high-voltage switch 6 for controlling the high-frequency switching of the circuit is also connected in series between the plasma actuator 3 and the positive electrode of the high-voltage power supply 5; The hot film sensor 4 is attached to the downstream of the plasma actuator 3 and arranged in a spanwise array. The spanwise measurement point position is consistent with the upstream double-wire wall thermal line 2 to evaluate the change of the surface shear stress τ after convection, corresponding to the benefit r t =1-τ actuation / τ baseline , where τ baseline and τ actuation Corresponding to the shear stress under the base and excitation conditions, respectively; The double-wire wall heat line 2 and the thermal film sensor 4 collect flow field data and transmit the signal to the control computer and FPGA controller 7.

5. The intelligent plasma turbulence friction reduction control system based on deep reinforcement learning as claimed in claim 4, characterized in that: The double-wire wall surface hot line 2 is distributed along the span direction array, and the array spacing range is 5-20mm; the normal position measured by the double-wire wall surface hot line 2 needs to be controlled in the viscous wall area y + <Within 50, y + Represents the dimensionless normal wall height, the height range of the double-wire wall hot line 2 is 10 <y + <50; The plasma actuator 3 is a comb-shaped array type, with an exposed electrode on the left and a ground electrode on the right; The high voltage power supply 5 is a high voltage sinusoidal AC power supply, a nanosecond pulse power supply or a pulse-DC power supply; The double-wire wall hot wire 2 and the hot film sensor 4 both use a constant temperature CTA wind speed hot wire instrument 8 to collect flow field data.

6. The intelligent plasma turbulence friction reduction control system based on deep reinforcement learning according to claim 5, characterized in that: Double wire wall hot wire 2 spacing L s =10mm;y + =15; The array spacing of the plasma actuator 3 is the span spacing L of the double-wire wall heat line 2 s Half of them are intertwined; The high voltage power supply 5 is a high voltage sinusoidal AC power supply, a nanosecond pulse power supply or a pulse-DC power supply.

7. The intelligent plasma turbulence friction reduction control system based on deep reinforcement learning as claimed in claim 4, characterized in that: The double-wire wall heating wire 2 comprises a heating wire base 21, a conical tip bracket 22, a welding point 23 and a tungsten wire 24; On the upper surface of the hot wire base 21, two conical tip brackets 22 are arranged along a diameter direction, and the whole is conical, and the welding point 23 arranged at the top thereof forms a thin and high cone; the two conical tip brackets 22 are arranged symmetrically about the center of the upper surface of the hot wire base 21; On the upper surface of the hot wire base 21, two low cones are arranged along another diameter direction, and the two low cones are symmetrically arranged about the center of the upper surface of the hot wire base 21; Two tungsten wires 24 are arranged on each double-wire wall heating wire 2. Each tungsten wire 24 is connected to a high and a low conical tip bracket 22 through a welding point 23. The two tungsten wires 24 are X-shaped along the flow direction and cross each other.

8. An intelligent plasma turbulent friction drag reduction method based on deep reinforcement learning, which is based on the intelligent plasma turbulent friction drag reduction control system based on deep reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: The method consists of two parts: a control loop and a learning loop. The control loop is used to interact with real-world scenarios, and the learning loop is based on the experience accumulated by the control loop. The details are as follows: The control cycle process is as follows: Step 1: The hot wire sensor monitors the state parameters s of the turbulent boundary layer in real time t =[(u(t),v(t)),(u(t-τ),v(t-τ)),(u(t-2τ),v(t-2τ))], and the state parameter s t Sent to the Actor network in the FPGA controller, where u(t) and v(t) represent the flow velocity and normal velocity at time t, respectively; Step 2: The Actor network will change the current state s t As the input of the neural network, make the corresponding action a t As the output of the network; Step 3: The deep deterministic policy gradient network DDPG adds a Gaussian noise function ε to the action strategy and then sends the corresponding control instruction to the high-voltage power supply, that is, the action in Represents U, f m , DC, which is a combination of abstract expression parameters, that is, action a is related to U, f m , DC related; U, f m , DC are voltage, modulation frequency and duty cycle respectively; Step 4: The high voltage power supply controls the plasma actuator to exert disturbance on the turbulent boundary layer in the specified excitation form, excitation intensity and frequency according to the instruction of the parameter combination transmitted by the Actor network; Step 5: The turbulent boundary layer is disturbed, and the flow field state changes further. After the plasma actuator is driven, the hot wire sensor monitors the new state parameter s' again. t , and use it as the input of the next round of Actor network; Step 6: The action of the hot film sensor on the plasma actuator t Evaluate and get the profit r t ; Step 7: FPGA controller will get the data information of this round [s t ,a t ,r t ,s' t ]Package and send to the host computer for learning cycle; The learning cycle process is as follows: The host computer obtains the data information obtained by multiple interactions between the FPGA controller and the flow field [s t ,a t ,r t ,s' t ] is stored in the experience pool. When the amount of data in the memory pool reaches the threshold, multiple sets of data information [s t ,a t ,r t ,s' t ] randomly extracts quantitative data as learning experience, calculates the loss function according to formula (1) and updates the parameters of the Actor network and the Critic network; The DDPG algorithm introduces two target networks based on the current network architecture. The data in the experience pool is used to update the four network parameters. The learning optimization goal of the Actor network is to make the action-state value function Q output by the Critic network μ (s t ,a t ) to maximize the optimal strategy, that is, to use the gradient descent method to make [-Q μ (s t ,a t )] is minimized; the goal of the Critic network is to minimize the error between the current network and the target Critic network; the loss functions of the two networks are as follows: Among them, L Actor and L Critic Represent the two loss functions of the Actor network and the Critic network respectively; N represents the number of selected experience groups; θ and μ represent the parameters of the Actor network and the Critic network respectively, corresponding to Q μ (s,a|θ) is the Q value output by the Critic network with the Actor network parameter θ and the parameter μ. s and a represent the current state s respectively. t and the current action a t , r(s,a) represents the corresponding benefit r under the current state and action t , Q μ (s,a) represents the Q value output by the critic network with parameter μ in the current state; γ∈[0,1] is the discount factor. When γ is close to 0, it means that the agent attaches more importance to short-term rewards; when γ is close to 1, the agent attaches more importance to long-term cumulative rewards. μ (s',a') is the target Critic network according to the next state s' in the experience library t The output value, s', a' respectively represents the next state s' t and the next action a' t ; After the network parameters θ and μ of the Actor network and the Critic network are updated, the current network parameters θ and μ are respectively transferred to the corresponding target network at regular time steps, and the network parameters θ' and μ' of the two target networks are updated using the sliding average update method; through continuous experience replay and learning, the control strategy of the Actor network is optimized, so that the intelligent agent trained by the FPGA controller can make the best action in the face of complex and changeable turbulent boundary layers and achieve the reduction of friction resistance.

9. The intelligent plasma turbulence friction drag reduction method based on deep reinforcement learning according to claim 8, characterized in that: The algorithm training and learning process is as follows: The first step is to establish a learning scenario. Hardware level: open the wind tunnel to establish a turbulent boundary layer, start the double-wire hot wire sensor, hot film sensor and plasma actuator, put the acquisition equipment in real-time monitoring mode, and put the control equipment in a standby state. Software level: define the neural network structure required by the DDPG algorithm and the hyperparameters corresponding to the learning control, use Gaussian random noise to initialize the network parameters, and send them to the four neural networks. The second step is to accumulate learning experience; the double-wire hot wire sensor obtains the current turbulent boundary layer flow field state s t = [(u(t), v(t)), (u(t-τ), v(t-τ)), (u(t-2τ), v(t-2τ))], sent to the Actor network, output the action in the current state The plasma actuator is controlled by a high voltage power supply and an IGBT high voltage switch. t The discharge waveform is changed to achieve excitation of different intensities and frequencies, and to regulate the turbulent boundary layer; after the excitation is completed, the double-wire hot wire sensor collects the next state s' t , the thermal film sensor evaluates the control yield r t =1-τ actuation / τ baseline ; The FPGA controller converts the data information obtained from this round of interaction into t ,a t ,r t ,s' t ]After packaging and sending to the control computer, enter the next round of interaction again; The third step is to replay the experience and update the network parameters. The control computer records the learning experience sent by the FPGA controller into the experience pool, and determines in real time whether the data in the experience pool reaches the set threshold. If the experience pool is full, the program automatically randomly samples from the experience pool and calculates the corresponding loss function for network parameter update. Otherwise, it continues to accumulate learning experience. Step 4: Learning ends. The control computer determines whether the drag reduction rate has reached convergence or the number of running steps has reached the set number of steps based on the data changes of the thermal film sensor. If the conditions are met, learning ends and the FPGA controller controls the turbulent boundary layer with the currently learned strategy. If the conditions are not met, the aforementioned learning process continues.

Images