An interpretable intelligent flow control method that integrates genetic programming and reinforcement learning

Abstract

This invention discloses an interpretable intelligent flow control method that integrates genetic programming and reinforcement learning: It introduces the automatic differentiation mechanism from neural networks into linear genetic programming, automatically adjusting the explicit expression obtained through gradient descent, which significantly reduces the time required for control law optimization. Explicit symbolic expression of the control law offers stronger interpretability compared to neural network control laws, facilitating researchers' understanding of control mechanisms and the discovery of new physical knowledge. Furthermore, this framework allows for the setting of state sensors and different reward functions based on different control objectives, demonstrating strong versatility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of active flow control, and in particular to an interpretable intelligent flow control method combining genetic programming and reinforcement learning. BACKGROUND

[0002] Active flow control (AFC) improves the overall performance of an aircraft by inputting additional energy to the flow field through actuators, thereby increasing lift and reducing flight resistance. Most of the research and application of AFC is open-loop control with fixed control parameters. Compared with open-loop control, closed-loop control can adjust control parameters or control strategies according to real-time feedback from state sensors, thereby improving the control benefits and robustness of AFC. Current intelligent closed-loop control methods represented by linear genetic programming (LGP) and deep reinforcement learning (DRL) have been widely used in the field of AFC, such as cylinder flow and airfoil separation control. However, there are still several problems to be solved in the existing intelligent flow control framework. First, although the LGP method is simple and easy to implement and can avoid falling into a local optimal strategy, the entire control law optimization process contains hundreds of iterations, often taking several hours of time (Li, R., Noack, B. R., Cordier, L., Borée, J., Harambat, F. (2017). Drag reduction of a car model by linear genetic programming control. Experiments in Fluids 58, 103. doi:10.1007 / s00348-017-2382-2), and the high time cost makes it difficult to apply to engineering practice. Second, although the DRL method can learn online to shorten the strategy optimization process to a few minutes and achieve control benefits comparable to LGP, since the control law in DRL is approximated by a deep neural network, the "black box nature" of the neural network makes it difficult for researchers to further explore the underlying physical explanation and flow mechanism behind the optimal control law (Yao Zhangyi, Shi Zhiwei, Dong Yizhang. Application of deep reinforcement learning in airfoil separation flow control [J]. Experiments in Fluid Mechanics, 2022, 36(03): 55-64.). In order to further improve the efficiency of closed-loop flow control, it is urgent to develop an intelligent learning framework with strong interpretability and fast learning speed. SUMMARY

[0003] In view of the slow optimization speed and poor interpretability of current intelligent control algorithms, the present application proposes an interpretable intelligent flow control method combining genetic programming and reinforcement learning, which includes two flow control strategy optimization stages, as follows:

[0004] Step 1: Genetic programming initial screening symbolic expression process;

[0005] The initial screening symbolic expression process for genetic programming LGD is as follows: LGP utilizes basic unary and binary function operators to generate a binary tree containing N by randomly generating a fixed-depth binary tree. i The function set F = {f1, f2, ..., f} of an initial control strategy Ni}, where f1, f2, ..., f Ni These represent different initial control strategies, f(s) t ) is a mapping from flow field state to action, i.e., a t =f(s) t ); where a t and s t These are the control commands from the exciter and the sensor signals, respectively, with the initial f(s) t The expression is randomly generated by LGP;

[0006] Each control strategy is applied to the flow field environment, and the corresponding fitness is obtained by measuring with sensors;

[0007] The LGP algorithm evaluates the fitness of each control law and performs operations such as copying, crossover, and mutation on the current generation's control policy according to the fitness ranking to obtain the new generation's control policy; that is, the greater the fitness of an individual, the greater the probability that the individual will be selected.

[0008] After preliminary screening of several generations of control laws, the explicit symbolic control law a with better performance was obtained. t * =f(s) t f(s), where θ is the numerical coefficient in the explicit expression. t ,θ) is the expression form after the initial screening stage of the control law, in which the numbers in the control law expression are used as parameters for further optimization; at this time, the corresponding numerical coefficients θ are definitely not the optimal values ​​and need to be further optimized.

[0009] Step 2: Reinforcement learning of the process of fine-tuning control law parameters;

[0010] Tuning the internal parameters using a reinforcement learning framework: achieving the best-performing expression a t * =f(s) t The operational relationship of ) and the control law a t * The coefficients θ are set as trainable weight coefficients; the gradient of the root mean square loss function with respect to θ is calculated; and the optimization of the parameters θ is achieved by combining an optimizer.

[0011] The reinforcement learning optimization framework includes an explicit symbolic expression and two neural networks; the explicit symbolic expression is the control policy, which establishes a bridge for communication with the fluid environment and determines the output of the actor.

[0012] The actor interacts with the real-time flow environment, based on the real-time flow field state. t Give control command a t The flow field migrates to a new state s t+1 And through reward r t As feedback information of the closed-loop system, [s] t ,a t ,r t ,s t+1 The sequence composed of these elements is stored in the experience replay pool.

[0013] The two neural networks inside the critic (Critic) analyze the current state s. t and control command a t The action value Q is evaluated and calculated by fitting the neural network inside the commentator, which guides the parameter update of the explicit symbolic expression control law;

[0014] Explicit symbolic expression control laws infer control commands based on the current state. The Q-network and Target Q-network are identical, the difference being that the Q-network is used to estimate the current value Q, while the Target Q-network, combined with reward, is used to estimate the expected value y. i The difference between the two is the network's loss function; the objective value of the value function in the current state is obtained:

[0015] y t =r t +γQ t (s t+1 ,f(s t+1 )) (1)

[0016] Where, r t Represents the action to be performed in the current state. t The reward received, Q t Here, γ represents the value calculated by the Target Q network, and γ represents the discount factor, which is the weight of the future reward. When training the Q network and the Target Q network, mini-batch data should be randomly drawn from the experience replay pool to update the weights and bias parameters of the Q network. The Target Q network is updated using a soft update method, i.e., updating only some parameters of the Target Q network. The loss function used for updating the Q network is:

[0017]

[0018] By combining the value gradient from the Q-network training, we obtain the policy gradient of the explicit symbolic expression control law during parameter updates.

[0019]

[0020] Where, N m s represents the number of experience sequences during training. i and a i Let represent the i-th state and action in the experience sequence, respectively, and let Q(s,a) represent the value of different actions a in state s. This represents the gradient of the value function with respect to action a in state s. This represents the gradient of the control law with respect to its coefficient θ, expressed in explicit symbols. Based on the obtained gradient information, the expression parameter θ is updated, and the parameter update process is repeated until the control effect no longer changes or the maximum time of the optimization process is reached.

[0021] In one embodiment of the present invention, the control law reasoning process is deployed on a field-programmable gate array (FPGA), and the control law optimization process is performed on a computer host CPU. The real-time control loop running on the FPGA and the strategy optimization loop running on the CPU are performed simultaneously. In the real-time control loop, the FPGA measures the velocity fluctuations within the shear layer using a hot-wire anemometer, and converts the measurement results using an analog-to-digital converter to obtain the current flow field state s. t ; will state s t The input is given to the explicit symbolic control law, resulting in control command a. t The control command a t The signal is converted into a control voltage signal by a digital-to-analog converter and applied to the plasma exciter, causing a disturbance in the flow field and leading to a new state s. t+1 .

[0022] In another embodiment of the invention, the fitness setting varies depending on the flow field and control objective. In actual flow control, parameters such as time-averaged lift, drag, area of ​​the separation zone, and surface pressure pulsation are used as fitness.

[0023] In one specific embodiment of the present invention, the method by which the LGP algorithm evaluates the fitness of each control law is to sort them in descending order of fitness values.

[0024] In another embodiment of the present invention, after preliminary screening of 3-5 generations of control laws, an explicit symbolic control law a with better performance is obtained. t * =f(s) t ,θ).

[0025] In another specific embodiment of the present invention, after preliminary screening of five generations of control laws, an explicit symbolic control law a with better performance is obtained. t * =f(s) t ,θ).

[0026] In another embodiment of the invention, the better-performing expression a is implemented using Python code. t * =f(s) t The operation relationship is determined; the gradient of the root mean square loss function with respect to θ is solved using numerical differentiation methods; and the optimization of parameter θ is achieved by combining Adam or RMSProp optimizers.

[0027] In another specific embodiment of the present invention, the parameter update of the explicit symbolic expression control law is specifically as follows: if the value of taking a certain action is high in the current state, the Q network inside the critic will guide the Actor to increase the probability of that action accordingly; conversely, if the value is low, the probability of the corresponding action will be reduced.

[0028] In another specific embodiment of the invention, the weights and bias parameters of the Q network are updated using the root mean square error loss.

[0029] This invention introduces the automatic differentiation mechanism from neural networks into linear genetic programming. By automatically adjusting the explicit expression obtained from linear genetic programming using a gradient descent algorithm, the time required for control law optimization can be significantly reduced. Explicit symbolic expressions of control laws offer greater interpretability compared to neural network control laws, facilitating researchers' understanding of control mechanisms and the discovery of new physical knowledge. This framework can be highly versatile, allowing for the configuration of state sensors and different reward functions based on varying control objectives.

[0030] The advantages of this invention are as follows:

[0031] 1. Traditional genetic programming and reinforcement learning closed-loop flow control have the disadvantages of long optimization time and poor interpretability, respectively. This invention can significantly reduce optimization time by combining the frameworks of the two, and the explicit symbolic expression of the control law has better interpretability than neural networks.

[0032] 2. This invention proposes optimizing the explicit symbolic expression control law within the actor-commentator framework, which can reduce the tens of thousands of training parameters of traditional neural network control by three orders of magnitude, thereby reducing the computation time and cost required for control law optimization.

[0033] 3. Furthermore, the explicit symbolic expression can improve the flow control benefit of the LGD control law by adjusting its parameter θ through the gradient descent algorithm within the reinforcement learning framework, and can overcome the problem of poor training stability caused by the random initial parameters of the DRL control law. Attached Figure Description

[0034] Figure 1 A schematic diagram illustrating an interpretable intelligent flow control method that integrates genetic programming and reinforcement learning is shown.

[0035] Figure 2 The parameter update process of an explicit symbolic representation control law is shown in the reinforcement learning framework;

[0036] Figure 3 A schematic diagram illustrating an implementation method for an interpretable intelligent flow control framework is shown. Detailed Implementation

[0037] The present invention will now be described in detail with reference to the accompanying drawings.

[0038] Figure 1 This invention illustrates an interpretable intelligent flow control method that integrates genetic programming and reinforcement learning. The method mainly comprises two flow control strategy optimization stages: a genetic programming process for initial screening of symbolic expressions and a reinforcement learning process for fine-tuning control law parameters.

[0039] Step 1: Initial screening of symbolic expressions using genetic programming;

[0040] The entire genetic programming (LGD) initial screening symbolic expression process is as follows: LGP utilizes basic unary and binary function operators (such as +, -, ×, sin, cos, x). 2 ,e x By randomly generating a binary tree of fixed depth (the structure of a binary tree is well known to those skilled in the art), a tree containing N is generated. i The function set F = {f1, f2, ..., f} of an initial control strategy Ni}, where f1, f2, ..., f Ni These represent different initial control strategies, f(s) t Essentially, it is a mapping from the state of the flow field to actions, i.e., a t =f(s) t ); where a t and s t These are the control commands from the exciter and the sensor signals, respectively, with the initial f(s) t The expression is randomly generated by LGP and may contain operators (addition, subtraction, multiplication, division) and basic functions (exponential function, logarithmic function, trigonometric function, etc.).

[0041] Each control strategy is applied to the flow field environment, and the corresponding fitness is obtained by measuring with sensors. It should be noted that the fitness setting varies depending on the flow field and control objective. In actual flow control, parameters such as time-averaged lift, drag, area of ​​the separation zone, and surface pressure pulsation are often used as fitness.

[0042] The LGP algorithm evaluates the fitness of each control law by sorting them from highest to lowest fitness value. Based on this fitness ranking, it performs operations such as copying, crossover, and mutation on the current generation's control policies to obtain a new generation's control policy. In other words, the higher the fitness of an individual, the greater its probability of being selected. Specifically, copying replicates the individual to the next generation, crossover swaps the function operations at the nodes of two individuals, and mutation generates new function operations at the nodes of an individual—this is known to those skilled in the art.

[0043] After preliminary screening using 3-5 generations of control laws (ideally 5 generations), the best-performing explicit symbolic control law a can be obtained. t * =f(s) t f(s), where θ is the numerical coefficient in the explicit expression. t The expression θ is the result of further optimization (emphasizing parameter optimization) after the initial screening stage of the control law (emphasizing the optimization of the functional relationship), where the numbers in the control law expression are used as parameters. This control law performed well in the first 3-5 generations, indicating that its functional form (binary tree structure) is reasonable. However, the corresponding numerical coefficients θ are definitely not optimal and require further optimization.

[0044] Step 2: Reinforcement learning of the process of fine-tuning control law parameters;

[0045] These internal parameters are tuned using a reinforcement learning framework. Specifically, the best-performing expression 'a' is implemented using Python code. t * =f(s) t The operational relationship of ) (this method is known to those skilled in the art), and the control law a t * The coefficients θ in the equation are set as trainable weight coefficients. The gradient of the root mean square loss function (the expression of the loss function is shown in equation (2) below) with respect to θ is solved using numerical differentiation. Furthermore, the optimization of the parameter θ is achieved by combining optimizers such as Adam and RMSProp (which are well known to those skilled in the art).

[0046] It is obvious that in the above-mentioned intelligent closed-loop control method, the total number of control strategies that need to be evaluated is far less than that required by the classic LGP method (20N). i -50N i Therefore, a more profitable control law can be found in a shorter time.

[0047] Figure 2Taking the classic DDPG algorithm as an example, this paper describes the update and iteration process of the explicit symbolic control policy under the reinforcement learning framework. Compared with the implicit neural network control law in traditional deep reinforcement learning, the control law in this invention is an explicit symbolic expression with fewer parameters, thus significantly reducing the time required for gradient calculation during training. The entire reinforcement learning optimization framework includes one explicit symbolic expression and two neural networks; the explicit symbolic expression is the control policy, which establishes a bridge for communication with the flow environment and determines the output of the actor. The actor interacts with the real flow environment and adjusts the output based on the real-time flow field state s. t Give control command a t The flow field migrates to a new state s t+1 And through reward r t As feedback information of the closed-loop system, [s] will ultimately be t ,a t ,r t ,s t+1 The sequence composed of [images] is stored in the experience replay pool. The two neural networks inside the critic process analyze the current state s. t and control command a t The action value Q (long-term reward) is evaluated in the current state. This action value is calculated by a neural network within the commentator and guides the parameter updates of the explicit symbolic control law. Specifically, if the value of taking an action in the current state is high, the Q-network within the commentator will guide the Actor to increase the probability of that action. Conversely, if the value is low, the probability of the corresponding action will be decreased. The explicit symbolic control law can infer control commands based on the current state. The Q-network and the Target Q-network are identical, the difference being that the Q-network is used to estimate the current value Q, while the Target Q-network, combined with the reward, is used to estimate the expected value y. i The difference between the two is the network's loss function. In practical implementation, the objective value of the value function in the current state is obtained as follows:

[0048] y t =r t +γQ t (s t+1 ,f(s t+1 )) (1)

[0049] Where, r t Represents the action to be performed in the current state. t The reward received, Q tγ represents the value calculated by the Target Q network, and γ represents the discount factor, which is the weight of the future reward. When training the Q network and the Target Q network, a small batch of data should be randomly drawn from the experience replay pool, and then the root mean square error loss should be used to update the weights and bias parameters of the Q network (parameter update methods are well known to those skilled in the art). The Target Q network should be updated using a soft update method, that is, updating only some parameters of the Target Q network to ensure slow updates and thus improve learning stability. The loss function used for updating the Q network is:

[0050]

[0051] By combining the value gradient from the Q-network training, we can obtain the policy gradient of the explicit symbolic control law during parameter updates.

[0052]

[0053] Where, N m s represents the number of experience sequences during training. i and a i Let represent the i-th state and action in the experience sequence, respectively, and let Q(s,a) represent the value of different actions a in state s. This represents the gradient of the value function with respect to action a in state s. This represents the gradient of the explicit symbolic control law with respect to its coefficients θ. In reinforcement learning optimization, both value and policy methods are combined. The explicit symbolic control law helps the agent handle continuous actions, while the value function helps update the control policy. Based on the obtained gradient information, the expression parameters θ are updated, and the parameter update process is repeated until the control effect no longer changes or the maximum time of the optimization process (preset) is reached.

[0054] Figure 3This diagram illustrates the hardware implementation scheme of the present invention for post-step co-enhancing. A hot-wire anemometer is selected as the sensor and is positioned inside the shear layer (at a location related to velocity fluctuations; the specific location can be determined using PIV particle image velocimetry) to monitor instantaneous velocity fluctuations. A plasma exciter is positioned at the corner edge of the post-step model in the wind tunnel test section to perturb the boundary layer through momentum or thermal effects. To reduce the control delay (time required for control law inference) between receiving the flow field state and issuing control commands, the control law inference process of the present invention is deployed on a Field Programmable Gate Array (FPGA) to reduce control latency (within 10µs) and improve control efficiency. The control law optimization process is performed on the CPU of the host computer. The real-time control loop running on the FPGA and the strategy optimization loop running on the CPU are performed simultaneously. In the real-time control loop, the FPGA first measures the velocity fluctuations inside the shear layer using the hot-wire anemometer and converts the measurement results using an analog-to-digital converter to obtain the current flow field state s. t . State s t Inputting this into an explicit symbolic control law will produce control command a. t The control command a t The signal is converted into a control voltage signal by a digital-to-analog converter and applied to the plasma exciter, causing a disturbance in the flow field and leading to a new state s. t+1 .

[0055] The policy optimization loop is completed collaboratively by a Python program and a LabVIEW program. The Python program is used to execute genetic programming and gradient descent algorithms, while the LabVIEW program essentially acts as an information bridge (containing empirical information and policy parameters). Specifically, LabVIEW continuously receives empirical information collected by the FPGA via a FIFO (First-In, First-Out) sequence. t and a t And from this information, the fitness or reward r is calculated. t Then, the fitness or empirical sequence [s] t ,a t ,r t ,s t+1 Transmitted to Python via TCP / IP protocol; Python updates the symbolic expression 'a' of the control policy using genetic programming or gradient descent algorithm. t * =f(s) tThe new control strategy symbolic expression or control law coefficient θ is sent to the FPGA chip via FIFO through LabVIEW to update the control strategy. Since the number of parameters in the explicit symbolic control law is much smaller than that in the neural network control law, the execution time of the control law will be greatly shortened, typically to the order of 10μs, which is more suitable for the needs of high-speed flow and high-frequency closed-loop control.

[0056] The present invention proposes an interpretable intelligent flow control framework that combines genetic programming and reinforcement learning. It can set the objective function of genetic programming and the reward of reinforcement learning according to different control objectives. It can be combined with various types of sensors and exciters, and has strong versatility and interpretability.

Claims

1. An interpretable intelligent flow control method combining genetic programming and reinforcement learning, characterized in that, The method comprises two flow control strategy optimization stages: a genetic programming preliminary screening symbolic expression process and a reinforcement learning fine-tuning control law parameter process; specifically as follows: Step 1: the genetic programming preliminary screening symbolic expression process; The genetic programming (LGP) initial screening symbolic expression process is as follows: the LGP uses basic unary and binary function operators to generate a function set containing N i initial control strategies in the form of a fixed depth binary tree generated by random generation F ={ f 1, f 2, …, f Ni}, wherein f 1, f 2, …, f Ni respectively represent different initial control strategies, f ( s t ) is a mapping from the flow field state to the action, i.e. a t =f ( s t ); wherein, a t and s t are control instructions and sensor signals of the exciter, respectively, and initial f ( s t ) expressions are randomly generated by the LGP. Each control strategy is applied to the flow field environment, and the corresponding fitness is obtained through sensor measurement; The LGP algorithm evaluates the fitness of each control law, and according to the fitness ranking, the control strategies of the current generation are copied, crossed, mutated and the like to obtain a new generation of control strategies; that is, the greater the fitness of an individual, the greater the probability of being selected by the individual; After several generations of preliminary screening of control laws, the explicit symbolic control law with better performance is obtained a t * =f ( s t , θ ), wherein θ is the numerical coefficient in the explicit expression, f ( s t , θ ) is the form of expression after the preliminary screening of control laws, in which the numbers in the control law expression are further optimized as parameters; at this time, the corresponding numerical coefficient θ is not necessarily the optimal value and needs to be further optimized; Step 2: the reinforcement learning fine-tuning control law parameter process; Adjusting internal parameters with reinforcement learning framework: Realizing better-performing expressions a t * =f ( s t ) operating relationship, and set the coefficients a t * in the control law θ as trainable weight coefficients; Solve the gradient of the root mean square loss function to θ ; Combine the optimizer to realize the optimization of parameters θ ; The reinforcement learning optimization framework comprises an explicit symbolic expression and two neural networks; the explicit symbolic expression is the control strategy, which establishes a bridge for communication with the flow environment and determines the output of the actor; Actor interacts with the real flow environment, according to the real-time flow field state s t Give control command a t , the flow field migrates to the new state s t+1 , and through the reward r t As feedback information of the closed-loop system, the sequence composed of [ s t , a t , r t , s t+1 ] is stored in the experience replay pool; two neural networks inside the critic s t and control commands a t under the action value Q is evaluated, which is computed by a neural network inside the critic that is fitted to guide the parameter update of the explicit symbolic representation of the control law; The explicit symbolic representation control law reasons the control command from the current state, the Q-network and the Target Q-network are identical, the difference is that the Q-network is used to estimate the current value Q , the Target Q-network is combined with the reward to estimate the expected value y i ; the difference between the two is the loss function of the network; the target value of the value function under the current state is obtained: （1） wherein, r t representing the action performed in the current state a t the reward obtained, Q t representing the value calculated by the Target Q network, γ representing the discount factor, which is the weight of future rewards When training the Q network and the Target Q network, a small batch of data should be randomly drawn from the experience replay pool, and the weight and bias parameters of the Q network are updated. When updating the Target Q network, a soft update method is used, that is, part of the parameters of the Target Q network is updated. The loss function used for Q network update is: （2） The strategy gradient of the explicit symbolic representation control law in the parameter update is obtained in combination with the value gradient of the Q network training : （3） where N m represents the number of experience sequences during training, s i and a i respectively represent the state and action in the i-th experience sequence, i respectively represent the state and action in the i-th experience sequence, Q s, a represents the value of different actions s under state a , represents the gradient of the value function with respect to action s under state a , represents the gradient of the explicit symbolic expression control law with respect to its coefficients θ ; based on the obtained gradient information, the expression parameters θ are updated, and the parameter updating process is executed in a loop until the control effect no longer changes or the maximum time of the optimization process is reached.​ 2. The genetically programmed and reinforcement learning integrated interpretable intelligent flow control method of claim 1, wherein, The control law reasoning process is deployed on a field programmable gate array (FPGA), and the optimization process of the control law is performed on a computer host CPU, and the real-time control loop running on the FPGA and the strategy optimization loop running on the CPU are performed simultaneously; in the real-time control loop, the FPGA measures the velocity fluctuation inside the shear layer through a hot-wire anemometer, and converts the measurement result through an analog-to-digital converter to obtain the current state of the flow field s t ; the state s t is input to an explicit symbolic control law to obtain a control instruction a t ； The control instruction a t is converted into a control voltage signal through a digital-to-analog converter and applied to a plasma exciter to generate a disturbance to the flow field, so that the flow field migrates to a new state s t+1 .

3. The genetically programmed and reinforcement learning integrated interpretable intelligent flow control method of claim 1, wherein, The fitness is set according to the flow field and the control target, and in actual flow control, the time-averaged lift, drag, separation area and surface pressure fluctuation are used as the fitness.

4. The genetically programmed and reinforcement learning integrated interpretable intelligent flow control method of claim 1, wherein, The method for the LGP algorithm to evaluate the fitness of each control law is to sort the values of the fitness from large to small.

5. The genetically programmed and reinforcement learning integrated interpretable intelligent flow control method of claim 1, wherein, After 3- 5 The preliminary screening of the explicit symbolic control law, the better performance of the explicit symbolic control law a t * =f ( s t , θ ).

6. The genetically programmed and reinforcement learning integrated interpretable intelligent flow control method of claim 5, wherein, After 5 generations of preliminary screening of control laws, the explicit symbolic control law with better performance is obtained a t * =f ( s t , θ ).

7. The genetically programmed and reinforcement learning integrated interpretable intelligent flow control method of claim 1, wherein, Implement a better-performing expression using Python code. a t * =f ( s t The operational relationship of ) is explained; the root mean square loss function is solved using numerical differentiation methods. θ The gradient; combined with the Adam or RMSProp optimizer to implement parameters. θ The search for optimization.

8. The genetically programmed and reinforcement learning integrated interpretable intelligent flow control method of claim 1, wherein, The parameter update of the guided explicit symbolic expression control law is as follows: if the value of a certain action under the current state is high, the Q network in the critic will guide the actor to adjust the probability of the action to be high; on the contrary, if the value is low, the corresponding action probability will be reduced.

9. The genetically programmed and reinforcement learning integrated interpretable intelligent flow control method of claim 1, wherein, The root mean square error loss is used to update the weight and bias parameters of the Q network.

Images