Flat steel box girder wind fairing aerodynamic optimization method based on deep reinforcement learning

Abstract

This invention discloses an aerodynamic optimization method for flat steel box girder nozzles based on deep reinforcement learning. First, an aerodynamic optimization model for the box girder nozzles is constructed, with the aerodynamic performance of the main girder cross-section as the objective and the constraints on the nozzle shape design variables as the constraints. Then, an agent is constructed, including a strategy model and a value function model. The strategy model selects actions to execute based on the current environment and reward function to generate new nozzle shape design variables. Simultaneously, the value function model evaluates the value of the current strategy. The agent is trained using the loss function of the value evaluation, ultimately obtaining the optimal strategy. The nozzle shape design variables obtained with the optimal strategy are the optimal design variables that satisfy the objective function and constraints. In other words, this invention, a deep reinforcement learning-based aerodynamic optimization method for flat steel box girder nozzles, achieves aerodynamic optimization design of flat steel box girder nozzles by building a deep reinforcement learning model.

Description

Technical Field

[0001] This invention belongs to the field of bridge wind-resistant design technology, specifically a method for aerodynamic optimization of wind nozzles for flat steel box girders based on deep reinforcement learning. Background Technology

[0002] Flat steel box girder air nozzles are a type of pneumatic device used in structures. In the aerodynamic design of structures, the aerodynamic performance of the air nozzles is crucial. Due to their simple structure, high strength, and low manufacturing cost, flat steel box girder air nozzles are widely used in large building structures. However, if the aerodynamic performance of the flat steel box girder air nozzles is problematic, it can lead to aerodynamic instability, noise issues, and other problems. Existing aerodynamic optimization methods mainly use computational fluid dynamics (CFD) combined with optimization algorithms, such as genetic algorithms and particle swarm optimization. While existing optimization methods can optimize air nozzles relatively accurately, they have some drawbacks. For example, CFD calculations require significant computational resources, and the optimization algorithms are inefficient, requiring a large number of calculations to achieve good optimization results. Therefore, current flat steel box girder air nozzle optimization methods still primarily rely on experience and trial-and-error approaches, resulting in complex designs, low efficiency, and difficulty in guaranteeing optimization results.

[0003] Deep reinforcement learning is a machine learning technique that maximizes long-term cumulative rewards by having an agent learn through interaction with its environment. It has achieved success in many fields, such as games, natural language processing, and robotics. Compared to traditional optimization methods, the biggest advantage of deep reinforcement learning is its ability to automatically learn and improve policies, making it suitable for complex nonlinear optimization problems. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an aerodynamic optimization method for flat steel box girder nozzles based on deep reinforcement learning, which can effectively improve the aerodynamic performance and optimization efficiency of the nozzles.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for aerodynamic optimization of flat steel box girder nozzles based on deep reinforcement learning includes the following steps:

[0007] Step 1: Design the objective function. The objective function is constrained by the limited range of the nozzle shape design variables and includes aerodynamic performance indicators to evaluate the optimization effect.

[0008] Step 2: Establish a simulation environment. The environment uses computational fluid dynamics numerical simulation to calculate the aerodynamic response of the flat steel box girder after the shape of the nozzle changes, and to obtain the designed aerodynamic performance indicators.

[0009] Step 3: Establish the intelligent agent. The intelligent agent adopts a deep reinforcement learning dual model, including a policy model and a value function model. The environment is a numerical simulation of the fluid dynamics of the main girder cross-section of the box girder. The policy model uses the design variables of the nozzle shape as the state, and selects the action to be executed based on the environment and reward value, generating new nozzle shape design variables. The value function model uses the linear combination of objective function values ​​caused by changes in the nozzle shape as the reward function, and evaluates the value of the action strategy selected by the policy simulation. Initialize the parameters of the policy model and the value function model.

[0010] Step 4: Train the agent and calculate the loss function between the value estimate of the value function model and the target value;

[0011] Step 5: Determine if the loss function is less than the set threshold: if yes, obtain the optimal strategy and proceed to Step 7; if no, proceed to Step 6.

[0012] Step 6: Update the parameters of the strategy model and value function model, and then execute Step 2;

[0013] Step 7: Obtain the design variables for the nozzle shape using the optimal strategy.

[0014] Furthermore, in step one, the objective function is:

[0015]

[0016] The constraints are:

[0017]

[0018]

[0019] Where I(θ) represents the objective function; θ represents the parameters of the objective function; C l and c m θ1 and θ2 represent the lift coefficient and lift moment coefficient of the box girder, respectively; λ represents the proportionality coefficient; θ1 represents the angle between the upper inclined surface of the vent and the top surface of the box girder; θ2 represents the angle between the lower inclined surface of the vent and the bottom surface of the box girder.

[0020] Furthermore, in step two, the reward function is obtained by linearly weighted summing of the values ​​of each aerodynamic parameter.

[0021] Furthermore, the reward function is a linear combination of the objective function values:

[0022] r = -|C l |-λ·|C m |

[0023] Where r represents the reward function; C l and C mλ represents the lift coefficient and lift moment coefficient of the box girder, respectively; λ represents the proportionality coefficient.

[0024] Furthermore, in step three, the value estimate of the value function model is:

[0025]

[0026] Where y represents the value estimate of the value function model; r represents the immediate reward obtained after performing action a′ in state s′; and γ represents the discount factor. s represents the output of the target network at the next time step; s represents the state; s′ represents the state at the next time step; a represents the action; a′ represents the action at the next time step; These are the parameters of the value function model.

[0027] Furthermore, in step four, the loss function is expressed as:

[0028]

[0029] Where Loss represents the loss function; y represents the value estimate of the value function model; This represents the target value.

[0030] Furthermore, in step six, the parameters of the policy model are updated using a DNN optimizer:

[0031]

[0032]

[0033] Where θ represents the parameters of the policy model; α represents the learning rate of the policy model. This represents the gradient with respect to the policy model parameters θ; N represents the number of samples in the experience pool. The gradient of action a; s represents the gradient with respect to the policy model parameters θ; s represents the state.

[0034] Furthermore, in step six, the parameters of the value function model are updated using the DDPG algorithm:

[0035]

[0036]

[0037] in, β represents the parameters of the value function model; β represents the learning rate of the value function model. Indicates the parameter The gradient; loss represents the loss function, used to measure the difference between the predicted value y and the actual value. The difference between them; y represents the value estimate of the value function model; s represents the output of the target network at the current time step; s represents the state; a represents the action.

[0038] Furthermore, step two includes the following steps:

[0039] (1) Parametric description of the cross section of the main girder of the box girder. Each shape of the cross section of the main girder of the box girder is represented by a set of parameters; the cross section shape of the main girder of the box girder is obtained by a set of parameters composed of new air nozzle shape design variables.

[0040] (2) The cross-sectional shape of the box girder is divided into grids to generate a flow field grid model. The flow field grid model is numerically simulated to update the fluid dynamics numerical simulation environment and obtain new aerodynamic parameters.

[0041] The beneficial effects of this invention are as follows:

[0042] This invention presents an aerodynamic optimization method for flat steel box girder nozzles based on deep reinforcement learning. First, an aerodynamic optimization model for box girder nozzles is constructed, with the aerodynamic performance of the main girder cross-section as the objective and the constraints on the nozzle shape design variables as the constraints. Then, an agent is constructed, comprising a strategy model and a value function model. The agent uses the fluid dynamics numerical simulation environment of the main girder cross-section as its environment, the nozzle shape design variables as its state, and the box girder nozzle aerodynamic optimization model to construct a reward function. The strategy model selects actions to execute based on the current environment and reward function to generate new nozzle shape design variables. Simultaneously, the value function model evaluates the value of the current strategy. The agent is trained using the loss function of the value evaluation to ultimately obtain the optimal strategy. The nozzle shape design variables obtained with the optimal strategy are the optimal design variables that satisfy the objective function and constraints. In other words, this invention, based on deep reinforcement learning, achieves aerodynamic optimization design of flat steel box girder nozzles by building a deep reinforcement learning model. Compared with traditional supervised learning, deep reinforcement learning has characteristics such as autonomous learning strategies and maximizing long-term rewards, making it closer to an intelligent optimization method.

[0043] The present invention also has the following advantages:

[0044] (1) This invention solves the problem that traditional aerodynamic optimization methods for flat steel box girder nozzles cannot accurately optimize the target. Traditional methods have certain limitations in the aerodynamic optimization of flat steel box girder nozzles, while this invention can perform aerodynamic optimization more accurately and maximize the optimization target.

[0045] (2) This invention solves the problem of time-consuming and inefficient manual adjustment of control parameters. By employing a deep reinforcement learning-based method, this invention achieves autonomous learning and intelligent control of nozzle shape design variables, thereby improving control efficiency and reducing the need for manual intervention. Attached Figure Description

[0046] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following figures are provided for illustration:

[0047] Figure 1 This is a flowchart of the aerodynamic optimization method for flat steel box girder nozzles based on deep reinforcement learning, as described in this invention.

[0048] Figure 2 This is a schematic diagram of the aerodynamic optimization method for flat steel box girder nozzles based on deep reinforcement learning, as proposed in this invention.

[0049] Figure 3 A schematic diagram of the design space for the main beam section of the box girder;

[0050] Figure 4 This is a grid division diagram of the main beam section of the box girder;

[0051] Figure 5 A schematic diagram showing the design parameters for the nozzle shape;

[0052] Figure 6 The aerodynamic performance optimization curves are shown for different phase lengths.

[0053] Figure 7 The surrounding pressure cloud map of the main beam section of the box girder before and after the aerodynamic optimization of the air nozzle. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0055] like Figure 1-2 As shown in the figure, the aerodynamic optimization method for the air nozzle of the flat steel box girder based on deep reinforcement learning in this embodiment includes the following steps.

[0056] Step 1: Using the aerodynamic performance index of the main beam section of the box girder as the objective function and the limitation range of the nozzle shape design variables as the constraint condition, construct the aerodynamic optimization model of the box girder nozzle.

[0057] Specifically, in this embodiment, the objective function is:

[0058]

[0059] The constraints are:

[0060]

[0061]

[0062] Where I(θ) represents the objective function; θ represents the parameters of the objective function; C l and C m θ1 represents the lift coefficient and lift moment coefficient of the box girder, respectively; λ represents the proportionality coefficient, which is 0.2 in this embodiment; θ1 represents the angle between the upper inclined surface of the vent and the top surface of the box girder; θ2 represents the angle between the lower inclined surface of the vent and the bottom surface of the box girder. The design space of the main beam section of the box girder is as follows: Figure 3 As shown, in this embodiment, parameters θ1 and θ2 are the design variables for the shape of the air nozzle, therefore the dimension of this problem is n = 2. When θ1 and θ2 are controlled within a certain range, the two rays will intersect at a point, thus forming a closed box girder main beam cross-section, as shown. Figure 4 As shown. Changing the values ​​of θ1 and θ2 will result in corresponding changes in the nozzle angle, shape, and main beam width.

[0063] Step Two: Establish a simulation environment using computational fluid dynamics (CFD) numerical simulation to calculate the aerodynamic response of the flat steel box girder after the nozzle shape change, and obtain the designed aerodynamic performance indicators. The CFD numerical simulation includes the following steps:

[0064] (1) Parametric description of the cross section of the main girder of the box girder. Each shape of the cross section of the main girder of the box girder is represented by a set of parameters. Under the condition that other parameters of the cross section of the main girder of the box girder remain unchanged, the cross section shape of the main girder of the box girder is obtained by combining a set of parameters composed of new air nozzle shape design variables.

[0065] (2) The cross-sectional shape of the box girder is meshed to generate a flow field mesh model. Numerical simulation of the flow field mesh model is then performed to update the fluid dynamics numerical simulation environment, and new aerodynamic parameters are obtained, such as... Figure 6 As shown. Specifically, in this embodiment, computational fluid dynamics is used to perform numerical simulation of the flow around the main beam section of the box girder. The main governing equation is the Reynolds-averaged Navier-Stokes equations. A turbulence model is used to generate a flow field mesh model. An open-source solver is used to solve the mathematical model of the flow field mesh model to update the fluid dynamics numerical simulation environment and obtain new aerodynamic parameters.

[0066] Step 3: Establish the agent. The agent employs a deep reinforcement learning dual-model approach, comprising a policy model (Actor) and a value function model (Critic). The environment is a numerical simulation of the fluid dynamics of the box girder main section. The policy model uses the design variables of the nozzle shape as its state, selecting actions based on the environment and reward values, and generating new nozzle shape design variables. The value function model uses a linear combination of objective function values ​​resulting from changes in the nozzle shape as its reward function, evaluating the value of the selected action strategy in the policy simulation. Initialize the policy model parameters θ0 and the value function model parameters.

[0067] In this embodiment, the policy model is used to output the action policy, while the value function model is used to estimate the value. Both models adopt an artificial neural network structure with two hidden layers. The policy model has 64 nodes in its hidden layer, and the value model has 128 nodes in its hidden layer. Both models use the ReLU (Rectified Linear Unit) activation function.

[0068] The reward function is obtained by linearly weighted summing of the aerodynamic parameter values. Specifically, in this embodiment, the reward function is a linear combination of the objective function values:

[0069] r = -|C l |-λ·|C m |

[0070] Where r represents the reward function; C l and C m λ represents the lift coefficient and lift moment coefficient of the box girder, respectively; λ represents the proportionality coefficient.

[0071] In this embodiment, the reward function is set to r = -I(θ).

[0072] The strategy model selects actions based on environment and reward values, generating new nozzle shape design variables; the value function model evaluates the value of the selected action strategy based on the strategy simulation. Specifically, the value estimate of the value function model is as follows:

[0073]

[0074] Where y represents the value estimate of the value function model; r represents the immediate reward obtained after performing action a′ in the current state s′; and γ represents the discount factor. s represents the output of the target network at the next time step; s represents the state; s′ represents the state at the next time step; a represents the action; a′ represents the action at the next time step; These are the parameters of the value function model.

[0075] Step 4: Train the agent and calculate the loss function between the value estimate of the value function model and the target value. For example... Figure 5 As shown, in this embodiment, the policy model is approximated using a deep neural network (parameter θ). Its input is the environment, which undergoes feature extraction and transformation through a series of layers in the neural network, ultimately outputting the numerical value of the action. During training, gradient ascent is used to maximize the expected cumulative reward value r to update the parameters of the policy model; the value function model also typically uses a deep neural network (parameter θ). The algorithm approximates the value of an action by taking its state and action as input. These are processed through a series of layers in a neural network for feature extraction and transformation, ultimately outputting a value estimate of the action. During training, the parameters of the value function model are updated by minimizing the error between the current policy and the target policy. Based on the state, action trajectory, and reward value obtained in each iteration, a loss function is constructed to optimize the policy model parameters θ and the value function model parameters. The optimization objective is to minimize the loss function. By using optimization algorithms, such as gradient descent (here, we are sampling the Adam algorithm, Adaptive Moment Estimation, or Adam), the parameters of the policy model and the value function model are updated to obtain a new model. In the DDPG algorithm, the mean squared error (MSE) is typically used as the loss function to measure the difference between the predicted value and the target value of the value function model. In this embodiment, the loss function is expressed as:

[0076]

[0077] Where Loss represents the loss function; y represents the value estimate of the value function model; This represents the target value.

[0078] Step 5: Determine if the loss function is less than the set threshold: if yes, obtain the optimal strategy and proceed to Step 7; if no, proceed to Step 6.

[0079] Step 6: Update the parameters of the strategy model and value function model, and then execute Step 2.

[0080] In this embodiment, the parameters of the policy model are updated using a DNN optimizer:

[0081]

[0082]

[0083] Where θ represents the parameters of the policy model; α represents the learning rate of the policy model. The gradient represents the gradient with respect to the policy model parameters θ, used to adjust the parameters to maximize or minimize a certain objective function; N represents the number of samples in the experience pool. The gradient of action a; s represents the gradient with respect to the policy model parameters θ; s represents the state.

[0084] In this embodiment, the DDPG algorithm is used to update the parameters of the value function model:

[0085]

[0086]

[0087] in, β represents the parameters of the value function model; β represents the learning rate of the value function model. Indicates the parameter The gradient is used to adjust the parameters to minimize the loss function; loss represents the loss function, which measures the difference between the predicted value y and the actual value y. The difference between them; y represents the value estimate of the value function model; s represents the output of the target network at the current time step; s represents the state; a represents the action.

[0088] Step 7: Obtain the design variables for the nozzle shape using the optimal strategy, such as... Figure 7 As shown, this is a contour map of the surrounding pressure on the main beam section of the box girder before and after aerodynamic optimization of the nozzle.

[0089] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for aerodynamic optimization of a flat steel box girder wind fairing based on deep reinforcement learning, characterized in that: Includes the following steps: Step 1: Design the objective function. The objective function is constrained by the limited range of the nozzle shape design variables and includes aerodynamic performance indicators to evaluate the optimization effect. Step 2: Establish a simulation environment. The environment uses computational fluid dynamics numerical simulation to calculate the aerodynamic response of the flat steel box girder after the shape of the nozzle changes, and to obtain the designed aerodynamic performance indicators. Step 3: Establish an intelligent agent. The intelligent agent adopts a deep reinforcement learning dual model, including a policy model and a value function model. The policy model uses the design variable of the nozzle shape as the state, selects the action to be executed based on the environment and reward value, and generates a new nozzle shape design variable. The value function model uses the linear combination of the objective function value caused by the change of the nozzle shape as the reward function, and evaluates the value of the action policy selected by the policy simulation. Initialize the policy model parameters and the value function model parameters. Step 4: Train the agent and calculate the loss function between the value estimate of the value function model and the target value; Step 5: Determine if the loss function is less than the set threshold: if yes, obtain the optimal strategy and proceed to Step 7; if no, proceed to Step 6. Step 6: Update the parameters of the strategy model and value function model, and then execute Step 2; Step 7: Obtain the design variables for the nozzle shape using the optimal strategy; In step one, the objective function is: The constraints are: in, Represent the objective function; Indicate the parameters of the objective function; and These represent the lift coefficient and lift moment coefficient of the box girder, respectively. Indicates the proportionality coefficient; This indicates the angle between the upper inclined surface of the air nozzle and the top surface of the box girder; This indicates the angle between the lower inclined surface of the air nozzle and the bottom surface of the box girder; In step three, the reward function is obtained by linearly weighted summation of the values ​​of each aerodynamic parameter. The reward function is a linear combination of the objective function values: in, Represents the reward function; and These represent the lift coefficient and lift moment coefficient of the box girder, respectively. Indicates the proportionality coefficient; In step three, the value estimate of the value function model is: in, This represents the value estimate from the value function model; Indicates the state Next action The immediate reward obtained afterward; Indicates the discount factor; It is the output of the target network in the next time step; Indicates state; Indicates the state of the next time step; Indicates an action; Indicates the action of the next time step; These are the parameters of the value function model; In step four, the loss function is expressed as: in, Represents the loss function; This represents the value estimate from the value function model; Indicates the target value; In step six, the parameters of the policy model are updated using a DNN optimizer: in, These are the parameters of the strategy model; The learning rate of the policy model; Indicates the parameters of the policy model The gradient; This indicates the number of samples in the experience pool; Action The gradient; Indicates the parameters of the policy model The gradient; Indicates state; In step six, the parameters of the value function model are updated using the DDPG algorithm: in, These are the parameters of the value function model; The learning rate represents the value function model; Indicates the parameter The gradient; This represents the loss function, used to measure the predicted value. and actual value The differences between them; This represents the value estimate from the value function model; This represents the output of the target network at the current time step; Indicates state; Indicates an action; Step two includes the following steps: (1) Parametric description of the cross section of the main beam of the box girder. Each shape of the cross section of the main beam of the box girder is represented by a set of parameters; the cross section shape of the main beam of the box girder is obtained by a set of parameters composed of new air nozzle shape design variables. (2) The cross-sectional shape of the box girder is divided into grids to generate a flow field grid model. The flow field grid model is numerically simulated to update the fluid dynamics numerical simulation environment and obtain new aerodynamic parameters.

Images