A method for suppressing unsteady pressure pulsation in an air intake based on deep reinforcement learning

Abstract

The application discloses an inlet non-steady pressure fluctuation suppression method based on deep reinforcement learning, and relates to the technical field of flow control. The application utilizes the real-time interaction and online learning of a deep reinforcement learning intelligent agent and a high-fidelity CFD environment, designs a compound reward function containing an adaptive weight factor and a strict physical constraint system, enables the controller to dynamically perceive the flow field state, automatically adjusts the control target and intensity, and thus realizes efficient and robust pressure fluctuation suppression under wide working conditions, significantly improves the adaptability, response speed and final suppression effect of the control, and guarantees the safety and engineering feasibility of the control process.

Description

Technical Field

[0001] This invention relates to the field of flow control technology, and in particular to a method for suppressing unsteady pressure pulsations in the inlet based on deep reinforcement learning. Background Technology

[0002] The air intake is a key aerodynamic component at the front end of a high-speed aircraft propulsion system. Its main function is to decelerate, pressurize, and guide high-speed incoming air (usually supersonic) to the engine with minimal flow losses. Through a series of complex internal flow channels and shock wave systems, the air intake effectively converts the kinetic energy of the air into pressure energy, providing a stable, uniform airflow with high total pressure recovery to the engine's compressor or combustion chamber. Its performance directly determines the engine's normal operation and the efficiency of the entire propulsion system. At specific flight Mach numbers and angles of attack, the air intake enters an unstable operating mode known as "surge" or "buzzing." This unstable flow manifests as strong self-excited oscillations, physically stemming from the complex, unsteady coupling between the shock wave train within the air intake and the downstream pressure (i.e., back pressure). This process is accompanied by severe, periodic pressure pulsations, i.e., unsteady pressure pulsation loads.

[0003] This unsteady pressure pulsation load poses an extremely severe challenge to high-speed aircraft: high-frequency, high-amplitude pressure pulsations can induce severe vibrations in the inlet duct panels and their supporting structures, leading to material fatigue, shortening component lifespan, and even causing structural damage in extreme cases; pressure pulsations propagate downstream, disrupting the intake stability of the combustion chamber, potentially causing combustion oscillations, or even engine shutdown, posing a direct threat to flight safety; strong pulsation loads may also be transmitted to the aircraft airframe, causing a decline in aircraft handling qualities and affecting its attitude control and passenger comfort.

[0004] To suppress these harmful oscillations within the air intake, researchers both domestically and internationally have proposed various flow control methods, mainly categorized into passive control and active control. Passive control techniques involve incorporating passive devices such as cavities and vortex generators into the air intake wall. While these methods are simple in structure and highly reliable, their control effects are typically fixed, exhibiting good performance only near the design point. They cannot adapt to a wide range of flight conditions, and their effectiveness in suppressing strongly nonlinear and unsteady shock wave oscillations is often limited, sometimes even introducing additional flow losses. Existing active control methods typically rely on controller designs based on physical models, such as classical PID control or modern control theory methods based on linearized models. Implementation involves creating vents or installing actuators on the lower wall of the air intake to disturb the flow field by adjusting the venting rate or jet flow in real time. However, these traditional active control methods have the following limitations: the flow in a supersonic air intake is highly nonlinear and unsteady, making it difficult to describe using precise, low-order mathematical models suitable for real-time control. Controllers based on simplified model designs are insufficient in robustness and control accuracy when facing real complex flows. Traditional control strategies are usually designed for steady-state or slowly changing conditions. For the rapid self-excited oscillations of shock trains, their response speed and control bandwidth are difficult to meet the requirements, resulting in control lag and possibly exacerbating the oscillations. Control laws (such as PID parameters) are fixed once set, and cannot learn and self-optimize online. They cannot adapt to continuous changes in flight conditions (such as Mach number and angle of attack) and lack intelligence. Summary of the Invention

[0005] The purpose of this invention is to provide a method for suppressing unsteady pressure pulsation in the air intake based on deep reinforcement learning, so as to improve the technical problems of traditional control methods, which rely on simplified mathematical models and have poor control effect, slow response and inability to adapt to changing operating conditions when dealing with highly nonlinear and unsteady air intake surge.

[0006] To achieve the above-mentioned objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A method for suppressing unsteady pressure fluctuations in the inlet based on deep reinforcement learning, comprising:

[0008] The controller is set up and based on the mesh file of the intake duct, the unsteady self-sustaining oscillation state is simulated by CFD simulation method and numerical solver to generate the reference pressure;

[0009] Construct a TD3 reinforcement learning model and set up a state vector and an intake control constraint system;

[0010] Based on the baseline pressure and intake control constraint system, combined with the composite adaptive reward function, the intake is suppressed by the TD3 reinforcement learning model.

[0011] Furthermore, the TD3 reinforcement learning model includes an Actor network, a first Critic network, and a second Critic network; the Actor network, the first Critic network, and the second Critic network all adopt fully connected neural networks, which include an input layer, a first hidden layer, a second hidden layer, and an output layer.

[0012] In the above-described scheme, this invention integrates the TD3 model (dual-Critic network structure) from deep reinforcement learning with a high-fidelity CFD simulation environment in a closed loop (features). It utilizes real-time interactive data between the agent and the complex flow field to learn control strategies, thus completely eliminating the reliance on precise, simplified fluid dynamics mathematical models. This results in adaptive and robust control capabilities, effectively addressing the highly nonlinear and unsteady inlet surge problem that traditional methods struggle to handle. Furthermore, the design of its composite reward function and physical constraint system ensures that the controller can automatically adhere to engineering safety boundaries while rapidly suppressing oscillations, achieving a balance between intelligence, efficiency, and safety. This provides a novel technical path for solving the cutting-edge challenge of wide-condition, adaptive flow control in high-speed aircraft inlets.

[0013] Furthermore, the generation of the reference pressure includes:

[0014] The mesh file of the air intake is collected and input into the numerical solver. Unsteady numerical simulation parameters are set to construct the unsteady numerical simulation environment for the current time step. The unsteady numerical simulation parameters include the calculation format, boundary conditions, total number of time steps, number of iteration steps in each time step, and report file definition for pressure monitoring points.

[0015] Based on the initial flow field data of the previous time step, the flow field simulation of the intake is carried out in an unsteady numerical simulation environment. The unsteady pressure values ​​of the pressure monitoring points are calculated empirically until the amplitude difference between two adjacent cycles meets the stopping condition, and the initial flow field data is updated.

[0016] A pressure outlet is set in a specific area of ​​the inlet compression ramp based on boundary conditions. The unsteady pressure value of the pressure outlet is dynamically updated by external commands through programming and defined as a variable pressure controller.

[0017] Based on the updated initial flow field data, the mean unsteady pressure value of all points in the wall region corresponding to the variable pressure controller is calculated and used as the reference pressure of the variable pressure controller.

[0018] Furthermore, the setting of the state vector and the intake duct control constraint system includes:

[0019] Based on the updated initial flow field data, the pressure pulses at each pressure monitoring point are analyzed to determine the core monitoring point at the current time step.

[0020] Based on the actions performed in the previous four time steps and the pressure coefficients of the core monitoring points, construct the state vector for the current time step;

[0021] Based on the physical limitations and engineering safety of the air intake, an air intake control constraint system is constructed; the air intake control constraint system includes absolute pressure constraint, pressure change constraint and oscillation energy constraint.

[0022] In the above-described scheme, this invention dynamically selects the core monitoring point most sensitive to surge and constructs a time-series state vector that integrates action and pressure history. This enables the agent to accurately perceive the phase and evolution trend of oscillations, solving the problems of misjudgment of control timing and slow response caused by incomplete or delayed state information in traditional controllers. Simultaneously, a constraint system containing multiple physical limitations is established, transforming prior engineering knowledge into safety rules that the agent must follow. This ensures strong robustness and engineering safety in the control process, fundamentally avoiding system instability or structural damage that may be caused by the agent exploring unsafe actions. This achieves an organic integration of data-driven control and engineering safety boundaries.

[0023] Furthermore, the suppression of the air intake through the TD3 reinforcement learning model includes:

[0024] Initialize the enhanced environment and experience playback buffer, and set the initial absolute back pressure of the variable pressure controller to the reference pressure;

[0025] The state vector of the current time step is output to the Actor network for processing to generate the action of the current time step;

[0026] Calculate the corresponding absolute back pressure based on the initial value and action of the variable pressure controller at the current time step;

[0027] Based on the absolute back pressure at the current time step, the boundary conditions of the flow field are updated using CFD simulation, the flow field data is updated, and the unsteady pressure values ​​at the core monitoring points are collected to calculate the corresponding pressure coefficients.

[0028] Based on the intake control constraint system, the absolute back pressure and pressure coefficient at the current time step, a composite adaptive reward function is calculated; the composite adaptive reward function includes target tracking reward, motion smoothing penalty, motion amplitude penalty and constraint violation penalty;

[0029] Based on the pressure coefficient and action at the current time step, the state vector is updated, and the TD3 reinforcement learning model is updated by combining the composite adaptive reward function.

[0030] Furthermore, the calculation of the composite adaptive reward function includes:

[0031] Obtain the first historical oscillation energy and the second historical oscillation energy, and calculate the historical pressure coefficient mean, first standard deviation and second standard deviation; based on the first historical oscillation energy, the second historical oscillation energy, the first standard deviation and the second standard deviation, calculate the adaptive weighting factor;

[0032] Calculate the target tracking reward based on the historical average pressure coefficient, adaptive weighting factor, and pressure coefficient at the current time step.

[0033] Calculate the motion smoothing penalty based on the absolute back pressure of the current time step and the absolute back pressure of the previous time step.

[0034] Calculate the motion amplitude penalty based on the absolute back pressure and reference pressure at the current time step;

[0035] Based on the pressure boundary value and the absolute back pressure at the current time step, the constraint violation penalty is calculated using an indicator function.

[0036] The composite adaptive reward function for the current time step is calculated based on target tracking reward, action smoothing penalty, action amplitude penalty, and constraint violation penalty.

[0037] In the above scheme, the composite adaptive reward function introduces a dynamic weighting factor, enabling the agent to autonomously adjust the focus of the control objective based on the real-time intensity of the flow field oscillations. This solves the problem of rigid control objectives and insufficient adaptability of traditional fixed reward functions when facing highly unsteady, multimodal surge processes. The composite structure integrating multi-objective constraints encourages effective oscillation suppression while comprehensively guiding and penalizing the smoothness, economy, and safety of control actions. This encodes complex engineering optimization objectives into a single reward signal, enabling the controller to achieve both efficient suppression and safe, robust intelligent decision-making capabilities. It overcomes the common problem of inefficient, dangerous, or convergent strategies often caused by improper reward signal design in complex physical systems using purely data-driven methods, achieving a balance between control performance and engineering practicality.

[0038] Furthermore, the updated TD3 reinforcement learning model includes:

[0039] Based on the pressure coefficient and action at the current time step, update the state vector to obtain the updated state vector;

[0040] Store the updated state vector and the composite adaptive reward function of the current time step into the experience replay buffer;

[0041] The updated state vector is input into the Actor network to generate the updated action vector;

[0042] The updated state vector and the updated action vector are input into the first Critic network and the second Critic network respectively to generate the corresponding first target Q value and second target Q value;

[0043] Based on the first objective Q value and the second objective Q value, with the goal of minimizing the mean squared error loss function, the network parameters of the first Critic network and the second Critic network are updated respectively by gradient descent.

[0044] The gradient strategy of the Actor network is calculated to maximize the first objective Q value, and the network parameters of the Actor network are updated through a gradient-based optimization algorithm.

[0045] The network parameters of the target Critic network and the target Actor network are updated using a soft update method.

[0046] Furthermore, the Actor network is updated every two time steps.

[0047] In the above-described scheme, this invention effectively solves the common problems of training instability, policy oscillation, and difficulty in convergence that are prevalent in deep reinforcement learning when applied to high-dimensional, nonlinear dynamic systems. It alleviates Q-value overestimation through conservative value estimation, avoids mutual interference between the policy and value functions through delayed updates, and ensures the stability of the learning objective through soft updates. Together, these measures guarantee that the agent can efficiently and reliably learn the optimal control policy that balances performance and robustness from high-dimensional data generated by interacting with complex flow fields in a safe simulation environment. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the method in an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] Please see Figure 1 This embodiment provides a method for suppressing unsteady pressure pulsations in the air intake based on deep reinforcement learning. Figure 1 The execution entity of the method shown can be a software and / or hardware device. The execution entity of this application can include, but is not limited to, at least one of the following: user equipment, network equipment, etc. User equipment can include, but is not limited to, computers, smartphones, personal digital assistants (PDAs), and the aforementioned electronic devices. Network equipment can include, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers. Cloud computing is a type of distributed computing, consisting of a super virtual computer composed of a group of loosely coupled computers. This embodiment does not impose any limitations on this.

[0052] A method for suppressing unsteady pressure fluctuations in the air intake based on deep reinforcement learning includes:

[0053] S1. Set up the controller and, based on the mesh file of the intake duct, simulate the unsteady self-sustaining oscillation state using CFD simulation method and numerical solver to generate the reference pressure;

[0054] S1 includes:

[0055] S1-1. Collect the mesh file of the air intake and input it into the CFD solver (numerical solver), set the unsteady numerical simulation parameters, and construct the unsteady numerical simulation environment for the current time step; the unsteady numerical simulation parameters include the calculation format, boundary conditions, total number of time steps and number of iteration steps in each time step, and the definition of the report file for pressure monitoring points;

[0056] S1-2. Based on the initial flow field data of the previous time step, the flow field simulation of the intake is carried out in an unsteady numerical simulation environment. The unsteady pressure value of the pressure monitoring point is calculated by empirical playback until the amplitude difference between two adjacent cycles meets the stopping condition, and the initial flow field data is updated.

[0057] Specifically, in the CFD software, an unsteady calculation model of the inlet is configured using the initial flow field data from the previous time step. By adjusting parameters such as back pressure, the flow field calculation continues until it triggers and enters a "surge" state. Using the pressure pulsation curves from real-time pressure monitoring points, when the pressure pulsation curves show stable periodicity in the pressure oscillations, and the difference in oscillation amplitude between two consecutive periods is less than 5% (stopping condition), it is determined that the flow field has reached a "stable self-sustaining oscillation state." Subsequently, the calculation is immediately interrupted, and the complete flow field data containing the physical information of all grid points at this moment is saved along with the simulation settings file, thus updating the initial flow field data.

[0058] S1-3. Based on boundary conditions, a pressure outlet is set in a specific area of ​​the inlet compression ramp. The unsteady pressure value of the pressure outlet can be dynamically updated by external commands through programming, which is defined as a variable pressure controller.

[0059] S1-4. Based on the updated initial flow field data, calculate the average unsteady pressure value of all points in the wall region corresponding to the variable pressure controller, and use it as the reference pressure of the variable pressure controller.

[0060] Specifically, CFD software is used to read the updated initial flow field data. This updated initial flow field data includes complete physical quantity information of all grid points at a specific instant (i.e., the time of data storage) when the flow field reaches a "steady self-sustaining oscillation state." Since the variable pressure controller is located in a specific region of the inlet compression ramp, the corresponding wall region on the inlet compression ramp is precisely selected using coordinate range or surface selection tools. This wall region is represented on the grid as a set of grid surfaces. For this selected set of grid surfaces, the static pressure value (unsteady pressure value) at the center point (or node) of each grid surface is extracted. This is a batch data reading operation, resulting in a data list containing N pressure values ​​(N equals the number of grid surfaces in the region). The average pressure value of all pressure values ​​in this wall region is calculated and used as the reference pressure for the variable pressure controller.

[0061] S2. Construct the TD3 reinforcement learning model and set up the state vector and intake control constraint system;

[0062] The TD3 reinforcement learning model includes an Actor network, a first Critic network, and a second Critic network. The Actor network, the first Critic network, and the second Critic network all adopt fully connected neural networks, which include an input layer, a first hidden layer, a second hidden layer, and an output layer.

[0063] The Actor network is responsible for selecting actions based on the state vector. The input to its input layer is the state vector. The first and second hidden layers each contain 256 neurons and use the ReLU activation function. The output of the output layer is the action and uses the Tanh activation function to restrict the output action to the range [-1, 1].

[0064] The first and second Critic networks have identical structures and are used to evaluate the value of state-action pairs to mitigate overestimation problems. The input layer of the first Critic network takes state-action pairs as input, and both its first and second hidden layers consist of 256 neurons using the ReLU activation function. The output layer consists of a single neuron.

[0065] S2 includes:

[0066] S2-1. Based on the updated initial flow field data, analyze the pressure pulses at each pressure monitoring point to determine the core monitoring point at the current time step;

[0067] S2-2. Based on the actions performed in the previous four time steps and the pressure coefficients of the core monitoring points, construct the state vector for the current time step. The corresponding expression is:

[0068] ;

[0069] in, , , , They represent the time steps respectively. , , , The action to be performed , , , They represent the time steps respectively. , , , The pressure coefficient below.

[0070] S2-3. Based on the physical limitations and engineering safety of the air intake, an air intake control constraint system is constructed; the air intake control constraint system includes absolute pressure constraint, pressure change constraint, and oscillation energy constraint.

[0071] The absolute pressure constraint means that the absolute back pressure of the variable pressure controller must be within the allowable range to protect the intake structure. The corresponding formula is:

[0072] ;

[0073] The back pressure change between adjacent time steps should not be too rapid to prevent impact on the intake duct or numerical instability. Therefore, the formula corresponding to the pressure change constraint is:

[0074] ;

[0075] Pressure pulsation should be controlled below a certain level to ensure intake stability. Therefore, the formula corresponding to the oscillation energy constraint is:

[0076] ;

[0077] in, Indicates the current time step The absolute back pressure of the variable pressure controller , These represent the minimum and maximum values ​​of the absolute back pressure, respectively. Represents absolute value. Indicates the previous time step The absolute back pressure of the variable pressure controller Indicates the maximum permissible rate of pressure change. This represents the summation function. Indicates the first Pressure coefficient at each time step Indicates the maximum allowable oscillation amplitude. This represents the average pressure coefficient within the sliding window. This indicates the size of the sliding window. In this embodiment, , The values ​​were taken as 5000 Pa and 20000 Pa, respectively. The value is 1000 Pa. The value is 0.01. The value is 50.

[0078] S3. Based on the reference pressure and intake control constraint system, combined with the composite adaptive reward function, the intake is suppressed by the TD3 reinforcement learning model.

[0079] S3 includes:

[0080] S3-1. Initialize the enhanced environment and experience playback buffer, and set the initial absolute back pressure of the variable pressure controller to the reference pressure;

[0081] S3-2. Output the state vector of the current time step to the Actor network for processing to generate the action of the current time step;

[0082] Specifically, the state vector at the current time step is used as input. It is first passed through the input layer to the first hidden layer, which contains 256 neurons. The input data is linearly weighted and summed, and a ReLU activation function is applied to introduce a non-linear feature transformation. The output of the first hidden layer is then fed into a second hidden layer with the same structure (also containing 256 neurons and using ReLU activation) for deeper feature abstraction and information integration. Finally, the processed signal reaches the output layer. After a linear transformation, the Tanh activation function strictly limits the final output value to the continuous interval [-1, 1]. This value represents the normalized action at the current time step. This forward computation process is executed once in real time at each time step, mapping the observed state to continuous control commands. The internal weights and bias parameters are continuously updated during training to learn the optimal control strategy.

[0083] S3-3. Based on the initial value and action of the absolute back pressure of the variable pressure controller at the current time step, calculate the corresponding absolute back pressure using the following formula:

[0084] ;

[0085] in, Indicates the action at the current time step. This represents the initial absolute back pressure value of the variable pressure controller at the current time step. Indicates absolute back pressure. Indicates the magnification factor; in this embodiment, the magnification factor takes the value... It is 200.

[0086] S3-4. Based on the absolute back pressure at the current time step, update the boundary conditions of the flow field using CFD simulation, update the flow field data using the same method as S1, collect the unsteady pressure values ​​at the core monitoring points, and calculate the corresponding pressure coefficients.

[0087] Specifically, the flow field converged in the previous time step is loaded as the initial state through the CFD solver, and the absolute back pressure of the current time step is read through a user-defined function to dynamically update the pressure outlet boundary conditions at the inlet compression surface (variable pressure controller). Subsequently, using the same unsteady numerical simulation method as S1, a flow field calculation for one time step is performed with these new boundary conditions to obtain the updated transient flow field. The pressure coefficients at the core monitoring points are collected from this updated flow field and automatically calculated and output by the CFD software, completing the closed loop from control command execution to flow field state observation, as well as the interaction between the reinforcement learning and CFD simulation methods used to construct the environment.

[0088] pressure coefficient The corresponding formula is:

[0089] ;

[0090] in, This indicates the static pressure value at the core monitoring point. , , These represent the far-field incoming static pressure, far-field incoming density, and far-field incoming velocity, respectively.

[0091] S3-5. Based on the intake control constraint system, the absolute back pressure and pressure coefficient of the current time step, calculate the composite adaptive reward function; the composite adaptive reward function includes target tracking reward, motion smoothing penalty, motion amplitude penalty and constraint violation penalty;

[0092] S3-5 includes:

[0093] S3-5-1. Obtain the first historical oscillation energy and the second historical oscillation energy, and calculate the historical pressure coefficient mean, first standard deviation and second standard deviation; based on the first historical oscillation energy, the second historical oscillation energy, the first standard deviation and the second standard deviation, calculate the adaptive weighting factor;

[0094] First historical oscillation energy This represents the average oscillation energy across all historical time steps. The second historical oscillation energy... This is the average oscillation energy over the most recent M historical time steps. The average historical pressure coefficient is the average historical pressure coefficient over all historical time steps. The first standard deviation... The standard deviation of the historical pressure coefficient across all historical time steps. The second standard deviation. This represents the standard deviation of the historical pressure coefficient over the most recent M historical time steps. In this embodiment, M is set to 20.

[0095] Adaptive weighting factor The corresponding formula is:

[0096] ;

[0097] in, , These represent the first weighting coefficient and the second weighting coefficient, respectively.

[0098] S3-5-2. Calculate the target tracking reward based on the historical average pressure coefficient, adaptive weighting factor, and pressure coefficient at the current time step.

[0099] The target tracking reward The corresponding formula is:

[0100] ;

[0101] in, Indicates the target tracking weight. This represents the pressure coefficient at the current time step. This represents the historical average pressure coefficient (target pressure coefficient). In this embodiment, the target tracking weight... The value is 1000.

[0102] S3-5-3. Calculate the motion smoothing penalty based on the absolute back pressure of the current time step and the absolute back pressure of the previous time step.

[0103] The action smoothing penalty The corresponding formula is:

[0104] ;

[0105] in, This represents the motion smoothing weight. In this embodiment, the motion smoothing weight... The value is .

[0106] S3-5-4. Calculate the motion amplitude penalty based on the absolute back pressure and reference pressure at the current time step;

[0107] The action amplitude penalty The corresponding formula is:

[0108] ;

[0109] in, This represents the motion amplitude coefficient. In this embodiment, the motion amplitude coefficient... The value is .

[0110] S3-5-5. Based on the pressure boundary value and the absolute back pressure at the current time step, calculate the constraint violation penalty using an indicator function;

[0111] Penalty for violating the constraints The corresponding formula is:

[0112] ;

[0113] in, , , Indicates the constraint penalty weight, Indicates an indicator function, Indicates the pressure boundary value on the side that exceeds the limit (i.e. or ), This represents the oscillation energy at the current time step. This represents the maximum permissible rate of pressure change. In this embodiment, the constraint penalty weight... , , The values ​​are 10, 5, and 8 respectively.

[0114] When the absolute back pressure at the current time step is greater than or When it is less than, then The value is 1; otherwise, it is 0. And so on. and The value of is the same as that of absolute back pressure.

[0115] S3-5-6. Based on target tracking reward, action smoothing penalty, action amplitude penalty and constraint violation penalty, calculate the composite adaptive reward function for the current time step.

[0116] The composite adaptive reward function The corresponding formula is:

[0117] .

[0118] S3-6. Based on the pressure coefficient and action at the current time step, update the state vector and update the TD3 reinforcement learning model using the composite adaptive reward function; the updated TD3 reinforcement learning model is used for suppression in the next time step.

[0119] S3-6 includes:

[0120] S3-6-1. Based on the pressure coefficient and action at the current time step, update the state vector to obtain the updated state vector. ;

[0121] S3-6-2. Store the updated state vector and the composite adaptive reward function of the current time step into the experience replay buffer;

[0122] The updated state vector, along with the current time step's state vector, action, and composite adaptive reward function, form an empirical tuple. Then it is stored in the experience replay buffer. This indicates the end of the round; it is True when the time step reaches 500.

[0123] S3-6-3. Input the updated state vector into the Actor network to generate the updated action vector. ;

[0124] S3-6-4. Input the updated state vector and the updated action vector into the first Critic network and the second Critic network respectively to generate the corresponding first target Q value and second target Q value.

[0125] The formula corresponding to S3-6-4 is:

[0126] ;

[0127] in, Describes the minimum value function. Indicates the discount factor. This represents noise added to the target policy. Represents the target Critic network. Represents the target Actor network. This represents the Q-value of the first objective or the Q-value of the second objective.

[0128] S3-6-5. Based on the first objective Q value and the second objective Q value, with the goal of minimizing the mean square error loss function, the network parameters of the first Critic network and the second Critic network are updated respectively by the gradient descent method.

[0129] The mean square error loss function The corresponding formula is:

[0130] .

[0131] S3-6-6 Calculate the gradient strategy of the Actor network to maximize the first objective Q value, and update the network parameters of the Actor network through a gradient-based optimization algorithm;

[0132] In S3-6-6, according to the formula:

[0133] ;

[0134] Calculate the policy gradient of the Actor network .in, Indicates the Q-value for the action gradient, This indicates that the Actor network outputs actions with respect to its own parameters. The gradient is used. Since gradient-based optimization algorithms (Adam algorithm) are existing technology, the parameter update process will not be elaborated upon. Furthermore, in practical applications, the Actor network is updated every two time steps (delayed update).

[0135] S3-6-7. Update the network parameters of the target Critic network and the target Actor network using a soft update method. The corresponding update formula is:

[0136] ;

[0137] ;

[0138] in, Indicates the soft update coefficient. This represents the updated network parameters of either the first or second Critic network. This represents the network parameters of the target Critic network at the current time step. This represents the updated network parameters of the target Critic network. This represents the updated parameters of the Actor network. This represents the network parameters of the target Actor network at the current time step. This represents the updated network parameters of the target Actor network.

[0139] Furthermore, when training the TD3 reinforcement learning model, each training round is set to contain a fixed number of interaction steps (e.g., 500 unsteady time steps). After a training round ends, an automated script is called in the main reinforcement learning program to reset the computational state of the CFD solver and reload the initial flow field data saved in step S1. This ensures that each new training round starts from the exact same "stable surge" initial state, guaranteeing the consistency of training conditions and thus fairly evaluating the progress of the agent's policy.

[0140] The training process continues for multiple rounds until any of the following convergence conditions are met:

[0141] The reward curve is stable: the average reward value of the composite adaptive reward function in the most recent 100 consecutive training rounds changes by less than a set threshold (e.g., 1%), indicating that the policy performance of the TD3 reinforcement learning model has stabilized.

[0142] Control performance met the target: The TD3 reinforcement learning model's strategy achieved the preset performance indicators in actual control. For example, the pressure coefficient variance (i.e., oscillation energy) of the core monitoring point was reduced by more than 80% compared with the uncontrolled baseline state, proving that pressure pulsation has been effectively suppressed.

[0143] The TD3 reinforcement learning model is trained using round-robin training and environment reset to ensure the stability and reproducibility of the training process. Training convergence is determined based on clear quantitative indicators, resulting in a reliable final control strategy. The final control strategy selects the action with the highest long-term benefit at each moment based on the current state. The pressure values ​​corresponding to these actions are applied to the intake manifold, and these pressure values ​​are still "temporarily" determined by the host computer based on the states returned by the sensors.

[0144] In summary, this invention utilizes real-time interaction and online learning between a deep reinforcement learning agent and a high-fidelity CFD environment. By designing a composite reward function with adaptive weighting factors and a rigorous physical constraint system, the controller can dynamically perceive the flow field state and automatically adjust the control objective and intensity, thereby achieving efficient and robust pressure pulsation suppression under a wide range of operating conditions. Compared to traditional active control methods based on fixed models, this invention significantly improves the adaptability, response speed, and final suppression effect of the control, while ensuring the safety and engineering feasibility of the control process. It provides a novel intelligent solution for suppressing supersonic inlet flow instability.

[0145] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0146] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0147] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for suppressing unsteady pressure pulsations in the air intake based on deep reinforcement learning, characterized in that, include: The controller is set up and based on the mesh file of the intake duct, the unsteady self-sustaining oscillation state is simulated by CFD simulation method and numerical solver to generate the reference pressure; Construct a TD3 reinforcement learning model and set up a state vector and an intake control constraint system; Based on the baseline pressure and intake control constraint system, combined with the composite adaptive reward function, the intake is suppressed by the TD3 reinforcement learning model; The aforementioned setting of the state vector and intake control constraint system includes: Based on the updated initial flow field data, the pressure pulses at each pressure monitoring point are analyzed to determine the core monitoring point at the current time step. Based on the actions performed in the previous four time steps and the pressure coefficients of the core monitoring points, construct the state vector for the current time step; Based on the physical limitations and engineering safety of the air intake, an air intake control constraint system is constructed; the air intake control constraint system includes absolute pressure constraint, pressure change constraint and oscillation energy constraint.

2. The method for suppressing unsteady pressure pulsations in the air intake based on deep reinforcement learning according to claim 1, characterized in that, The generation of the reference pressure includes: The mesh file of the air intake is collected and input into the numerical solver. Unsteady numerical simulation parameters are set to construct the unsteady numerical simulation environment for the current time step. The unsteady numerical simulation parameters include the calculation format, boundary conditions, total number of time steps, number of iteration steps in each time step, and report file definition for pressure monitoring points. Based on the initial flow field data of the previous time step, the flow field simulation of the intake is carried out in an unsteady numerical simulation environment. The unsteady pressure value of the pressure monitoring point is calculated until the amplitude difference between two adjacent cycles meets the stopping condition, and the initial flow field data is updated. A pressure outlet is set in a specific area of ​​the inlet compression ramp based on boundary conditions. The unsteady pressure value of the pressure outlet is dynamically updated by external commands through programming and defined as a variable pressure controller. Based on the updated initial flow field data, the mean value of unsteady pressure at all points in the wall region corresponding to the variable pressure controller is calculated and used as the reference pressure for the variable pressure controller.

3. The method for suppressing unsteady pressure pulsations in the inlet based on deep reinforcement learning according to claim 1, characterized in that, The TD3 reinforcement learning model includes an Actor network, a first Critic network, and a second Critic network. The Actor network, the first Critic network, and the second Critic network all adopt fully connected neural networks, which include an input layer, a first hidden layer, a second hidden layer, and an output layer.

4. The method for suppressing unsteady pressure pulsations in the inlet based on deep reinforcement learning according to claim 3, characterized in that, The method of suppressing the air intake through the TD3 reinforcement learning model includes: Initialize the enhanced environment and experience playback buffer, and set the initial absolute back pressure of the variable pressure controller to the reference pressure; The state vector of the current time step is output to the Actor network for processing to generate the action of the current time step; Calculate the corresponding absolute back pressure based on the initial value and action of the variable pressure controller at the current time step; Based on the absolute back pressure at the current time step, the boundary conditions of the flow field are updated using CFD simulation, the flow field data is updated, and the unsteady pressure values ​​at the core monitoring points are collected to calculate the corresponding pressure coefficients. Based on the intake control constraint system, the absolute back pressure and pressure coefficient at the current time step, a composite adaptive reward function is calculated; the composite adaptive reward function includes target tracking reward, motion smoothing penalty, motion amplitude penalty and constraint violation penalty; Based on the pressure coefficient and action at the current time step, the state vector is updated, and the TD3 reinforcement learning model is updated by combining the composite adaptive reward function.

5. The method for suppressing unsteady pressure pulsations in the inlet based on deep reinforcement learning according to claim 4, characterized in that, The calculation of the composite adaptive reward function includes: Obtain the first and second historical oscillation energies, and calculate the historical pressure coefficient mean, first standard deviation and second standard deviation; Based on the first historical oscillation energy, the second historical oscillation energy, the first standard deviation, and the second standard deviation, the adaptive weighting factor is calculated. Calculate the target tracking reward based on the historical average pressure coefficient, adaptive weighting factor, and pressure coefficient at the current time step. Calculate the motion smoothing penalty based on the absolute back pressure of the current time step and the absolute back pressure of the previous time step. Calculate the motion amplitude penalty based on the absolute back pressure and reference pressure at the current time step; Based on the pressure boundary value and the absolute back pressure at the current time step, the constraint violation penalty is calculated using an indicator function. The composite adaptive reward function for the current time step is calculated based on target tracking reward, action smoothing penalty, action amplitude penalty, and constraint violation penalty.

6. The method for suppressing unsteady pressure pulsations in the inlet based on deep reinforcement learning according to claim 4, characterized in that, The updated TD3 reinforcement learning model includes: Based on the pressure coefficient and action at the current time step, update the state vector to obtain the updated state vector; Store the updated state vector and the composite adaptive reward function of the current time step into the experience replay buffer; The updated state vector is input into the Actor network to generate the updated action vector; The updated state vector and the updated action vector are input into the first Critic network and the second Critic network respectively to generate the corresponding first target Q value and second target Q value; Based on the first objective Q value and the second objective Q value, with the goal of minimizing the mean squared error loss function, the network parameters of the first Critic network and the second Critic network are updated respectively by gradient descent. The gradient strategy of the Actor network is calculated to maximize the first objective Q value, and the network parameters of the Actor network are updated through a gradient-based optimization algorithm. The network parameters of the target Critic network and the target Actor network are updated using a soft update method.

7. The method for suppressing unsteady pressure pulsations in the inlet based on deep reinforcement learning according to claim 6, characterized in that, The Actor network is updated every two time steps.

Images