Wind tunnel test integration method based on deep reinforcement learning

Abstract

The invention relates to the technical field of aerodynamic tests, in particular to a wind tunnel test integration method based on deep reinforcement learning. The method comprises the following steps: firstly, integrating an equipment communication protocol commonly used in a wind tunnel test into a unified adapter through Python coding, and realizing seamless communication and data interaction with equipment through a standardized write-in and read interface; secondly, the method integrates various code libraries of a deep reinforcement learning technology, and supports a user to freely call various deep reinforcement learning algorithms including PPO, SAC and TD3 and corresponding optimization functions; and finally, through a standard gym environment interface, smooth interaction between the adapter and the deep reinforcement learning algorithm is ensured, and real-time data transmission and closed-loop control among the actuator, the sensor, the controller and the DRL algorithm are realized. Through the method, automatic control and parameter optimization in the wind tunnel test are further improved, and the experiment efficiency and the system performance are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of aerodynamic test technology, and in particular to a wind tunnel test integration method based on deep reinforcement learning. Background Art

[0002] The impact of wind loads and vibrations on structural safety is an important consideration in the fields of architecture, automobiles, aviation, etc., especially in the context of rapid development of science and technology and transportation. With the continuous increase in people's requirements for the speed of cars and airplanes, and the increasing number of high-rise buildings, such risks are particularly prominent, further highlighting the importance of wind tunnel tests. Wind tunnel tests can not only explore effective flow control methods, reduce these risks, and improve the overall performance of the system, but also complete wind characteristics research and visualization, etc., providing a basis and support for aerodynamic research.

[0003] In recent years, deep reinforcement learning (DRL) has become an important tool for solving complex sequential decision-making problems in closed-loop active control. By combining reinforcement learning with deep learning, DRL uses the powerful computing power of deep neural networks to significantly improve its applicability and effectiveness in various control systems. This method has demonstrated excellent performance in multiple control fields.

[0004] Combining DRL technology with wind tunnel tests and applying them to aerodynamic research provides a promising flow control solution. However, this combination faces multiple challenges, including software system integration, hardware compatibility, and the complexity of data interaction. Therefore, developing a wind tunnel test integration method based on deep reinforcement learning can provide researchers with a concise and easy-to-use tool, which has become an urgent need in the current field of aerodynamic research. The establishment of this method will greatly simplify the process of related research, improve research efficiency, and provide strong support for further innovation in the field of aerodynamics. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art, meet the actual needs in reality, and provide a wind tunnel test integration method based on deep reinforcement learning. This method combines deep reinforcement learning (DRL) with traditional wind tunnel test technology to provide an efficient solution for automated and intelligent aerodynamic experiments. This method realizes intelligent control and optimization of the aerodynamic experiment process by combining the emerging reinforcement learning technology with traditional wind tunnel test methods. By utilizing the adaptability and powerful optimization capabilities of the deep reinforcement learning algorithm, it is possible to dynamically adjust the test parameters, improve the test efficiency, and obtain the best control strategy, thereby providing more powerful tools and technical support for aerodynamic research.

[0006] The specific technical solution adopted by the present invention is as follows:

[0007] A wind tunnel test integration method based on deep reinforcement learning, which integrates reinforcement learning technology (DRL), deep learning technology, PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), TD3 (Twin Delayed Deep Deterministic Policy Gradient) and other algorithms, and supports multiple communication methods such as TCP (Transmission Control Protocol), UDP (User Datagram Protocol), serial communication (Serial), Modbus communication protocol, etc. In addition, it is also deployed based on the gym environment to achieve efficient interaction with wind tunnel equipment.

[0008] First, this method controls the communication protocols of the equipment commonly used in wind tunnel tests (pressure scanning valves, force balances, Arduino, motors, laser displacement meters, etc.) through Python code, which is mainly divided into three parts: initialization, writing, and reading. Initialization: Establish a communication channel with the device based on the communication parameters (baud rate, port) entered by the user. Writing: Used to send control instructions and test parameters to ensure that the device performs operations as set. Reading: Obtain real-time data and status from sensors or actuators for subsequent processing and analysis. All codes have been tested with actual equipment to ensure their reliability and effectiveness in wind tunnel tests.

[0009] Secondly, this method integrates all codes into a unified adapter, and provides a unified initialization, writing and reading interface through the adapter to simplify the communication process between different devices and improve the flexibility and scalability of the system. Initialization function: Based on the communication parameters (such as communication mode, baud rate, port number, etc.) input by the user, automatically establish a data transmission channel to ensure smooth and stable communication between devices. This function supports multiple communication protocols and can be flexibly configured according to the specific needs of the user to adapt to different devices and communication environments. Write function: Allows users to modify the operating instructions and parameters of the device in real time, and package these instructions or parameters into messages that conform to the corresponding communication protocol format for sending. Through this function, users can dynamically adjust the operating status of the device and realize flexible control and management of various types of equipment during the wind tunnel test. Read function: Obtain measurement data from the device's sensor in real time, or read the current set value and status information from the actuator. The read data will be fed back to the user immediately to help him monitor the operation of the device in real time and provide an accurate reference for subsequent analysis and optimization.

[0010] Next, this method integrates several popular DRL libraries, such as Tianshou and Stable Baselines3. Users can freely choose suitable DRL algorithms (such as PPO, SAC, TD3, etc.) from these libraries and flexibly use the optimization tools in these libraries to provide more intelligent control solutions for experiments.

[0011] Finally, this method provides a universal interface compatible with the gym environment, through which users can freely control the adapter of the method to achieve precise operation of the wind tunnel test equipment. At the same time, the system supports two-way data transmission between actuators, sensors, controllers and DRL algorithms, thereby achieving closed-loop data feedback and control optimization.

[0012] The basic principles of each algorithm in the present invention are as follows:

[0013] Adapter: The adapter integrates multiple communication protocols commonly used in wind tunnel experiments, and implements communication management of the equipment through control codes written in Python. The adapter functions are divided into three core modules: initialization, writing, and reading. Initialization: Establish a stable communication channel based on the communication parameters entered by the user (such as communication method, baud rate, port, etc.) to ensure smooth data transmission between different devices. Write: Responsible for converting user-defined instructions and parameters into a format that conforms to the communication protocol, and sending it to the device to achieve control and parameter adjustment of the wind tunnel test equipment. Read: Read real-time data or the current status of the equipment from the sensors or actuators in the wind tunnel equipment to provide users with real-time feedback and equipment status monitoring.

[0014] Deep reinforcement learning algorithm (DRL): This method integrates multiple current mainstream deep reinforcement learning algorithm libraries, including Tianshou and Stable Baselines3. Users can flexibly select and call deep reinforcement learning algorithms in these libraries (such as PPO, SAC, TD3, etc.). This method provides comprehensive functional support, including training optimization and parameter adjustment, which can greatly improve the training efficiency of the algorithm and accelerate the convergence process of the model. In addition, users can customize and adjust the optimization strategy according to specific experimental needs to further improve the accuracy and reliability of experimental results.

[0015] Gym environment: The Gym environment is a standard framework in deep reinforcement learning research, providing a unified input and output interface to ensure seamless interaction with the DRL algorithm. This environment allows users to freely use adapters in methods to control wind tunnel test equipment and realize real-time data transmission between actuators, sensors, controllers and DRL algorithms. Through this method, users can easily collect, process and control aerodynamic experimental data, improve experimental efficiency, and provide a stable testing environment for the application of deep reinforcement learning in flow control.

[0016] Specifically, the following steps are included:

[0017] Step 1: Control the communication protocol of the equipment commonly used in wind tunnel tests with Python coding. It is generally divided into three parts: initialization, writing and reading. Initialization is responsible for establishing a communication channel using the communication parameters input by the user, writing is responsible for sending data and parameters, and reading is responsible for reading data and parameters. All Python codes have been tested in wind tunnel test equipment.

[0018] Step 2. Under the communication protocol, write three functions, namely initialization, writing and reading. Initialization can use the communication parameters set by the user to establish a communication channel and prepare for data transmission. The writing function allows users to freely change the behavior or parameters of the device. Reading can read data from sensors or read current settings from actuators, etc.

[0019] Step 3: Several popular DRL libraries (Tianshou, Stablebaselines3) are also added to this method. Users can freely choose and use different DRL algorithms according to experimental requirements, such as PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), TD3 (Twin Delayed DDPG), etc. The goal of the DRL algorithm is to obtain a strategy that maximizes the return on the sampled trajectory:

[0020]

[0021] τ={s0,a0,s1,a1,...,s t ,a t )

[0022]

[0023] Among them, γ is the discount factor, r represents the reward value at different time steps, and the “discount factor is used to reduce the reward value obtained at a distant moment;

[0024] The parameters are updated using the gradient descent method:

[0025]

[0026] Among them, θ represents the network parameters, α is the learning rate, and J(θ) is the loss function.

[0027] The update formula of the PPO algorithm is:

[0028]

[0029] The update formula of the SAC algorithm is:

[0030] Q θ (s,a)←Q θ (s,a)+α(r t +γmin i＝1,2 Q θ' (s t+1 ,a t+1 )-Q θ (s t ,a t )-βlogπ θ (a t |s t ))

[0031] The update formula of the TD3 algorithm is:

[0032] Q θ (s,a)←Q θ (s,a)+α(r+γmin i＝1,2 Q θ' (s',π θ' (s'))-Q θ (s t ,a t ))

[0033] Step 4: Provide a general gym environment interface, in which users can freely use the method's Adapter to control the wind tunnel test equipment and freely transmit data between actuators, sensors, controllers and DRLs; so that the equipment operation status and control strategy can be closely combined to form a complete closed-loop control system:

[0034]

[0035] Beneficial effects of the present invention: The present invention not only provides an interactive channel between deep reinforcement learning and wind tunnel environment, but also integrates a universal adapter, making the writing and reading of actuators, controllers, and sensors more free; the present invention also simplifies the implementation steps of wind tunnel tests based on deep reinforcement learning, reduces the challenges brought by interaction problems, and greatly reduces the cost of test layout. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0037] In order to deepen the understanding of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The embodiments are only used to explain the present invention and do not limit the protection scope of the present invention.

[0038] Example: Figure 1 As shown, a wind tunnel test integration method based on deep reinforcement learning includes the following steps:

[0039] Step 1: Encode and control the equipment communication protocol commonly used in wind tunnel tests through Python language; the core of this step is to modularly design the equipment initialization, data writing and reading functions, and build them into independent function modules for subsequent calls and operations.

[0040] Step 2: Integrate the communication protocols used by different devices into a unified adapter; the adapter has the ability to be compatible with multiple protocols and provides standardized write and read interfaces to ensure seamless communication and data interaction between different types of devices.

[0041] Step 3: Integrate deep reinforcement learning (DRL) technology. This method supports multiple current mainstream DRL libraries (such as Stablebaselines3, Tianshou, etc.) and provides corresponding algorithm interfaces for parameter adjustment and real-time feedback mechanism during automated control and optimization experiments.

[0042] Step 4: By introducing the universal interface of the gym environment, the adapter is used as the action output module of the DRL algorithm, so that it can control the equipment to perform corresponding operations in the wind tunnel test; at the same time, the interface supports real-time reading of data from the wind tunnel test equipment and feeds it back to the reinforcement learning algorithm to form a data closed loop, providing a basis for further model optimization.

[0043] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A wind tunnel test integration method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Code and control the equipment used in the wind tunnel test through Python language; Step 2: Integrate the communication protocols used by the equipment used in the wind tunnel test into a unified adapter; Step 3: Use integrated deep reinforcement learning technology and integrated DRL library, use the optimization tools in the DRL library, and experiment to provide a more intelligent control solution; Step 4: By introducing the universal interface of the gym environment, the adapter is used as the action output module of the DRL algorithm, so that it can control the equipment to perform corresponding operations in the wind tunnel test. At the same time, the interface supports real-time reading of data from the wind tunnel test equipment and feeds it back to the reinforcement learning algorithm to form a data closed loop, providing a basis for further model optimization.

2. The wind tunnel test integration method based on deep reinforcement learning according to claim 1, characterized in that: In the step 1, the equipment used in the wind tunnel test includes a pressure scanning valve, a force measuring balance, an Arduino, a motor, and a laser displacement meter.

3. The wind tunnel test integration method based on deep reinforcement learning according to claim 2 is characterized in that: The specific process of step 1 is: the communication protocol of the commonly used equipment in the wind tunnel test is controlled by Python code, and the whole process is divided into three parts: initialization, data writing and data reading; Initialization: Responsible for establishing a communication channel with the wind tunnel equipment according to the communication parameters input by the user to ensure that the equipment can communicate stably; Write: used to transfer user-specified data or parameters to the device so that the device can perform corresponding operations according to the new settings; Read: Read real-time data and status from the device’s sensors or actuators for subsequent analysis and processing.

4. The wind tunnel test integration method based on deep reinforcement learning according to claim 3 is characterized in that: The specific process of step 2 is: integrating the communication protocols used by different devices into a unified adapter, which is compatible with multiple protocols and provides a unified standard initialization, writing and reading interface to ensure seamless communication and data interaction between different types of devices. Users only need to set the corresponding parameters.

5. The wind tunnel test integration method based on deep reinforcement learning according to claim 4 is characterized in that: The specific process of step three is as follows: multiple deep reinforcement learning libraries are integrated into this method, including Tianshou and StableBaselines3; users can freely choose and use different DRL algorithms according to experimental requirements. In addition, users can freely call the optimization tools provided in these libraries, so that users can quickly iterate and adjust algorithm parameters in actual wind tunnel experiments to improve experimental efficiency and model performance.

6. The wind tunnel test integration method based on deep reinforcement learning according to claim 5, characterized in that: The DRL algorithm used in step 3 is the PPO algorithm, the SAC algorithm, or the TD3 algorithm. The specific algorithm is as follows: τ={s 0, a0,s 1, a1,...,s t, a t ) Among them, γ is the discount coefficient, r represents the reward value at different time steps, and the discount coefficient is used to reduce the reward value obtained at a distant moment; The parameters are updated using the gradient descent method: Among them, θ represents the network parameters, α is the learning rate, and J(θ) is the loss function; The update formula of the PPO algorithm is: The update formula of the SAC algorithm is: Q θ (s,a)←Q θ (s,a)+α(r t +γmin i＝1,2 Q θ' (s t+1 ,a t+1 )-Q θ (s t ,a t )-blog θ (a t |s t )) The update formula of the TD3 algorithm is: Q θ (s,a)←Q θ (s,a)+α(r+γmin i＝1,2 Q θ' (s',π θ' (s'))-Q θ (s t ,a t ))。 7. The wind tunnel test integration method based on deep reinforcement learning according to claim 6, characterized in that: The specific process of step 4 is as follows: a general interface compatible with the gym environment is provided, through which the user can freely control the Adapter in this method to achieve efficient management and operation of the wind tunnel test equipment; the interface supports real-time data transmission between actuators, sensors, controllers and deep reinforcement learning algorithms, so that the equipment operation status and control strategy can be closely combined to form a complete closed-loop control system:

Images