Reinforcement learning semi-active suspension intelligent control method based on Bayesian optimization

Abstract

The invention is suitable for the technical field of vehicle intelligent control, and provides a reinforcement learning semi-active suspension intelligent control method based on Bayesian optimization. According to the method, vehicle state data are collected and processed, then a near-end strategy optimization (PPO) algorithm is adopted and combined with a non-parametric award function to design a deep reinforcement learning model, an optimal hyper-parameter set of the model is searched through a Bayesian optimization algorithm, the optimized hyper-parameter training model is utilized, a semi-active suspension intelligent control strategy is generated, and the semi-active suspension intelligent control strategy is obtained. And evaluating the action result by the vehicle state award. According to the method, multi-target collaborative optimization is realized by adopting a non-parametric award function in a PPO algorithm, a complex control problem is solved by utilizing a three-layer deep neural network in a data driving mode, and Bayesian optimization is introduced to automatically search hyper-parameters so as to improve efficiency and performance; meanwhile, an optimization objective function is designed to be a combination of a trapezoidal numerical integral and a simple moving average value so as to stabilize a reward value and accelerate convergence.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent control of vehicles, and particularly relates to a semi-active suspension intelligent control method based on reinforcement learning of Bayesian optimization. BACKGROUND

[0002] With the continuous progress of automobile technology, the performance, comfort and safety boundaries of vehicles are constantly expanding, and the suspension system, as the key interface connecting the vehicle frame and the wheels, plays a core role in it. It relieves the impact of road unevenness on the vehicle, enhances vehicle handling, and ensures a smooth driving experience. With the increasing demand for vehicle performance improvement, ride comfort and safety directly related to the suspension system are increasingly valued. Semi-active suspension systems have found a balance between the simplicity of passive suspension systems and the control adaptability of active suspension systems. Through intelligent control strategies, it can adjust the damping force according to real-time driving conditions, improving ride comfort and handling stability without consuming as much energy as active suspension systems, and its control effect is close to that of active suspension systems, becoming one of the general technologies for intelligent vehicles.

[0003] Currently, the research and development goal of suspension system control strategies focuses on effectively suppressing vehicle vertical vibration in various driving environments while improving handling stability to ensure driving safety. Existing control strategies mainly fall into two categories: methods based on classical control theory, which are easy to implement basic control of the suspension system, but are based on the assumption of a linear time-invariant system. However, the actual suspension system has nonlinear characteristics, and the difference between the simplified model and the real system leads to insufficient adaptability in multiple working conditions. Methods based on modern control theory can implement more complex control logic, but rely on complex algorithms and models. Even with high-order mathematical methods, it is difficult to completely describe the real dynamics of the suspension system, such as damping hysteresis, component wear, and inertia parameter fluctuations caused by changes in vehicle load, etc. Therefore, it is difficult to cope with diverse and complex working conditions, and the application range is limited.

[0004] In recent years, the explosive development of artificial intelligence technology has brought innovation to the field of intelligent driving, and the breakthrough of deep reinforcement learning technology has provided a new solution for the optimization of semi-active suspension control strategies. However, the efficiency of deep reinforcement learning is highly dependent on the configuration of hyperparameters such as learning rate, neural network size, discount factor, etc., and there is a problem of hyperparameter sensitivity. Therefore, the present application proposes a semi-active suspension intelligent control method based on reinforcement learning of Bayesian optimization. SUMMARY

[0005] The purpose of the present application is to provide a semi-active suspension intelligent control method based on reinforcement learning of Bayesian optimization, aiming to solve the problems raised in the background technology.

[0006] The purpose of the application is realized by the following technical solutions: A semi-active suspension intelligent control method based on Bayesian optimization and reinforcement learning, comprising the following steps: Step 1: data acquisition and processing; Real-time acquisition of vehicle state data through a sensor network arranged on the vehicle, and pre-processing of data from different sensors of the vehicle using a preset method; Step 2: design of deep reinforcement learning model based on non-parametric reward function; A deep reinforcement learning model is constructed based on the proximal policy optimization algorithm; the pre-processed vehicle state data is used as the state space, and the control current output of the vehicle semi-active suspension is used as the action space, and a non-parametric reward function is designed; the vehicle state data includes vehicle vertical acceleration, vehicle roll angle, vehicle roll angular velocity, and vehicle yaw angular velocity; Step 3: parameter optimization of deep reinforcement learning model based on Bayesian optimization algorithm; A set of hyperparameters to be optimized for the proximal policy optimization algorithm is constructed, and the set of hyperparameters is searched by the Bayesian optimization method; Step 4: training of deep reinforcement learning model based on non-parametric reward function and generation of semi-active suspension intelligent control strategy based on Bayesian optimization; Initialize the experience buffer, policy network parameters, and value network parameters, train the deep reinforcement learning model using the set of hyperparameters obtained by Bayesian optimization, and obtain the semi-active suspension intelligent control strategy based on the deep reinforcement learning model based on non-parametric reward function and Bayesian optimization.

[0007] Further, in step 1, the data collected by the sensor is sixteen-dimensional data, including vehicle vertical acceleration , vehicle roll angle , vehicle roll angular velocity , vehicle yaw angular velocity , four sets of suspension deflection , four sets of tire dynamic travel , four sets of suspension relative displacement velocity ; the multi-dimensional dynamic data collected by the sensor is pre-processed to ensure data accuracy and quality, a sensor type-physical quantity dimension mapping table is established, and the original output data of each sensor is converted to a dynamic unit system to unify the dimension standard.

[0008] Further, step 2 includes the following sub-steps: Step 2.1: state observation selection; The state observation of the PPO algorithm is defined as , wherein represents State observation at time t, including vehicle body vertical acceleration , vehicle roll angle , vehicle roll angle velocity , vehicle yaw rate ; Step 2.2: Action selection Define action as , where denotes action at time t, i.e. control current acting on semi-active suspension ; Step 2.3: Non-parametric reward function setting Define non-parametric reward function as , where denotes reward function of environment at time t; vehicle body vertical acceleration is used to stabilize vibration in vehicle body vertical direction; vehicle roll angle is used to ensure vehicle handling stability ; Step 2.4: Suspension physical constraints Take suspension dynamic deflection not exceeding maximum stroke and vehicle tire dynamic load less than static load as constraints Step 2.5: Network architecture building

[0009] Further, the step 3 includes the following sub-steps: Step 3.1: Building hyperparameter set Define candidate hyperparameter set as , where denotes candidate hyperparameter set, is policy network learning rate, is value network learning rate, is discount factor, is experience region, is batch sampling size, is learning round number, is entropy loss weight Step 3.2: Building proxy function Select Gaussian process as proxy model to make posterior prediction on target function, where input and output are mean and standard deviation of target function at candidate point ; Step 3.3: Select the acquisition function; choose As an acquisition function, it guides the exploration behavior of the optimization process in the hyperparameter space; represents the expected improvement function; represents the posterior distribution of the Gaussian process; represents the expected value; Represents the current best GP function value; Step 3.4: Construct the optimization objective function; The optimization objective function is defined as the trapezoidal numerical integral based on the average reward of deep reinforcement learning, and the expression is ;in, represents the trapezoidal numerical integration function; Indicates the use of candidate hyperparameter sets during deep reinforcement learning training Simple moving average of post-system rewards; is a fixed incremental parameter used to shift the overall reward to the positive range; is the time window size of the moving average; In order to use the candidate hyperparameter set in the deep reinforcement learning training process t Moment system rewards.

[0010] Furthermore, step 4 includes the following sub-steps: Step 4.1: Initialize the optimal hyperparameter set obtained by Bayesian optimization search; Step 4.2: Initialize the experience buffer to store the state, action, reward, next state, and termination signal trajectory data generated by the interaction between the agent and the environment; Step 4.3: Randomly initialize policy network parameters and value network parameters ; Step 4.4: Set the number of PPO algorithm training rounds , and the maximum number of training steps in each round T ; Step 4.5: Perform training, using the current policy network to interact with the environment for the maximum number of steps in each training round T ,exist Always monitor the current system status , according to the policy network The sampling action controls the current, and the damping force input is obtained according to the current and the compression speed of the shock absorber. The system generates Reward feedback at all times and the state at the next moment ; The system's empirical data Stored in the experience buffer; the generalized advantage estimation method is used to calculate the advantage value ; Step 4.6: the collected whole trajectory data is optimized and updated , the trajectory data is randomly shuffled, and is divided according to the batch sampling size ; the old policy network is used to calculate the action probability , the policy ratio is calculated , and the target function with clipping is calculated , wherein is the clipping coefficient, The function limits the policy update by clipping the importance weight, and is specifically defined as follows: ; After one round of update, the old policy network parameters are updated to the current policy network parameters; Step 4.7: the value network is updated by minimizing the mean square error between the predicted state value and the actual return, and the value loss function is defined as , wherein is the value loss, is the value prediction of the value network to the state ; Step 4.8: steps 4.5, 4.6 and 4.7 are repeatedly executed; if the maximum number of iterations is reached, the training is ended; otherwise, the training is continued back to step 4.5; Step 4.9: after step 4.8 is ended, the experience buffer is reinitialized in step 4.2, the Bayesian optimization training is performed, until the maximum number of iterations of the Bayesian optimization is reached, the training is stopped, the optimal hyperparameter set is obtained, and the deep reinforcement learning model trained based on the optimal hyperparameter set is used to generate the intelligent control strategy of the semi-active suspension.

[0011] Compared with the prior art, the present application has the following advantages: 1. The present application directly links the vehicle body vertical acceleration (ride comfort index) and the vehicle roll angle (handling stability index) by using a non-parametric reward function instead of a traditional parametric design in the PPO algorithm, reduces the influence of parameter uncertainty, makes the reward mechanism dynamically adapt to complex working conditions, realizes multi-objective collaborative optimization, and solves the multi-objective collaborative optimization problem of vehicle safety, ride comfort and handling stability.

[0012] 2. The present application uses a three-layer deep neural network to construct the policy network and the value network, processes the vehicle state space (vehicle body vertical acceleration, vehicle roll angle, etc.) and the action space (semi-active suspension control current) in a data-driven manner, and effectively solves the complex control problem in a large state space without relying on high-precision mathematical modeling of traditional control algorithms.

[0013] 3. The present application solves the problem of deep reinforcement learning super parameter sensitivity by introducing a Bayesian optimization automatic search PPO algorithm super parameter set, reducing the randomness and inefficiency of manual parameter adjustment, saving time and improving performance.

[0014] 4. The present application designs the optimization objective function as the combination of trapezoidal numerical integration and simple moving average based on the average reward of deep reinforcement learning, which helps to accelerate the model convergence process while ensuring the stability of the reward function value. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The flow chart of the method of the present application.

[0016] Figure 2 The lateral angle (Roll) of the vehicle corresponding to different suspension control strategies changes with time (Time) under the condition of 80km / h snake-shaped pile around the pile.

[0017] Figure 3 The vertical acceleration of the vehicle body corresponding to different suspension control strategies changes with time (Time) under the condition of 50km / h B-grade road straight driving. DETAILED DESCRIPTION

[0018] In order to have a clearer understanding of the technical features, objectives and beneficial effects of the present application, the technical solutions of the present application will be described in detail as follows, but it cannot be understood as limiting the scope of implementation of the present application.

[0019] The present application provides a reinforcement learning semi-active suspension intelligent control method based on Bayesian optimization, and its flow chart is shown in Figure 1 The method comprises the following steps: Step 1: data acquisition and processing; Collect multi-dimensional state data of the vehicle through a sensor network, and perform preprocessing and standardization to provide high-quality input for subsequent model training. Specifically, it includes: Data acquisition: through the accelerometer, displacement sensor and IMU module arranged on the vehicle to form a sensor network, real-time acquisition of sixteen-dimensional real vehicle state data, including: vertical acceleration of the vehicle body , vehicle roll angle , vehicle roll angle velocity , vehicle yaw rate , four sets of suspension deflection , four sets of tire dynamic travel , four sets of suspension relative displacement velocity .

[0020] ​Data processing: The data from different sensors of the vehicle are preprocessed using the method of interpolation correction to ensure the accuracy and applicability of the collected real vehicle data. A sensor type-physical quantity dimension mapping table is established to convert the raw output data of the accelerometer, displacement sensor and IMU module into a dynamic unit system, ensuring that the data have consistent measurement standards and enhancing the adaptability of the algorithm.

[0021] Step 2: Design of deep reinforcement learning model based on non-parametric reward function A deep reinforcement learning model is constructed based on the Proximal Policy Optimization (PPO) algorithm. The preprocessed vehicle state data (vehicle vertical acceleration, vehicle roll angle, vehicle roll angular velocity, and vehicle yaw angular velocity) are used as the state space, and the control current output of the vehicle semi-active suspension is used as the action space. A non-parametric reward function is designed as an extension and generalization of the current mainstream parametric reward function. The core of the PPO algorithm includes the policy network (outputting action decisions) and the value network (evaluating state values), which jointly support the decision-making and optimization process of reinforcement learning.

[0022] The sub-steps of Step 2 are as follows: Step 2.1: Selection of state observations Considering the indicators of vehicle operational stability and ride comfort for the semi-active suspension intelligent control method, the state observations of the PPO algorithm are defined as where represents the state observations at time t, including the vehicle vertical acceleration , the vehicle roll angle , the vehicle roll angular velocity , and the vehicle yaw angular velocity .

[0023] Step 2.2: Selection of action quantities The action quantities are defined as where represents the action quantities at time t, i.e., the control current applied to the semi-active suspension.

[0024] Step 2.3: Setting of non-parametric reward function The main purpose of the semi-active suspension control system is to maintain the vehicle vertical acceleration within a reasonable range and improve the suppression of vehicle roll under different driving conditions. Therefore, the non-parametric reward function is defined as where represents The reward function of the environment at the moment; the vertical acceleration of the vehicle body included in the reward function The purpose is to stabilize the vertical vibration of the vehicle body, avoid the interference of road excitation on passengers, and ensure the ride comfort of the vehicle; the vehicle roll angle It is to ensure the vehicle's handling stability.

[0025] Step 2.4: Suspension physical limitations; To avoid damaging vehicle components, two physical limitations must be considered: first, suspension dynamic deflection must be within its maximum travel limits; second, the dynamic load on the vehicle tires must be less than the static load to ensure constant contact between the road surface and the wheels. When the input motion exceeds these limits, only the maximum input value is used.

[0026] Step 2.5: Network architecture construction; The policy network and the value network use the same deep neural network structure. Considering the system complexity and real-time requirements, a three-layer network is adopted. The number of neurons in each layer is 100, and the activation function uses the linear rectification function (ReLU).

[0027] Step 3: Optimize the parameters of the deep reinforcement learning model based on the Bayesian optimization algorithm; Construct a set of hyperparameters to be optimized for the PPO algorithm and search using the Bayesian optimization method (Bayesian optimization is a model-based sequential optimization technique that can be used to adjust the hyperparameters of any noisy black box function).

[0028] The sub-steps of step 3 are as follows: Step 3.1: Construct a set of hyperparameters; Define the candidate hyperparameter set as ,in represents the candidate hyperparameter set, is the policy network learning rate, is the value network learning rate, is the discount factor, For the experience area, is the batch sampling size, is the number of learning rounds, is the entropy loss weight.

[0029] Step 3.2: Construct the proxy function; Choose Gaussian Process As a proxy model, it is used to make a posteriori prediction of the target function, where the input and output are the candidate points The mean of the objective function at and standard deviation .

[0030] Step 3.3: Select the acquisition function; choose As a collection function, it guides the exploration behavior of the optimization process in the hyperparameter space; where, represents the expected improvement function; represents the posterior distribution of the Gaussian process; represents the expected value; while the best GP function value obtained so far is denoted as .

[0031] Step 3.4: Constructing the optimization objective function; The optimization objective function is defined as the trapezoidal numerical integral based on the average reward of deep reinforcement learning, expressed as ; where, represents the trapezoidal numerical integral function; represents the simple moving average of system rewards after using the candidate hyperparameter set during the deep reinforcement learning training process; is a fixed increment parameter used to shift the rewards as a whole to the positive interval; is the time window size of the moving average; represents the system reward at time t after using the candidate hyperparameter set during the deep reinforcement learning training process.

[0032] Step 4: Parameter-free reward function deep reinforcement learning model training based on Bayesian optimization and semi-active suspension intelligent control strategy generation; Initialize the experience buffer, policy network parameters, and value network parameters, and train the deep reinforcement learning model using the hyperparameter set obtained by Bayesian optimization to obtain the parameter-free reward function deep reinforcement learning semi-active suspension intelligent control strategy based on Bayesian optimization.

[0033] The sub-steps of Step 4 are as follows: Step 4.1: Initialize the optimal hyperparameter set obtained by Bayesian optimization search; Step 4.2: Initialize the experience buffer to store trajectory data such as state, action, reward, next state, and termination signal generated by the interaction between the agent and the environment; Step 4.3: Randomly initialize the policy network parameters and the value network parameters ; Step 4.4: Set the number of PPO algorithm training rounds and the maximum step number in each round T ; Step 4.5: Training, in each training round, use the current policy network to interact with the environment to perform the maximum step number T , and observe the current system state at time , according to the policy network The sampling action controls the current, and the damping force input is obtained according to the current and the shock absorber compression speed (obtained by looking up the table according to the characteristics of the suspension). The system generates Reward feedback at all times and the state at the next moment . The system's empirical data Stored in the experience buffer. The advantage value is calculated using the generalized advantage estimation method , used for subsequent policy network updates.

[0034] Step 4.6: Perform the following operations on the collected trajectory data: Round optimization update, randomly disrupt the trajectory data, and follow the batch sampling size Use the old policy network to calculate the action probability , calculate the strategy ratio After that, calculate the objective function with clipping ,in is the crop factor, The function limits the policy update to a "reasonable range" (no more than , not less than ), ensuring that the difference between the new strategy and the old strategy is not too large, thereby improving sample utilization efficiency and training stability. Its specific definition is as follows: ; After a round of update is completed, the old policy network parameters are updated to the current policy network parameters.

[0035] Step 4.7: The value network is updated by minimizing the mean squared error between the predicted state value and the actual return, defining the value loss function ,in For loss of value, For the value network state value prediction.

[0036] Step 4.8: Repeat steps 4.5, 4.6, and 4.7. If the maximum number of iterations is reached, the training ends. Otherwise, return to step 4.5 to continue training.

[0037] Step 4.9: After step 4.8 is completed, return to step 4.2 to reinitialize the experience buffer area and perform Bayesian optimization training until the maximum number of iterations of Bayesian optimization is reached. Then, the training is stopped to obtain the optimal hyperparameter set. Based on the deep reinforcement learning model obtained by training with the optimal hyperparameter set, a semi-active suspension intelligent control strategy is generated.

[0038] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0039] Example 1: Under the conditions of 80km / h of snake-shaped driving around the pile and 50km / h of B-grade straight-line driving on the road, the comprehensive control effect of handling stability and ride comfort is experimentally verified. The parameter-free reward function reinforcement learning algorithm based on Bayesian optimization (the method proposed in the present application, BO-NRPPO) is compared with passive suspension, parameter reward function reinforcement learning algorithm (PRPPO), Bayesian optimization-based parameter reward function reinforcement learning algorithm (BO-PRPPO), and parameter-free reward function reinforcement learning algorithm (NRPPO).

[0040] From Figure 2 It can be seen that the BO-NRPPO strategy performs best among all the compared strategies and passive suspension, effectively suppressing the peak change of roll angle, and its suppression effect on roll angle is significantly better than other control methods. Specifically, compared with passive suspension, the roll control performance of BO-NRPPO is improved by 9.6%; compared with BO-PRPPO, NRPPO and PRPPO strategies, it is improved by 3.09%, 15.6% and 15.93% respectively, which reflects the obvious advantages of the method proposed in the present application in handling stability.

[0041] From Figure 3 It can be seen that in the evaluation of ride comfort, although the root mean square value of acceleration of BO-NRPPO does not reach the lowest, its performance is improved by 9.75% compared with BO-PRPPO; compared with passive suspension, the improvement reaches 32.29%. It is worth noting that although BO-NRPPO is not necessarily the optimal strategy in terms of ride comfort, NRPPO performs significantly worse than BO-NRPPO in handling stability, even lower than passive suspension, which shows that strategies without the introduction of Bayesian optimization technology are difficult to effectively balance the two goals of ride comfort and handling stability. The BO technology systematically explores the hyperparameter space, combines the performance evaluation of the PPO algorithm, optimizes the key hyperparameter configuration, and thus significantly improves the overall control performance of the BO-NRPPO strategy.

[0042] From the comprehensive performance indicators, it can be seen that the BO-NRPPO strategy proposed in the present application exhibits strong comprehensive optimality and practical application potential under various driving conditions.

[0043] The above is only the preferred embodiment of the present application, and it should be noted that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which should be considered as the protection scope of the present application, and these will not affect the effect and practicality of the present application.

Claims

1. A reinforcement learning semi-active suspension intelligent control method based on Bayesian optimization, characterized in that: The following steps are involved: Step 1: Data collection and processing; The vehicle status data is collected in real time through the sensor network arranged on the vehicle, and the data from different sensors of the vehicle are pre-processed using a preset method; Step 2: Design of deep reinforcement learning model based on non-parametric reward function; Build a deep reinforcement learning model based on the proximal policy optimization algorithm; The preprocessed vehicle state data is used as the state space, the control current output of the vehicle's semi-active suspension is used as the action space, and a non-parametric reward function is designed. The vehicle state data includes the vertical acceleration of the vehicle body, the vehicle roll angle, the vehicle roll angular velocity, and the vehicle yaw angular velocity. Step 3: Optimize the parameters of the deep reinforcement learning model based on the Bayesian optimization algorithm; Construct a set of hyperparameters to be optimized for the proximal policy optimization algorithm and search using the Bayesian optimization method; Step 4: Training the parameter-free reward function deep reinforcement learning model based on Bayesian optimization and generating the intelligent control strategy for the semi-active suspension; Initialize the experience buffer, policy network parameters, and value network parameters, and use the hyperparameter set obtained by Bayesian optimization to train the deep reinforcement learning model, and obtain a parameter-free reward function deep reinforcement learning semi-active suspension intelligent control strategy based on Bayesian optimization.

2. The Bayesian optimization-based reinforcement learning semi-active suspension intelligent control method according to claim 1 is characterized in that: In step 1, the data collected by the sensor is 16-dimensional data, including the vertical acceleration of the vehicle body , vehicle roll angle , vehicle roll angular velocity , vehicle yaw rate , four sets of suspension dynamic deflection , Four sets of tire travel , relative displacement speed of the four suspension groups ; Preprocess the multi-dimensional dynamic data collected by the sensor to ensure data accuracy and quality, establish a sensor type-physical quantity dimension mapping table, dynamically convert the unit system of the original output data of each sensor, and unify the measurement standards.

3. The Bayesian optimization-based reinforcement learning semi-active suspension intelligent control method according to claim 1 is characterized in that: The step 2 includes the following sub-steps: Step 2.1: state observation selection; The state observation of the PPO algorithm is defined as ,in, express The state observation at the moment, including the vertical acceleration of the vehicle body , vehicle roll angle , vehicle roll angular velocity , vehicle yaw rate ; Step 2.2: Action volume selection; Define the action amount as ,in express The action amount at the moment, that is, the control current acting on the semi-active suspension ; Step 2.3: Non-parametric reward function setting; Define the non-parametric reward function: ,in, express The reward function of the environment at the moment; the vertical acceleration of the vehicle body included in the reward function Used to stabilize the vibration of the vehicle body in the vertical direction; vehicle roll angle Used to ensure the vehicle's handling stability; Step 2.4: Suspension physical limitations; The constraints are that the suspension dynamic deflection does not exceed the maximum travel and the dynamic load on the vehicle tires is less than the static load; Step 2.5: Network architecture construction; The policy network and the value network use the same three-layer deep neural network, with 100 neurons in each layer and a linear rectification function as the activation function.

4. The Bayesian optimization-based reinforcement learning semi-active suspension intelligent control method according to claim 1, characterized in that: Step 3 includes the following sub-steps: Step 3.1: Construct a set of hyperparameters; Define the candidate hyperparameter set as ,in represents the candidate hyperparameter set, is the policy network learning rate, is the value network learning rate, is the discount factor, For the experience area, is the batch sampling size, is the number of learning rounds, is the entropy loss weight; Step 3.2: Build the proxy function; Choose Gaussian Process As a proxy model, it is used to make a posteriori prediction of the target function, where the input and output are the candidate points The mean of the objective function at and standard deviation ; Step 3.3: Select the acquisition function; choose As an acquisition function, it guides the exploration behavior of the optimization process in the hyperparameter space; represents the expected improvement function; represents the posterior distribution of the Gaussian process; represents the expected value; Represents the current best GP function value; Step 3.4: Construct the optimization objective function; The optimization objective function is defined as the trapezoidal numerical integral based on the average reward of deep reinforcement learning, and the expression is ;in, represents the trapezoidal numerical integration function; Indicates the use of candidate hyperparameter sets during deep reinforcement learning training Simple moving average of post-system rewards; is a fixed incremental parameter used to shift the overall reward to the positive range; is the time window size of the moving average; In order to use the candidate hyperparameter set in the deep reinforcement learning training process t Moment system rewards.

5. The Bayesian optimization-based reinforcement learning semi-active suspension intelligent control method according to claim 1 is characterized in that: The step 4 includes the following sub-steps: Step 4.1: Initialize the optimal hyperparameter set obtained by Bayesian optimization search; Step 4.2: Initialize the experience buffer to store the state, action, reward, next state, and termination signal trajectory data generated by the interaction between the agent and the environment; Step 4.3: Randomly initialize policy network parameters and value network parameters ; Step 4.4: Set the number of PPO algorithm training rounds , and the maximum number of training steps in each round T ; Step 4.5: Perform training, using the current policy network to interact with the environment for the maximum number of steps in each training round T ,exist Always monitor the current system status , according to the policy network The sampling action controls the current, and the damping force input is obtained according to the current and the compression speed of the shock absorber. The system generates Reward feedback at all times and the state at the next moment ; The system's empirical data Stored in the experience buffer; the advantage value is calculated using the generalized advantage estimation method ; Step 4.6: Perform the following operations on the collected trajectory data: Round optimization update, randomly disrupt the trajectory data, and follow the batch sampling size Use the old policy network to calculate the action probability , calculate the strategy ratio After that, calculate the objective function with clipping ,in is the crop factor, The function limits the policy update by clipping the importance weights. It is defined as follows: ； After a round of update is completed, the old policy network parameters are updated to the current policy network parameters; Step 4.7: The value network is updated by minimizing the mean squared error between the predicted state value and the actual return, defining the value loss function ,in For loss of value, For the value network state Value prediction; Step 4.8: Repeat steps 4.5, 4.6, and 4.

7. If the maximum number of iterations is reached, the training ends. Otherwise, return to step 4.5 to continue training; Step 4.9: After step 4.8 is completed, return to step 4.2 to reinitialize the experience buffer area and perform Bayesian optimization training until the maximum number of iterations of Bayesian optimization is reached. Then, the training is stopped to obtain the optimal hyperparameter set. Based on the deep reinforcement learning model obtained by training with the optimal hyperparameter set, a semi-active suspension intelligent control strategy is generated.

Images