Energy recovery full-active suspension control method based on deep reinforcement learning

Abstract

The invention discloses an energy recovery full-active suspension control method based on deep reinforcement learning. The method comprises the following steps: 1, constructing an energy recovery full-active suspension simulation model and a simulation pavement model; 2, constructing a state quantity and an action quantity, and processing the action quantity to control suspension output control force; 3, designing a reward function; and 4, constructing a strategy-evaluation network, and training a neural network based on a PPO algorithm so as to dynamically generate a suspension control strategy. By optimizing the dynamic response of the suspension system, the performance of the vehicle on various road surfaces and under various driving conditions can be improved, so that collaborative optimization of energy recovery and suspension performance can be realized, and multiple requirements of the intelligent vehicle on comfort, safety and energy efficiency can be met.

Description

Technical Field

[0001] The present invention belongs to the full - active suspension control technology in the field of intelligent driving, and is particularly applicable to vehicles with high requirements for driving smoothness and ride comfort. Background Art

[0002] An active suspension system is an automotive suspension system that can instantaneously adjust the suspension stiffness and damping according to the real - time road conditions and driver's demands. The active suspension system adjusts the suspension stiffness and damping through an actuator, and uses a controller to calculate the control strategy of the suspension system in real - time according to the signals of sensors, and controls the output state of the actuator.

[0003] The full - active suspension system significantly improves the vibration and bump phenomena during vehicle driving by actively adjusting the suspension force, and enhances the comfort of passengers. Traditional passive and semi - active suspension systems are limited by their fixed parameters and are difficult to maintain the best performance under various road conditions. By introducing reinforcement learning algorithms, adaptive control of complex road conditions can be achieved, further improving ride comfort. In addition, with the increasing demand for energy conservation and environmental protection, electromagnetic energy recovery technology has gradually become an important development direction for suspension systems. Electromagnetic energy recovery technology converts the vibration energy of the suspension system into electrical energy through electromagnetic devices for storage, which not only improves the overall energy efficiency of the vehicle, but also provides new possibilities for the intelligent control of the suspension system.

[0004] Traditional suspension control methods, such as PID control, fuzzy control, etc., have limited performance in complex and non - linear environments. Reinforcement learning, as an emerging intelligent control method, can effectively handle complex and variable control tasks through interaction and learning with the environment. However, there is a significant technical contradiction between energy recovery and ride comfort in the full - active suspension system: to maximize the capture efficiency of vibration energy, it is necessary to increase the suspension damping or stiffness to enhance energy conversion, but this will lead to an increase in the vertical acceleration of the vehicle body and a decrease in ride comfort; to reduce the transmission of road shocks, it is necessary to reduce the suspension damping or stiffness to improve compliance, but this will weaken the response efficiency of the energy recovery device. Traditional control methods are difficult to dynamically balance this contradiction. Summary of the Invention

[0005] The present invention is to solve the above - mentioned deficiencies of the prior art, and proposes an energy - recovery full - active suspension control method based on deep reinforcement learning, in order to optimize the dynamic response of the suspension system, improve the performance of the vehicle under various road surfaces and driving conditions, and thus achieve the collaborative optimization of energy recovery and suspension performance to meet the multiple requirements of intelligent vehicles for comfort, safety, and energy efficiency.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The characteristics of a method for controlling an energy recovery fully active suspension based on deep reinforcement learning according to the present invention include the following steps:

[0008] Step 1: Construct a simulation dynamics model of an energy recovery 1 / 2 vehicle four-degree-of-freedom fully active suspension and a random road surface model;

[0009] Step 2: Define the state parameter set of the suspension , , is the total number of steps, is the normalized state quantity at the th step, and there is , where is the vehicle body acceleration at the th step, , are the front and rear wheel suspension dynamic deflections at the th step respectively, is the vehicle body displacement at the th step, is the remaining battery power at the th step;

[0010] Define the action space set of the suspension , where is the suspension stiffness, and is the suspension damping coefficient , is the magnitude of the external current connected to the motor, is the PWM duty cycle of the H-bridge in the energy recovery circuit;

[0011] Step 3: Construct the reward number at the th step;

[0012] Step 4: Construct a deep reinforcement learning network, which consists of an input layer with 5 neurons, a hidden layer with fully connected layers, an action network with an output layer of 4 neurons, and an input layer with 5 neurons, a hidden layer with fully connected layers, and a value network with an output layer of 1 neuron;

[0013] Step 5: Input into the action network for processing, and obtain the action quantity of the suspension at the th step. After the simulation dynamics model executes the action quantity at the th step, obtain the state at the +1th step, thereby forming the th sample, denoted as , and then obtain m samples and store them in the sample pool D;

[0014] Step 6: Based on the sample pool D, use the PPO algorithm to train the deep reinforcement learning network to obtain the optimal deep reinforcement learning model;

[0015] Step 7: Use the optimal action network in the optimal deep reinforcement learning model to process the real-time input state information and output the corresponding control instructions, so as to realize the active control of the energy recovery full active suspension.

[0016] The feature of the energy recovery full active suspension control method based on deep reinforcement learning according to the present invention is also that in the step 3, the reward number of the ith step is constructed by using the formula (1) ;

[0017] (1)

[0018] In the formula (1), represents the energy recovery reward value of the ith step, and there is:

[0019] (2)

[0020] In the formula (2), represents the battery power threshold, represents the energy recovery efficiency of the ith step, and there is:

[0021] (3)

[0022] (4)

[0023] In the formula (3), is the battery power of the ith step, is the maximum available battery power; when the remaining battery power > 90%, the energy recovery circuit is open and the energy recovery stops;

[0024] In the formula (4), is the energy recovery energy of the suspension in the ith step, is the total energy absorbed by the suspension in the ith step.

[0025] Furthermore, in the step 5, the output value of the ith step of the action network is obtained by using the formula (5) ;

[0026] (5)

[0027] In the formula (5), are the parameters of the action network; represents the The output of the fully-connected layer The corresponding activation value, with Tanh as the output function, is the bias matrix of the output layer, is the bias vector of the output layer, and the activation value corresponding to the output of the first fully-connected layer is calculated using Equation (6) The corresponding activation value For the output of the k-th fully-connected layer The corresponding activation value is as follows:

[0028] (6)

[0029] In Equation (6), is the bias matrix of the first fully-connected layer, is the bias matrix of the k-th fully-connected layer, ReLU is the activation function, is the bias vector of the first fully-connected layer, is the bias vector of the k-th fully-connected layer; ;

[0030] The action amount of the suspension at the step is obtained using Equation (7) where, is the suspension stiffness at the step, is the suspension damping coefficient at the step, is the magnitude of the external current applied to the motor at the step; is the PWM duty cycle of the H-bridge in the energy recovery circuit at the step;

[0031] (7)

[0032] In Equation (8), represents the random noise at the step.

[0033] Furthermore, Step 6 includes the following steps:

[0034] Step 6.1: Calculate the priority weight of the th sample using Equation (8) as follows:

[0035] (8)

[0036] In Equation (8), , , , are 4 weight coefficients;

[0037] If the vertical body acceleration in the step exceeds the comfort threshold or the dynamic deflections of the front and rear wheel suspensions in the step are close to the mechanical limit or the battery power in the step reaches the protection limit value, then the value of will be increased; otherwise, it remains unchanged;

[0038] Step 6.2: Calculate the probability that the th sample is selected by using Equation (9), so as to obtain the probabilities that all samples are selected;

[0039] (9)

[0040] In Equation (9), α is the priority strength coefficient for controlling the sampling bias degree; is the priority weight of the

[0041] th sample; Step 6.3: Define the current iteration number as c and initialize c = 1; define the update frequency as and the maximum iteration number as ; define the parameters of the action network at the c-th iteration as , define the environment interaction parameters of the action network at the c-th iteration as , and define the parameters of the value network at the c-th iteration as

[0042] Step 6.4: Calculate the policy probability ratio of the th sample at the c-th iteration by using Equation (10):

[0043] (10)

[0044] In Equation (10), is the policy output by the action network at the c-th iteration, is the environment interaction policy output by the action network at the c-th iteration; is the probability of selecting action by policy in state ; is the probability of selecting action by policy in state ;

[0045] Draw a batch of samples for the c-th iteration from the sample pool D according to the probability that all samples are selected , based on a batch of samples for the c-th iteration , use Equation (11) to construct the priority-weighted action objective function of the action network at the c-th iteration :

[0046] (11)

[0047] In Equation (11), is the clipping threshold,[[]] is the expected value of the -th sample at the c-th iteration,[[]] is the clipping function,[[]] represents the advantage function of the -th sample at the c-th iteration, and there is:

[0048] (12)

[0049] In Equation (12), , represent the predicted values of the value network at the c-th iteration for and respectively;

[0050] Step 6.5: Use Equation (13) to construct the priority-weighted value function loss of the value network at the c-th iteration :

[0051] (13)

[0052] In Equation (13), represents the discount factor,[[]] , represent the predicted values of the value network at the c-th iteration for and respectively;

[0053] Step 6.6: If the remainder of the ratio of to is 1, use Equations (14) and (15) to update , respectively to obtain the parameters of the policy network at the (c + 1)-th step and the parameters of the value network at the (c + 1)-th step , otherwise, do not update , :

[0054] (14)

[0055] In formula (14), is the learning rate, is the gradient operator of the policy network parameters ;

[0056] (15)

[0057] In formula (15), are the parameters of the value network is the gradient operator of

[0058] Step 6.7: After assigning to , judge whether holds. If it holds, it means that the optimal deep reinforcement learning model is obtained; otherwise, return to Step 6.4 and execute sequentially.

[0059] An electronic device according to the present invention includes a memory and a processor, characterized in that the memory is used to store a program that supports the processor to execute the energy recovery full active suspension control method, and the processor is configured to execute the program stored in the memory.

[0060] A computer-readable storage medium according to the present invention, characterized in that a computer program stored on the computer-readable storage medium executes the steps of the energy recovery full active suspension control method when run by a processor.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] 1. The present invention innovatively applies the deep reinforcement learning method to the full active suspension control. This method first conducts a large number of trainings in a simulation environment and is applied to the HIL bench test and real vehicle test after meeting the requirements. Since this application uses the reinforcement learning method of deep neural network, the suspension can ensure good comfort and road adaptability under different road conditions while ensuring safety. At the same time, by introducing an electromagnetic energy recovery device, the present invention can convert the vibration energy of the suspension system into electrical energy for storage, significantly improving the overall energy efficiency of the vehicle and realizing the collaborative optimization of energy recovery and suspension performance.

[0063] 2. When training the neural network in the present invention, it is assumed that the samples are independent and identically distributed. However, there is a temporal correlation between the data collected through reinforcement learning. Directly using this data for sequential training may lead to biases and instabilities in the policy optimization process. To address the issue of sample independence and improve sample efficiency, a trajectory pool of finite size is created in the PPO method of the present invention to store the sequence of experiences of the agent interacting in the environment. Before each step of training, a certain number of fresh experiences are collected according to the current policy and added to the trajectory pool. Subsequently, in each update cycle, PPO randomly samples multiple mini-batches from the trajectory pool to approximately calculate the advantage function and adjust the policy parameters. This approach not only makes more efficient use of the samples but also reduces the correlation between samples through the random selection of mini-batches, helps avoid local optima, promotes more robust policy learning, and thus ensures the convergence and stability of the training process.

[0064] 3. Through the design of the multi-objective reward function of the PPO algorithm in the present invention, the real-time energy recovery amount of the electromagnetic energy recovery device is taken as one of the important optimization objectives, while also taking into account indicators such as vehicle body vertical acceleration, suspension dynamic deflection (and vehicle body vertical displacement. Under different driving conditions (such as bumpy roads, turning, straight driving), the suspension parameters can be dynamically adjusted to maximize the energy recovery efficiency while ensuring comfort, significantly improving the overall performance of the vehicle.

[0065] 4. The present invention is highly intelligent and adaptable. It can automatically adjust the suspension control strategy according to real-time road conditions and driving demands, adapting to different driving styles and environmental changes. For example, it prioritizes optimizing comfort and energy recovery on bumpy roads, ensures ride comfort first when turning, and maximizes energy recovery when driving straight. This intelligent control method not only enhances the driving experience of the vehicle but also reduces energy consumption, conforming to the development trend of energy conservation and environmental protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is the deep reinforcement learning control framework diagram for energy recovery full-active suspension;

[0067] Figure 2 is the four-quadrant operation principle diagram of the motor;

[0068] Figure 3 is the schematic diagram of the energy recovery circuit;

[0069] Figure 4 is the schematic diagram of the deep reinforcement learning algorithm based on the PPO network. DETAILED DESCRIPTION OF THE INVENTION

[0070] The following will clearly and completely describe the technical solutions in conjunction with the accompanying drawings in the embodiments of the present invention.

[0071] In this embodiment, a method for controlling an energy recovery fully active suspension based on deep reinforcement learning includes the following steps:

[0072] Step 1: Construct a simulation dynamics model of an energy recovery 1 / 2 vehicle four-degree-of-freedom fully active suspension and a random road surface model, and select Class A, Class B, and Class C road surfaces as the random road surface models under low-speed, medium-speed, and high-speed conditions respectively;

[0073] As Figure 1 shown, the fully active suspension reinforcement learning control framework of this embodiment includes the following parts: the main body of the fully active suspension reinforcement learning controller, the fully active suspension simulation model, the simulation road surface model, the electromagnetic vibration energy recovery device, the state observation quantity, the fully active suspension action quantity, and the reward. The controller obtains the state observation quantities (suspension dynamic deflection, body acceleration, body vertical displacement) from the fully active suspension system, and uses the controller strategy to control the suspension stiffness, suspension damping coefficient, motor current magnitude, and H-bridge PWM duty cycle of the energy recovery circuit for the fully active suspension and its electromagnetic actuator at each state. The suspension then changes the current suspension state quantity according to the active control force output by the controller, and the actuator performs energy recovery according to the current and load resistance values output by the controller, and generates a reward based on this state to evaluate the pros and cons of the strategy and update it.

[0074] The four-degree-of-freedom model of the fully active suspension can describe the vertical movement of the body, the pitching movement, and the vertical movement of the front and rear wheels, can more comprehensively reflect the dynamic characteristics of the vehicle in the vertical direction, and can avoid the complexity and computational burden brought by multi-directional coupling, focusing on the core vertical dynamics problem.

[0075] The main purpose of the present invention is to design a suspension control method when the vehicle is driving on a good road surface. Therefore, a random road surface model is adopted: generally, the statistical characteristic parameter of the spatial frequency power spectral density and its time-domain form are used to describe the unevenness of the random road surface, and the road surface power spectral density The fitting expression is as shown in Equation (1):

[0076] (1)

[0077] Where: is the power spectral density of the vertical displacement ; is the spatial frequency , which defines the number of wavelengths per meter of the road surface; is the reference spatial frequency , = 0.1 ; is the road surface unevenness coefficient ; 𝑊 is the frequency exponent, usually taken as 𝑊 = 2.

[0078] The road surface roughness is divided into eight levels according to the road surface power spectrum density, as shown in Table 1. The table gives the road surface roughness of each level when 𝑊=2. The upper and lower limits and their geometric mean are also listed. < 𝑛 <2.83 The corresponding root mean square value of road roughness within the range The geometric mean of .

[0079] Table 1 Standard classification table for road roughness

[0080]

[0081] According to the requirements, Class A, Class B, and Class C roads are taken as random road models under low-speed, medium-speed, and high-speed conditions, respectively, to simulate highways, urban roads, and old roads with bumps.

[0082] Step 2: Considering the influence of different driving conditions on suspension control force, define the action space and state space parameters of the suspension according to the main performance evaluation indicators of the suspension system:

[0083] ① The vertical acceleration of the vehicle body is used to characterize the smoothness and ride comfort of the vehicle;

[0084] ② The dynamic deflection of the suspension is the displacement change from the equilibrium position (static deflection position) to the maximum compression or extension position of the suspension system under dynamic conditions (such as when the vehicle passes through bumpy roads or turns). Appropriate dynamic deflection helps reduce the vibration transmitted to the vehicle body and improve the riding comfort of passengers. If the dynamic deflection is too small, it may cause the vehicle to feel too "hard" when encountering bumps during driving, affecting the riding experience; while excessive dynamic deflection may make the vehicle appear loose and lack handling stability;

[0085] ③Vertical displacement of the vehicle body. Changes in the vertical displacement of the vehicle body will affect the center of gravity height and roll angle of the vehicle, thereby affecting ride comfort;

[0086] ④Battery remaining power: The battery remaining power is used as the state parameter of the deep reinforcement learning network to achieve the coordinated optimization of ride comfort and energy recovery;

[0087] Define the state parameter set of the suspension , , is the total number of steps, For the The normalized state quantity of the step, and ,in, For the The vehicle body acceleration of the step, , are the front and rear wheel suspension dynamic deflections at the th step respectively, is the vehicle body displacement at the th step, is the remaining battery power at the th step;

[0088] The energy recovery full active suspension of the present invention is optimized on the basis of the original passive suspension structure. The motor type electromagnetic actuator is placed inside the spring and outside the damper, and has the same dynamic stroke as the spring and the damper, forming a parallel structure. When the electromagnetic actuator acts as a motor and the suspension system is in the active control mode, various sensors installed on the vehicle body collect signals such as the displacement and acceleration of the vehicle and transmit them to the control system. After receiving the signals, the control system calculates the required current output to control the output of the Ampere force, and generates the corresponding acting force through the coil in the electromagnetic actuator to offset the road surface input excitation resistance, so as to suppress the vehicle body vibration. When the electromagnetic actuator works as a generator and the suspension system is in the energy recovery mode, the coil of the actuator moves with the suspension, cuts the magnetic induction line and generates current. The current converts the alternating current into direct current through the energy recovery circuit and stores it in the energy storage device, realizing the process of converting the mechanical energy of vibration into electrical energy.

[0089] The four-quadrant operation principle of the voice coil motor is as Figure 2 shown. Taking the x-axis to represent the motor torque direction and the y-axis to represent the mover speed direction, the rotary motor is torque and speed. The torque and speed of the motor form the xoy plane coordinate system. When the motor torque is positive and the motion speed is positive, the motor works in the first quadrant and is in the forward rotation electric state. Similarly, when the motor works in the third quadrant, it is in the reverse rotation electric state; when the motor torque is negative and the motion speed is positive, the motor works in the second quadrant and is in the forward rotation power generation state. Similarly, when the motor works in the fourth quadrant, it is in the reverse rotation power generation state. Vibration energy recovery utilizes the four-quadrant operation and regenerative braking principle of the motor. That is, when the thrust of the electromagnetic actuator is in the same direction as the relative speed between the mover and the stator (the difference between the sprung mass speed and the unsprung mass speed), that is, the motor operates in the electric state (the first and third quadrants), the electromagnetic actuator consumes energy and its acting force simultaneously suppresses the suspension vibration; when the thrust of the electromagnetic actuator is opposite to its speed, that is, the motor operates in the power generation state (the second and fourth quadrants), the suspension does work on the motor, and the electrical energy flows through the actuator to the power supply, and the vibration energy is converted into electrical energy and collected by the energy storage part.

[0090] This energy recovery fully active suspension selects an electromagnetic actuator as a moving coil voice coil linear motor. This type of voice coil motor is a single-phase two-pole device, and its driving method can use an integrated inverter-rectifier H-bridge circuit. The driving circuit connects the motor and the DC-DC circuit, and realizes four-quadrant operation through the inverter-type H-bridge driving circuit, enabling the motor to switch working quadrants according to the working conditions, realizing the active control and energy recovery of the actuator. The feedback energy is stored in the energy storage module with a super capacitor as the energy storage element through the DC-DC control circuit. The principle is as Figure 3 shown.

[0091] Select the stiffness of the suspension, the damping coefficient, the magnitude of the current connected to the motor, and the PWM duty cycle of the H-bridge in the energy recovery circuit as the action space of the deep reinforcement learning network: control the active control force output by the suspension by outputting the stiffness of the suspension, the damping coefficient, and the magnitude of the current connected to the motor through the network; control the energy recovery efficiency by outputting the PWM duty cycle of the H-bridge in the energy recovery circuit through the network.

[0092] Define the action space set of the suspension , where, is the suspension stiffness, and is the suspension damping coefficient , is the magnitude of the external current connected to the motor, is the PWM duty cycle of the H-bridge in the energy recovery circuit;

[0093] Step 3. Use Equation (2) to construct the reward number at the th step;

[0094] (2)

[0095] In Equation (2), represents the energy recovery reward value at the i-th step, and there is:

[0096] (3)

[0097] In Equation (3), represents the battery power threshold, represents the energy recovery efficiency at the i-th step, and there is:

[0098] , (4)

[0099] (5)

[0100] In Equation (4), is the battery power at the i-th step, is the maximum available battery power; when the remaining battery power > 90%, the energy recovery circuit is open and the energy recovery stops;

[0101] In Equation (5), is the energy recovered by the suspension at the i-th step, is the total energy absorbed by the suspension at the i-th step;

[0102] Among them, negative rewards are given to the squared values of the four state variables of vehicle body acceleration, front and rear wheel suspension dynamic deflections, and vehicle body displacement respectively to penalize the states that affect ride comfort; the design of the energy recovery reward value enables the control method to prioritize energy recovery at low SOC and reduce the energy recovery efficiency at high SOC to achieve the collaborative optimization of energy recovery and ride comfort;

[0103] Step 4: Construct a deep reinforcement learning network, which consists of an input layer with 5 neurons, a hidden layer with a fully connected layer, an action network composed of an output layer with 4 neurons, and an input layer with 5 neurons, a hidden layer with a fully connected layer, and a value network composed of an output layer with 1 neuron; among them, the number of neurons in the input layer of the action network and the value network corresponds to the number of state parameters, and the number of neurons in the output layer of the action network corresponds to the number of action parameters;

[0104] Step 5: Input into the action network for processing, and use Equation (6) to obtain the output value of the action network at the i-th step ;

[0105] (6)

[0106] In Equation (5), are the parameters of the action network; represents the corresponding activation value output by the layer of the fully connected layer, Tanh is the output function, is the bias matrix of the output layer, is the bias vector of the output layer, and use Equation (7) to calculate the corresponding activation value output by the first layer of the fully connected layer , the corresponding activation value output by the k-th layer of the fully connected layer :

[0107] (7)

[0108] In Equation (6), is the bias matrix of the first layer of the fully connected layer, is the bias matrix of the k-th layer of the fully connected layer, ReLU is the activation function, is the bias vector of the first layer of the fully connected layer, is the bias vector of the k-th fully connected layer; .

[0109] The action amount of the suspension at the -th step is obtained by using Equation (8), where , is the suspension stiffness at the -th step, is the suspension damping coefficient at the -th step, is the magnitude of the external current connected to the motor at the -th step; is the PWM duty cycle of the H-bridge in the energy recovery circuit at the -th step;

[0110] (8)

[0111] In Equation (8), represents the random noise at the -th step.

[0112] After the simulation dynamics model executes the action amount at the -th step, the state at the +1-th step is obtained, thus forming the -th sample, denoted as , and then m samples are obtained and stored in the sample pool D.

[0113] Step 6: As shown in the schematic diagram of the deep reinforcement learning algorithm based on the PPO network in Figure 4 , using this principle, based on the sample pool D, the deep reinforcement learning network is trained using the PPO algorithm on the A-level, B-level, and C-level simulation road surface models under three different working conditions of low speed, medium speed, and high speed to obtain the optimal deep reinforcement learning model:

[0114] Step 6.1: Calculate the priority weight of the -th sample using Equation (9):

[0115] (9)

[0116] In Equation (8), , , , are 4 weight coefficients; by using the dynamic weight , the learning efficiency for key states such as large-amplitude vibration and high-energy recovery scenarios is enhanced, which is different from traditional uniform sampling.

[0117] If the vertical acceleration of the vehicle body at the -th step Exceeding the comfort threshold or the front and rear wheel suspension dynamic deflections at the approach the mechanical limit or the battery power at the reaches the protection limit value, then the value of will be increased; otherwise, it remains unchanged.

[0118] Step 6.2: Calculate the probability that the k-th sample is selected using Equation (10) , so as to obtain the probabilities that all samples are selected;

[0119] (10)

[0120] In Equation (10), α is the priority strength coefficient that controls the sampling bias degree; is the priority weight of the k-th sample.

[0121] Step 6.3: Define the current iteration number as c and initialize c = 1; Define the update frequency as , and the maximum iteration number as ; Define the parameters of the action network at the c-th iteration as , define the environment interaction parameters of the action network at the c-th iteration as , and define the parameters of the value network at the c-th iteration as ;

[0122] Step 6.4: Calculate the policy probability ratio of the k-th sample at the c-th iteration using Equation (11) :

[0123] (11)

[0124] In Equation (10), is the policy output by the action network at the c-th iteration, is the environment interaction policy output by the action network at the c-th iteration; is the probability of selecting action in state by policy at the c-th iteration; is the probability of selecting action in state by policy at the c-th iteration.

[0125] According to the probabilities that all samples are selected, draw a batch of samples for the c-th iteration from the sample pool D ​​, a batch of samples based on the c-th iteration , use Equation (12) to construct the priority-weighted action objective function of the action network at the c-th iteration :

[0126] (12)

[0127] In Equation (12), is the clipping threshold, is the expected value of the -th sample at the c-th iteration, is the clipping function, which is updated by weighted gradient to preferentially optimize samples that have a significant impact on suspension performance (such as severe vibration or high energy recovery scenarios), represents the advantage function of the -th sample at the c-th iteration, and there is:

[0128] (13)

[0129] In Equation (13), , represent the predicted values of the value network at the c-th iteration for and respectively.

[0130] Step 6.5: Use Equation (14) to construct the priority-weighted value function loss of the value network at the c-th iteration :

[0131] (14)

[0132] In Equation (13), represents the discount factor, , represent the predicted values of the value network at the c-th iteration for and respectively.

[0133] Step 6.6: Update , respectively to obtain the parameters of the policy network at the (c + 1)-th step and the parameters of the value network at the (c + 1)-th step , otherwise, do not update , :

[0134] (15)

[0135] In Equation (15), is the learning rate, is the parameter of the policy network Gradient operator of

[0136] (16)

[0137] In formula (16), is the parameter of the value network Gradient operator of

[0138] Step 6.7: After assigning to , judge whether holds. If it holds, it means that the optimal deep reinforcement learning model is obtained; otherwise, return to Step 6.4 and execute sequentially;

[0139] Step 7: Use the optimal action network in the optimal deep reinforcement learning model to process the real-time input state information and output the corresponding control instructions, so as to realize the active control of the energy recovery full active suspension.

[0140] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0141] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is run by a processor, it executes the steps of the above method.

Claims

1. A fully active suspension control method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1, construct a simulation dynamics model and a random road surface model of a four-degree-of-freedom fully active suspension of an energy recovery half vehicle; Step 2: Define the state parameter set of the suspension , , is the total number of steps, For the The normalized state quantity of the step, and ,in, For the The vehicle body acceleration of the step, , Respectively The dynamic deflection of the front and rear wheel suspension, For the Step body displacement, For the The remaining battery power of the step; Define the action space set of the suspension ,in, is the suspension stiffness, is the suspension damping coefficient , is the external current connected to the motor, is the PWM duty cycle of the H-bridge in the energy recovery circuit; Step 3: Build Number of rewards per step ; Step 4: Build a deep reinforcement learning network, including an input layer with 5 neurons, The action network consists of a hidden layer of fully connected layers, an output layer of 4 neurons, and an input layer of 5 neurons. A value network consisting of a hidden layer of fully connected layers and an output layer of 1 neuron; Step 5: Input the action network for processing, and obtain the suspension in the first The amount of movement of the step The simulation dynamics model performs the The amount of movement of the step After that, get +1 step status , thus forming the Samples, denoted as , and then obtain m samples and store them in the sample pool D; Step 6: Based on the sample pool D, the PPO algorithm is used to train the deep reinforcement learning network to obtain the optimal deep reinforcement learning model; Step 7: Use the optimal action network in the optimal deep reinforcement learning model to process the real-time input state information and output corresponding control instructions to achieve active control of the energy recovery fully active suspension.

2. The energy recovery fully active suspension control method based on deep reinforcement learning according to claim 1 is characterized in that: In step 3, formula (1) is used to construct the Number of rewards per step ; (1) In formula (1), represents the energy recovery reward value of the i-th step, and has: (2) In formula (2), Indicates the battery level threshold. represents the energy recovery efficiency of the i-th step, and: (3) (4) In formula (3), is the battery power at step i, The maximum available power of the battery; when the remaining power of the battery is >90%, the energy recovery circuit is open and energy recovery stops; In formula (4), is the energy recovery energy of the suspension at step i, is the total energy absorbed by the suspension at step i.

3. The energy recovery fully active suspension control method based on deep reinforcement learning according to claim 2 is characterized in that: In step 5, the output value of the i-th step of the action network is obtained by using formula (5) ; (5) In formula (5), are the parameters of the action network; Indicates The output of the fully connected layer The corresponding activation value, Tanh is the output function, is the bias matrix of the output layer, is the bias vector of the output layer, and the output of the first fully connected layer is calculated using formula (6): The corresponding activation value , the output of the kth fully connected layer The corresponding activation value : (6) In formula (6), is the bias matrix of the first fully connected layer, is the bias matrix of the kth fully connected layer, ReLU is the activation function, is the bias vector of the first fully connected layer, is the bias vector of the kth fully connected layer; ; Using formula (7), we can get the suspension The amount of movement of the step ,in, For the The suspension stiffness of the step, For the The suspension damping coefficient of the step, For the The external current connected to the stepping motor; For the The PWM duty cycle of the H-bridge in the energy recovery circuit; (7) In formula (8), Indicates The random noise of the step.

4. The energy recovery fully active suspension control method based on deep reinforcement learning according to claim 3 is characterized in that: The step 6 comprises the following steps: Step 6.1: Use equation (8) to calculate Priority weight of the sample : (8) In formula (8), , , , are 4 weight coefficients; Jordi The vertical acceleration of the vehicle body Exceeding the comfort threshold or Dynamic deflection of front and rear wheel suspension Approaching mechanical limits or Battery level of the step If the protection limit is reached, the The value of ; otherwise, remain unchanged; Step 6.2: Use equation (9) to calculate the The probability of a sample being selected , thus obtaining the probability of all samples being selected; (9) In formula (9), α is the priority strength coefficient that controls the degree of sampling bias; is the priority weight of the k-th sample; Step 6.3: Define the current number of iterations as c, and initialize c=1; define the update frequency as , the maximum number of iterations is ; Define the parameters of the action network at the cth iteration as , define the environment interaction parameter of the action network at the cth iteration as , define the parameters of the value network at the cth iteration as ; Step 6.4: Use formula (10) to calculate the cth iteration The strategy probability ratio of the sample : (10) In formula (10), is the strategy output by the action network in the cth iteration, is the environment interaction strategy output by the action network at the cth iteration; The strategy for the cth iteration In Status Select action The probability of The strategy for the cth iteration In Status Select action The probability of According to the probability of all samples being selected, a batch of samples for the cth iteration are drawn from the sample pool D , based on a batch of samples from the cth iteration , using formula (11) to construct the priority-weighted action objective function of the action network at the cth iteration : (11) In formula (11), is the clipping threshold, For the The expected value of the samples at the cth iteration, is the clipping function, Indicates The advantage function of the samples at the cth iteration is: (12) In formula (12), , Indicates that the value network is respectively and The predicted value of Step 6.5: Use formula (13) to construct the priority-weighted value function loss of the value network at the cth iteration : (13) In formula (13), represents the discount factor, , Indicates that the value network is respectively and The predicted value of Step 6.6: If and When the remainder of the ratio is 1, use equations (14) and (15) to calculate , Update to get the parameters of the policy network at step c+1 and the parameters of the value network at step c+1 Otherwise, do not update , : (14) In formula (14), is the learning rate, is the policy network parameter The gradient operator of (15) In formula (15), is the parameter of the value network The gradient operator of Step 6.7: Assign to After that, judge Is it true? If so, it means that the optimal deep reinforcement learning model is obtained; otherwise, return to step 6.4 and execute sequentially.

5. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the energy recovery full-active suspension control method described in any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the energy recovery full active suspension control method according to any one of claims 1 to 4 are executed.

Images