Intelligent suspension control method, medium and device combined with deterministic experience tracking

Abstract

The present invention relates to the field of vehicle dynamics control technology, and more particularly to an intelligent suspension control method, medium, and device incorporating deterministic empirical tracking. The method comprises pre-training two evaluation networks and a control strategy network; using an initial control strategy model obtained from the pre-training to determine a first control force output by the intelligent suspension based on a first state of the vehicle; applying the first control force to the vehicle to obtain the vehicle's next first state; determining a first reward and an auxiliary reward group based on the first control force and the first state to obtain a second sample; re-training the two pre-trained initial evaluation models and initial control strategy models using multiple groups of second samples; and adding a perturbation to the first control force during the training process to obtain two corresponding final evaluation models and final control strategy models. The present invention effectively addresses existing issues such as low sample utilization efficiency, poor data stability, and difficulty in mining high-value data.

Description

Technical Field

[0001] The present invention belongs to the technical field of automobile dynamics control, and in particular relates to an intelligent suspension control method, medium and device combined with deterministic empirical tracking. Background Art

[0002] Deep reinforcement learning (RL) technology has become a hot topic in autonomous vehicle research, demonstrating significant potential in key areas such as visual perception, decision-making, and control. Based on a data-driven learning paradigm, this technology offers numerous advantages for autonomous vehicle control strategies. Deep RL generates a rich data resource through continuous interaction with the environment, which it then uses to self-optimize control behavior. This technology demonstrates strong adaptability to new environments and is particularly well-suited for solving complex and dynamic control challenges. Furthermore, it allows developers to guide efficient exploration and exploitation of the intelligent agent (i.e., controller) by setting clear control objectives, significantly reducing the time-consuming burden of manual parameter tuning required in traditional methods. Within the control architecture of autonomous driving technology, vertical control technology for intelligent suspensions is crucial for enhancing ride comfort, a key factor influencing public acceptance of autonomous vehicles.

[0003] When applying deep reinforcement learning control algorithms to intelligent suspension systems, existing technologies face a core challenge: sample estimation and fitting in high-dimensional space, the so-called "sample crisis." This crisis is mainly reflected in the efficiency and quality control of sample data.

[0004] The Chinese patent publication number is CN112078318A, the publication date is December 15, 2020, and the patent name is "An intelligent control method for automobile active suspension based on deep reinforcement learning algorithm", and

[0005] Taking the invention patent application with Chinese patent publication number CN111487863A and publication date August 4, 2020, and the patent name "A reinforcement learning control method for active suspension based on deep Q neural network" as an example, the current technology generally adopts the method of processing the data of each independent moment and single control as a single sample. This method ignores the temporal logic and correlation between data in the control process, resulting in a large amount of valuable temporal information not being fully mined and utilized, significantly reducing the overall utilization of the data.

[0006] Furthermore, the capacity limitations of a single data sample not only weaken the data's stability but also make it more susceptible to external environmental interference, thereby increasing inter-data variability. This poses a significant challenge to the convergence speed and stability of the control strategy. In extreme cases, the control strategy may struggle to converge to the optimal solution due to excessive data fluctuations. More seriously, when a large amount of low-value, low-information data floods the training process, truly high-value data crucial to control strategy optimization is often overwhelmed, making it difficult to effectively identify and utilize. This situation severely impacts the effectiveness, robustness, and ultimately optimality of intelligent suspension control methods. Summary of the Invention

[0007] In view of this, the present invention aims to provide an intelligent suspension control method, medium and device combined with deterministic experience tracking to assist in rewarding the completion of the experience tracking memory mechanism. This mechanism is specifically designed to integrate and process information generated in a dense reward environment, effectively identify and amplify high-value data that have a significant contribution to the optimization of the control strategy, thereby accelerating the optimization iteration process of the value and strategy network; in addition, according to the network training process, a disturbance is added to the first control force to further promote the convergence of the training. The method provided by the present invention effectively solves the problems of low sample utilization efficiency, poor data stability and difficulty in mining high-value data in the prior art, thereby improving the overall performance and robustness of the intelligent suspension control method.

[0008] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0009] An intelligent suspension control method combined with deterministic empirical tracking, comprising:

[0010] S1: Obtain multiple groups of first samples, and use the first samples to pre-train the constructed first evaluation network, second evaluation network, and control strategy network to obtain an initial first evaluation model, an initial second evaluation model, and an initial control strategy model respectively;

[0011] S2: The initial control strategy model obtained in step S1 determines a second control force output by the intelligent suspension at the current moment based on the first state of the vehicle at the current moment; applies the second control force to the vehicle to obtain a next second state of the vehicle; determines a second reward and an auxiliary reward group based on the second control force and the second state; and uses the second state, the second control force, the second reward, the auxiliary reward group, and the next second state as a second sample.

[0012] S3: Repeat step S2 multiple times to obtain multiple sets of second samples, and use the multiple sets of second samples to retrain the initial first evaluation model, initial second evaluation model, and initial control strategy model obtained in S1; during the training process, add a disturbance to the second control force, and change the disturbance according to the training effect; after training, obtain the corresponding final first evaluation model, final second evaluation model, and final control strategy model;

[0013] S4: Run the intelligent suspension and control the intelligent suspension using the final first evaluation model, the final second evaluation model, and the final control strategy model trained in step S4.

[0014] Furthermore, in step S1, each group of first samples includes: a first state of the vehicle at a certain moment, a first control force output by the intelligent suspension at the same moment, a next first state of the vehicle after the intelligent suspension applies the first control force to the vehicle, and a first reward obtained based on the first state, the first control force, and the first state.

[0015] Furthermore, the first state ,in, and denote the acceleration and velocity of the unsprung mass of the vehicle at time t, represents the dynamic travel of the smart suspension at time t, represents the vertical displacement of the vehicle at time t, represents the unsprung displacement of the vehicle at time t; represents the velocity difference between the sprung and unsprung masses at time t, represents the unsprung speed of the vehicle at time t;

[0016] The first reward is:

[0017] ;

[0018] in, represents the first reward at time t, 、 '、 and represents the first reward coefficient, represents the dynamic wheel load of the vehicle at time t, , represents the road height incentive information at time t, represents the first control force, P represents the control trigger coefficient, Indicates the ultimate dynamic deflection of the intelligent suspension.

[0019] Furthermore, the pre-training in step S1 includes:

[0020] In each pre-training time step, the first sample is extracted and the corresponding first target value is calculated by the following formula:

[0021] ;

[0022] in, represents the first target value, represents the discount factor, represents the j-th evaluation network, represents the network weight of the j-th evaluation network; represents the control strategy network, represents the network weight of the control strategy network, Indicates the next first state at time t;

[0023] Combined with the first target value, the loss function for training the two evaluation networks is calculated:

[0024] ;

[0025] in, represents the loss function of the j-th evaluation network, Represents the time step of pre-training; the two evaluation networks are pre-trained using their loss functions; the control strategy network is pre-trained using the following formula:

[0026] ;

[0027] in, Represents the first state at time t.

[0028] Furthermore, in step S2, the second state The second reward is:

[0029] ;

[0030] in, represents the second reward at time t, 、 、 and represents the second reward coefficient, represents the second control force at time t;

[0031] The auxiliary reward groups are:

[0032] ;

[0033] in, represents the auxiliary reward group at time t, represents the auxiliary reward, k represents the step of obtaining the auxiliary reward; auxiliary reward for:

[0034] ;

[0035] in, 、 and Represents the auxiliary reward coefficient.

[0036] Furthermore, during the training process of step S3:

[0037] In each training time step, multiple sets of second samples , calculate the corresponding second target value by the following formula:

[0038] ;

[0039] in, Indicates the Get step, represents the second target value, represents the deterministic experience-assisted reward discount factor, represents the initial j-th evaluation model, represents the model weight of the initial j-th evaluation model; represents the initial control strategy model, represents the model weight of the initial control strategy model, Indicates the next second state at time t;

[0040] Combined with the second target value, the loss function for training the two initial evaluation models is calculated:

[0041] ;

[0042] in, Represents the loss function of the initial j-th evaluation model, and N represents the time step of training; the two initial evaluation models are trained using their loss functions;

[0043] The initial control strategy model is trained using the following formula:

[0044] ;

[0045] During training, if the second reward grows slowly, increase the perturbation; if the second reward grows steadily, reduce the perturbation.

[0046] Furthermore, in step S1, it further includes:

[0047] Increase the number of network layers in the first evaluation network and the second evaluation network, as well as the number of neurons in each network layer; reduce the number of network layers in the control strategy network, as well as the number of neurons in each network layer; pre-train the modified first evaluation network, the second evaluation network and the control strategy network.

[0048] Furthermore, between step S3 and step S4, the following steps are further included:

[0049] The final control strategy model trained in step S3 is used to control the intelligent suspension of other vehicles to obtain multiple sets of second samples again; the training process of step S3 is repeated, and the final first evaluation model, the final second evaluation model, and the final control strategy model are trained again using the new multiple sets of second samples.

[0050] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the intelligent suspension control method combined with deterministic empirical tracking provided by the present invention.

[0051] An electronic device, comprising:

[0052] memory for storing computer programs;

[0053] The processor is configured to implement the steps of the intelligent suspension control method combined with deterministic experience tracking provided by the present invention when executing a computer program.

[0054] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0055] (1) The present invention innovatively proposes a deterministic experience tracking mechanism using auxiliary rewards in the intelligent suspension control method combined with deterministic experience tracking. As a general off-policy method, it can accurately identify and amplify valuable data, thereby effectively promoting the rapid optimization and iteration of the value and policy network. This mechanism significantly improves data utilization efficiency and accelerates the learning process of the algorithm. In addition, the present invention uses an exploration strategy optimization mechanism based on reinforcement learning to add a perturbation to the first control force during the training process, further promoting the convergence of training and ensuring the universality and robustness of the model.

[0056] (2) The present invention creates an intelligent suspension control method that combines deterministic experience tracking. By introducing a transfer learning approach, the final control strategy model is used to control the intelligent suspension of other vehicles and is trained based on the conditions of other vehicles. This allows the resulting control strategy model to have good generalization and strong adaptability. This means that when faced with the complex and ever-changing control requirements of autonomous vehicles, the present invention can quickly adapt and provide the optimal control strategy, laying a solid foundation for the widespread application of autonomous driving technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0058] Figure 1 A flow chart of an intelligent suspension control method combined with deterministic experience tracking according to an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a framework of an intelligent suspension control method combined with deterministic empirical tracking according to an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a control strategy network according to an embodiment of the present invention;

[0061] Figure 4 Schematic diagram of two evaluation networks according to an embodiment of the present invention;

[0062] Figure 5 A schematic diagram of a dynamic model of an intelligent suspension system according to an embodiment of the present invention;

[0063] Figure 6 A schematic structural diagram of an electronic device according to an embodiment of the present invention.

[0064] Description of reference numerals:

[0065] 1. Electronic device; 2. External device; 3. Processing unit; 4. Bus; 5. Network adapter; 6. Display; 7. (I / O) interface; 8. System memory; 9. Random access memory; 10. Cache memory; 11. Storage system; 12. Utility; 13. Program module. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0067] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0068] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first", "second" and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0069] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art can understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0070] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0071] like Figures 1 to 2 As shown, the intelligent suspension control method combined with deterministic experience tracking described in the embodiment of the present invention includes:

[0072] S1: Obtain multiple groups of first samples, and use the first samples to pre-train the constructed first evaluation network, second evaluation network, and control strategy network to obtain an initial first evaluation model, an initial second evaluation model, and an initial control strategy model.

[0073] In one embodiment, the control strategy network is as follows: Figure 3 As shown in the figure, the vehicle state is input into a fully connected layer composed of 128 neurons for processing. The processed data is processed by the ReLU activation function and then input into a fully connected layer composed of 128 neurons. The output data is processed by the ReLU activation function and then input into a fully connected layer composed of 1 neuron. Finally, the output data is activated by the Tanh activation layer and scaled by the scaling layer to obtain the control force output by the intelligent suspension. The structures of the first evaluation network and the second evaluation network are shown in the figure. Figure 4As shown in the figure, the vehicle state is input into a fully connected layer composed of 128 neurons for processing; the processed data is processed by the ReLU activation function and then enters the fully connected layer composed of 200 neurons together with the control information output by the control strategy network. The output features are processed by the ReLU activation function and then input into the fully connected layer composed of 1 neuron to output the evaluation results. The initialization process of the three networks includes: The network weights in , Second Evaluation Network The network weights in , and control strategy network The network weight Perform random initialization, s represents the state of the vehicle, and a represents the control force output by the intelligent suspension.

[0074] In some embodiments, the intelligent suspension system dynamics model is as follows: Figure 5 As shown, Figure 5 in represents the vertical displacement of the vehicle at time t, represents the unsprung displacement of the vehicle at time t, Indicates the vehicle's body mass, represents the unsprung mass, represents the road displacement of the vehicle at time t, Indicates the vehicle's tire stiffness, represents the suspension spring stiffness of the smart suspension, Indicates the damping coefficient of the smart suspension.

[0075] Each group of first samples includes: the first state of the vehicle at time t , the first control force output by the intelligent suspension at time t , the intelligent suspension applies the first control force to the next first state of the vehicle after the vehicle , and according to the first state , first control force and the first state The first reward received Specifically, the first state Includes the acceleration of the vehicle's unsprung mass at time t and speed , dynamic travel of the intelligent suspension at time t , and the velocity difference between the sprung and unsprung masses of the vehicle at time t ,Right now Among them, the acceleration Characterizes the vehicle's ride comfort, acceleration and speed The vertical displacement of the vehicle body Calculate the second and first order derivatives to get; dynamic stroke Characterizes the dynamic deflection of the intelligent suspension, which is an important indicator of safety; speed difference , represents the unsprung speed of the vehicle at time t, and the unsprung speed Unsprung displacement Obtained by a single derivation.

[0076] In some embodiments, the first reward for:

[0077] ;

[0078] in, represents the dynamic wheel load of the vehicle at time t, ,wheel Dynamic loads characterize the handling stability of actively driven vehicles; 、 '、 and Indicates the first reward coefficient, the first reward coefficient 、 '、 and It can effectively balance multi-objective optimization problems. Its value is adaptively adjusted according to human experience or actual conditions. P represents the control trigger coefficient. The control trigger coefficient P is a fixed value used to prevent unsafe control strategies during training. The value of the control trigger coefficient P is adaptively selected and adjusted according to the actual situation. The above formula can be understood as follows: if the dynamic travel of the intelligent suspension at time t is Exceeding the suspension limit dynamic deflection , in the first reward In one embodiment, the control trigger coefficient P is set to -500, and the suspension limit dynamic deflection is , the first reward coefficient 、 '、 and The values are 0.7, 0.1, 0.1 and 0.1 respectively.

[0079] In one embodiment, the first control force is expressed as follows: Make a hard constraint:

[0080] ;

[0081] in, represents the interception function, and Respectively represent the limited first control force The minimum and maximum values of . and maximum value Obtained based on the maximum force limit that the intelligent suspension can output, a minimum value is set in a certain embodiment , maximum value .

[0082] In some embodiments, the process of pre-training the first evaluation network, the second evaluation network, and the control policy network includes:

[0083] In each pre-training time step, the first sample is extracted and the corresponding first target value is calculated by the following formula:

[0084] ;

[0085] in, represents the first target value, represents the discount factor, represents the j-th evaluation network, represents the network weight of the j-th evaluation network;

[0086] Combined with the first target value, the loss function for training the two evaluation networks is calculated:

[0087] ;

[0088] in, represents the loss function of the j-th evaluation network, Represents the time step of pre-training; the two evaluation networks are pre-trained using their loss functions; the control strategy network is pre-trained using the following formula:

[0089] .

[0090] In one embodiment, the discount factor The value is 0.99, the number of pre-training episodes is 2000, and the time step of each pre-training episode is set to ; Use the stochastic gradient descent method to train the two evaluation networks, and use the soft update method to adjust the network weights of the two evaluation networks during the training process. and To update, that is:

[0091] ;

[0092] in, and Represent the updated network weights of the two evaluation networks, Indicates the soft update frequency, the value is 0.001;

[0093] The control strategy network is trained using the stochastic gradient ascent method, and the network weights of the control strategy network are also updated using the soft update method during the training process. To update, that is:

[0094] ;

[0095] in, Indicates the network weight after the control strategy network is updated. The soft update frequency here is The value is still 0.001.

[0096] S2: The initial control strategy model obtained in step S1 determines the second control force output by the intelligent suspension at the current moment based on the second state of the vehicle at the current moment; applies the second control force to the vehicle to obtain the next second state of the vehicle; determines the second reward and auxiliary reward group based on the second control force and the second state; and uses the second state, the second control force, the second reward, the auxiliary reward group, and the next second state as the second sample.

[0097] In some embodiments, the second state , the second reward is:

[0098] ;

[0099] in, represents the second reward at time t, 、 、 and Indicates the second reward coefficient, the second reward coefficient 、 、 and It can also effectively balance multi-objective optimization problems, and its values are adaptively adjusted according to human experience or actual conditions. The above formula can be understood as follows: if the dynamic travel of the smart suspension at time t is Exceeding the suspension limit dynamic deflection , in the second reward Add a control trigger coefficient P to force the current round of training to stop directly.

[0100] The auxiliary reward groups are:

[0101] ;

[0102] in, represents the auxiliary reward group at time t, represents the auxiliary reward, k represents the step of obtaining the auxiliary reward; auxiliary reward for:

[0103] ;

[0104] in, 、 and Indicates the auxiliary reward coefficient, and its value is adaptively adjusted according to human experience or actual conditions. In a certain embodiment, the control trigger coefficient P is still set to -500, and the suspension limit dynamic deflection , the second reward coefficient 、 、 and The values are 0.7, 0.1, 0.1 and 0.1 respectively, and the auxiliary reward coefficient 、 and The values are 0.7, 0.1, and 0.1, respectively. The acquisition step k is set to k = [logN], where N represents the training time step in step S3, and [·] represents the rounding function. In one embodiment, an experience buffer pool is further provided for storing samples. The second state, second control force, second reward, and next second state are stored as intermediate samples in the experience buffer pool. The auxiliary reward group is determined and calculated in the experience buffer pool. The intermediate sample and the corresponding auxiliary reward group are combined to form the second sample, which is then stored in the experience buffer pool.

[0105] S3: Repeat step S2 multiple times to obtain multiple sets of second samples. The initial first evaluation model, initial second evaluation model, and initial control strategy model obtained in S1 are retrained using the multiple sets of second samples. During the training process, a disturbance is added to the second control force, and the disturbance is varied based on the training results. After training, the corresponding final first evaluation model, final second evaluation model, and final control strategy model are obtained. In one embodiment, the multiple sets of second samples are stored in a time-sequential buffer pool.

[0106] In some embodiments, the training process of step S3 includes:

[0107] In each training time step, multiple sets of second samples are used The corresponding second target value is calculated by the following formula:

[0108] ;

[0109] in, Indicates the Get step, represents the second target value, represents the deterministic experience-assisted reward discount factor, represents the initial j-th evaluation model, represents the model weight of the initial j-th evaluation model; represents the initial control strategy model, represents the model weight of the initial control strategy model, In one embodiment, multiple groups of first samples are randomly drawn from the experience buffer pool. , and calculate the corresponding first target value.

[0110] In one embodiment, the second control force is expressed as follows: Make a hard constraint:

[0111] ;

[0112] in, and Respectively represent the limited second control force The minimum and maximum values of . and maximum value It is also obtained based on the maximum force limit that the intelligent suspension can output, and in a certain embodiment, a minimum value is set. , maximum value .

[0113] Combined with the first target value, the loss function for training the two initial evaluation models is calculated:

[0114] ;

[0115] in, represents the loss function of the initial j-th evaluation model; the two initial evaluation models are trained using their loss functions;

[0116] The initial control strategy model is trained using the following formula:

[0117] ;

[0118] During the training process, if the second reward grows slowly, the perturbation amount is increased so that the final model obtained has a wider adaptability; if the second reward grows steadily, the perturbation amount is reduced to promote the convergence of the model training. The present invention introduces an exploration strategy optimization mechanism based on reinforcement learning. During the training process, a perturbation amount is added to the second control force, and the perturbation amount is changed according to the training effect, thereby improving the adaptability of the model in different environments and promoting the training of the model. Since the second training of the present invention adopts the training method of reinforcement learning, the data involved in the training is acquired in episodes, and each episode can be adjusted before the start, and modified before the start of the next episode based on the situation of the previous episode.

[0119] During training, this method adds a perturbation to the second control force and changes the perturbation based on the training results. By carefully analyzing the changes in the second reward, the variance of the Gaussian noise used as the perturbation is dynamically adjusted (with a mean of 0) to optimize the training effect and adaptability of the model. The specific adjustment strategy is as follows:

[0120] First, define a threshold parameter for measuring the change in the second reward , and a basic Gaussian noise variance .

[0121] The judgment on the growth of the second reward includes:

[0122] If in two consecutive training iterations, the second reward obtained in the latter training The second reward obtained from the previous training The difference between satisfy , the second reward is considered to be growing slowly. This means that under the current training state, the model's performance has not improved significantly, and it may be stuck in a local optimum or convergence is too slow. In this case, to enable the model to explore a wider range of state space, increase the variance of the Gaussian noise to improve the model's adaptability.

[0123] like If the second reward shows a continuous upward trend over multiple (e.g., 5) training iterations, the second reward is considered to be growing steadily. This indicates that the model is able to effectively learn and improve performance under the current training settings. To help the model converge to the optimal solution faster, the variance of the Gaussian noise is reduced.

[0124] Strategies for adjusting the Gaussian noise variance include:

[0125] When it is determined that the second reward increases slowly, a linear increase method is used to increase the variance of the Gaussian noise, that is, represents the new variance, Is a fixed incremental value used to control the amplitude of the variance increase. Increasing the variance gradually can increase the fluctuation range of Gaussian noise, guide the model to explore more different control strategies, and thus improve the generalization ability and adaptability of the model. In one embodiment, set .

[0126] When the second reward is considered to be growing steadily, an exponential decay method is used to reduce the variance of the Gaussian noise, that is, ,in is an attenuation coefficient less than 1, is an exponential factor related to the number of training iterations, i.e. As the training progressed, Gradually increases, making the variance decrease at an increasingly faster rate, thereby reducing noise interference and prompting the model to converge to the optimal solution faster.

[0127] Through the above precise judgment of the changes in the second reward and the dynamic adjustment strategy of the Gaussian noise variance as the disturbance quantity, it is possible to flexibly balance the model's exploration ability and convergence speed according to actual conditions during the model training process. The resulting model has both broad adaptability and the ability to quickly and accurately converge to the optimal intelligent suspension control strategy.

[0128] In one embodiment, the deterministic experience assistance reward discount factor The value is 0.9, the discount factor The same value is set to 0.99, the number of training episodes is still 2000, and the time step of each training episode is set , the perturbation amount is a Gaussian random number The process of adding a disturbance to the second control force and changing the disturbance according to the training effect takes effect in the first 1000 training episodes. The two initial evaluation models are trained using the stochastic gradient descent method. During the training process, the model weights of the two initial evaluation models are adjusted using the soft update method. and To update, that is:

[0129] ;

[0130] in, and Respectively represent the updated model weights of the two initial evaluation models, The value is 0.001;

[0131] The initial control strategy model is trained using the stochastic gradient ascent method. During the training process, the model weights of the initial control strategy model are also adjusted using the soft update method. To update, that is:

[0132] ;

[0133] in, Represents the network weight after the initial control strategy model is updated. The soft update frequency here is The value is still 0.001.

[0134] In some embodiments, step S3 further includes:

[0135] In step S1, it also includes:

[0136] Increase the number of network layers in the first evaluation network and the second evaluation network, as well as the number of neurons in each network layer; simultaneously reduce the number of network layers in the control strategy network, as well as the number of neurons in each network layer; and pre-train the modified first evaluation network, second evaluation network, and control strategy network. In one embodiment, the number of network layers in the two evaluation networks, as well as the number of neurons in each network layer, are increased by a factor of 2. In this case, the number of network layers in the control strategy network, as well as the number of neurons in each network layer, need to be reduced by a factor of 2. The present invention achieves compression pruning of the control strategy network through the above operations, greatly reducing the parameters and computational complexity of the network model, making it easier to deploy on resource-constrained hardware platforms.

[0137] S4: Run the intelligent suspension and control the intelligent suspension using the final first evaluation model, the final second evaluation model, and the final control strategy model trained in step S4.

[0138] In some embodiments, between step S3 and step S4, the method further includes:

[0139] The final control strategy model trained in step S3 is used to control the intelligent suspension of other vehicles, regenerating multiple sets of second samples. The training process in step S3 is repeated, and the final first evaluation model, the final second evaluation model, and the final control strategy model are retrained using the new sets of second samples. By utilizing transfer learning, the trained final control strategy model is used to control the intelligent suspension of other vehicles and train them, allowing the resulting model to quickly adapt to new scenarios, such as other similar vehicle models or road conditions, thereby reducing retraining time and data requirements.

[0140] The intelligent suspension control method combined with deterministic experience tracking provided by the present invention is compared with the deep deterministic policy gradient algorithm, the dual-delay deterministic policy gradient algorithm and the model predictive control algorithm. The method proposed by the present invention achieves a significant improvement of 74.92%, 64.20% and 54.64% in control performance, respectively. This data fully demonstrates the superiority and efficiency of the present invention in the field of intelligent suspension control. In addition, the intelligent suspension control method combined with deterministic experience tracking provided by the present invention is experimented under different speed conditions. On random roads of grades A, B and C, the optimization effect of the present invention on ride comfort is close to 90%. Even on the grade D road with extremely poor road conditions, the optimization amount is stable at about 85%. This result not only verifies the robustness of the present invention, but also highlights its excellent generalization performance under complex road conditions.

[0141] Based on the provided intelligent suspension control method combined with deterministic experience tracking, the present invention also provides an intelligent suspension control system that takes into account the time delay of the suspension system. The system includes an intelligent agent, a car or test bench equipped with an intelligent suspension and related sensors, a state observer and state estimator, and an intrinsic reward function. Among them, the intelligent agent controls the intelligent suspension, and the intelligent suspension control method combined with deterministic experience tracking provided by the present invention is integrated into the intelligent agent. The related sensors include acceleration sensors and displacement sensors and inertial measurement units (IMUs). The operation process of the intelligent suspension control system is as follows:

[0142] The intelligent agent obtains data such as vehicle body acceleration, vehicle body speed, suspension dynamic deflection and derivative of suspension dynamic deflection from the entire vehicle system through sensors, and combines these data into the current state information of the vehicle through a state observer and a state estimator. The intelligent agent then decides how much control force the intelligent suspension should output in the current state based on this state information. Under the action of the control force, the state of the entire vehicle changes. The intelligent agent generates a reward value to evaluate its control action based on the new system state and the intrinsic reward function. The intelligent agent performs self-iteration and control strategy optimization with reference to the reward value in accordance with the intelligent suspension control method combined with deterministic empirical tracking provided by the present invention.

[0143] Figure 6 FIG. 1 is a structural diagram of an electronic device 1 provided in an embodiment of the present invention. Figure 6 A block diagram of an exemplary electronic device 1 suitable for implementing embodiments of the present invention is shown. Figure 6 The electronic device 1 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0144] like Figure 6 As shown, electronic device 1 is represented in the form of a general-purpose computing device. Electronic device 1 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0145] The components of the electronic device 1 may include, but are not limited to: one or more processors or processing units 3, a system memory 8, and a bus 4 connecting different system components (including the system memory 8 and the processing unit 3).

[0146] Bus 4 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0147] The electronic device 1 typically includes a variety of computer system readable media, which can be any available media that can be accessed by the electronic device 1, including volatile and non-volatile media, removable and non-removable media.

[0148] The system memory 8 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 9 and / or cache memory 10. The electronic device 1 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 11 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 Not shown, often called a "hard drive"). Although Figure 6 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 4 via one or more data medium interfaces. System memory 8 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0149] A program / utility 12 having a set (at least one) of program modules 13 may be stored, for example, in system memory 8. Such program modules 13 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 13 generally implement the functions and / or methods of the embodiments described herein.

[0150] The electronic device 1 may also communicate with one or more external devices 2 (e.g., a keyboard, a pointing device, a display 6, etc.), one or more devices that enable a user to interact with the electronic device 1, and / or any device that enables the electronic device 1 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 7. Furthermore, the electronic device 1 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 5. Figure 6 As shown, the network adapter 5 communicates with other modules of the electronic device 1 via the bus 4. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0151] The processing unit 3 executes various functional applications and data processing by running programs stored in the system memory 8 , such as implementing the intelligent suspension control method combined with deterministic experience tracking provided by an embodiment of the present invention.

[0152] An embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored. When the program is executed by a processor, the intelligent suspension control method combined with deterministic empirical tracking provided in all the inventive embodiments of this application is implemented.

[0153] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus or device.

[0154] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0155] The program code that comprises on the computer-readable medium can be transmitted with any appropriate medium, includes but not limited to wireless, electric wire, optical cable, RF etc., or above-mentioned any suitable combination.Can write the computer program code that is used to carry out the operation of the present invention with one or more programming languages or its combination, described programming language comprises object-oriented programming language such as Java, Smalltalk, C++, also comprises conventional procedural programming language--such as " C " language or similar programming language.Program code can be carried out on user's computer completely, partly on user's computer, carry out as an independent software package, partly on user's computer partly on remote computer, or carry out completely on remote computer or server.In the situation that relates to remote computer, remote computer can comprise local area network (LAN) or wide area network (WAN) to be connected to user's computer by the network of any kind, perhaps, can be connected to external computer (for example, utilize Internet service provider to come to connect by Internet).

[0156] An embodiment of the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above-mentioned intelligent suspension control method combined with deterministic empirical tracking.

[0157] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. This is not limited herein.

[0158] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. An intelligent suspension control method combined with deterministic empirical tracking, characterized in that: include: S1: Acquire multiple groups of first samples, and use the first samples to pre-train the constructed first evaluation network, second evaluation network, and control strategy network to obtain an initial first evaluation model, an initial second evaluation model, and an initial control strategy model respectively; In step S1, the method further includes: increasing the number of network layers in the first evaluation network and the second evaluation network, and the number of neurons in each network layer; and reducing the number of network layers in the control strategy network, and the number of neurons in each network layer; and pre-training the modified first evaluation network, the second evaluation network, and the control strategy network. S2: The initial control strategy model obtained in step S1 determines a second control force output by the intelligent suspension at the current moment based on the first state of the vehicle at the current moment; applies the second control force to the vehicle to obtain a next second state of the vehicle; determines a second reward and an auxiliary reward group based on the second control force and the second state; and uses the second state, the second control force, the second reward, the auxiliary reward group, and the next second state as a second sample. S3: Repeat step S2 multiple times to obtain multiple groups of second samples, and use the multiple groups of second samples to retrain the initial first evaluation model, initial second evaluation model, and initial control strategy model obtained in S1; during the training process, add a disturbance amount to the second control force, and change the disturbance amount according to the training effect; after training, obtain the corresponding final first evaluation model, final second evaluation model, and final control strategy model; the process of adjusting the disturbance amount includes: defining a basic Gaussian noise variance; when it is determined that the second reward is growing slowly, using a linear increase method to increase the variance of the Gaussian noise; when it is determined that the second reward is growing steadily, using an exponential decay method to reduce the variance of the Gaussian noise; S4: running the intelligent suspension and controlling the intelligent suspension using the final first evaluation model, the final second evaluation model, and the final control strategy model trained in step S4; Between step S3 and step S4, the method further includes: using the final control strategy model trained in step S3 to control the intelligent suspension of other vehicles to obtain multiple groups of second samples again; repeating the training process of step S3, and using the new multiple groups of second samples to train the final first evaluation model, the final second evaluation model and the final control strategy model again.

2. The intelligent suspension control method combined with deterministic empirical tracking according to claim 1, characterized in that: In step S1, each group of the first samples includes: the first state of the vehicle at a certain moment, the first control force output by the intelligent suspension at the same moment, the next first state of the vehicle after the intelligent suspension applies the first control force to the vehicle, and a first reward obtained based on the first state, the first control force, and the first state.

3. The intelligent suspension control method combined with deterministic empirical tracking according to claim 2, characterized in that: The first state ,in, and denote the acceleration and velocity of the unsprung mass of the vehicle at time t, represents the dynamic travel of the smart suspension at time t, represents the vertical displacement of the vehicle at time t, represents the unsprung displacement of the vehicle at time t; represents the velocity difference between the sprung mass and the unsprung mass at time t, represents the unsprung speed of the vehicle at time t; The first reward is: ； in, represents the first reward at time t, 、 '、 and represents the first reward coefficient, represents the dynamic wheel load of the vehicle at time t, , represents the road height incentive information at time t, represents the first control force, P represents the control trigger coefficient, It represents the ultimate dynamic deflection of the intelligent suspension.

4. The intelligent suspension control method combined with deterministic empirical tracking according to claim 3, characterized in that: The pre-training in step S1 includes: In each pre-training time step, the first sample is extracted and the corresponding first target value is calculated by the following formula: ； in, represents the first target value, represents the discount factor, represents the j-th evaluation network, represents the network weight of the j-th evaluation network; represents the control strategy network, represents the network weight of the control strategy network, Indicates the next first state at time t; Combined with the first target value, the loss function for training the two evaluation networks is calculated: ； in, represents the loss function of the j-th evaluation network, represents the time step of pre-training; the two evaluation networks are pre-trained using the loss functions of the two evaluation networks; the control strategy network is pre-trained using the following formula: ； in, Represents the first state at time t.

5. The intelligent suspension control method combined with deterministic empirical tracking according to claim 4, characterized in that: In step S2, The second state ; The second reward is: ； in, represents the second reward at time t, 、 、 and represents the second reward coefficient, represents the second control force at time t; The auxiliary reward groups are: ； in, represents the auxiliary reward group at time t, represents the auxiliary reward, k represents the acquisition step of the auxiliary reward; the auxiliary reward for: ； in, 、 and Represents the auxiliary reward coefficient.

6. The intelligent suspension control method combined with deterministic empirical tracking according to claim 5, characterized in that: During the training process of step S3: In each training time step, multiple sets of second samples , calculate the corresponding second target value by the following formula: ； in, Indicates the Get step, represents the second target value, represents the deterministic experience-assisted reward discount factor, represents the initial j-th evaluation model, represents the model weight of the initial j-th evaluation model; represents the initial control strategy model, represents the model weight of the initial control strategy model, Indicates the next second state at time t; Combined with the second target value, the loss function of training the two initial evaluation models is calculated: ； in, Represents the loss function of the initial j-th evaluation model, and N represents the time step of training; the two initial evaluation models are trained using their loss functions; The initial control strategy model is trained using the following formula: ； During the training process, if the second reward increases slowly, the disturbance amount is increased; if the second reward increases steadily, the disturbance amount is reduced.

7. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the intelligent suspension control method combined with deterministic empirical tracking as claimed in any one of claims 1 to 6.

8. An electronic device, characterized in that: include: memory for storing computer programs; A processor is configured to implement the steps of the intelligent suspension control method combined with deterministic empirical tracking as claimed in any one of claims 1 to 6 when executing the computer program.

Images