Suspension System Control Method, Medium and Electronic Device for Adjusting Rewards in Combination with RND

Abstract

The present invention relates to the field of automotive control technology, and particularly to a suspension system control method, medium and electronic device that combines RND to adjust rewards. The method includes: determining multiple optimization objectives for controlling the suspension system; constructing a TD3 architecture with an RND reward adjustment structure and training the TD3 architecture according to the multiple optimization objectives; in the RND reward adjustment structure, calculating the difference between the outputs of the Critic network and the CriticT network in the TD3 architecture, and generating a physical value correlation factor according to the multiple optimization objectives; combining the obtained difference and the physical value correlation factor with the reward corresponding to the current environmental state to obtain the current reward, and training the TD3 architecture with the current reward to obtain a TD3 model; finally, inputting the current environmental state that meets the optimization objectives into the TD3 model to predict the output action of the suspension system. The RND reward adjustment structure provided by the present invention, through the combination with external rewards, enables the suspension system to better adapt to different tasks and environments, improving the overall performance of reinforcement learning.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle control, and particularly relates to a suspension system control method, medium, and electronic device that combine RND to adjust rewards. Background Art

[0002] In 2019, Random Network Distillation (RND) proposed by Schaul et al. is an exploration mechanism based on intrinsic rewards, aiming to solve the exploration problem when external rewards are sparse in reinforcement learning. The core idea of RND is to generate intrinsic rewards by using a randomly initialized fixed target network and a trainable prediction network.

[0003] In the application of deep reinforcement learning (DRL), the RND algorithm has received wide attention for its unique intrinsic reward mechanism. This mechanism is based on prediction error and state novelty, aiming to guide the agent to explore more unknown states. However, the actual application of the suspension system control system needs to strictly follow a series of physical constraints, such as safety, stability, and ride comfort. These constraints are crucial for ensuring the driving experience and vehicle performance.

[0004] If these physical limitations are not explicitly modeled in the suspension system control, RND may cause the agent to over-explore. Specifically, the agent may try to perform some actions that do not meet physical or safety requirements. This situation may not only affect the performance of the suspension system but also pose serious safety hazards. For example, during the exploration process, the agent may choose overly aggressive control strategies, causing the suspension system to face severe vibrations and instability, resulting in passenger discomfort and potential vehicle damage. Summary of the Invention

[0005] In view of this, the present invention aims to provide a suspension system control method, medium, and electronic device that combine RND to adjust rewards. By introducing an RND reward adjustment structure, the obtained TD3 model can adjust the rewards according to the degree of compliance between the physical value correlation factor and physical constraints in the RND reward adjustment structure, so that the suspension system can maintain sensitivity to physical reality while pursuing intrinsic rewards.

[0006] To achieve the above object, the technical solution of the present invention is realized as follows:

[0007] A suspension system control method that combines RND to adjust rewards, including:

[0008] S1: Determine multiple optimization goals for controlling the suspension system;

[0009] S2: Construct a TD3 architecture with an RND reward adjustment structure, and train the TD3 architecture according to multiple optimization objectives in step S1;

[0010] In the RND reward adjustment structure, calculate the difference between the outputs of the Critic network and the CriticT network in the TD3 architecture, and generate a physical value correlation factor according to multiple optimization objectives; Combine the obtained difference and the physical value correlation factor with the reward corresponding to the current environmental state to obtain the current reward, and train the TD3 architecture with the current reward to obtain the TD3 model;

[0011] S3: Input the current environmental state that meets the optimization objectives in step S1 into the TD3 model obtained in step S2 to predict the output action of the suspension system.

[0012] Furthermore, the RND reward adjustment structure includes a maximization module, a difference output module, a physical value correlation factor module, and a reward output module; Among them,

[0013] The maximization module receives the evaluation value output by the Critic network in the TD3 architecture and outputs the maximum value in the evaluation value;

[0014] The difference output module receives the maximum value and the minimum value in the target evaluation value output by the CriticT network in the TD3 architecture, and calculates the difference through the following formula:

[0015] ;

[0016] Among them, represents the difference, represents the maximum value, represents the minimum value;

[0017] The physical value correlation factor module combines multiple optimization objectives and the negative gradient function to generate a physical value correlation factor;

[0018] The reward output module multiplies the difference and the physical value correlation factor by the reward corresponding to the current environmental state to obtain the current reward.

[0019] Furthermore, the TD3 architecture in step S2 also includes an Actor network, an ActorT network, a Critic1 network, a Critic2 network, a CriticT1 network, and a CriticT2 network; Among them,

[0020] The Actor network receives the current environmental state and applies the generated execution action to the environment to generate the next environmental state; the ActorT network receives the next environmental state and generates the target action; the CriticT1 network and the CriticT2 network simultaneously receive the target action and the next environmental state, and input the minimum target evaluation value among the output target evaluation values into the RND reward adjustment structure; the Critic1 network and the Critic2 network simultaneously receive the current environmental state and the next environmental state; the Critic1 network and the Critic2 network input the evaluation value corresponding to the next environmental state into the RND reward adjustment structure; the Critic1 network and the Critic2 network generate the evaluation values participating in the training according to the current environmental state.

[0021] Further, the training process in step S2 includes:

[0022] S21: Initialize the TD3 architecture;

[0023] S22: In the current environmental state, the Actor network generates an action and controls the suspension system to execute the action to obtain the next environmental state, and the corresponding reward generated by the action in the optimization target; the ActorT network receives the next environmental state and generates the target action;

[0024] S23: The Critic1 network and the Critic2 network simultaneously receive the current environmental state and the next environmental state; the Critic1 network and the Critic2 network input the evaluation value generated according to the next environmental state into the RND reward adjustment structure;

[0025] S24: The CriticT1 network and the CriticT2 network simultaneously receive the target action and the next environmental state, and input the minimum target evaluation value among the generated target evaluation values into the RND reward adjustment structure;

[0026] S25: The RND reward adjustment structure receives the evaluation value in step S23 and the minimum target evaluation value in step S24, and generates the current reward in combination with the reward obtained according to step S22;

[0027] S26: Repeat steps S22 - S25 multiple times, and integrate the obtained action, the next environmental state, the current environmental state and the current reward into experience data and store them in the experience pool;

[0028] S27: Randomly sample a batch of experience data from the experience pool, and calculate the target value using the Bellman expectation formula; calculate the loss functions of the Critic1 network and the Critic2 network in combination with the target value and the target action;

[0029] S28: Based on the loss function obtained in step S27, perform parameter update on the TD3 architecture using a delayed update strategy, and repeat steps S21 - S28 with the updated TD3 architecture until the training of the TD3 architecture is completed.

[0030] Further, in step S27, the target value is obtained through the following formula:

[0031] ;

[0032] where, represents the target value, represents the discount factor, represents the CriticT1 network, represents the CriticT2 network, represents the network parameters of the CriticT1 network, represents the network parameters of the CriticT2 network, represents the target action, represents the next environmental state, and r represents the current reward.

[0033] Further, in step S27, the loss functions for training the Critic1 network and the Critic2 network are:

[0034] ;

[0035] ;

[0036] where, represents the loss function of the Critic1 network, represents the loss function of the Critic2 network, represents the result obtained by inputting the action and the current environmental state into the Critic1 network, represents the network parameters of the Critic1 network, represents the result obtained by inputting the action and the current environmental state into the said Critic2 network, represents the network parameters of the Critic2 network, MSE represents the mean square error loss function, represents the target value of the Critic1 network, represents the target value of the Critic2 network.

[0037] A readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the suspension system control method for adjusting rewards in combination with RND provided by the present invention.

[0038] An electronic device, comprising:

[0039] A memory for storing a computer program;

[0040] A processor for implementing the steps of the suspension system control method combining RND adjustment of rewards provided by the present invention when executing the computer program.

[0041] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0042] In the suspension system control method combining RND adjustment of rewards of the present invention, by introducing an RND reward adjustment structure with a physical value correlation factor, the relationship between exploration and physical constraints is effectively balanced, thereby prompting the suspension control system to implement a more robust and safe control strategy. This can not only improve the stability of the suspension system under various driving conditions but also enhance the comfort of passengers. In practical applications, this method may significantly improve the response ability of the vehicle, enabling the intelligent agent to make more accurate and safe decisions in complex environments.

[0043] Integrating the physical value correlation factor into the RND algorithm can not only improve the safety and stability of suspension control but also provide a new idea for the application of deep reinforcement learning in complex engineering control tasks. This combination provides a more reasonable exploration guidance for the intelligent agent, ensuring that it always follows the physical laws in a changing environment, thereby improving the performance of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0045] Figure 1 is a schematic flow chart of the suspension system control method combining RND adjustment of rewards according to an embodiment of the present invention;

[0046] Figure 2 is a schematic structural diagram of the TD3 architecture according to an embodiment of the present invention;

[0047] Figure 3 is a schematic structural diagram of the RND reward adjustment structure according to an embodiment of the present invention;

[0048] Figure 4 is a schematic structural diagram of the electronic device according to an embodiment of the present invention.

[0049] Description of the reference numerals:

[0050] 1, Electronic device; 2, External device; 3, Processing unit; 4, Bus; 5, Network adapter; 6, Display; 7, (I / O) Interface; 8, System memory; 9, Random access memory; 10, Cache memory; 11, Storage system; 12, Utility; 13, Program module. Detailed implementation

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the following further details the present invention in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation to the present invention.

[0052] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0053] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise stated, the meaning of "plurality" is two or more.

[0054] In the description of the present invention, it should be noted that unless otherwise clearly defined and limited, the terms "installed", "connected", "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood through specific situations.

[0055] The following will detail the present invention with reference to the accompanying drawings and in conjunction with the embodiments.

[0056] As Figures 1 to 3 shown, the suspension system control method combining RND adjustment rewards described in the embodiments of the present invention includes:

[0057] S1: Determine multiple optimization objectives for controlling the suspension system.

[0058] In a certain embodiment, the optimization objectives for controlling the suspension system are respectively the displacement of the unsprung mass, the velocity of the unsprung mass, the acceleration of the unsprung mass; the displacement of the sprung mass, the velocity of the sprung mass, the acceleration of the sprung mass.

[0059] S2: Construct a TD3 architecture with an RND reward adjustment structure, and train the TD3 architecture according to the multiple optimization objectives in step S1.

[0060] Specifically, the TD3 architecture includes an Actor network, an ActorT network, a Critic1 network, a Critic2 network, a CriticT1 network, a CriticT2 network, and an RND reward adjustment structure. The Actor network receives the current environmental state and applies the generated execution action to the environment to generate the next environmental state. The ActorT network receives the next environmental state and generates a target action. The CriticT1 network and the CriticT2 network simultaneously receive the target action and the next environmental state, and input the minimum target evaluation value among the output target evaluation values into the RND reward adjustment structure. The Critic1 network and the Critic2 network simultaneously receive the current environmental state and the next environmental state; the Critic1 network and the Critic2 network input the evaluation value corresponding to the next environmental state into the RND reward adjustment structure. In the RND reward adjustment structure, calculate the difference between the outputs of the Critic network and the CriticT network in the TD3 architecture, and generate a physical value correlation factor according to the multiple optimization objectives; combine the obtained difference and the physical value correlation factor with the reward corresponding to the current environmental state to obtain the current reward. The Critic1 network and the Critic2 network generate evaluation values for participating in the training according to the current environmental state. Train the TD3 architecture with the current reward to obtain the TD3 model.

[0061] In a specific embodiment, the Actor network, the Actor-T network, the two CriticT networks, and the two Critic networks generally adopt a structure of 3 to 4 layers. Among them, the structures of the Actor network and the Actor-T network are the same, and the structures of the two CriticT networks and the two Critic networks correspond to each other. That is, the structure of the CriticT1 network is the same as that of the CriticT1 network, and the structure of the CriticT2 network is the same as that of the CriticT2 network. The input layer of the Actor network can receive the state with one node, pass through 2 to 3 hidden layers (for example, the number of nodes is 300 and 200, and the ReLU activation function is used), and the output layer generates actions. Since the generated actions need to be restricted within the range of the action space (for example, [−1,1]), the tanh activation function is usually used. If the range of the action space is larger, the output value can be extended to the target range through a linear transformation. The input layers of the two Critic networks both receive the concatenation result of the current environmental state and the action, and there should be two nodes in the input layer. The hidden layer settings are similar to those of the Actor network (also using the ReLU activation function), and the output layer is a linear output for calculating the evaluation value of the generated action. The soft update TD3 model is adopted to balance performance and computational cost while avoiding the problems of gradient disappearance and overfitting.

[0062] In some embodiments, the RND reward adjustment structure is as Figure 3 shown, including a maximization module, a difference output module, a physical value correlation factor module, and a reward output module. Among them, the maximization module receives the evaluation values corresponding to the next environmental states of the Critic1 network and the Critic2 network, and outputs the maximum value among the evaluation values. The difference output module receives the maximum value and the minimum value among the target evaluation values output by the CriticT1 network and the CriticT2 network, and calculates the mean square error of the minimum value and the maximum value to obtain the difference between the outputs of the Critic network and the CriticT network. The physical value correlation factor module generates a physical value correlation factor by combining multiple optimization objectives and a negative gradient function. The reward output module multiplies the difference and the physical value correlation factor by the reward corresponding to the current environmental state to obtain the current reward.

[0063] It should be noted that the maximization module is the connection structure between the TD3 and the RND algorithms, which is one of the innovation points of the present invention. Its function is to receive the outputs of the Critic1 and Critic2 networks, output the maximum value of the two to calculate the difference in the RND algorithm, and at the same time complete the connection between the TD3 algorithm model and the RND algorithm model.

[0064] It can be understood that in the difference output module, the difference is calculated by the following formula:

[0065] ;

[0066] Among them, represents the difference, represents the maximum value, represents the minimum value.

[0067] In the reward output module, the current reward is obtained through the following formula:

[0068] ;

[0069] Among them, represents the current reward, represents the physical value correlation factor, represents the reward corresponding to the current environmental state. The reward output module can achieve that when the suspension system is performing an action, it will not only consider the rewards of the external environment (such as reaching the goal, obtaining items, etc.), but also internally regulate the rewards according to the exploration of unknown states, so as to balance the roles of exploration and exploitation.

[0070] It should be noted that the traditional RND algorithm can be put into training after calculating the output difference L. However, in actual engineering problems, especially in the vertical suspension control problem, we must control the Actor network not to explore some action spaces that do not conform to physical reality, so as to reduce the waste of computing resources and improve the exploration efficiency. Therefore, to achieve this goal, the physical value correlation factor module designed in the present invention changes the ordinary hyperparameters in the traditional RND algorithm to physical value correlation factors, and changes its form to a negative gradient function for state variables , that is:

[0071] ;

[0072] Among them, represents the optimization goal, and n represents the number of optimization goals. It can be understood that the negative gradient function satisfies the following conditions:

[0073] ;

[0074] Among them, represents taking the partial derivative of each term (T1, T2,..., T in the negative gradient function n ).

[0075] Because in the vertical control of the suspension, the state variables of each component of the suspension will not have very large values (too large state variable values often mean situations such as hitting the limit block).

[0076] For the physical value correlation factor The definition adopts the generalized function definition method. Any negative gradient function that satisfies the above conditions can be used as the expression form of the actual physical value correlation factor η. The present invention does not limit the specific form of the negative gradient function. This condition means that when the agent installed on the suspension system explores an action space that does not conform to the physical reality, it will not bring great benefits. The larger the state quantity, the more unrealistic it is, and the smaller the physical value correlation factor η, the smaller the current reward, that is, the exploration intensity is restricted.

[0077] In a specific embodiment, the selected negative gradient function is an exponential function. The agent used consists of a processor and a motor and is located between the sprung mass and the unsprung mass. It can output a vertical force to the environment / system according to the current state by using the Actor network in the processor.

[0078] The RND reward adjustment structure provided by the present invention provides intrinsic rewards for the agent by combining the differences between the outputs of the Critic network and the CriticT network, encourages the exploration of unencountered states, effectively alleviates the sparse reward problem, enables the agent to more actively explore states in complex environments, and thus improves the learning efficiency. The RND reward adjustment structure provided by the present invention better adapts to different tasks and environments and improves the overall performance of reinforcement learning by combining with extrinsic rewards (i.e., the corresponding rewards generated by actions in the optimization target).

[0079] It can be understood that in the TD3 architecture provided by the present invention, the Actor network receives the current environmental state , and the generated execution action acts on the environment to generate the next environmental state . The ActorT network receives the next environmental state , and generates the target action . The CriticT1 network and the CriticT2 network simultaneously receive the target action and the next environmental state , and input the minimum target evaluation value and in the output target evaluation values into the RND reward adjustment structure. represents the network parameters of the CriticT1 network, represents the network parameters of the CriticT2 network. The Critic1 network and the Critic2 network simultaneously receive the current environmental state and the next environmental state . The Critic1 network and the Critic2 network output the evaluation values corresponding to the next environmental state and Input into the RND reward adjustment structure, the Critic1 network and the Critic2 network generate evaluation values for training according to the current environmental state , generating evaluation values for participating in training and , denotes the network parameters of the Critic1 network, denotes the network parameters of the Critic2 network.

[0080] In some embodiments, the training process in step S2 includes:

[0081] S21: Initialize the TD3 architecture. In some embodiments, the following parameters are randomly initialized using the Xavier initialization method, and the initialization includes:

[0082] Initialize the environmental state; the state of the suspension is the state corresponding to the optimization target, that is, including the displacement, velocity, and acceleration of the sprung mass, and the displacement, velocity, and acceleration of the unsprung mass;

[0083] Randomly initialize the Critic1 network and the Critic2 network. Specifically, randomly initialize the network parameters of the Critic1 network and the network parameters

[0084] of the Critic2 network; For the two target Critic networks, namely the CriticT1 network and the CriticT2 network, initialize. Specifically, initialize the network parameters of the CriticT1 network equal to the network parameters , and the initialized network parameters equal to the network parameters ;

[0085] Randomly initialize the Actor network. Specifically, randomly initialize the network parameters of the Actor network;

[0086] For the target Actor network, namely the ActorT network, initialize. Specifically, initialize the network parameters of the ActorT network equal to the network parameters ;

[0087] Initialize the hyperparameters for updating the TD3 architecture. The hyperparameters include the discount factor , the soft update frequency , the policy update frequency policy_delay, and the learning rates of the Critic1 network and the Critic2 network. In one embodiment, the learning rate of the Actor network is typically set to 10 -4 , and the learning rates of both Critic networks are 10 -3 , and the soft update frequency is set in [0.005, 0.01].

[0088] S22: In the current environmental state, the Actor network generates an action and controls the suspension system to execute the action, obtaining the next environmental state and the corresponding reward generated by the action in the optimization objective. The ActorT network receives the next environmental state and generates a target action.

[0089] In some embodiments, in the current environmental state s, the Actor network generates an action added with exploration noise of , that is:

[0090] ;

[0091] Among them, represents the Actor network, and the exploration noise is Gaussian noise.

[0092] Controlling the suspension system to execute the action to obtain the next environmental state , and obtaining the action through the following formula The corresponding rewards generated in each optimization objective , that is:

[0093] ;

[0094] Among them, j represents the number of optimization objectives, represents the next environmental state corresponding to the jth optimization objective, represents the reward corresponding to the jth optimization objective.

[0095] The ActorT network receives the next environmental state , generates a target action , that is:

[0096] ;

[0097] Among them, represents the ActorT network, represents the noise, and Gaussian noise is used in one embodiment.

[0098] S23: The Critic1 network and the Critic2 network simultaneously receive the current environmental state and the next environmental state; the minimum evaluation value among the evaluation values generated by the Critic1 network and the Critic2 network according to the next environmental state is input into the RND reward adjustment structure.

[0099] It can be understood that the Critic1 network and the Critic2 network obtain corresponding evaluation values based on the evaluation values generated according to the next environmental state and , the evaluation value and the evaluation value are input into the RND reward adjustment structure.

[0100] S24: The CriticT1 network and the CriticT2 network simultaneously receive the target action and the next environmental state, and input the minimum target evaluation value among the generated target evaluation values into the RND reward adjustment structure.

[0101] It can be understood that the CriticT1 network and the CriticT2 network simultaneously receive the target action and the next environmental state , and respectively generate the target evaluation value and the target evaluation value . The target evaluation value and the target evaluation value are obtained by the following formula, and the minimum target evaluation value is:

[0102] ;

[0103] The minimum target evaluation value is input into the RND reward adjustment structure.

[0104] S25: The RND reward adjustment structure receives the evaluation value in step S23 and the minimum target evaluation value in step S24, and combines the reward obtained according to step S22 to generate the current reward.

[0105] Combined with the above description of the RND reward adjustment structure, it can be obtained that the evaluation value and the evaluation value are input into the maximization module in the RND reward adjustment structure, and the maximum value is obtained by the following formula:

[0106] .

[0107] The minimum target evaluation value is input into the difference output module in the RND reward adjustment structure, and the difference output module combines the maximum value and the minimum target evaluation value Output the difference L. The reward output module combines the difference L and the physical value correlation factor generated by the physical value correlation factor module , and according to the reward corresponding to the current environmental state obtain the current reward r.

[0108] It can be understood that the current reward r is obtained by the following formula:

[0109] ;

[0110] where represents the physical value correlation factor corresponding to the jth optimization objective, and n represents the total number of optimization objectives.

[0111] S26: Repeat steps S22 - S25 multiple times, and integrate the obtained actions, next environmental state, current environmental state, and current reward into experience data and store them in the experience pool.

[0112] It can be understood that by repeating steps S22 - S25 multiple times, the obtained actions , next environmental state , current environmental state and the current reward r are integrated into experience data and stored in the experience pool.

[0113] S27: Randomly sample a batch of experience data from the experience pool, and use the Bellman expectation formula to calculate the target value; combine the target value and the target action to calculate the loss functions of the Critic1 network and the Critic2 network.

[0114] It can be understood that randomly sample a batch of experience data from the experience pool , and according to the experience data use the Bellman expectation formula to calculate the target value , that is:

[0115] .

[0116] The loss functions for training the Critic1 network and the Critic2 network are:

[0117] ;

[0118] ;

[0119] where represents the loss function of the Critic1 network, represents the loss function of the Critic2 network, MSE represents the mean squared error loss function, Denote the target value of Critic1 network , Denote the target value of Critic2 network .

[0120] S28: According to the loss function obtained in step S27, update the parameters of the TD3 architecture using a delayed update strategy, and repeat steps S21 - S28 with the updated TD3 architecture until the training of the TD3 architecture is completed. Among them, the delayed update strategy specifically includes:

[0121] During each round of training, update the parameters and parameter by the following formula:

[0122] ;

[0123] ;

[0124] where denotes taking the gradient of the loss function with respect to in it, denotes taking the gradient of the loss function with respect to in it, denotes the gradient coefficient.

[0125] Update the parameters and parameter respectively by the following formula: and parameter :

[0126] ;

[0127] ;

[0128] Every policy_delay rounds of training, update the parameters and parameter , including:

[0129] Calculate the policy loss by the following formula:

[0130] ;

[0131] According to the policy loss , update the parameter by the following formula:

[0132] ;

[0133] where Denote the policy loss in to calculate the gradient;

[0134] Then, update the parameter through the following formula:

[0135] ;

[0136] Update the current environmental state to the next environmental state .

[0137] S3: Input the current environmental state that meets the optimization objective of step S1 into the TD3 model obtained in step S2 to predict the output action of the suspension system.

[0138] Correspondingly, according to the embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0139] Figure 4 It is a schematic structural diagram of an electronic device 1 provided in the embodiments of the present invention. Figure 4 It shows a block diagram of an exemplary electronic device 1 suitable for implementing the embodiments of the present invention. Figure 4 The shown electronic device 1 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0140] As Figure 4 shown, the electronic device 1 is presented in the form of a general-purpose computing device. The electronic device 1 is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0141] The components of the electronic device 1 may include but are not limited to: one or more processors or processing units 3, a system memory 8, and a bus 4 connecting different system components (including the system memory 8 and the processing unit 3).

[0142] The bus 4 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, a processor bus, or a local bus using any of the several bus architectures. By way of example, and not limitation, these architectures include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0143] The electronic device 1 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 1, including both volatile and nonvolatile media, removable and non-removable media.

[0144] The system memory 8 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 9 and / or cache memory 10. The electronic device 1 may further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, a storage system 11 can be provided for reading from and writing to a non-removable, nonvolatile magnetic medium ( Figure 4 not shown and typically called a “hard disk drive”). Although Figure 4 not shown in FIG. 1, a disk drive for reading from and writing to a removable nonvolatile disk (e.g., a “floppy disk”), and an optical disk drive for reading from and writing to a removable nonvolatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to the bus 4 by one or more data media interfaces. The system memory 8 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the present invention.

[0145] A program / utility 12 having a set (at least one) of program modules 13 can be stored, for example, in the system memory 8, such program modules 13 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a network environment. The program modules 13 typically carry out the functions and / or methods of the embodiments described herein.

[0146] The electronic device 1 can also communicate with one or more external devices 2 (such as a keyboard, a pointing device, a display 6, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 1, and / or communicate with any device that enables the electronic device 1 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 7. Moreover, the electronic device 1 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 5. As Figure 4 shown, the network adapter 5 communicates with other modules of the electronic device 1 through a bus 4. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 1, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0147] The processing unit 3 executes various functional applications and data processing by running programs stored in the system memory 8, for example, implementing the suspension system control method combining RND adjustment rewards provided by the embodiments of the present invention.

[0148] The embodiments of the present invention also provide a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored. Among them, when the program is executed by a processor, the suspension system control method combining RND adjustment rewards provided by all the embodiments of the present application is implemented.

[0149] The computer storage medium of the embodiments of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0150] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0151] The program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing. The computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user computer, partially on the user computer, executed as a stand-alone software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0152] An embodiment of the present invention also provides a computer program product, including a computer program, which when executed by a processor implements the suspension system control method for adjusting rewards in combination with RND as described above.

[0153] It should be understood that various forms of the flow shown above can be used, reordering, adding, or deleting steps. For example, the steps described in the disclosure of the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and no limitation is made herein.

[0154] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A suspension system control method that combines RND to adjust rewards, characterized in that, Including: S1: Determine multiple optimization objectives for controlling the suspension system; S2: Construct a TD3 architecture with an RND reward adjustment structure, and train the TD3 architecture according to the multiple optimization objectives in step S1; The RND reward adjustment structure includes a maximization module, a difference output module, a physical value association factor module, and a reward output module; where The maximization module receives the evaluation value output by the Critic network in the TD3 architecture and outputs the maximum value among the evaluation values; The difference output module receives the maximum value and the minimum value among the target evaluation values output by the CriticT network in the TD3 architecture, and calculates the difference between the outputs of the Critic network and the CriticT network in the TD3 architecture through the following formula: ； Among them, represents the said difference, represents the said maximum value, represents the said minimum value; The physical value association factor module generates a physical value association factor according to multiple optimization objectives and a negative gradient function, that is: ； Among them, represents a physical value correlation factor, represents an optimization objective, and n represents the number of optimization objectives, represents a negative gradient function, and the negative gradient function satisfies the following conditions: ； Among them, represents taking the partial derivative of each term (T1, T2,..., T ) in the negative gradient function n , and the negative gradient function is an exponential function; The reward output module multiplies the obtained difference by the reward corresponding to the physical value association factor and the current environmental state to obtain the current reward, and trains the TD3 architecture with the current reward to obtain a TD3 model; S3: Input the current environmental state that meets the optimization objectives in step S1 into the TD3 model obtained in step S2 to predict the output action of the suspension system.

2. The suspension system control method for adjusting rewards in combination with RND according to claim 1, characterized in that, The TD3 architecture in step S2 further includes an Actor network, an ActorT network, a Critic1 network, a Critic2 network, a CriticT1 network, and a CriticT2 network; where The Actor network receives the current environmental state and applies the generated execution action to the environment to generate the next environmental state; The ActorT network receives the next environmental state and generates the target action; The CriticT1 network and the CriticT2 network simultaneously receive the target action and the next environmental state, and input the minimum target evaluation value among the output target evaluation values into the RND reward adjustment structure; The Critic1 network and the Critic2 network simultaneously receive the current environmental state and the next environmental state; the Critic1 network and the Critic2 network input the evaluation value corresponding to the next environmental state into the RND reward adjustment structure; the Critic1 network and the Critic2 network generate evaluation values participating in training according to the current environmental state.

3. The suspension system control method for adjusting rewards in combination with RND according to claim 2, characterized in that, The training process in step S2 includes: S21: Initialize the TD3 architecture; S22: In the current environmental state, the Actor network generates an action and controls the suspension system to execute the action to obtain the next environmental state and the corresponding reward generated by the action in the optimization objective; the ActorT network receives the next environmental state and generates the target action; S23: The Critic1 network and the Critic2 network simultaneously receive the current environmental state and the next environmental state; the evaluation values generated by the Critic1 network and the Critic2 network based on the next environmental state are input into the RND reward adjustment structure; S24: The CriticT1 network and the CriticT2 network simultaneously receive the target action and the next environmental state, and input the minimum target evaluation value among the generated target evaluation values into the RND reward adjustment structure; S25: The RND reward adjustment structure receives the evaluation value in step S23 and the minimum target evaluation value in step S24, and combines with the reward obtained according to step S22 to generate the current reward; S26: Repeat steps S22 to S25 multiple times, and integrate the obtained action, the next environmental state, the current environmental state, and the current reward into experience data and store them in the experience pool; S27: Randomly sample a batch of experience data from the experience pool, and calculate the target value using the Bellman expectation formula; combine the target value and the target action to calculate and train the loss functions of the Critic1 network and the Critic2 network; S28: According to the loss functions obtained in step S27, perform parameter update on the TD3 architecture using a delayed update strategy, and repeat steps S21 to S28 with the updated TD3 architecture until the training of the TD3 architecture is completed.

4. The suspension system control method for adjusting rewards in combination with RND according to claim 3, characterized in that, In step S27, the target value is obtained through the following formula: ； Among them, represents the target value, represents the discount factor, represents the CriticT1 network, represents the CriticT2 network, represents the network parameters of the CriticT1 network, represents the network parameters of the CriticT2 network, represents the target action, represents the next environmental state, and r represents the current reward.

5. The suspension system control method for adjusting rewards in combination with RND according to claim 4, characterized in that, In step S27, the loss functions for training the Critic1 network and the Critic2 network are: ； ； Among them, represents the loss function of the Critic1 network, represents the loss function of the Critic2 network, represents the action and the current environmental state input into the Critic1 network to obtain the result, represents the network parameters of the Critic1 network, represents the action and the current environmental state input into the Critic2 network to obtain the result, represents the network parameters of the Critic2 network, and MSE represents the mean squared error loss function, represents the target value of the Critic1 network, represents the target value of the Critic2 network.

6. A readable storage medium, characterized in that, The computer program is stored on the readable storage medium, and when the computer program is executed by the processor, it implements the steps of the suspension system control method combining RND to adjust the reward as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, Including: A memory for storing a computer program; A processor for implementing the steps of the suspension system control method combining RND to adjust the reward as described in any one of claims 1 to 5 when executing the computer program.

Images