Suspension Control Method, Storage Medium and Electronic Device

Abstract

The present invention relates to the technical field of vehicle vibration control, and in particular to a suspension control method, a storage medium and an electronic device. The method includes: First, build a basic TD3 architecture, use the current state of the suspension as the input, and the active control action of the suspension as the output, pre-train the basic TD3 architecture, and during the pre-training process, determine the conflict state; introduce an adversarial network into the basic TD3 architecture, that is, add an Actor-against network and a Critic-against network to the structure of the basic TD3 architecture to obtain an adversarial-TD3 architecture; Finally, use the conflict state to train the adversarial-TD3 architecture to obtain a control model for controlling the suspension. The present invention realizes the adversarial correction of the action space in the form of dual-network collaborative optimization, which not only reduces the complexity of the reward function design, but also greatly improves the convergence speed and training efficiency of the network, thus effectively solving the target preference problem in the suspension control system and significantly improving the response ability and stability of the system.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle vibration control, and particularly relates to a suspension control method, a storage medium, and an electronic device. Background Art

[0002] In existing deep reinforcement learning technologies for practical engineering applications such as suspension control, there is a problem that it is difficult to balance multiple objectives. Specifically, current methods usually rely on precise reward function weights to balance the priorities of each objective, but this approach has the following main deficiencies:

[0003] (1) Complexity of target preference design: Existing methods need to rely on precise and fixed reward weight settings to adjust the priorities of multiple objectives. However, the interaction between different performance indicators is complex and dynamic, and it is difficult to accurately describe it through static weight settings, resulting in ineffective resolution of conflicts between objectives and poor adaptability of the system under different road conditions.

[0004] (2) Low convergence and training efficiency: Existing algorithms often have difficulty converging in multi-objective optimization tasks, and the training process is time-consuming. Especially when the weight design is unreasonable, it is more likely to fall into local optima or exhibit instability of solutions.

[0005] (3) Limited response speed and stability: Traditional methods fail to effectively handle the complex trade-off between objectives, resulting in difficulty in synchronously optimizing the response speed and stability of the suspension control system. This makes the vehicle prone to a decline in control performance when facing complex road conditions and difficult to meet the requirements of stability and fast response in practical applications. Summary of the Invention

[0006] In view of this, the present invention aims to provide a suspension control method, a storage medium, and an electronic device, which combine the idea of adversarial networks with the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm to achieve adversarial correction of the action space in the form of dual-network collaborative optimization, enabling the suspension to achieve dynamic balance in the environment. At the same time, the present invention reduces the complexity of reward function design and significantly improves the convergence speed and training efficiency of the network, thus effectively solving the problem of target preference in the suspension control system and significantly enhancing the response ability and stability of the system.

[0007] To achieve the above object, the technical solution of the present invention is realized as follows:

[0008] A suspension control method, comprising:

[0009] S1: Build a basic TD3 architecture, use the current state of the suspension as the input, and the active control action of the suspension as the output, and pre-train the basic TD3 architecture; during the pre-training process, determine the conflict state;

[0010] S2: Add the Actor-against network and the Critic-against network to the basic TD3 architecture to obtain the adversarial-TD3 architecture;

[0011] The Actor-against network receives the conflict state and generates a corresponding action against it , and combines the action against it with the action generated by the basic TD3 architecture to obtain the final control action of the suspension;

[0012] The Critic-against network receives the action against it and the conflict state, and guides the Actor-against network to generate a new action against it ;

[0013] S3: Use the conflict state to train the adversarial-TD3 architecture to obtain a control model for controlling the suspension.

[0014] Furthermore, in step S1, the pre-training process of the basic TD3 architecture includes:

[0015] S11: Initialize the basic TD3 architecture;

[0016] S12: In the current state , the Actor network in the basic TD3 architecture generates a first action with noise , and controls the suspension to execute the first action , to obtain a first state and the corresponding first reward ; Store the experience data in the experience pool;

[0017] S13: Input the first action and the current state into the Critic1 network in the basic TD3 architecture to obtain a result , representing the network parameters of the Critic1 network; Input the first action and the current state into the Critic2 network in the basic TD3 architecture to obtain a result , representing the network parameters of the Critic2 network;

[0018] S14: Randomly sample a batch of samples from the experience pool, and according to the samples , use the Actor-T network in the basic TD3 architecture to generate a first target action with noise ; Using the CriticT1 network and the CriticT2 network in the basic TD3 architecture, calculate the first target action through the following formula The corresponding first target value:

[0019] ;

[0020] Wherein, Represents the first target value, Represents the discount factor, Represents the CriticT1 network, Represents the CriticT2 network, Represents the network parameters of the CriticT1 network, Represents the network parameters of the CriticT2 network;

[0021] S15: Combine the first target value Obtained in step S14 and the calculation result obtained in step S13, and calculate the loss functions of the Critic1 network and the Critic2 network through the following formula:

[0022] ;

[0023] ;

[0024] Wherein, Represents the loss function of the Critic1 network, Represents the loss function of the Critic2 network;

[0025] S16: According to the loss functions obtained in step S15, perform parameter update on the basic TD3 architecture using the delayed update strategy, and repeat steps S12 - S16 with the updated basic TD3 architecture to complete the pre-training of the basic TD3 architecture.

[0026] Furthermore, in step S2, for the action Combine with the action generated by the basic TD3 architecture through the following formula to obtain the final control action:

[0027] ;

[0028] Wherein, Represents the final control action, Represents the action generated by the basic TD3 architecture, and K represents the weighting ratio.

[0029] Furthermore, in step S3, the training process of the adversarial - TD3 architecture includes:

[0030] S31: Initialize the adversarial - TD3 architecture;

[0031] S32: Extract the conflict state from the current state . The Actor network generates a second action in the current state , and the Actor-against network generates a countermeasure action against the conflict state . Obtain the corresponding countermeasure reward for the countermeasure action ; The Critic-against network receives the countermeasure action and the conflict state , and obtains the output of the Critic-against network , where represents the parameters of the Critic-against network;

[0032] S33: Combine the countermeasure action in step S32 with the second action to obtain the final control action ; Control the suspension to execute the final control action , and obtain the second state and the second reward ; Store the experience data in the experience pool;

[0033] S34: Input the second action and the current state into the Critic1 network to obtain a result ; Input the second action and the current state into the Critic2 network to obtain a result ;

[0034] S35: Randomly sample a batch of samples from the experience pool , and use the Actor-T network to generate a second target action with noise according to the samples ; Use the CriticT1 network and the CriticT2 network to calculate the second target value corresponding to the second target action through the following formula: :

[0035] ;

[0036] S36: Combine the second target value obtained in step S35 and the calculation result obtained in step S33, and calculate the new loss functions of the Critic1 network and the Critic2 network through the following formula:

[0037] ;

[0038] ;

[0039] Among them, represents the new loss function of the Critic1 network corresponding to the second target value, represents the new loss function of the Critic2 network corresponding to the second target value, and MSE represents the mean squared error loss function;

[0040] Using the gradient descent method, for the network parameters of the Critic1 network and the network parameters of the Critic2 network are updated to minimize the two new loss functions;

[0041] S37: Use the action in step S32 and the output of the Critic-against network , and the second target value obtained in step S35 , use the following formula to update the parameters of the Actor-agains network :

[0042] ;

[0043] S38: Adopt a delayed update strategy to update the parameters of the adversarial-TD3 architecture, and repeat steps S32 to S38 with the updated adversarial-TD3 architecture.

[0044] Furthermore, the delayed update strategy in step S38 includes:

[0045] In each training:

[0046] Using the network parameters of the Critic1 network , through the following formula, update the network parameters of the CriticT1 network :

[0047] ;

[0048] Among them, represents the soft update rate;

[0049] Using the network parameters of the Critic2 network , through the following formula, update the network parameters of the CriticT2 network :

[0050] ;

[0051] After every policy_delay training rounds, calculate the loss of the Actor network using the following formula :

[0052] ;

[0053] where represents the parameters of the Actor network, represents the action generated by the Actor network in the current state ;

[0054] Using the parameters of the Actor network , update the network parameters of the Actor-T network using the following formula:

[0055] .

[0056] Furthermore, the suspension control method further includes:

[0057] S4: Periodically update the Actor network, Actor-T network, and Actor-against network in the adversarial-TD3 architecture using the following formula:

[0058] ;

[0059] where E represents the mean squared error, represents the generated action of the Actor network, Actor-T network, or Actor-against network in the current state , represents taking the minimum value of the action a gradient, represents taking the result after inputting the action a by the Critic1 network in the adversarial-TD3 architecture in the current state and the result after inputting the action a by the Critic2 network in the current state , that is:

[0060] ;

[0061] represents taking the gradient of the network parameters of the Actor network, Actor-T network, or Actor-against network

[0062] ​A readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the steps of the suspension control method provided by the present invention are implemented.

[0063] An electronic device includes:

[0064] A memory for storing a computer program;

[0065] A processor for implementing the steps of the suspension control method provided by the present invention when executing the computer program.

[0066] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0067] In the suspension control method of the present invention, the concept of an adversarial network is introduced into the deep reinforcement learning algorithm structure based on TD-3, and action adversarial networks (i.e., Actor-against network and Critic-against network) are used for final action correction, so that the finally calculated action can optimize the target suppressed by it to a certain extent on the basis of optimizing the advantage target, and bypass the difficult and highly non-explainable reward function design link. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0069] Figure 1 is a schematic flow chart of the suspension control method described in the embodiment of the present invention;

[0070] Figure 2 is a schematic diagram of the basic TD3 architecture described in the embodiment of the present invention;

[0071] Figure 3 is a schematic diagram of the adversarial-TD3 architecture described in the embodiment of the present invention;

[0072] Figure 4 is a schematic diagram of the structure of the electronic device described in the embodiment of the present invention.

[0073] Description of the reference numerals:

[0074] 1. Electronic device; 2. External device; 3. Processing unit; 4. Bus; 5. Network adapter; 6. Display; 7. (I / O) interface; 8. System memory; 9. Random access memory; 10. Cache memory; 11. Storage system; 12. Utility; 13. Program module. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the following further details the present invention in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation to the present invention.

[0076] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0077] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation to the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more.

[0078] In the description of the present invention, it should be noted that unless otherwise clearly specified and limited, the terms "installed", "connected", "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood through specific situations.

[0079] The following will detail the present invention with reference to the accompanying drawings and in conjunction with embodiments.

[0080] As Figures 1 to 3 shown, the suspension control method described in the embodiment of the present invention includes:

[0081] S1: Build a basic TD3 architecture, use the current state of the suspension as the input, and the active control action of the suspension as the output, and pre-train the basic TD3 architecture; during the pre-training process, determine the conflict state.

[0082] In one embodiment, the states of the suspension include the displacement, velocity, and acceleration of the sprung mass, and the displacement, velocity, and acceleration of the unsprung mass.

[0083] The conflicting states are explained here: Suppose it is desired that state S1 always converges to the feasible region of -5 to +5 in the end. If it occurs that when state S1 converges within the feasible region, state S2 does not always converge within the feasible region of S2, then it is determined that the two objectives of state S1 and state S2 are in conflict.

[0084] In some embodiments, in step S1, the pre-training process of the basic TD3 architecture includes:

[0085] S11: Initialize the basic TD3 architecture. The basic TD3 architecture includes an Actor network, an Actor-T network, a Critic1 network, a Critic2 network, a CriticT1 network, and a CriticT2 network. In a specific embodiment, the Actor network, the Actor-T network, the two CriticT networks, and the two Critic networks generally adopt a structure of 3 to 4 layers, where the structures of the Actor network and the Actor-T network are the same, and the structures of the two CriticT networks and the two Critic networks correspond to each other, that is, the structure of the CriticT1 network is the same as that of the CriticT1 network, and the structure of the CriticT2 network is the same as that of the CriticT2 network. The input layer of the Actor network can receive the state with one node, pass through 2 to 3 hidden layers (for example, the number of nodes is 300 and 200, and the ReLU activation function is used), and the output layer generates the action. Since the generated action needs to be restricted within the range of the action space (for example, [-1, 1]), the tanh activation function is usually used. If the range of the action space is larger, the output value can be extended to the target range through a linear transformation. The input layers of the two Critic networks both receive the concatenated result of the current state and the action, and there should be two nodes in the input layer. The hidden layer settings are similar to those of the Actor network (also using the ReLU activation function), and the output layer is a linear output for calculating the evaluation value of the generated action. The learning rate of the Actor network is usually set to 10 -4 and the learning rates of the two Critic networks are both 10 -3 , and the strategy of soft-updating the target network is adopted. The weight update coefficient τ is usually in [0.005, 0.01]. This design achieves a balance between performance and computational cost, and at the same time avoids the problems of gradient disappearance and overfitting. The initialization specifically includes: randomly initializing the following parameters using the Xavier initialization method: the network parameters of the Actor network and the network parameters of the Actor-T network , Network parameters of Critic1 network , Network parameters of Critic2 network , Network parameters of CriticT1 network , and network parameters of CriticT2 network . In addition, the experience pool and soft update rate are randomly initialized , discount factor and update frequency policy_delay.

[0086] S12: In the current state , the Actor network generates a first action with noise , that is:

[0087] ;

[0088] where represents the Actor network, represents the noise.

[0089] The actuator in the suspension executes the first action to obtain the first state , and the corresponding first reward is obtained through the following formula , that is:

[0090] ;

[0091] where n represents the dimension number of the first state , represents the weight of the reward corresponding to different dimensions i of the first state , which can be determined by the researchers themselves. Equal weight can be preferentially selected, that is .

[0092] The current state , the first action , the first state and the first reward are used as a set of data, experience data , and stored in the experience pool in the form of samples.

[0093] S13: Input the first action and the current state into the Critic1 network to obtain the result , input the first action and the current state into the Critic2 network to obtain the result .

[0094] S14: Randomly sample a batch of samples from the experience pool , and based on the sample , use the Actor-T network to generate the first target action with noise , that is:

[0095] ;

[0096] Among them, represents the Actor-T network, represents the noise, and Gaussian noise is used in a certain embodiment.

[0097] Utilize the CriticT1 network and the CriticT2 network to calculate the first target value corresponding to the first target action through the following formula: :

[0098] ;

[0099] Among them, represents the first target value, represents the CriticT1 network, represents the CriticT2 network.

[0100] S15: Combine the first target value obtained in step S14 and the calculation result obtained in step S13, and calculate the loss functions of the Critic1 network and the Critic2 network through the following formula:

[0101] ;

[0102] ;

[0103] Among them, represents the loss function of the Critic1 network, represents the loss function of the Critic2 network, and MSE represents the mean square error loss function.

[0104] S16: According to the loss functions obtained in step S15, adopt a delayed update strategy to update the parameters of the basic TD3 architecture, and repeat steps S12 - S16 with the updated basic TD3 architecture to complete the pre-training of the basic TD3 architecture.

[0105] It should be noted that this pre-training is only for discovering the adversarial target, which depends on the specific research project of the researcher. If a researcher has conducted multiple training experiments in their research field and has already discovered the adversarial target, then there is no need for pre-training.

[0106] Specifically, the parameter update in step S16 occurs during the pre-training process of the basic TD3 architecture. During each round of pre-training, the parameters are updated by the following formula and parameter :

[0107] ;

[0108] ;

[0109] where represents taking the gradient of the loss function with respect to in it, represents taking the gradient of the loss function with respect to in it.

[0110] According to parameter and parameter respectively update parameter and parameter :

[0111] ;

[0112] ;

[0113] Every policy_delay rounds of training, update the parameters and parameter , including:

[0114] Calculate the policy loss by the following formula:

[0115] ;

[0116] According to the policy loss , update the parameter by the following formula:

[0117] ;

[0118] where represents taking the gradient of the policy loss with respect to in it;

[0119] Then update the parameter by the following formula:

[0120] ;

[0121] Update the current state to the first state .

[0122] It should be noted that before adding the action adversarial network (i.e., Actor-against network and Critic-against network), it is necessary to pre-train the original network under the original algorithm structure framework (i.e., the basic TD3 architecture), observe the differences in the convergence speed and convergence degree between different optimization objectives. The objective with fast convergence and good optimization will be defined as the dominant state, while the objective in conflict with it, that is, the objective with slow convergence and poor optimization, will be defined as the conflict state. The conflict state is the object processed by the action adversarial network (i.e., Actor-against network and Critic-against network).

[0123] S2: Add the Actor-against network and Critic-against network on the basis of the basic TD3 architecture to obtain the adversarial-TD3 architecture. Among them, the Critic-against network receives the action and the conflict state, and guides the Actor-against network to generate a new action ; the Actor-against network receives the conflict state and generates the corresponding action , combines the action with the action generated by the basic TD3 architecture to obtain the final control action of the suspension. In a specific embodiment, the action is combined with the action generated by the basic TD3 architecture through the following formula to obtain the final control action:

[0124] ;

[0125] where, represents the final control action, represents the action generated by the basic TD3 architecture, and K represents the weighting ratio.

[0126] S3: Use the conflict state to train the adversarial-TD3 architecture to obtain a control model for controlling the suspension.

[0127] In some embodiments, in step S3, the training process of the adversarial-TD3 architecture includes:

[0128] S31: Initialize the adversarial-TD3 architecture.

[0129] In a specific embodiment, the initialization specifically includes: random initialization: the network parameters of the Critic1 network, the network parameters of the Critic2 network, , Network parameters of the CriticT1 network , Network parameters of the CriticT2 network , Network parameters of the Critic-against network , and network parameters of the Actor-against network . Among them, in the embodiments of the present invention, let the initial value of the network parameter be equal to the initial value of the network parameter , let the initial value of the network parameter be equal to the initial value of the network parameter , and let the initial value of the network parameter be equal to the initial value of the network parameter . In addition, the experience pool, soft update rate , discount factor , learning rate of the Actor network , and weighting ratio K are also randomly initialized.

[0130] S32: Extract the conflict state from the current state , that is:

[0131] ;

[0132] In one embodiment, is to extract the dimension where the conflict state in the current state is located.

[0133] The Actor-against network generates a countermeasure action for the conflict state , and obtains the countermeasure reward corresponding to the countermeasure action , that is:

[0134] ;

[0135] ;

[0136] Among them, represents a function that obtains the corresponding state and reward according to the input action.

[0137] The Actor network generates a second action in the current state , that is:

[0138] ;

[0139] The Critic-against network receives the countermeasure action and the conflict state , obtain the output of the Critic-against network ;

[0140] S33: Combine the action in step S32 with the second action using the method of step S2 to obtain the final control action .

[0141] Control the suspension to execute the final control action , obtain the second state and the second reward , that is:

[0142] ;

[0143] Take the current state , the final control action , the second state and the second reward as a set of data, experience data , and store it in the experience pool in the form of samples.

[0144] S34: Input the second action and the current state into the Critic1 network to obtain the result ; Input the second action and the current state into the said Critic2 network to obtain the result .

[0145] S35: Randomly sample a batch of samples from the experience pool , and according to the samples , use the Actor-T network to generate a second target action with noise , that is:

[0146] ;

[0147] wherein represents the local noise intercepted from the noise [-c, c], in one embodiment, the noise is Gaussian noise.

[0148] Use the CriticT1 network and the CriticT2 network to calculate the second target value corresponding to the second target action :

[0149] 。

[0150] S36: Combine the second target value obtained in step S35 and the calculation result obtained in step S33, and calculate the new loss functions of the Critic1 network and the Critic2 network through the following formula:

[0151] ;

[0152] ;

[0153] where, represents the new loss function of the Critic1 network corresponding to the second target value, represents the new loss function of the Critic2 network corresponding to the second target value;

[0154] Update the network parameters and network parameters using the gradient descent method to minimize the two new loss functions.

[0155] S37: Use the reward in step S32 and the output of the Critic-against network , and the second target value obtained in step S35 , and use the following formula to update the parameters of the Actor-agains network using the gradient descent method:

[0156] ;

[0157] S38: Adopt a delayed update strategy to update the parameters of the adversarial-TD3 architecture, and repeat steps S32 - S38 with the updated adversarial-TD3 architecture. The number of repetitions can be selected according to actual needs and the experience of researchers.

[0158] In a specific embodiment, repeat steps S32 - S38 600 - 650 times with the updated adversarial-TD3 architecture. The delayed update strategy in step S38 includes:

[0159] In each training:

[0160] Use the network parameters of the Critic1 network to update the network parameters of the CriticT1 network through the following formula:

[0161] ;

[0162] where, represents the soft update rate;

[0163] Using the network parameters of the Critic2 network , update the network parameters of the CriticT2 network through the following formula :

[0164] ;

[0165] After every policy_delay training rounds, calculate the loss of the Actor network through the following formula :

[0166] ;

[0167] where represents the parameters of the Actor network, represents the action generated by the Actor network in the current state ;

[0168] According to the loss , update the parameters of the Actor network through the following formula

[0169] ;

[0170] where represents taking the gradient of in the loss ;

[0171] Using the parameters of the Actor network, update the network parameters of the Actor-T network through the following formula

[0172] .

[0173] Furthermore, the suspension control method further includes:

[0174] S4: Periodically update the Actor network, Actor-T network, and Actor-against network in the adversarial-TD3 architecture through the following formula

[0175] ;

[0176] where E represents the mean square error, represents the action generated by the Actor network, Actor-T network, or Actor-against network in the current state , represents taking the minimum value of the action a gradient, Indicates the current state of the Critic1 network in the adversarial-TD3 architecture The result after entering action a and Critic2 network in its current state The result after entering action a The minimum value in , that is:

[0177] ;

[0178] Indicates the network parameters for an Actor network, an Actor-T network, or an Actor-against network Find the gradient.

[0179] Accordingly, according to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0180] Figure 4 FIG. 1 is a structural diagram of an electronic device 1 provided in an embodiment of the present invention. Figure 4 A block diagram of an exemplary electronic device 1 suitable for implementing embodiments of the present invention is shown. Figure 4 The electronic device 1 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0181] like Figure 4 As shown, electronic device 1 is represented in the form of a general-purpose computing device. Electronic device 1 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0182] The components of the electronic device 1 may include, but are not limited to: one or more processors or processing units 3, a system memory 8, and a bus 4 connecting different system components (including the system memory 8 and the processing unit 3).

[0183] The bus 4 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an Accelerated Graphics Port, a processor bus, or a local bus using any of the several bus architectures. By way of example, and not limitation, these architectures include the Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0184] The electronic device 1 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 1, including both volatile and nonvolatile media, removable and non-removable media.

[0185] The system memory 8 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 9 and / or cache memory 10. The electronic device 1 can further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, a storage system 11 can be provided for reading from and writing to a non-removable, nonvolatile magnetic medium ( Figure 4 not shown and typically called a "hard disk drive"). Although Figure 4 not shown in FIG. 1, a disk drive for reading from and writing to a removable nonvolatile disk (e.g., a "floppy disk"), and an optical disk drive for reading from and writing to a removable nonvolatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) can be provided. In these instances, each drive can be connected to the bus 4 by one or more data media interfaces. The system memory 8 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the present invention.

[0186] A program / utility 12 having a set (at least one) of program modules 13 can be stored, for example, in the system memory 8, such program modules 13 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may include an implementation of a networking environment. The program modules 13 typically carry out the functions and / or methods of the embodiments described herein.

[0187] The electronic device 1 can also communicate with one or more external devices 2 (such as a keyboard, a pointing device, a display 6, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 1, and / or communicate with any device that enables the electronic device 1 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 7. Moreover, the electronic device 1 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 5. As Figure 4 shown, the network adapter 5 communicates with other modules of the electronic device 1 through the bus 4. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 1, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0188] The processing unit 3 executes various functional applications and data processing by running programs stored in the system memory 8, for example, implementing the suspension control method provided by the embodiments of the present invention.

[0189] Embodiments of the present invention also provide a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein when the program is executed by a processor, the suspension control method provided by all embodiments of the present application is implemented.

[0190] The computer storage medium of the embodiments of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or device.

[0191] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0192] The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. The computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0193] An embodiment of the present invention also provides a computer program product, including a computer program, which when executed by a processor implements the suspension control method according to the foregoing.

[0194] It should be understood that various forms of the flow shown above may be used, with steps reordered, added, or deleted. For example, the steps recited in the disclosure of the present invention may be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and no limitation is imposed herein.

[0195] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A suspension control method, characterized in that, Including: S1: Build a basic TD3 architecture, use the current state of the suspension as the input, and the active control action of the suspension as the output, and pre-train the basic TD3 architecture; During the pre-training process, determine the conflict state; The pre-training process of the basic TD3 architecture includes: S11: Initialize the basic TD3 architecture; S12: In the current state the Actor network in the basic TD3 architecture generates a first action with noise and controls the suspension to execute the first action to obtain a first state and the corresponding first reward ; store the experience data into the experience pool; S13: Input the first action and the current state into the Critic1 network in the basic TD3 architecture to obtain a result , representing the network parameters of the Critic1 network; Input the first action and the current state into the Critic2 network in the basic TD3 architecture to obtain a result , representing the network parameters of the Critic2 network; S14: Randomly sample a batch of samples from the experience pool , and based on the samples , use the Actor-T network in the basic TD3 architecture to generate a first target action with noise ; Utilize the CriticT1 network and the CriticT2 network in the basic TD3 architecture to calculate the first target value corresponding to the first target action through the following formula: ； Among them, represents the first target value, represents the discount factor, represents the CriticT1 network, represents the CriticT2 network, represents the network parameters of the CriticT1 network, represents the network parameters of the CriticT2 network; S15: Combine the first target value obtained in step S14 and the calculation result obtained in step S13, and calculate the loss functions for training the Critic1 network and the Critic2 network through the following formula: ； ； Among them, represents the loss function of the Critic1 network, represents the loss function of the Critic2 network; S16: According to the loss function obtained in step S15, update the parameters of the basic TD3 architecture using a delayed update strategy, and repeat steps S12 - S16 with the updated basic TD3 architecture to complete the pre-training of the basic TD3 architecture; S2: Add an Actor-against network and a Critic-against network to the basic TD3 architecture to obtain an adversarial-TD3 architecture; The Actor-against network receives the conflict state and generates a corresponding action against it , combines the action against it with the action generated by the basic TD3 architecture to obtain the final control action of the suspension; The Critic-against network receives the action against and the conflict state, and guides the Actor-against network to generate a new action against ; S3: Use the conflict state to train the adversarial-TD3 architecture to obtain a control model for controlling the suspension.

2. The suspension control method according to claim 1, wherein In step S2, the action for is combined with the action generated by the basic TD3 architecture through the following formula to obtain the final control action: ； Among them, represents the said final control action, represents the action generated by the said basic TD3 architecture, and K represents the weighting ratio.

3. The suspension control method according to claim 1, wherein, In step S3, the training process of the adversarial-TD3 architecture includes: S31: Initialize the adversarial-TD3 architecture; S32: Extract the conflict state from the current state ; the Actor network generates a second action in the current state ; the Actor-against network generates a countermeasure action against the conflict state ; obtain the corresponding countermeasure reward for the countermeasure action ; the Critic-against network receives the countermeasure action and the conflict state ; obtain the output of the Critic-against network ; represents the parameters of the Critic-against network ; ; ; ​ S33: Combine the action in step S32 with the second action to obtain the final control action ; Control the suspension to execute the final control action to obtain the second state and the second reward ; Store the experience data into the experience pool; S34: Input the second action and the current state into the Critic1 network to obtain a result ; Input the second action and the current state into the Critic2 network to obtain a result ; S35: Randomly sample a batch of samples from the experience pool , and based on the samples , use the Actor-T network to generate a second target action with noise ; Use the CriticT1 network and the CriticT2 network to calculate the second target value corresponding to the second target action through the following formula : ； S36: Combine the second target value obtained in step S35 and the calculation result obtained in step S33, and calculate the new loss functions of the Critic1 network and the Critic2 network through the following formula: ； ； Among them, represents the new loss function of the Critic1 network corresponding to the second target value, represents the new loss function of the Critic2 network corresponding to the second target value, and MSE represents the mean square error loss function; Using the gradient descent method, the network parameters of the Critic1 network and the network parameters of the Critic2 network are updated to minimize the two new loss functions; S37: Use the reward in step S32 and the output of the Critic-against network , and the second target value obtained in step S35 , and use the gradient descent method to update the parameters of the Actor-against network using the following formula : ； S38: Update the parameters of the adversarial-TD3 architecture using a delayed update strategy, and repeat steps S32 - S38 with the updated adversarial-TD3 architecture.

4. The suspension control method according to claim 3, characterized in that The delayed update strategy in step S38 includes: In each training: Using the network parameters of the Critic1 network , the network parameters of the CriticT1 network are updated by the following formula: ； Among them, represents the soft update rate; Using the network parameters of the Critic2 network , update the network parameters of the CriticT2 network through the following formula: ； After every training round with a policy_delay, calculate the loss of the Actor network using the following formula : ； Among them, represents the parameters of the Actor network, represents the action generated by the Actor network in the current state ; Using the parameters of the Actor network , the network parameters of the Actor-T network are updated by the following formula: 。 5. The suspension control method according to claim 1, wherein The suspension control method further includes: S4: Periodically update the Actor network, Actor-T network, and Actor-against network in the adversarial-TD3 architecture through the following formula: ； where E represents the mean squared error, represents the current state the generated action of the Actor network, the Actor-T network or the Actor-against network described below, represents taking the minimum value with respect to the action a of the gradient, represents taking the result after inputting the action a by the Critic1 network in the adversarial-TD3 architecture at the current state and the result after inputting the action a by the Critic2 network in the current state and taking the minimum value among them, that is: the result after inputting the action a, ​ ； represent the network parameters of the said Actor network, the said Actor-T network or the said Actor-against network calculate the gradient.

6. A readable storage medium, characterized in that, A computer program is stored on the readable storage medium, and when the computer program is executed by a processor, it implements the steps of the suspension control method according to any one of claims 1 to 5.

7. An electronic device, characterized in that, Including: A memory for storing a computer program; A processor for implementing the steps of the suspension control method according to any one of claims 1 to 5 when executing the computer program.

Images