Suspension control method, storage medium and electronic equipment

Abstract

The invention relates to the technical field of vehicle vibration control, in particular to a suspension control method, a storage medium and electronic device.The method comprises the steps that firstly, a basic TD3 framework is built, the current state of a suspension serves as input, the active control action of the suspension serves as output, the basic TD3 framework is pre-trained, and in the pre-training process, the conflict state is determined; the method comprises the following steps: introducing an adversarial network into a basic TD3 architecture, namely, adding an Actor-Aaginst network and a Crimic-Aaginst network on the structure of the basic TD3 architecture to obtain an adversariation-TD3 architecture; and finally, training the adversariation-TD3 architecture by using the conflict state to obtain a control model for controlling the suspension. According to the method, the antagonism correction of the action space is realized in a dual-network collaborative optimization form, the complexity of reward function design is reduced, and the convergence speed and the training efficiency of the network are greatly improved, so that the target preference problem in the suspension control system is effectively solved, and the response capability and the stability of the system are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle vibration control, and particularly relates to a suspension control method, a storage medium, and an electronic device. Background Art

[0002] In existing deep reinforcement learning technologies for practical engineering applications such as suspension control, there is a problem of difficulty in balancing multiple objectives. Specifically, current methods usually rely on precise reward function weights to weigh the priorities of each objective, but this approach has the following main deficiencies: (1) Complexity of target preference design: Existing methods need to rely on precise and fixed reward weight settings to adjust the priorities of multiple objectives. However, the interaction between different performance indicators is complex and dynamic, making it difficult to accurately describe through static weight settings, resulting in ineffective resolution of conflicts between objectives and poor adaptability of the system under different road conditions; (2) Low convergence and training efficiency: Existing algorithms often have difficulty converging in multi-objective optimization tasks, and the training process is time-consuming. Especially when the weight design is unreasonable, it is more likely to fall into local optima or exhibit instability of solutions; (3) Limited response speed and stability: Traditional methods, due to the failure to effectively handle the complex trade-off between objectives, make it difficult to synchronously optimize the response speed and stability of the suspension control system, resulting in a decline in control performance of the vehicle when facing complex road conditions and difficulty in meeting the requirements of stability and fast response in practical applications. Summary of the Invention

[0003] In view of this, the present invention aims to provide a suspension control method, a storage medium, and an electronic device, which combine the idea of an adversarial network with the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm to achieve adversarial correction of the action space in the form of collaborative optimization of dual networks, enabling the suspension to achieve dynamic balance in the environment; at the same time, the present invention reduces the complexity of reward function design and significantly improves the convergence speed and training efficiency of the network, thereby effectively solving the problem of target preference in the suspension control system and significantly enhancing the response ability and stability of the system.

[0004] To achieve the above object, the technical solution of the present invention is realized as follows: A suspension control method, comprising: S1: Build a basic TD3 architecture, use the current state of the suspension as input, and the active control action of the suspension as output, and pre-train the basic TD3 architecture; during the pre-training process, determine the conflict state; S2: Add an Actor-against network and a Critic-against network to the basic TD3 architecture to obtain an adversarial-TD3 architecture; The Actor-against network receives the conflict state and generates corresponding actions against it The actions against it are combined with the actions generated by the basic TD3 architecture to obtain the final control actions for the suspension; The Critic-against network receives the actions against it and the conflict state, and guides the Actor-against network to generate new actions against it ; S3: Use the conflict state to train the adversarial-TD3 architecture to obtain a control model for controlling the suspension.

[0005] Furthermore, in step S1, the pre-training process of the basic TD3 architecture includes: S11: Initialize the basic TD3 architecture; S12: In the current state , the Actor network in the basic TD3 architecture generates a first action with noise , and controls the execution of the first action in the suspension to obtain a first state and the corresponding first reward ; Store the experience data in the experience pool; S13: Input the first action and the current state into the Critic1 network in the basic TD3 architecture to obtain a result , representing the network parameters of the Critic1 network; Input the first action and the current state into the Critic2 network in the basic TD3 architecture to obtain a result , representing the network parameters of the Critic2 network; S14: Randomly sample a batch of samples from the experience pool , and according to the samples , use the Actor-T network in the basic TD3 architecture to generate a first target action with noise ; Use the CriticT1 network and CriticT2 network in the basic TD3 architecture to calculate the first target value corresponding to the first target action through the following formula: ; Among them, represents the first target value, represents the discount factor, Denote the CriticT1 network, Denote the CriticT2 network, Denote the network parameters of the CriticT1 network, Denote the network parameters of the CriticT2 network; S15: Combine the first target value obtained in step S14 and the calculation result obtained in step S13, and calculate the loss functions of the Critic1 network and the Critic2 network through the following formula: ; ; where, Denote the loss function of the Critic1 network, Denote the loss function of the Critic2 network; S16: According to the loss functions obtained in step S15, adopt a delayed update strategy to update the parameters of the basic TD3 architecture, and repeat steps S12 - S16 with the updated basic TD3 architecture to complete the pre-training of the basic TD3 architecture.

[0006] Further, in step S2, for the action Combine it with the action generated by the basic TD3 architecture through the following formula to obtain the final control action: ; where, Denote the final control action, Denote the action generated by the basic TD3 architecture, and K represents the weighting ratio.

[0007] Further, in step S3, the training process of the adversarial - TD3 architecture includes: S31: Initialize the adversarial - TD3 architecture; S32: Extract the conflict state in the current state , the Actor network generates a second action in the current state , the Actor - against network generates a countermeasure action for the conflict state , obtain the countermeasure reward corresponding to the countermeasure action ; The Critic - against network receives the countermeasure action and the conflict state , and obtains the output of the Critic - against network, Represent the parameters of the Critic-against network; S33: Combine the action in step S32 with the second action to obtain the final control action ; Control the suspension to execute the final control action to obtain the second state and the second reward ; Store the experience data into the experience pool; S34: Input the second action and the current state into the Critic1 network to obtain the result ; Input the second action and the current state into the said Critic2 network to obtain the result ; S35: Randomly sample a batch of samples from the experience pool , and according to the samples , use the Actor-T network to generate a second target action with noise ; Use the CriticT1 network and the CriticT2 network to calculate the second target value corresponding to the second target action : ; S36: Combine the second target value obtained in step S35 and the calculation result obtained in step S33, and calculate the new loss functions of the Critic1 network and the Critic2 network through the following formula: ; ; where, represents the new loss function of the Critic1 network corresponding to the second target value, represents the new loss function of the Critic2 network corresponding to the second target value, and MSE represents the mean square error loss function; Adopt the gradient descent method to update the network parameters of the Critic1 network and the network parameters of the Critic2 network to make the two new loss functions reach the minimum value; S37: Use the action in step S32 and the output of the Critic-against network , as well as the second target value obtained in step S35 , the parameters of the Actor - against network are updated using the gradient descent method with the following formula : ; S38: The parameters of the adversarial - TD3 architecture are updated using a delayed update strategy, and steps S32 - S38 are repeated with the updated adversarial - TD3 architecture.

[0008] Furthermore, the delayed update strategy in step S38 includes: In each training: Using the network parameters of the Critic1 network , the network parameters of the CriticT1 network are updated through the following formula: ; where represents the soft update rate; Using the network parameters of the Critic2 network , the network parameters of the CriticT2 network are updated through the following formula: ; After every policy_delay training rounds, the loss of the Actor network is calculated through the following formula : ; where represents the parameters of the Actor network, represents the action generated by the Actor network in the current state ; Using the parameters of the Actor network , the network parameters of the Actor - T network are updated through the following formula: .

[0009] Furthermore, the suspension control method further includes: S4: The Actor network, Actor - T network, and Actor - against network in the adversarial - TD3 architecture are periodically updated through the following formula: ; where E represents the mean square error, represents the generated action of the Actor network, Actor - T network, or Actor - against network in the current state , represents taking the minimum value of the action a The gradient of represents the result after inputting the action a in the current state of the Critic1 network in the adversarial-TD3 architecture and the result after inputting the action a in the current state of the Critic2 network The minimum value among them, that is: ; ; represents taking the gradient of the network parameters of the Actor network, Actor-T network, or Actor-against network Find the gradient.

[0010] A readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the suspension control method provided by the present invention.

[0011] An electronic device includes: A memory for storing a computer program; A processor for implementing the steps of the suspension control method provided by the present invention when executing the computer program.

[0012] Compared with the prior art, the present invention can achieve the following beneficial effects: In the suspension control method of the present invention, the concept of an adversarial network is introduced into the structure of the deep reinforcement learning algorithm based on TD-3, and an action adversarial network (i.e., Actor-against network and Critic-against network) is used for final action correction, so that the finally calculated action can, on the basis of optimizing the advantage objective, optimize to a certain extent the objective suppressed by it, and bypass the difficult and highly non-explainable reward function design link. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings: Figure 1 is a schematic flow chart of the suspension control method described in the embodiment of the present invention; Figure 2 is a schematic diagram of the basic TD3 architecture described in the embodiment of the present invention; Figure 3 is a schematic diagram of the adversarial-TD3 architecture described in the embodiment of the present invention; Figure 4Schematic diagram of the structure of the electronic device according to the embodiment of the present invention

[0014] Description of the reference numerals in the drawings: 1. Electronic device; 2. External device; 3. Processing unit; 4. Bus; 5. Network adapter; 6. Display; 7. (I / O) interface; 8. System memory; 9. Random access memory; 10. Cache memory; 11. Storage system; 12. Utility tool; 13. Program module. Detailed implementation manners

[0015] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention.

[0016] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other.

[0017] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, "a plurality" means two or more.

[0018] In the description of the present invention, it should be noted that, unless otherwise clearly defined and limited, the terms "installed", "connected", "connected" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected, or indirectly connected through an intermediate medium, and it may be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood through specific situations.

[0019] The present invention will be described in detail below with reference to the drawings and in combination with embodiments.

[0020] AsFigures 1 to 3 As shown in Figures 1 to 3 , the suspension control method described in the embodiment of the present invention includes: S1: Build a basic TD3 architecture, use the current state of the suspension as the input, and the active control action of the suspension as the output, and pre-train the basic TD3 architecture; during the pre-training process, determine the conflict state.

[0021] In one embodiment, the state of the suspension includes the displacement, velocity, and acceleration of the sprung mass, and the displacement, velocity, and acceleration of the unsprung mass.

[0022] Here is an explanation of the conflict state: Assume that it is desired that state S1 always converges to the feasible region of -5 to +5 in the end. If it occurs that when state S1 converges within the feasible region, state S2 does not always converge within the feasible region of S2, then it is determined that the two targets of state S1 and state S2 are in conflict.

[0023] In some embodiments, in step S1, the pre-training process of the basic TD3 architecture includes: S11: Initialize the basic TD3 architecture. The basic TD3 architecture includes an Actor network, an Actor-T network, a Critic1 network, a Critic2 network, a CriticT1 network, and a CriticT2 network. In a specific embodiment, the Actor network, the Actor-T network, the two CriticT networks, and the two Critic networks usually adopt a structure of 3 to 4 layers. Among them, the structures of the Actor network and the Actor-T network are the same, and the structures of the two CriticT networks and the two Critic networks correspond to each other, that is, the structure of the CriticT1 network is the same as that of the CriticT1 network, and the structure of the CriticT2 network is the same as that of the CriticT2 network. The input layer of the Actor network can receive the state with one node, pass through 2 to 3 hidden layers (for example, the number of nodes is 300 and 200, and the activation function uses ReLU), and the output layer generates an action. Since the generated action needs to be restricted within the range of the action space (for example, [-1,1]), the tanh activation function is usually used. If the range of the action space is larger, the output value can be extended to the target range through a linear transformation. The input layers of the two Critic networks both receive the concatenated result of the current state and the action, and there should be two nodes in the input layer. The hidden layer settings are similar to those of the Actor network (also using the ReLU activation function), and the output layer is a linear output for calculating the evaluation value of the generated action. The learning rate of the Actor network is usually set to 10 -4 , and the learning rates of the two Critic networks are both 10 -3, and adopt the strategy of soft-updating the target network. The weight update coefficient τ is usually in the range of [0.005, 0.01]. This design achieves a balance between performance and computational cost while avoiding the problems of gradient disappearance and overfitting. The initialization specifically includes: randomly initializing the following parameters using the Xavier initialization method: the network parameters of the Actor network , the network parameters of the Actor-T network , the network parameters of the Critic1 network , the network parameters of the Critic2 network , the network parameters of the CriticT1 network , and the network parameters of the CriticT2 network . In addition, the experience pool, soft update rate , discount factor , and update frequency policy_delay are also randomly initialized.

[0024] S12: In the current state , the Actor network generates a first action with noise , that is: ; Among them, represents the Actor network, represents the noise.

[0025] The actuator in the control suspension executes the first action to obtain the first state , and the corresponding first reward is obtained through the following formula, that is: ; Among them, n represents the dimension number of the first state , represents the weight of the reward corresponding to different dimensions i of the first state , which can be determined by the researchers themselves. Equal weight can be preferentially selected, that is .

[0026] The current state , the first action , the first state , and the first reward are used as a set of data experience data , and stored in the experience pool in the form of samples.

[0027] S13: Input the first action and the current state into the Critic1 network to obtain the result , input the first action and the current state Input the Critic2 network to obtain the result .

[0028] S14: Randomly sample a batch of samples from the experience pool , and according to the samples , use the Actor-T network to generate the first target action with noise , that is: ; wherein, represents the Actor-T network, represents the noise, and Gaussian noise is adopted in a certain embodiment.

[0029] Utilize the CriticT1 network and the CriticT2 network to calculate the first target value corresponding to the first target action through the following formula: ; ; wherein, represents the first target value, represents the CriticT1 network, represents the CriticT2 network.

[0030] S15: Combine the first target value obtained in step S14 and the calculation result obtained in step S13, and calculate the loss functions of the Critic1 network and the Critic2 network through the following formula: ; ; wherein, represents the loss function of the Critic1 network, represents the loss function of the Critic2 network, and MSE represents the mean square error loss function.

[0031] S16: According to the loss functions obtained in step S15, adopt a delayed update strategy to update the parameters of the basic TD3 architecture, and repeat steps S12 - S16 with the updated basic TD3 architecture to complete the pre-training of the basic TD3 architecture.

[0032] It should be noted that this pre-training is only for discovering the adversarial target, depending on the specific research project of the researcher. If a researcher has conducted multiple training experiments in his / her research field and has already discovered the adversarial target, then there is no need for pre-training.

[0033] Specifically, the parameter update in step S16 occurs during the pre-training process of the basic TD3 architecture. During each round of pre-training, the parameters are updated according to the following formula and the parameter : ; ; wherein, denotes taking the gradient of the loss function with respect to ; denotes taking the gradient of the loss function with respect to ;

[0034] According to the parameter and the parameter respectively update the parameter and the parameter : ; ; Every policy_delay rounds of training, update the parameter and the parameter , including: Calculate the policy loss through the following formula: ; According to the policy loss , update the parameter through the following formula: ; wherein, denotes taking the gradient of the policy loss with respect to ; Then update the parameter through the following formula: ; Update the current state to the first state .

[0035] It should be noted that before adding the action adversarial network (i.e., the Actor-against network and the Critic-against network), it is necessary to pre-train the original network under the original algorithm structure framework (i.e., the basic TD3 architecture), observe the differences in the convergence speed and convergence degree between different optimization objectives, the objective with fast convergence and good optimization will be defined as the dominant state, and the objective conflicting with it, that is, the objective with slow convergence and poor optimization will be defined as the conflicting state, and the conflicting state is the object processed by the action adversarial network (i.e., the Actor-against network and the Critic-against network). S2: Add the Actor-against network and the Critic-against network on the basis of the basic TD3 architecture to obtain the adversarial-TD3 architecture. Among them, the Critic-against network receives the actions and conflicting states to guide the Actor-against network to generate new actions ; the Actor-against network receives the conflicting state and generates corresponding actions ; combine the actions with the actions generated by the basic TD3 architecture to obtain the final control action of the suspension. In a specific embodiment, the actions and the actions generated by the basic TD3 architecture are combined through the following formula to obtain the final control action: ; ; where represents the final control action, represents the action generated by the basic TD3 architecture, and K represents the weighting ratio.

[0036] S3: Use the conflicting state to train the adversarial-TD3 architecture to obtain a control model for controlling the suspension.

[0037] In some embodiments, in step S3, the training process of the adversarial-TD3 architecture includes: S31: Initialize the adversarial-TD3 architecture.

[0038] In a specific embodiment, the initialization specifically includes: random initialization: the network parameters of the Critic1 network, the network parameters of the Critic2 network, the network parameters of the Actor network, , the network parameters of the Critic-against network , and the network parameters of the Actor-against network . Among them, in the embodiments of the present invention, the initial value of the network parameter is equal to the initial value of the network parameter , the initial value of the network parameter is equal to the initial value of the network parameter , and the initial value of the network parameter is equal to the initial value of the network parameter . In addition, the experience pool, soft update rate , discount factor , learning rate of the Actor network , and weighted ratio K are randomly initialized.

[0039] S32: Extract the conflict state from the current state , that is: ; In one embodiment, is the dimension where the conflict state in the current state is located.

[0040] The Actor-against network generates an action for the conflict state , and obtains the corresponding reward for the action , that is: ; ; Among them, represents a function that obtains the corresponding state and reward according to the input action.

[0041] The Actor network generates a second action in the current state , that is: ; The Critic-against network receives the action and the conflict state , and obtains the output of the Critic-against network; S33: Combine the action in step S32 with the second action using the method in step S2 to obtain the final control action 。

[0042] The control suspension performs the final control action to obtain the second state and the second reward , that is: ; Use the current state , the final control action , the second state and the second reward as a set of data (experience data) , and store it in the experience pool in the form of samples.

[0043] S34: Input the second action and the current state into the Critic1 network to obtain the result ; Input the second action and the current state into the said Critic2 network to obtain the result .

[0044] S35: Randomly sample a batch of samples from the experience pool , and according to the samples , use the Actor-T network to generate a second target action with noise , that is: ; wherein represents the local noise intercepted from the noise [-c, c], and in one embodiment, the noise is Gaussian noise.

[0045] Use the CriticT1 network and the CriticT2 network to calculate the second target value corresponding to the second target action through the following formula: .

[0046] S36: Combine the second target value obtained in step S35 and the calculation results obtained in step S33, and calculate the new loss functions of the Critic1 network and the Critic2 network through the following formula: ; ; wherein represents the new loss function of the Critic1 network corresponding to the second target value Represents the new loss function of the Critic2 network corresponding to the second target value; Update the network parameters using the gradient descent method and the network parameters to minimize the two new loss functions.

[0047] S37: Use the reward in step S32 and the output of the Critic-against network , and the second target value obtained in step S35 , and use the following formula to update the parameters of the Actor-agains network using the gradient descent method : ; S38: Adopt a delayed update strategy to update the parameters of the adversarial-TD3 architecture, and repeat steps S32~S38 with the updated adversarial-TD3 architecture. The number of repetitions can be selected according to actual needs and the experience of researchers.

[0048] In a specific embodiment, repeat steps S32~S38 600~650 times with the updated adversarial-TD3 architecture. The delayed update strategy in step S38 includes: In each training: Utilize the network parameters of the Critic1 network , and update the network parameters of the CriticT1 network through the following formula: ; where represents the soft update rate; Utilize the network parameters of the Critic2 network , and update the network parameters of the CriticT2 network through the following formula: ; After every policy_delay training rounds, calculate the loss of the Actor network through the following formula: ; where represents the parameters of the Actor network, represents the action generated by the Actor network in the current state ; According to the loss , update the parameters of the Actor network through the following formula: ; Among them, represents the gradient calculation of the loss in ; Using the parameters of the Actor network , the network parameters of the Actor-T network are updated through the following formula: .

[0049] Furthermore, the suspension control method further includes: S4: Periodically update the Actor network, Actor-T network, and Actor-against network in the adversarial-TD3 architecture through the following formula: ; where E represents the mean square error, represents the generated action of the Actor network, Actor-T network, or Actor-against network in the current state , represents the minimum value of the action a gradient of, represents taking the minimum value of the result after inputting the action a by the Critic1 network in the adversarial-TD3 architecture in the current state and the result after inputting the action a by the Critic2 network in the current state , that is: and the result after inputting the action a by the Critic2 network in the current state , namely: ; ; represents the gradient calculation of the network parameters of the Actor network, Actor-T network, or Actor-against network .

[0050] Correspondingly, according to the embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0051] Figure 4 is a schematic structural diagram of an electronic device 1 provided in the embodiments of the present invention. Figure 4 shows a block diagram of an exemplary electronic device 1 suitable for implementing the embodiments of the present invention. Figure 4 The displayed electronic device 1 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0052] Such as Figure 4As shown, the electronic device 1 is presented in the form of a general-purpose computing device. The electronic device 1 is intended to represent various forms of digital computers, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the invention described and / or claimed herein.

[0053] The components of the electronic device 1 may include, but are not limited to: one or more processors or processing units 3, a system memory 8, and a bus 4 that couples different system components (including the system memory 8 and the processing unit 3).

[0054] The bus 4 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0055] The electronic device 1 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 1, including both volatile and nonvolatile media, removable and non-removable media.

[0056] The system memory 8 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 9 and / or cache memory 10. The electronic device 1 may further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, a storage system 11 may be used for reading from and writing to a non-removable, nonvolatile magnetic medium ( Figure 4 not shown and typically referred to as a "hard disk drive"). Although Figure 4 not shown in the figures, a disk drive for reading from and writing to a removable nonvolatile disk (e.g., a "floppy disk") and an optical disk drive for reading from and writing to a removable nonvolatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 4 by one or more data media interfaces. The system memory 8 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of the embodiments of the present invention.

[0057] A program / utilities 12 having a set (at least one) of program modules 13 can be stored in, for example, the system memory 8. Such program modules 13 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment. The program modules 13 generally execute the functions and / or methods in the embodiments described in the present invention.

[0058] The electronic device 1 can also communicate with one or more external devices 2 (such as a keyboard, a pointing device, a display 6, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 1, and / or communicate with any device that enables the electronic device 1 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 7. Moreover, the electronic device 1 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 5. As Figure 4 shown, the network adapter 5 communicates with other modules of the electronic device 1 through the bus 4. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 1, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0059] The processing unit 3 executes various functional applications and data processing by running the programs stored in the system memory 8, such as implementing the suspension control method provided in the embodiments of the present invention.

[0060] In the embodiments of the present invention, a non-transitory computer-readable storage medium storing computer instructions is also provided, on which a computer program is stored. When the program is executed by a processor, the suspension control methods provided in all the embodiments of the present application are implemented.

[0061] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable media may be computer-readable signal media or computer-readable storage media. More specific examples (a non-exhaustive list) of the computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0062] The computer-readable signal media may include data signals propagated in a baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal media may also be any computer-readable media other than the computer-readable storage media, and the computer-readable media may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0063] The program code contained on the computer-readable media may be transmitted by any appropriate medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination of the foregoing. The computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof, and the programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0064] The embodiments of the present invention further provide a computer program product, including a computer program, and the computer program realizes the suspension control method according to the foregoing when being executed by a processor.

[0065] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the disclosure of the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and no limitation is imposed herein.

[0066] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A suspension control method, characterized in that: include: S1: building a basic TD3 architecture, and pre-training the basic TD3 architecture with the current state of the suspension as input and the active control action of the suspension as output; during the pre-training process, determining the conflict state; S2: Adding Actor-against network and Critic-against network to the basic TD3 architecture to obtain the adversarial-TD3 architecture; The Actor-against network receives the conflict state and the conflict state, and generates corresponding actions against , the action Combined with the action generated by the basic TD3 architecture, the final control action of the suspension is obtained; The critic-against network receives the action against and the conflict state, guiding the Actor-against network to generate new targeted actions ; S3: Using the conflict state to train the adversarial-TD3 architecture to obtain a control model for controlling the suspension.

2. The suspension control method according to claim 1, characterized in that: In step S1, the pre-training process of the basic TD3 architecture includes: S11: Initializing the basic TD3 architecture; S12: In the current state The Actor network in the basic TD3 architecture generates the first action with noise , and controlling the suspension to perform the first action , get the first state And the corresponding first reward ;Store experience data to the experience pool; S13: The first action and the current state Input the Critic1 network in the basic TD3 architecture and get the result , represents the network parameters of the Critic1 network; the first action and the current state Input the Critic2 network in the basic TD3 architecture and get the result , represents the network parameters of the Critic2 network; S14: Randomly sample a batch of samples from the experience pool , and according to the sample , using the Actor-T network in the base TD3 architecture to generate the first target action with noise ; Using the CriticT1 network and CriticT2 network in the basic TD3 architecture, the first target action is calculated by the following formula The corresponding first target value: ； in, represents the first target value, represents the discount factor, represents the CriticT1 network, represents the CriticT2 network, represents the network parameters of the CriticT1 network, represents the network parameters of the CriticT2 network; S15: Combined with the first target value obtained in step S14 The calculation result obtained in step S13 is used to calculate the loss function of training the Critic1 network and the Critic2 network by the following formula: ； ； in, represents the loss function of the Critic1 network, represents the loss function of the Critic2 network; S16: According to the loss function obtained in step S15, the basic TD3 architecture is updated with a delayed update strategy, and steps S12 to S16 are repeated with the updated basic TD3 architecture to complete the pre-training of the basic TD3 architecture.

3. The suspension control method according to claim 1 or 2, characterized in that: In step S2, the action The final control action is obtained by combining the action generated by the basic TD3 architecture with the following formula: ； in, represents the final control action, represents the action generated by the basic TD3 architecture, and K represents the weighted ratio.

4. The suspension control method according to claim 2, characterized in that: In step S3, the training process of the adversarial-TD3 architecture includes: S31: Initializing the adversarial-TD3 architecture; S32: In the current state Extract the conflict status , the Actor network is in the current state Generate the second action , the Actor-against network generates a conflict state for Targeted Action , get the action Corresponding rewards ; The Critic-against network receives the action against and the conflicting state , and get the output of the Critic-against network , Represents the parameters of the Critic-against network; S33: The targeted action in step S32 is With the second action Combined to get the final control action ; Control the suspension to perform the final control action , and get the second state and the second reward ;Store experience data into said pool of experience; S34: The second action and the current state Input the Critic1 network and get the result ; The second action and the current state Input the Critic2 network and get the result ; S35: Randomly sample a batch of samples from the experience pool , and according to the sample , using the Actor-T network to generate a second target action with noise ; Using the CriticT1 network and the CriticT2 network, the second target action is calculated by the following formula The corresponding second target value : ； S36: Combining the second target value obtained in step S35 and the calculation result obtained in step S33, a new loss function of the Critic1 network and the Critic2 network is calculated by the following formula: ； ； in, represents the new loss function of the Critic1 network corresponding to the second target value, represents a new loss function of the Critic2 network corresponding to the second target value, and MSE represents the mean square error loss function; The network parameters of the Critic1 network are adjusted using the gradient descent method. and the network parameters of the Critic2 network Update so that the two new loss functions reach the minimum value; S37: Use the targeted action in step S32 and the output of the critic-against network , and the second target value obtained in step S35 , use the following formula to update the parameters of the Actor-against network using the gradient descent method : ； S38: Adopting a delayed update strategy to update the parameters of the adversarial-TD3 architecture, and repeating steps S32 to S38 with the updated adversarial-TD3 architecture.

5. The suspension control method according to claim 4, characterized in that: The delayed update strategy in step S38 includes: During each training session: Using the network parameters of the Critic1 network , the network parameters of the CriticT1 network are given by To update: ； in, Indicates the soft update rate; Using the network parameters of the Critic2 network , the network parameters of the CriticT2 network are given by To update: ； After each training round policy_delay, the loss of the Actor network is calculated by the following formula : ； in, represents the parameters of the Actor network, Indicates the current state of the Actor network The action generated below; Using the parameters of the Actor network , the network parameters of the Actor-T network are given by To update: 。 6. The suspension control method according to claim 1, characterized in that: The suspension control method further includes: S4: The Actor network, Actor-T network and Actor-against network in the adversarial-TD3 architecture are periodically updated by the following formula: ； Where E represents the mean square error, Indicates the current status the generation action of the Actor network, the Actor-T network, or the Actor-against network, Indicates finding the minimum value for action a The gradient of Indicates that the Critic1 network in the adversarial-TD3 architecture is in the current state The result after entering action a and Critic2 network in its current state The result after entering action a The minimum value in , that is: ； Indicates the network parameters for the Actor network, the Actor-T network, or the Actor-against network Find the gradient.

7. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the suspension control method according to any one of claims 1 to 6 are implemented.

8. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the suspension control method according to any one of claims 1 to 6 when executing the computer program.

Images