Clutch position control method and device of AMT vehicle and electronic equipment
The parameters of the AMT system clutch position control model are dynamically adjusted through the Actor-Critic deep reinforcement learning algorithm network, solving the problems of low clutch position control accuracy and insufficient robustness in the prior art, and achieving higher control accuracy and adaptability.
Patent Information
- Application Number
- CN202411222528.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-29
AI Technical Summary
The clutch position control accuracy of existing AMT systems is low and has insufficient robustness, making it difficult to effectively deal with different working conditions and external interference.
The Actor-Critic deep reinforcement learning algorithm network is adopted to dynamically adjust the parameters based on the proportional-integral-differential control model, and the network parameters are iteratively updated, control strategies are optimized, and the accuracy and robustness of clutch position control are improved.
It significantly improves the accuracy and robustness of clutch position control in AMT system, can better deal with different working conditions and external interference, and achieves fine adjustment of clutch position.
Smart Images

Figure CN119042250A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vehicle clutch control, and in particular to a clutch position control method, device and electronic equipment for an AMT vehicle. Background Art
[0002] As a key component of the commercial vehicle transmission system, the Automated Mechanical Transmission (AMT) has been widely used in the industry due to its advantages of low cost, high efficiency and easy manufacturing. Based on the traditional gear transmission, AMT realizes the automation of speed change by adding an automatic shifting mechanism. Among them, clutch control is the core technology of the AMT system. The precise control of the clutch position not only affects the service life of the clutch, but also directly affects the comfort of AMT shifting.
[0003] At present, the clutch position control mainly adopts the conventional PID (proportional-integral-differential) algorithm. Although the algorithm is simple and practical, it is sensitive to the changes in the parameters of the controlled object and lacks the ability to predict the control effect. Therefore, in practical applications, the robustness and control accuracy of the algorithm need to be improved.
[0004] Therefore, how to effectively improve the clutch position control accuracy of the AMT system is a technical problem that needs to be solved urgently. Summary of the invention
[0005] In view of the above problems existing in the prior art, the present invention provides a clutch position control method, device and electronic equipment for an AMT vehicle, which are used to effectively improve the clutch position control accuracy of the AMT system.
[0006] The present invention provides a clutch position control method for an AMT vehicle, comprising the following steps.
[0007] Obtain a target position of a clutch of a target vehicle; wherein the target position of the clutch is obtained by learning the requested torque of the target vehicle; obtain a current actual position of the clutch of the target vehicle; obtain a control signal by using a clutch position control model according to a clutch error value between the target position of the clutch of the target vehicle and the current actual position; wherein the clutch position control model is constructed based on a proportional-integral-differential control model, and a proportional parameter, an integral parameter and a differential parameter of the clutch position control model are obtained by an Actor-Critic deep reinforcement learning algorithm network; adjust the position of the clutch of the target vehicle according to the control signal, and obtain the adjusted actual position of the clutch; update the parameters of the Actor-Critic deep reinforcement learning algorithm network according to the adjusted actual position of the clutch and the current actual position; use the adjusted actual position of the clutch as the new current actual position of the clutch, and repeatedly perform the clutch error value between the target position of the clutch of the target vehicle and the current actual position, and obtain the position control signal at the current moment and subsequent steps by using the clutch position control model, until the clutch error value between the target position of the clutch of the target vehicle and the current actual position of the clutch is less than a preset threshold.
[0008] According to a clutch position control method for an AMT vehicle provided by the present invention, the clutch position control model at least includes a term consisting of the product of the proportional parameter and the first-order backward difference of the clutch error value, a term consisting of the product of the differential parameter and the clutch error value, and a term consisting of the product of the integral parameter and the second-order backward difference of the clutch error value; the clutch error value between the target position and the current actual position of the clutch of the target vehicle is used to obtain a control signal using the clutch position control model, including: obtaining the proportional parameter, integral parameter and differential parameter of the clutch position control model at the current moment according to the clutch error value, the first-order backward difference of the clutch error value and the second-order backward difference of the clutch error value using the Actor-Critic deep reinforcement learning algorithm network; substituting the proportional parameter, integral parameter and differential parameter, the clutch error value, the first-order backward difference of the clutch error value and the second-order backward difference of the clutch error value into the clutch position control model to obtain the control signal.
[0009] According to a clutch position control method for an AMT vehicle provided by the present invention, the actor network and the critic network of the Actor-Critic deep reinforcement learning algorithm network share the same radial basis function network; the proportional parameter, integral parameter and differential parameter of the clutch position control model at the current moment are obtained by using the Actor-Critic deep reinforcement learning algorithm network according to the clutch error value between the target position of the clutch of the target vehicle and the current actual position, the first-order backward difference of the clutch error value and the second-order backward difference of the clutch error value, including: taking a vector including the clutch error value, the first-order backward difference of the clutch error value and the second-order backward difference of the clutch error value as the state at the current moment, inputting each element of the vector into the corresponding neurons of the input layer of the Actor-Critic deep reinforcement learning algorithm network, and obtaining the proportional parameter, integral parameter and differential parameter output by the Actor-Critic deep reinforcement learning algorithm network at the current moment, as well as the value function at the current moment.
[0010] According to a clutch position control method for an AMT vehicle provided by the present invention, the output layer of the radial basis function network includes four neural units: a first neural unit, a second neural unit, a third neural unit and a fourth neural unit; the elements of the vector are respectively input into corresponding neurons of the input layer of the Actor-Critic deep reinforcement learning algorithm network to obtain the proportional parameter, integral parameter and differential parameter of the Actor-Critic deep reinforcement learning algorithm network at the current moment, as well as the value function of the current moment, including: the elements of the vector are respectively input into corresponding neurons of the input layer of the Actor-Critic deep reinforcement learning algorithm network, the first neural unit outputs the proportional parameter, the second neural unit outputs the The integral parameter, the third neural unit outputs the differential parameter, and the fourth neural unit outputs the value function at the current moment; the updating of the parameters of the actor network according to the scale parameter, integral parameter and differential parameter at the current moment, and the value function at the current moment, includes: updating the weight value associated with the first neural unit, the weight value associated with the second neural unit, and the weight value associated with the third neural unit according to the scale parameter, integral parameter and differential parameter at the current moment, and the value function at the current moment; the updating of the parameters of the critic network according to the TD error at the current moment, and the value function at the current moment, includes: updating the weight value associated with the fourth neural unit according to the TD error at the current moment, and the value function at the current moment.
[0011] According to a clutch position control method for an AMT vehicle provided by the present invention, the updating of the parameters of the Actor-Critic deep reinforcement learning algorithm network according to the adjusted actual position of the clutch and the current actual position includes: determining the reward function of the Actor-Critic deep reinforcement learning algorithm network at the current moment according to the clutch error value at the current moment and the clutch error value at the previous moment, as well as the overshoot and convergence time of the clutch position control; obtaining the state of the Actor-Critic deep reinforcement learning algorithm network at the next moment according to the adjusted actual position of the clutch, and obtaining the proportional parameter, integral parameter and differential parameter at the next moment, and the value function at the next moment by using the Actor-Critic deep reinforcement learning algorithm network; obtaining the TD error at the current moment according to the value function at the next moment and the value function at the current moment, and the reward function at the current moment; updating the parameters of the critic network according to the TD error at the current moment and the value function at the current moment; updating the parameters of the actor network according to the proportional parameter, integral parameter and differential parameter at the current moment, and the value function at the current moment.
[0012] The present invention also provides a clutch position control device for an AMT vehicle, comprising the following modules: The first acquisition module is used to acquire the target position of the clutch of the target vehicle; wherein the target position of the clutch is obtained by learning the requested torque of the target vehicle; the second acquisition module is used to acquire the current actual position of the clutch of the target vehicle; the third acquisition module is used to obtain the control signal using the clutch position control model according to the clutch error value between the target position and the current actual position of the clutch of the target vehicle; wherein the clutch position control model is constructed based on the proportional-integral-differential control model, and the proportional parameter, integral parameter and differential parameter of the clutch position control model are obtained through the Actor-Critic deep reinforcement learning algorithm network; the adjustment module is used to adjust the control signal according to the control signal , adjust the position of the clutch of the target vehicle and obtain the actual position of the adjusted clutch; an update module, used to update the parameters of the Actor-Critic deep reinforcement learning algorithm network according to the adjusted actual position of the clutch and the current actual position; an iteration module, used to use the adjusted actual position of the clutch as the new current actual position of the clutch, repeatedly execute the clutch error value between the target position of the clutch of the target vehicle and the current actual position, and use the clutch position control model to obtain the position control signal at the current moment and subsequent steps, until the clutch error value between the target position of the clutch of the target vehicle and the current actual position of the clutch is less than a preset threshold.
[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a clutch position control method for an AMT vehicle as described above is implemented.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the clutch position control method of an AMT vehicle as described in any one of the above-mentioned methods is implemented.
[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the clutch position control method of the AMT vehicle as described above is implemented.
[0016] The present invention also provides a vehicle storing a set of instruction sets, wherein the instruction sets are executed by the vehicle to implement any of the above-mentioned clutch position control methods for the AMT vehicle.
[0017] The clutch position control method, device and electronic device for an AMT vehicle provided by the present invention utilize an Actor-Critic deep reinforcement learning algorithm network to dynamically adjust the parameters of a clutch position control model constructed based on a proportional-integral-differential control model according to the deviation between the actual position of the clutch and the target position, so that the clutch position control model has stronger adaptability and robustness, and achieves fine adjustment of the clutch position and improves control accuracy. By continuously iteratively updating the parameters of the Actor-Critic network and performing feedback learning based on the actual control effect, the Actor-Critic deep reinforcement learning algorithm network can continuously optimize its control strategy and effectively respond to different working conditions and external interference. This can effectively improve the clutch position control accuracy of the AMT system. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 It is a flow chart of the clutch position control method of the AMT vehicle provided by the present invention.
[0020] Figure 2 It is a flow chart of a method for obtaining a control signal by using a clutch position control model provided by the present invention.
[0021] Figure 3 It is a flowchart of a method for updating parameters of an Actor-Critic deep reinforcement learning algorithm network provided by the present invention.
[0022] Figure 4 It is a schematic diagram of the clutch position control process of the AMT vehicle provided by the present invention.
[0023] Figure 5 It is a schematic diagram of the structure of the radial basis function network provided by the present invention.
[0024] Figure 6 It is a schematic diagram of the clutch position principle of the AMT vehicle provided by the present invention.
[0025] Figure 7 It is a schematic structural diagram of the clutch position control device for an AMT vehicle provided by the present invention.
[0026] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] Combine the following Figure 1-Figure 6 The clutch position control method of the AMT vehicle of the present invention is described.
[0029] Figure 1 : is a flow chart of the clutch position control method of the AMT vehicle provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Obtain a target position of a clutch of a target vehicle.
[0030] The target vehicle is a vehicle whose clutch needs to be automatically controlled, for example, a vehicle at the starting stage or a vehicle that needs to change speed while moving.
[0031] In the specific implementation process, the target position of the clutch is obtained through self-learning of the clutch request torque. The clutch request torque refers to the engine torque that the clutch end needs to carry and transmit in order to overcome the resistance encountered during the vehicle starting or driving. The magnitude of the clutch request torque mainly depends on the resistance that needs to be overcome when the vehicle starts or drives, and its value can represent the engagement depth of the clutch. The speed of the change of the clutch request torque directly affects the speed of the change of the clutch target position.
[0032] The resistance that a vehicle needs to overcome when starting or driving It is the sum of slope resistance, rolling resistance and air resistance. The calculation formula is as follows.
[0033] (1)
[0034] in: , , They are slope resistance, rolling resistance and air resistance.
[0035] The clutch engagement process is mainly divided into the torque engagement stage and the torque-free engagement stage. Among them, the torque engagement stage is the key stage that affects the vehicle's shift comfort, which is further subdivided into the first stage of the slip zone and the second stage of the synchronization zone. In the slip zone, the change rate of the requested torque needs to be strictly controlled to ensure the smoothness and comfort of the shift process; while in the torque-free engagement stage and the torque engagement stage of the synchronization zone, the clutch can be engaged quickly to achieve efficient shifting operations.
[0036] In the specific implementation process, the clutch request torque can be calculated using the collected accelerator pedal opening, the slope of the target vehicle, the equivalent vehicle weight of the target vehicle, the clutch state and other related parameters. The target position of the clutch is determined by the clutch request torque using the following formula.
[0037] (2)
[0038] in, q is the target position of the clutch, K is the torque position conversion coefficient, which changes with self-learning. Request torque for the clutch.
[0039] In some embodiments, the torque position conversion coefficient K is determined by a least square method, which is used to minimize the square of the error between the actual target position of the clutch and the clutch position calculated based on the clutch request torque. The formula is as follows.
[0040] (3)
[0041] in: is the actual target position value of the clutch, is the clutch position calculated based on the clutch requested torque.
[0042] Step 102: Obtain the current actual position of the clutch of the target vehicle.
[0043] In a specific implementation process, the current actual position of the clutch of the target vehicle may be acquired through a position sensor installed on the clutch actuator.
[0044] Step 103: According to the clutch error value between the target position and the current actual position of the clutch of the target vehicle, a control signal is obtained using a clutch position control model.
[0045] The clutch position control model is constructed based on the proportional-integral-derivative control (PID) model. The proportional (P) parameter, integral (I) parameter and differential (D) parameter of the clutch position control model are obtained through the Actor-Critic deep reinforcement learning algorithm network.
[0046] A PID control system includes: input signal, feedback signal, clutch error value, PID control model, control signal and execution object.
[0047] The input signal is the target position of the clutch. The feedback signal is the current actual position of the clutch obtained by feedback and conversion through the position sensor installed on the clutch actuator. The clutch error value is the difference between the target position of the clutch and the current actual position: (4)
[0048] in, is the target position of the clutch, is the current actual position of the clutch.
[0049] The PID control model includes the P parameter control model, the I parameter control model and the D parameter control model, and the final output signal is the sum of the three. It can be expressed as follows.
[0050] (5)
[0051] in, is the actual position of the clutch, is the proportional coefficient (i.e., proportional parameter), is the integration time constant (i.e., integration parameter), is the differential time constant (i.e., differential parameter). The PID control model can accurately control the voltage duty cycle of the clutch motor by selecting appropriate proportional, integral, and differential coefficients, thereby achieving effective control of the clutch position by the solenoid valve.
[0052] In the specific implementation process, since the clutch position control is a sampling control for the computer, the PID control algorithm needs to be discretized. This involves using a series of sampling time points kT to represent the continuous time t, while using matrix numerical integration to approximate the integral term, and using the first-order backward difference to approximate the differential term. This processing method enables the PID control algorithm to be effectively applied in a discretized sampling control system. The formula is as follows.
[0053] (6)
[0054] Right now
[0055] (7)
[0056] in, , , T is the sampling period, k is the sampling number, k = 0, 1, 2, ..., is the clutch error value at the k-1th moment, is the clutch error value at the kth moment.
[0057] The final design equations for the clutch position control model are as follows: (8)
[0058] in, The adjustment amount of the clutch position at time k can be converted into a control signal; is the first-order backward difference of the clutch error value; is the second-order backward difference of the clutch error value.
[0059] like Figure 6 As shown, formula (8) is equivalent to the error signal (clutch error value) , through the state converter Converted into a discrete PID control model (clutch position control model), the state converter is written in vector form as:
[0060] in: , .
[0061] The Actor-Critic deep reinforcement learning algorithm network is a reinforcement learning algorithm that contains two neural networks: the actor network (Actor network) and the critic network (Critic network). The actor network represents the policy neural network that performs actions. Its output is related to the current state and past action sequences. It is used to learn strategies and select actions based on the current state. The critic network is responsible for evaluating the quality of the environment state after the action is performed. Its input is the current environment state after the action is performed, and its output is the evaluation value corresponding to the state. The actor network dynamically adjusts the probability of outputting a certain action in a certain state based on the evaluation of the critic network, while the critic network determines whether rewards or penalties should be given by evaluating the current state. In this adaptive algorithm, the actor network and the critic network learn collaboratively, where the actor network learns good strategies and action selections, while the critic network evaluates strategies and provides reinforcement signals.
[0062] In the specific implementation process, the proportional parameters, differential parameters and integral parameters of the clutch position control model can be determined according to the clutch error value using the Actor-Critic deep reinforcement learning algorithm network; the control signal can be obtained by using the clutch position control model shown in formula (8) according to the clutch error value and the determined proportional parameters, differential parameters and integral parameters.
[0063] For a detailed description of the clutch position control model to obtain the control signal, see Figure 2 The relevant content in will not be repeated here.
[0064] Step 104: Adjust the position of the clutch of the target vehicle according to the control signal, and obtain the actual position of the adjusted clutch.
[0065] During the specific implementation process, the generated control signal is transmitted to the clutch actuator to drive the solenoid valve of the clutch motor; the solenoid valve adjusts its working state according to the duty cycle of the control signal, thereby controlling the magnitude and direction of the current entering the clutch drive motor; the clutch drive motor generates corresponding torque and speed according to the change of current passing through the solenoid valve, and drives the clutch actuator (such as a clutch release bearing or a diaphragm spring) to adjust the clutch engagement position.
[0066] Step 105: Update the parameters of the Actor-Critic deep reinforcement learning algorithm network according to the adjusted actual position of the clutch and the current actual position.
[0067] For a detailed description of updating the parameters of the Actor-Critic deep reinforcement learning algorithm network based on the adjusted actual position of the clutch and the current actual position, see Figure 3 The related operations in will not be repeated here.
[0068] In a specific implementation process, the adjusted actual position of the clutch is used as the new current actual position of the clutch, and steps 103 to 105 are repeatedly performed until the clutch error value between the target position of the clutch of the target vehicle and the current actual position of the clutch is less than a preset threshold value. The preset threshold value is a smaller value set based on experience or experimental results.
[0069] Figure 2 FIG. 1 is a flow chart of a method for obtaining a control signal by using a clutch position control model provided by the present invention, such as Figure 2 As shown, the method includes the following: Step 201: According to the clutch error value, the first-order backward difference of the clutch error value and the second-order backward difference of the clutch error value, the proportional parameter, the integral parameter and the differential parameter of the clutch position control model at the current moment are obtained by using the Actor-Critic deep reinforcement learning algorithm network.
[0070] The clutch position control model shown in formula (8) includes a term consisting of the product of a proportional parameter and a first-order backward difference of a clutch error value, a term consisting of a differential parameter and a product of a clutch error value, and a term consisting of an integral parameter and a second-order backward difference of a clutch error value.
[0071] In the specific implementation process, a vector containing the clutch error value, the first-order backward difference of the clutch error value, and the second-order backward difference of the clutch error value can be used as the state at the current moment, and each element of the vector can be input into the corresponding neurons of the input layer of the Actor-Critic deep reinforcement learning algorithm network to obtain the proportional parameters, integral parameters and differential parameters of the Actor-Critic deep reinforcement learning algorithm network output at the current moment, as well as the value function at the current moment.
[0072] In some embodiments, the actor network and the critic network of the Actor-Critic deep reinforcement learning algorithm network share the same radial basis function network.
[0073] The network structure of the radial basis function network is as follows Figure 5 As shown in the figure, it can simultaneously realize the learning of the strategy function of the actor network and the value function of the critic network, and has the advantages of strong global approximation ability, simple structure, and fast and easy training method.
[0074] The actor network and the critic network share the input layer and hidden layer resources of the radial basis function network, which not only reduces the storage space requirements of the learning system, but also avoids repeated calculations of the hidden layer node outputs, thereby improving the learning efficiency of the system.
[0075] In a specific implementation process, the clutch error value, the first-order backward difference of the clutch error value, and the second-order backward difference of the clutch error value can be input as the current state into the corresponding neurons of the input layer of the radial basis function network.
[0076] The basis function of the second layer (hidden layer) nodes of the radial basis function network uses a Gaussian kernel function. j The output of a hidden node is: (9)
[0077] in, For the The center vector of the node, For the The width parameter of each node, is the number of hidden nodes.
[0078] The third layer (output layer) of the radial basis function network is composed of two parts: the actor network and the critic network. The actor network outputs the control parameters of the clutch position control model : Scale parameter , integration parameters And the differential parameters . Critic network output value function The formula is as follows.
[0079] (10)
[0080] (11)
[0081] in, is the weight between the jth neuron in the hidden layer and the kth output node in the output layer.
[0082] In some embodiments, the output layer of the radial basis function network includes four neural units: a first neural unit, a second neural unit, a third neural unit and a fourth neural unit; each element of the vector can be input into the corresponding neurons of the input layer of the Actor-Critic deep reinforcement learning algorithm network, the first neural unit outputs a proportional parameter, the second neural unit outputs an integral parameter, the third neural unit outputs a differential parameter, and the fourth neural unit outputs the value function at the current moment.
[0083] In some embodiments, in order to obtain more precise control parameters , which can be used to control the output parameters of the actor network Make the corrections as shown below and set the corrected control parameters , as the proportional parameter, integral parameter and differential parameter of the clutch position control model at the current moment.
[0084] = + (12)
[0085] in: is a Gaussian interference that follows a normal distribution, with an expected value of 0 and a variance of related to .
[0086] Step 202: Substitute the proportional parameter, the integral parameter, the differential parameter, the clutch error value, the first-order backward difference of the clutch error value, and the second-order backward difference of the clutch error value into the clutch position control model to obtain a control signal.
[0087] In the specific implementation process, the proportional parameter, the integral parameter and the differential parameter, the clutch error value, the first-order backward difference of the clutch error value, and the second-order backward difference of the clutch error value are substituted into the clutch position control model shown in formula (8). The clutch position control model outputs the adjustment amount of the clutch position, and the adjustment amount of the clutch position is converted into a control signal.
[0088] Figure 3 It is a flow chart of the method for updating the parameters of the Actor-Critic deep reinforcement learning algorithm network provided by the present invention, such as Figure 3 As shown, the method includes the following: Step 301: Determine the reward function of the Actor-Critic deep reinforcement learning algorithm network at the current moment according to the clutch error value at the previous moment, the overshoot and convergence time of the clutch position control.
[0089] When designing the reward function of the Actor-Critic deep reinforcement learning algorithm network, we focus on two aspects: one is the impact of the clutch position error and its rate of change on the performance of the control system to ensure the accuracy and stability of the clutch position control; the other is the overshoot and convergence time of the clutch position control to ensure the response speed of the clutch control system. This design aims to make the clutch position control system have good control accuracy, stability and fast response characteristics. The reward function is as follows: (13)
[0090] in, and are the enhanced signal coefficients of the error change rate, and are the reinforcement signals of error and error change rate, respectively. is the overshoot, is the expected overshoot, is the adjustment value of the overshoot, and its value satisfies , is the convergence time, is the expected convergence time, is the adjustment value of the convergence time, and its value satisfies .
[0091] error and error rate of change The definition is as follows: (14)
[0092] (15)
[0093] in, is the allowable error band.
[0094] like Figure 6 As shown, the reward function at the current moment is based on the control signal at the current moment (the adjustment amount of the clutch position at the current moment The clutch error value after the clutch actuator is converted) , and the control signal at the previous moment (the adjustment amount of the clutch position at the previous moment The clutch error value after the clutch actuator is converted Therefore, the reward function can be used to realize instant feedback on the control effect of the control signal obtained by the clutch position control model, so that the Actor-Critic deep reinforcement learning algorithm network can quickly respond to changes in the clutch position based on the reward function and adjust the action generation strategy in time, thereby improving the control accuracy and response speed of the clutch position control model.
[0095] Step 302: According to the actual position of the adjusted clutch, the state of the Actor-Critic deep reinforcement learning algorithm network at the next moment is obtained, and the proportional parameter, integral parameter and differential parameter at the next moment, as well as the value function at the next moment are obtained by using the Actor-Critic deep reinforcement learning algorithm network.
[0096] For the specific implementation of this step, please refer to the relevant content in step 201 to step 202, which will not be repeated here.
[0097] Step 303: Obtain the TD error at the current moment according to the value function at the next moment, the value function at the current moment, and the reward function at the current moment.
[0098] TD error (Temporal Difference Error) reflects the difference between the immediate reward and the expected reward for taking an action in a given state. If it is positive, it means that the actual return is higher than the expected return, and the action should be increased. The probability of taking If it is negative, it means that the actual return is lower than the expected return, and the probability of taking this action should be reduced. The calculation formula is as follows.
[0099] (16)
[0100] in, is the reward function at the current moment obtained in step 301; is the value function at the next moment; is the value function at the current moment; is the discount factor.
[0101] In the subsequent steps, the critic network uses the TD error signal to update the state value function, making the evaluation results of the critic network more accurate; the actor network uses the policy gradient method to update the policy parameters according to the value function of the critic network to reduce the error signal , making it more likely that future strategies will produce higher expected cumulative rewards.
[0102] Step 304: Update the parameters of the critic network according to the TD error at the current moment and the value function at the current moment.
[0103] In the specific implementation process, the parameters of the critic network are updated by the following formula: (17)
[0104] in, is the critic network learning rate; is the weight parameter of the critic network at time t; is the weight parameter of the critic network at time t+1; w represents the parameter of the critic network, which needs to be initialized at the beginning.
[0105] In some embodiments, the output layer of the radial basis function network includes four neural units: a first neural unit, a second neural unit, a third neural unit, and a fourth neural unit; the weight value associated with the fourth neural unit can be updated according to the TD error at the current moment and the value function at the current moment using formula (17).
[0106] Step 305: Update the parameters of the actor network according to the current scale parameter, integral parameter and differential parameter, and the value function at the current moment.
[0107] In the specific implementation process, the parameters of the actor network are updated by the policy gradient method shown below: (18)
[0108] in, Represents the parameters of the actor network, which needs to be initialized at the beginning; is the actor network learning rate; is the state of the environment, i.e. Status The following actions are taken, is the action strategy taken, is the action value function, are the parameters of the critic network.
[0109] In some embodiments, the output layer of the radial basis function network includes four neural units: a first neural unit, a second neural unit, a third neural unit, and a fourth neural unit; Formula (18) can be used to calculate the proportional parameter, integral parameter, and differential parameter (as value), and the value function at the current moment, to update the weight value associated with the first neural unit, the weight value associated with the second neural unit, and the weight value associated with the third neural unit.
[0110] In the specific implementation, after updating the parameters of the actor network and the critic network, the center vector and width parameters of the radial basis function network are calculated.
[0111] In the embodiments provided by the present invention, Figure 4 As shown in the figure, by comprehensively considering the change of clutch error value, overshoot and convergence time, the reward function can more comprehensively evaluate the effect of the current control strategy, thereby guiding the algorithm to make more refined adjustments and improve the accuracy of clutch position control; the adjusted actual position of the clutch is used to update the state of the Actor-Critic network, and the proportional parameter, integral parameter, differential parameter and value function of the next moment are predicted accordingly, which enables the control system to dynamically adapt to the change of clutch position and achieve more stable control; by calculating the TD error at the current moment and updating the parameters of the critic network in combination with the value function at the current moment, this mechanism provides a clear learning direction for the algorithm, helps to speed up learning and improve control performance; at the same time, the parameters of the actor network and the critic network are updated, realizing the coordinated optimization of the control strategy (actor network) and the value evaluation (critic network), further improving the overall effect of clutch position control.
[0112] The clutch position control device of the AMT vehicle provided by the present invention is described below. The clutch position control device of the AMT vehicle described below and the clutch position control method of the AMT vehicle described above can be referred to each other.
[0113] Figure 7 Schematic diagram of the structure of the clutch position control device of the AMT vehicle provided by the present invention. Figure 7 As shown, the device 700 includes the following modules.
[0114] A first acquisition module 710 is used to acquire a target position of a clutch of a target vehicle; wherein the target position of the clutch is obtained by learning a requested torque of the target vehicle; A second acquisition module 720, used to acquire the current actual position of the clutch of the target vehicle; A third acquisition module 730 is used to obtain a control signal using a clutch position control model according to a clutch error value between the target position and the current actual position of the clutch of the target vehicle; wherein the clutch position control model is constructed based on a proportional-integral-differential control model, and the proportional parameter, integral parameter and differential parameter of the clutch position control model are obtained through an Actor-Critic deep reinforcement learning algorithm network; An adjustment module 740, configured to adjust the position of the clutch of the target vehicle according to the control signal, and obtain the actual position of the clutch after the adjustment; An updating module 750, configured to update the parameters of the Actor-Critic deep reinforcement learning algorithm network according to the adjusted actual position of the clutch and the current actual position; The iterative module 760 is used to use the adjusted actual position of the clutch as the new current actual position of the clutch, repeatedly execute the clutch error value between the target position of the clutch of the target vehicle and the current actual position, and use the clutch position control model to obtain the position control signal at the current moment and subsequent steps until the clutch error value between the target position of the clutch of the target vehicle and the current actual position of the clutch is less than a preset threshold.
[0115] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830 and a communications bus 840, wherein the processor 810, the communications interface 820 and the memory 830 communicate with each other via the communications bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the clutch position control method of the AMT vehicle, the method comprising: obtaining the target position of the clutch of the target vehicle; wherein the target position of the clutch is obtained by learning the requested torque of the target vehicle; obtaining the current actual position of the clutch of the target vehicle; according to the clutch error value between the target position and the current actual position of the clutch of the target vehicle, using the clutch position control model, obtaining a control signal; wherein the clutch position control model is constructed based on the proportional-integral-differential control model, and the proportional parameter, integral parameter and differential parameter of the clutch position control model are obtained by the Actor-Critic deep reinforcement learning algorithm. network; according to the control signal, the position of the clutch of the target vehicle is adjusted, and the actual position of the adjusted clutch is obtained; according to the adjusted actual position of the clutch and the current actual position, the parameters of the Actor-Critic deep reinforcement learning algorithm network are updated; the actual position of the adjusted clutch is used as the new current actual position of the clutch, and the clutch error value between the target position of the clutch of the target vehicle and the current actual position is repeatedly executed, and the position control signal at the current moment and subsequent steps are obtained by using the clutch position control model, until the clutch error value between the target position of the clutch of the target vehicle and the current actual position of the clutch is less than a preset threshold.
[0116] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0117] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the clutch position control method of the AMT vehicle provided by the above-mentioned methods, the method including: obtaining the target position of the clutch of the target vehicle; wherein the target position of the clutch is obtained by learning the requested torque of the target vehicle; obtaining the current actual position of the clutch of the target vehicle; according to the clutch error value between the target position and the current actual position of the clutch of the target vehicle, using a clutch position control model, obtaining a control signal; wherein the clutch position control model is constructed based on a proportional-integral-differential control model, and the proportional parameter of the clutch position control model is , integral parameters and differential parameters are obtained through the Actor-Critic deep reinforcement learning algorithm network; according to the control signal, the position of the clutch of the target vehicle is adjusted, and the actual position of the adjusted clutch is obtained; according to the adjusted actual position of the clutch and the current actual position, the parameters of the Actor-Critic deep reinforcement learning algorithm network are updated; the actual position of the adjusted clutch is used as the new current actual position of the clutch, and the clutch error value between the target position of the clutch of the target vehicle and the current actual position is repeatedly executed, and the position control signal at the current moment and subsequent steps are obtained by using the clutch position control model, until the clutch error value between the target position of the clutch of the target vehicle and the current actual position of the clutch is less than a preset threshold.
[0118] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the clutch position control method of the AMT vehicle provided by the above methods, the method comprising: obtaining a target position of the clutch of the target vehicle; wherein the target position of the clutch is obtained by learning the requested torque of the target vehicle; obtaining the current actual position of the clutch of the target vehicle; and obtaining a control signal using a clutch position control model according to a clutch error value between the target position and the current actual position of the clutch of the target vehicle; wherein the clutch position control model is constructed based on a proportional-integral-differential control model, and the proportional parameter, integral parameter and differential parameter of the clutch position control model are obtained by Ac The method is to obtain the clutch position of the target vehicle by an Actor-Critic deep reinforcement learning algorithm network; adjust the position of the clutch of the target vehicle according to the control signal, and obtain the actual position of the adjusted clutch; update the parameters of the Actor-Critic deep reinforcement learning algorithm network according to the adjusted actual position of the clutch and the current actual position; use the adjusted actual position of the clutch as the new current actual position of the clutch, repeatedly execute the clutch error value between the target position of the clutch of the target vehicle and the current actual position, and use the clutch position control model to obtain the position control signal at the current moment and subsequent steps, until the clutch error value between the target position of the clutch of the target vehicle and the current actual position of the clutch is less than a preset threshold.
[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A clutch position control method for an AMT vehicle, characterized in that: include: Acquiring a target position of a clutch of a target vehicle; wherein the target position of the clutch is obtained by learning a requested torque of the target vehicle; Obtaining a current actual position of a clutch of the target vehicle; According to the clutch error value between the target position and the current actual position of the clutch of the target vehicle, a control signal is obtained by using a clutch position control model; wherein the clutch position control model is constructed based on a proportional-integral-differential control model, and the proportional parameter, integral parameter and differential parameter of the clutch position control model are obtained by an Actor-Critic deep reinforcement learning algorithm network; According to the control signal, adjusting the position of the clutch of the target vehicle and obtaining the actual position of the adjusted clutch; updating the parameters of the Actor-Critic deep reinforcement learning algorithm network according to the adjusted actual position of the clutch and the current actual position; The adjusted actual position of the clutch is used as the new current actual position of the clutch, and the clutch error value between the target position of the clutch of the target vehicle and the current actual position is repeatedly executed. The position control signal at the current moment and subsequent steps are obtained using the clutch position control model, until the clutch error value between the target position of the clutch of the target vehicle and the current actual position of the clutch is less than a preset threshold.
2. The clutch position control method of an AMT vehicle according to claim 1, characterized in that: The clutch position control model includes at least a term consisting of a product of the proportional parameter and a first-order backward difference of the clutch error value, a term consisting of a product of the differential parameter and the clutch error value, and a term consisting of a product of the integral parameter and a second-order backward difference of the clutch error value; The step of obtaining a control signal by using a clutch position control model according to a clutch error value between the target position and the current actual position of the clutch of the target vehicle comprises: According to the clutch error value, the first-order backward difference of the clutch error value, and the second-order backward difference of the clutch error value, using the Actor-Critic deep reinforcement learning algorithm network, obtaining the proportional parameter, integral parameter, and differential parameter of the clutch position control model at the current moment; Substitute the proportional parameter, integral parameter and differential parameter, the clutch error value, the first-order backward difference of the clutch error value and the second-order backward difference of the clutch error value into the clutch position control model to obtain the control signal.
3. The clutch position control method of an AMT vehicle according to claim 2, characterized in that: The actor network and the critic network of the Actor-Critic deep reinforcement learning algorithm network share the same radial basis function network; The clutch error value between the target position of the clutch of the target vehicle and the current actual position, the first-order backward difference of the clutch error value, and the second-order backward difference of the clutch error value are used to obtain the proportional parameter, integral parameter, and differential parameter of the clutch position control model at the current moment by using the Actor-Critic deep reinforcement learning algorithm network, including: A vector including the clutch error value, the first-order backward difference of the clutch error value, and the second-order backward difference of the clutch error value is used as the state at the current moment, and each element of the vector is input into the corresponding neurons of the input layer of the Actor-Critic deep reinforcement learning algorithm network to obtain the proportional parameter, integral parameter, differential parameter and value function at the current moment output by the Actor-Critic deep reinforcement learning algorithm network.
4. The clutch position control method of an AMT vehicle according to claim 3, characterized in that: The updating of the parameters of the Actor-Critic deep reinforcement learning algorithm network according to the adjusted actual position of the clutch and the current actual position includes: Determine the reward function of the Actor-Critic deep reinforcement learning algorithm network at the current moment according to the clutch error value at the previous moment, and the overshoot and convergence time of the clutch position control; According to the actual position of the adjusted clutch, the state of the Actor-Critic deep reinforcement learning algorithm network at the next moment is obtained, and the proportional parameter, integral parameter and differential parameter at the next moment, as well as the value function at the next moment are obtained by using the Actor-Critic deep reinforcement learning algorithm network; Obtaining the TD error at the current moment according to the value function at the next moment, the value function at the current moment, and the reward function at the current moment; Update the parameters of the critic network according to the TD error at the current moment and the value function at the current moment; The parameters of the actor network are updated according to the scale parameter, the integral parameter and the differential parameter at the current moment and the value function at the current moment.
5. The clutch position control method of an AMT vehicle according to claim 4, characterized in that: The output layer of the radial basis function network includes four neural units: a first neural unit, a second neural unit, a third neural unit and a fourth neural unit; The step of inputting each element of the vector into the corresponding neuron of the input layer of the Actor-Critic deep reinforcement learning algorithm network to obtain the current proportional parameter, integral parameter and differential parameter output by the Actor-Critic deep reinforcement learning algorithm network, as well as the value function at the current moment, includes: Input each element of the vector into the corresponding neuron of the input layer of the Actor-Critic deep reinforcement learning algorithm network, the first neural unit outputs the proportional parameter, the second neural unit outputs the integral parameter, the third neural unit outputs the differential parameter, and the fourth neural unit outputs the value function at the current moment; The updating of the parameters of the actor network according to the current scale parameter, the integral parameter and the differential parameter, and the value function at the current moment, comprises: According to the proportional parameter, integral parameter and differential parameter at the current moment and the value function at the current moment, updating the weight value associated with the first neural unit, the weight value associated with the second neural unit and the weight value associated with the third neural unit; The updating of the parameters of the critic network according to the TD error at the current moment and the value function at the current moment includes: The weight value associated with the fourth neural unit is updated according to the TD error at the current moment and the value function at the current moment.
6. A clutch position control device for an AMT vehicle, characterized in that: include: A first acquisition module is used to acquire a target position of a clutch of a target vehicle; wherein the target position of the clutch is obtained by learning a requested torque of the target vehicle; A second acquisition module, used to acquire a current actual position of the clutch of the target vehicle; A third acquisition module is used to obtain a control signal using a clutch position control model according to a clutch error value between the target position and the current actual position of the clutch of the target vehicle; wherein the clutch position control model is constructed based on a proportional-integral-differential control model, and a proportional parameter, an integral parameter and a differential parameter of the clutch position control model are obtained through an Actor-Critic deep reinforcement learning algorithm network; An adjustment module, used to adjust the position of the clutch of the target vehicle according to the control signal, and obtain the actual position of the adjusted clutch; An updating module, configured to update the parameters of the Actor-Critic deep reinforcement learning algorithm network according to the adjusted actual position of the clutch and the current actual position; An iterative module is used to use the adjusted actual position of the clutch as the new current actual position of the clutch, repeatedly execute the clutch error value between the target position of the clutch of the target vehicle and the current actual position, and use the clutch position control model to obtain the position control signal at the current moment and subsequent steps until the clutch error value between the target position of the clutch of the target vehicle and the current actual position of the clutch is less than a preset threshold.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the clutch position control method of the AMT vehicle as described in any one of claims 1 to 5 is implemented.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the clutch position control method of the AMT vehicle as claimed in any one of claims 1 to 5 is implemented.
9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the clutch position control method of the AMT vehicle as claimed in any one of claims 1 to 5 is implemented.
10. A vehicle, characterized in that: A set of instruction sets is stored, and the instruction sets are executed by the vehicle to implement the clutch position control method described in any one of claims 1-5.
Citation Information
Cited By
Control method and control system of motorcycle clutch
CN120868152A
