Trajectory tracking model training method and device, equipment and medium

Abstract

The invention discloses a trajectory tracking model training method and device, equipment and a medium, and the method comprises the steps: obtaining a standard motion parameter and a standard state parameter in a process that a target user drives a target distributed electric vehicle to run on a target road, and forming a pre-training data set; performing initialization training on the original depth deterministic strategy gradient model according to a pre-training data set to obtain an initial depth deterministic strategy gradient model; updating the experience pool, and training the initial depth deterministic strategy gradient model until the initial depth deterministic strategy gradient model meets a preset trajectory tracking requirement; and taking the initial depth deterministic strategy gradient model meeting a trajectory tracking preset requirement as a target trajectory tracking model suitable for the target distributed electric vehicle. According to the invention, the target trajectory tracking model suitable for the target distributed electric vehicle is obtained, so that trajectory tracking of the target distributed electric vehicle is realized.

Description

Technical Field

[0001] The present invention relates to the field of automobile automatic driving technology, and in particular to a trajectory tracking model training method, device, equipment and medium. Background Art

[0002] With the continuous improvement of social needs, autonomous driving has emerged. Autonomous driving technology relies on trajectory tracking control methods, while traditional trajectory tracking control methods usually determine the reasonable steering wheel angle and vehicle speed based on the vehicle state, thereby achieving vehicle trajectory tracking and autonomous driving.

[0003] However, the trajectory tracking model provided in the related art cannot be used for trajectory tracking of distributed electric vehicles. Therefore, how to train a trajectory tracking model suitable for distributed electric vehicles is a problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present application solve the technical problem that the trajectory tracking model provided in the prior art cannot be used for trajectory tracking of distributed electric vehicles by providing a trajectory tracking model training method, device, equipment and medium, and achieve the technical effect of training a trajectory tracking model suitable for distributed electric vehicles.

[0005] In a first aspect, the present application provides a trajectory tracking model training method, the method comprising:

[0006] Obtain standard action parameters and standard state parameters of a target user driving a target distributed electric vehicle on a target road, and form a pre-training data set; the target distributed electric vehicle refers to an electric vehicle with front-wheel steering, four-wheel independent drive, and distributed stability torque algorithm;

[0007] Initialize and train the original deep deterministic policy gradient model according to the pre-trained dataset to obtain an initial deep deterministic policy gradient model;

[0008] According to the experience pool containing multiple state data training samples, the initial deep deterministic policy gradient model and the preset target vehicle environment model, the experience pool is updated, and the initial deep deterministic policy gradient model is trained until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking;

[0009] The initial deep deterministic policy gradient model that meets the preset requirements of trajectory tracking is used as the target trajectory tracking model suitable for target distributed electric vehicles.

[0010] In a second aspect, the present application provides a trajectory tracking model training device, the device comprising:

[0011] A pre-training set acquisition module is used to acquire standard action parameters and standard state parameters of a target user driving a target distributed electric vehicle on a target road, and to form a pre-training data set; the target distributed electric vehicle refers to an electric vehicle with front-wheel steering, four-wheel independent drive, and distributed stability torque algorithm;

[0012] The model initial training module is used to initialize the training of the original deep deterministic policy gradient model according to the pre-training data set to obtain the initial deep deterministic policy gradient model;

[0013] A model update training module is used to update the experience pool according to the experience pool containing multiple state data training samples, the initial deep deterministic policy gradient model and the preset target vehicle environment model, and to train the initial deep deterministic policy gradient model until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking;

[0014] The target model determination module is used to use an initial deep deterministic policy gradient model that meets preset requirements of trajectory tracking as a target trajectory tracking model suitable for a target distributed electric vehicle.

[0015] In a third aspect, the present application provides an electronic device, including:

[0016] processor;

[0017] a memory for storing processor-executable instructions;

[0018] Wherein, the processor is configured to execute to implement a trajectory tracking model training method provided in the first aspect.

[0019] In a fourth aspect, the present application provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to implement a trajectory tracking model training method as provided in the first aspect.

[0020] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0021] This embodiment obtains standard action parameters and standard state parameters in the process of a target user driving a target distributed electric vehicle on a target road, and forms a pre-trained data set; initializes and trains the original deep deterministic policy gradient model according to the pre-trained data set to obtain an initial deep deterministic policy gradient model; updates the experience pool according to an experience pool containing multiple state data training samples, an initial deep deterministic policy gradient model, and a preset target vehicle environment model, and trains the initial deep deterministic policy gradient model until the initial deep deterministic policy gradient model meets the preset requirements for trajectory tracking; and uses the initial deep deterministic policy gradient model that meets the preset requirements for trajectory tracking as a target trajectory tracking model suitable for the target distributed electric vehicle. It can be seen that this embodiment trains a trajectory tracking model for an electric vehicle with front-wheel steering, four-wheel independent drive, and a distributed stability torque algorithm, and obtains a target trajectory tracking model suitable for the target distributed electric vehicle, so as to achieve trajectory tracking of the target distributed electric vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A schematic diagram of a flow chart of a trajectory tracking model training method provided in this embodiment;

[0024] Figure 2 A schematic diagram of vehicle trajectory tracking provided in this embodiment;

[0025] Figure 3 A schematic diagram of the conversion between the absolute coordinate system and the relative coordinate system provided in this embodiment;

[0026] Figure 4 A schematic diagram of the architecture of the deep deterministic policy gradient model provided for this embodiment;

[0027] Figure 5 A schematic diagram of the structure of a trajectory tracking model training device provided in this embodiment;

[0028] Figure 6 A schematic diagram of the structure of an electronic device provided in this embodiment. DETAILED DESCRIPTION

[0029] The embodiment of the present application solves the technical problem that the trajectory tracking model provided in the prior art cannot be used for trajectory tracking of distributed electric vehicles by providing a trajectory tracking model training method.

[0030] The technical solution of the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0031] This embodiment obtains standard action parameters and standard state parameters in the process of a target user driving a target distributed electric vehicle on a target road, and forms a pre-trained data set; initializes and trains the original deep deterministic policy gradient model according to the pre-trained data set to obtain an initial deep deterministic policy gradient model; updates the experience pool according to an experience pool containing multiple state data training samples, an initial deep deterministic policy gradient model, and a preset target vehicle environment model, and trains the initial deep deterministic policy gradient model until the initial deep deterministic policy gradient model meets the preset requirements for trajectory tracking; and uses the initial deep deterministic policy gradient model that meets the preset requirements for trajectory tracking as a target trajectory tracking model suitable for the target distributed electric vehicle. It can be seen that this embodiment trains a trajectory tracking model for an electric vehicle with front-wheel steering, four-wheel independent drive, and a distributed stability torque algorithm, and obtains a target trajectory tracking model suitable for the target distributed electric vehicle, so as to achieve trajectory tracking of the target distributed electric vehicle.

[0032] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0033] First of all, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0034] This embodiment provides Figure 1 A trajectory tracking model training method is shown, and the method includes steps S11 to S14.

[0035] Step S11, obtaining standard action parameters and standard state parameters of a target user driving a target distributed electric vehicle on a target road, and forming a pre-training data set; the target distributed electric vehicle refers to an electric vehicle with front-wheel steering, four-wheel independent drive and distributed stability torque algorithm;

[0036] Step S12, initializing the training of the original deep deterministic policy gradient model according to the pre-trained data set to obtain an initial deep deterministic policy gradient model;

[0037] Step S13, updating the experience pool and training the initial deep deterministic policy gradient model according to the experience pool containing multiple state data training samples, the initial deep deterministic policy gradient model and the preset target vehicle environment model, until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking;

[0038] Step S14, using the initial deep deterministic policy gradient model that meets the preset requirements of trajectory tracking as a target trajectory tracking model suitable for the target distributed electric vehicle.

[0039] The trajectory tracking model training method provided in this embodiment can be executed by a device for model training, and the specific method can be selected according to actual conditions.

[0040] Regarding step S11, standard action parameters and standard state parameters of a target user driving a target distributed electric vehicle on a target road are obtained to form a pre-training data set.

[0041] The target distributed electric vehicle mentioned in this embodiment refers to an electric vehicle with front-wheel steering (including four-wheel steering), four-wheel independent drive and distributed stability torque algorithm. The trajectory tracking model finally trained in this embodiment is also applicable to electric vehicles with front-wheel steering, four-wheel independent drive and distributed stability torque algorithm.

[0042] The target user mentioned in this embodiment refers to a human driver who can independently complete the driving task of tracking the trajectory of the target distributed electric vehicle. That is to say, in the process of the target user driving the target distributed electric vehicle to achieve trajectory tracking, relevant parameters of the target user in the process of driving the target distributed electric vehicle are collected, specifically including standard action parameters and standard state parameters, and a pre-training data set is formed based on these standard action parameters and standard state parameters.

[0043] Among them, the standard action parameters include the front wheel angle change rate of the target distributed electric vehicle during driving and the output torque change rate of the drive motor of each tire respectively. The front wheel angle change rate refers to the change rate of the front wheel angle per unit time during driving, and the output torque change rate refers to the change rate of the output torque of the drive motor of each tire per unit time during driving. The length of the unit time can be selected according to the actual situation, for example, it can be 0.1 seconds, it can also be 0.5 seconds, etc.

[0044] The standard state parameters include the trajectory characteristics of the target road and the return vector, longitudinal velocity, lateral velocity and yaw angular velocity of the target distributed electric vehicle during driving.

[0045] It should be noted that the action parameters involved in the following are all parameters including the rate of change of the front wheel angle and the rate of change of the output torque corresponding to the drive motor of each tire, and the state parameters are all parameters including trajectory characteristics and return vector, longitudinal speed, lateral speed and yaw rate. However, due to different scenarios, the prefixes of "action parameters" and "state parameters" are different, such as standard action parameters, standard state parameters, t-th action parameters, t-th state parameters, etc., where "standard" and "t-th" are prefixes to distinguish them.

[0046] In the related art, the control parameters of the path tracking output are discrete quantities, so that the vehicle relies on the discrete control parameters to track the path during the autonomous driving process, and the vehicle has a sense of frustration, which affects the autonomous driving experience. For example, in the related art during the automatic tracking driving process, when the steering wheel angle is calculated once every 1 second, the steering wheel angle calculated in the previous second is 5°, and the steering wheel angle calculated in the next second is 10°, there is a 1 second interval in between, and the steering wheel angle output jumps from 5° to 10°, which is a discrete quantity control, which leads to a sense of frustration in the vehicle.

[0047] The action parameters used in the training model of this embodiment are all change rates, specifically including the parameters of the change rate of the front wheel angle and the change rate of the output torque corresponding to the drive motor of each tire. The control of the change rate can greatly reduce the sense of frustration, and in the best case, the sense of frustration of the vehicle can be avoided. For example, during automatic tracking driving, the change rate of the front wheel angle is 1° per second, and the current front wheel angle is 10°. The front wheel angle is increased at a speed of 1° per second, so that the control amount is continuous in time, the transition is smooth, and the sense of frustration of the vehicle is reduced or avoided.

[0048] The trajectory characteristics can be expressed as follows:

[0049] Target trajectory L on the absolute coordinate plane t (that is, the target road of step S11) can be expressed by a parametric equation, see formula (1).

[0050]

[0051] Among them, L t is the parameter equation of the target road, X t (k) and Y t (k) is a projection parameter function of the target road in two perpendicular directions, X and Y. t represents time. k is a parameter used to measure the speed of the trajectory. It is a monotonically increasing function of time t. Without loss of generality, the range of k is [0, 1].

[0052] Assume the relationship between k and time t is:

[0053] k=f0(t), k∈[0,1], t∈[t min , t max ] Formula (2)

[0054] The slope of this function can, under certain circumstances, characterize the speed of the vehicle at each point along the path.

[0055] Interpolation characterization:

[0056] In the interval [t min , t max ] uniformly select h+1 points and obtain h trajectory segments:

[0057] {k(i),X t [k(i)]}, i=1,...,h+1 Formula (3)

[0058] {k(i), Y t [k(i)]}, i=1,...,h+1 Formula (4)

[0059] Perform h+1 degree polynomial interpolation on the two parameter functions of formula (1) respectively to obtain parameter vectors of the interpolation polynomial function, such as formulas (5) and (6).

[0060] X c =(a1, ..., a h ) Formula (5)

[0061] Y c =(a1, ..., a h ) Formula (6)

[0062] The interpolated parameter vector is used as a state quantity to characterize the characteristics of the trajectory.

[0063] This embodiment uses k to decouple time t from the path, that is, the parametric equation of k is used to measure only the trajectory of the path, but not the time it takes to pass through the path, and thus does not limit the speed at which the vehicle passes through the path. The 0-1 of this k can indicate that this section of the path can be completed in 1 second, 2 seconds, or 3 seconds, and the specific passing time can be set according to actual conditions. In this way, the path is independent of speed, and k is equivalent to a parameter that decouples the parametric equation of the path from time. k represents the length of the path that lasts for time t, and k is specifically in the range of 0-1, while t is in the range of minimum time and maximum time.

[0064] The return vector is explained as follows, which is a discrete representation of the preview and lateral deviation. The return vector is the adjustment vector of the target trajectory interpolation point corresponding to the turning trajectory of the vehicle at the current position:

[0065] Assume that the current position of the vehicle is expressed as

[0066] Calculate the vehicle's current position and h+1 target trajectory interpolation points {X t [k(i)],Y t [k(i)]}, and select the interpolation point closest to the current position. Assume that the nearest interpolation point is {X t [k(i0)],Y t [k(i0)]};

[0067] Calculate the position deviation vector between the current position and the nearest interpolation point:

[0068]

[0069] Determine the trajectory direction of the nearest interpolation point

[0070] The return vector is:

[0071] λ back =λ location +λ target Formula (8)

[0072] Vehicle speed

[0073] The return vector and the vehicle's velocity vector are used as state quantities to characterize the tracking correction effect of the vehicle. The return vector is a discrete representation of the preview model and the lateral deviation of the vehicle position, as shown in the figure. Figure 2 As shown. The vehicle is not necessarily on the track. The points on the track are interpolation points. First find the nearest interpolation point. The interpolation point itself has a corresponding forward direction. The vehicle has a forward direction from the interpolation point to the interpolation point. Adding these two directions together can get the return vector. The vehicle needs to follow the return vector to get close to this track. The ultimate goal of the vehicle is to get back on the track, but in order to get back on the track, it first needs to get close to the interpolation point. In addition, it should not be too hasty in the process of getting close to the interpolation point. It is hoped that the vehicle will get close to it more naturally to ensure a good riding experience.

[0074] The longitudinal velocity of the vehicle V x , lateral speed V y And yaw angular velocity γ are used as state quantities to characterize the motion state and stability of the vehicle.

[0075] Regarding step S12, the original deep deterministic policy gradient model is initialized and trained according to the pre-trained data set to obtain an initial deep deterministic policy gradient model.

[0076] The deep deterministic policy gradient model, also known as the DDPG model (Deep Deterministic Policy Gradient), is a type of reinforcement learning algorithm.

[0077] This embodiment collects control information such as the output torque and front wheel angle of the four-wheel drive motor as standard action parameters, extracts the standard action parameters as the output labels of the actor network (also called Actor network) of reinforcement learning, collects the standard state parameters of the vehicle as the input of the Actor network, and forms paired data of state and action to construct an Actor network training database. For the data collected in the Actor training database, a Critic network training database is constructed by maximizing the probability of taking the action in the corresponding state, that is, a pre-trained data set is constructed. Based on the pre-trained data set, the original deep deterministic policy gradient model is initialized and trained to obtain an initial deep deterministic policy gradient model with initial internal parameters. The training process can refer to the training process in the relevant technology, and this embodiment will not be repeated.

[0078] This embodiment will subsequently further perform model training on the basis of the initial deep deterministic policy gradient model.

[0079] Regarding step S13, based on the experience pool containing multiple state data training samples, the initial deep deterministic policy gradient model and the preset target vehicle environment model, the experience pool is updated and the initial deep deterministic policy gradient model is trained until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking.

[0080] The target vehicle environment model corresponding to the target distributed electric vehicle is constructed. The target vehicle environment model includes the vehicle three-degree-of-freedom dynamics model, the vehicle tire lateral force model, the vehicle state and tire angle model, the relative motion and absolute motion conversion model, and the tire longitudinal force model. Therefore, it specifically includes the following five construction processes. The target vehicle environment model conforms to the characteristics of the distributed control vehicle with front-wheel steering.

[0081] Construction process 1: construct a vehicle three-degree-of-freedom dynamic model corresponding to the target distributed electric vehicle.

[0082] The three-degree-of-freedom dynamic model of the vehicle is built, and the longitudinal, lateral speed and yaw degrees of freedom are taken to obtain the following formulas (9)-(11):

[0083]

[0084] in, is the derivative of the vehicle's longitudinal velocity (i.e., acceleration), is the derivative of the vehicle's lateral velocity, is the derivative of the vehicle's yaw rate, F xfl 、F xfr 、F xrl 、F xrr are the longitudinal forces of the left front wheel, right front wheel, left rear wheel, and right rear wheel tires, respectively, and F yfl 、F yfr 、F yrl 、F yrr are the lateral forces of the left front wheel, right front wheel, left rear wheel, and right rear wheel tires, δ f is the front wheel steering angle, l f , l r , l s The distance from the front wheel to the vehicle's center of mass, the distance from the rear wheel to the vehicle's center of mass, and half of the wheelbase, m and I are respectively Z are the mass and moment of inertia of the vehicle respectively.

[0085] Construction process two: constructing a vehicle tire lateral force model corresponding to the target distributed electric vehicle.

[0086] The present invention adopts the Fiala tire model to describe the tire lateral force. When the tire slip angle α is very small, tanα≈α. In order to simplify the modeling and ensure the linear relationship of the model, this embodiment omits the quadratic and higher terms related to the tire slip angle α in the formula, and then the tire model can be approximated as:

[0087]

[0088] Where C is the tire cornering stiffness, μ is the road adhesion coefficient, and F Z is the tire vertical load, and sgn is the sign function.

[0089] Construction process three, constructing a vehicle state and tire angle model corresponding to the target distributed electric vehicle, including formulas (13)-(15).

[0090]

[0091] Among them, α f is the front wheel slip angle, α r is the rear wheel slip angle, and β is the vehicle center of mass slip angle.

[0092] Construction process four: constructing a relative motion and absolute motion conversion model corresponding to the target distributed electric vehicle.

[0093] This embodiment models the trajectory of vehicle motion in an absolute coordinate system, and for this purpose converts the relative motion of the vehicle into absolute motion, including formula (16) to formula (20).

[0094]

[0095] in, is the derivative of the vehicle's yaw angle, are the derivatives of the longitudinal and lateral velocities in the absolute coordinate system, The derivatives of the X and Y axis coordinates in the absolute coordinate system. The conversion between the absolute coordinate system and the relative coordinate system is as follows Figure 3 shown.

[0096] Construction process five, constructing a tire longitudinal force model corresponding to the target distributed electric vehicle, including formulas (21)-(22).

[0097]

[0098] T <F Z μR e Formula (22)

[0099] Where T is the torque of the tire drive motor, R e The target distributed electric vehicle involved in this embodiment includes four wheel hub motor torques T fl 、T fr 、T rl 、T rr and the front wheel turning angle δ f is the control variable. Formula (22) is used as a control constraint to ensure that the tire does not slip.

[0100] An experience pool containing multiple state data training samples is constructed, and the construction method includes:

[0101] Control the target vehicle environment model to perform driving simulation on a preset road containing multiple working condition characteristics; collect multiple state transition process data of the target vehicle environment model during the driving simulation on the preset road; use each state transition process data of the target vehicle environment model during the driving simulation on the preset road as a state data training sample; fill each state data training sample into a corresponding storage space to construct an experience pool.

[0102] The preset road is a simulated road, which can be represented by the related methods of the aforementioned trajectory characteristics.

[0103] The preset road is divided into multiple path segments. The state transfer process data includes the upper simulation state parameters and upper simulation action parameters corresponding to the previous path segment of the target vehicle environment model, and the lower simulation state parameters corresponding to the next path segment adjacent to the previous path segment. The previous path segment and the next path segment corresponding to each state transfer process data are different. The action parameters include the front wheel steering angle change rate and the output torque change rate corresponding to the drive motor of each tire; the state parameters include the return vector, longitudinal speed, lateral speed and yaw angular velocity.

[0104] Specifically, step S13 may include steps S131 to S134.

[0105] Step S131, updating an experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model;

[0106] Step S132, whenever the experience pool is updated once, a target training sample is randomly selected from the experience pool containing multiple state data training samples, and the initial deep deterministic policy gradient model is trained once according to the target training sample;

[0107] Step S133, determining whether the trained initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking;

[0108] Step S134, when the trained initial deep deterministic policy gradient model does not meet the preset requirements of trajectory tracking, based on the trained initial deep deterministic policy gradient model, return to execute the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking.

[0109] Regarding step S131, the experience pool containing multiple state data training samples is updated according to the initial deep deterministic policy gradient model and the preset target vehicle environment model.

[0110] For the current preset road, the preset road is divided into N path segments, and t is traversed from 1 to N in sequence, where N is the number of path segments of the preset road, and both N and t are positive integers, 0<t≤N.

[0111] like Figure 4 As shown, it is a schematic diagram of the architecture of the deep deterministic policy gradient model, which specifically includes an actor network and a critic network. The actor network includes an online actor network and a target actor network (the two have the same internal structure), and the critic network also includes an online critic network and a target critic network (the two have the same internal structure). The process of training the model in this embodiment is to train these four networks.

[0112] The input to the actor network is the state parameter, and the output of the actor network is at least one set of action parameters that lead to the state parameter and its corresponding probability, that is, the probability of using a certain action parameter to form the state parameter. By inputting the corresponding state parameter and action parameter into the critic network, the probability of using the action parameter to obtain the state parameter can be obtained. The online actor network and the online critic network correspond to the state and action at the current moment or the current stage, while the target actor network and the target critic network predict the state and action corresponding to the next moment or the next stage.

[0113] The online actor network and the online critic network calculate the probabilities corresponding to the actions and states of the current stage, and the quality of the actions and states of the current stage affects the quality of the actions and states of the next stage (i.e., the future stage). If the rewards of the actions and states of the current stage are very high, but the rewards of the corresponding actions and states of the next stage are very poor, it means that there are major flaws in the actions and states of the current stage, and the vehicle cannot achieve long-term good path tracking. Therefore, it is necessary to reduce the rewards corresponding to the actions and states of the current stage. If the rewards of the actions and states of the current stage are very high, and the rewards of the corresponding actions and states of the next stage are also very high, it means that the actions and states of the current stage can improve the effect of path tracking and enable the vehicle to achieve long-term good path tracking. Therefore, it is necessary to appropriately increase the rewards corresponding to the actions and states of the current stage. In other words, the online actor network and the online critic network, as well as the target actor network and the target critic network, need to constrain each other to achieve long-term good path tracking for the vehicle.

[0114] The input of the target vehicle environment model is the action of controlling the vehicle in the current stage, and the output is the state and corresponding reward of the vehicle in the next stage caused by the action in the current stage. The larger the reward, the better the effect of the action in the current stage on vehicle tracking, and the smaller the reward, the worse the effect of the action in the current stage on vehicle tracking.

[0115] The reward strategy provided in this embodiment includes three aspects, namely, the degree of return to the right position, speed and motion stability, as follows:

[0116]

Return degree

[0117] In this embodiment, the distance between the vehicle speed vector and the return vector is used to measure the degree of return, and the return difference between two consecutive states is normalized as a reward. The formulas are shown in (23)-(24).

[0118] J back (t)=||λ v (t)-λ back (t)||2 Formula (23)

[0119]

[0120] Interpolate the recovery effect of the next moment and the previous moment. If the recovery is getting better and better, a higher reward will be given, and R1 will continue to increase. The denominator +1 is divided by 1, which is normalization to avoid excessive changes in the reward difference between two adjacent moments and ensure better stability.

[0121]

speed

[0122]

[0123] V max It is the maximum speed that the vehicle is allowed to pass through on the current path. The faster the vehicle passes through this trajectory, the greater the reward R2. For example, if the maximum speed is 100 kilometers per hour, the greater the reward will be if the vehicle can pass through this trajectory as much as possible within this range.

[0124] [Motion stability]

[0125]

[0126] It is hoped that the lateral speed and yaw are as small as possible, which indicates a higher R3.

[0127] The reward function is composed of the above three reward objectives, which can ensure the maximum passing speed and vehicle motion stability while guiding the vehicle to complete trajectory tracking. The following is the reward function formula, where ψ is the weight coefficient, which is used to adjust the proportion of each reward.

[0128] R=R1+R2+R3 Formula (27)

[0129] The update of the experience pool needs to rely on the initial deep deterministic policy gradient model and the target vehicle environment model.

[0130] For each path segment, the following steps are performed:

[0131] Step A1, inputting the t-th state parameter of the target vehicle environment model at the t-th path into the online actor network of the initial deep deterministic policy gradient model, and obtaining the action probability of taking the t-th action parameter at the t-th state parameter at the t-th path; t is a positive integer;

[0132] Step A2, inputting the t-th action parameter into the preset target vehicle environment model, obtaining the t-th reward corresponding to the t-th action parameter and the t+1-th state parameter corresponding to the t+1-th path segment;

[0133] Step A3, taking the tth state parameter, tth action parameter, tth reward corresponding to the tth path segment and the t+1th state parameter corresponding to the t+1th path segment as a state data training sample and putting them into the experience pool to update the experience pool containing multiple state data training samples.

[0134] Regarding step A1, when t is 1, the corresponding first state parameter can be a preset initial state parameter. When t is not 1, the corresponding tth state parameter S t , which can be determined by the target vehicle environment model according to the action a of the t-1th path segment t The state parameter S of the tth path segment obtained by running t and its corresponding action probability and reward r t .

[0135] The control quantity (i.e., action parameter) of the distributed electric vehicle includes the output torque of the four wheel drive motors and the front wheel steering angle, which are (T lr , T fr , T rl , T rr , δ f ). To ensure the continuity of the control quantity, this embodiment uses the change rates of the above five control quantities as the control actions of the reinforcement learning algorithm, that is, the output of the actor network is

[0136] Regarding step A2, the t-th action parameter a t Input the preset target vehicle environment model, and you can get the target vehicle environment model output to take the tth action parameter a t The corresponding t-th reward r t The t+1th state parameter S corresponding to the t+1th path t+1 .

[0137] Regarding step A3, (S t , a t , r t , S t+1 ) as a state data training sample and put it into the experience pool to update the experience pool containing multiple state data training samples.

[0138] That is to say, the experience pool is updated in sequence according to the path sequence of the preset roads. For example, for the first preset road, the length is 300 meters, N is 500, and the experience pool is updated for each path segment in sequence from the beginning to the end of 300 meters. After the training process corresponding to the first preset road is completed, the experience pool is updated for each path segment in sequence from the beginning to the end of 500 meters for the second preset road, the length is 500 meters, N is 300, and the experience pool is updated for each path segment in sequence from the beginning to the end of 500 meters. After each update of the experience pool, step S132 needs to be executed once to perform a model training process.

[0139] The description of step S132 will now continue.

[0140] Regarding step S132, every time the experience pool is updated, a target training sample is randomly selected from the experience pool containing multiple state data training samples, and the initial deep deterministic policy gradient model is trained once according to the target training sample.

[0141] The training process specifically includes step B1 to step B7.

[0142] Step B1, input the target state and target action in the target training sample into the online critic network in the initial deep deterministic policy gradient model to obtain the actual probability of selecting the target action under the target state;

[0143] Step B2, adjusting the online critic internal parameters of the online critic network according to the difference between the actual probability and the preset target probability;

[0144] Step B3, adjusting the target critic internal parameters of the target critic network according to the target critic internal parameters of the target critic network in the initial deep deterministic policy gradient model and the adjusted online critic internal parameters of the online critic network;

[0145] Step B4, determining the optimal action corresponding to the target state according to the target state in the target training sample and the adjusted online critic network;

[0146] Step B5, input the target state into the online actor network in the initial deep deterministic policy gradient model to obtain the actual action corresponding to the target state;

[0147] Step B6, adjusting the internal parameters of the online actor network according to the difference between the actual action and the optimal action;

[0148] Step B7, adjusting the target actor internal parameters of the target actor network according to the target actor internal parameters of the target actor network in the initial deep deterministic policy gradient model and the online actor internal parameters adjusted by the online actor network.

[0149] Among them, the critic network in the DDPG algorithm is trained, and its loss function is defined as formula (28)-(29).

[0150]

[0151] Where n is bachsize, R(t) is the environmental reward at time t (Formula 27), γ1 is the reward attenuation factor (valued at 0.9), is the output of the target critic network.

[0152] The parameter update of the target actor network refers to formula (30), and the parameter update of the target critic network refers to formula (31).

[0153]

[0154] in, is the target critic internal parameter after the target critic network updates, θ Q is the internal parameter of the online critic network corresponding to the current online critic, θ Q′ Update the internal parameters of the target critic before the target critic network; is the internal parameter of the target actor after the target actor network updates, θ μ is the internal parameter of the online actor corresponding to the online actor network, θ μ′ is the internal parameter of the target actor before the target actor network is updated; τ is the update factor (the value is 0.01).

[0155] That is to say, according to the last updated θ Q′ and θ Q , you can get the current updated According to the last updated θ μ′ and θ μ , you can get the current updated

[0156] After the initial deep deterministic policy gradient model is trained once according to the target training sample, before determining whether the trained initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking, the method also includes steps C1 to C4.

[0157] Step C1, determine whether the trajectory tracking error corresponding to the M consecutive state data training samples most recently updated in the experience pool exceeds the first preset distance; M is a positive integer; for example, M is the number of corresponding path segments within 5 consecutive seconds, each path segment corresponds to a state data training sample, and the first preset distance can be 1 meter.

[0158] Step C2, when the trajectory tracking error corresponding to the most recently updated M consecutive state data training samples in the experience pool does not exceed the first preset distance, it means that the training process of the current preset road has been completed, and the next preset road can be returned to execute the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model.

[0159] When the trajectory tracking error corresponding to the most recently updated M consecutive state data training samples in the experience pool exceeds the first preset distance, it means that the training process on the current preset road should continue, that is, continue to execute steps 131-S132 until the execution condition of step C2 is met.

[0160] Step C3, determining whether the distance between the actual position of the vehicle corresponding to the latest state data training sample and the target end position of the current preset road is less than or equal to a second preset distance; the second preset distance may be 2 meters.

[0161] Step C4, when the distance between the actual position of the vehicle corresponding to the latest state data training sample and the target terminal position of the current preset road is less than or equal to the second preset distance, it means that the training process of the current preset road has been completed. For the next preset road, return to execute the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model.

[0162] When the distance between the actual vehicle position corresponding to the latest status data training sample and the target terminal position of the current preset road is greater than the second preset distance, it means that the training process on the current preset road should continue, that is, continue to execute steps 131-step S132 until the execution condition of step C4 is met.

[0163] Regarding step S133, it is determined whether the trained initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking.

[0164] Specifically, it may include step D1 to step D3.

[0165] Step D1, determine whether the total training distance of the initial deep deterministic policy gradient model is as long as the preset total distance; the total training distance can be set according to actual conditions and can be hundreds of thousands of meters.

[0166] Step D2, after the initial deep deterministic policy gradient model has been trained for a preset total distance, controlling the initial deep deterministic policy gradient model to perform path tracking on a target road with a third preset distance;

[0167] Step D3, after the initial deep deterministic policy gradient model completes the path tracking of the target road, determine whether the maximum trajectory tracking error of the initial deep deterministic policy gradient model during the path tracking process is less than or equal to a fourth preset distance.

[0168] Regarding step S134, when the trained initial deep deterministic policy gradient model does not meet the preset requirements of trajectory tracking, based on the trained initial deep deterministic policy gradient model, return to execute the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking.

[0169] Regarding step S14, the initial deep deterministic policy gradient model that meets the preset requirements of trajectory tracking is used as a target trajectory tracking model suitable for the target distributed electric vehicle.

[0170] When the maximum trajectory tracking error of the initial deep deterministic policy gradient model in the path tracking process is less than or equal to the fourth preset distance, the current initial deep deterministic policy gradient model is used as a target trajectory tracking model suitable for the target distributed electric vehicle.

[0171] When the maximum trajectory tracking error of the initial deep deterministic policy gradient model during the path tracking process is greater than the fourth preset distance, return to execute the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model.

[0172] For example, the vehicle corresponding to the initial deep deterministic policy gradient model will be trained multiple times. The path length of the preset road corresponding to each round of training is between 300-500 meters, and the initial position of the vehicle is within 10m of the path starting point. When the vehicle's trajectory tracking error does not exceed 1m for 5 consecutive seconds, or the distance from the target end point is no more than 2m, or it is more than 50 meters away from the route, it is determined that the vehicle has completed a target trajectory tracking task, the target route and the initial state of the vehicle are updated, and training is restarted. A training effect test is performed every 1,000 trainings. The set path length of the target road is between 300-500 meters, the vehicle starting point is at the starting point of the path, and the initial direction is consistent with the target path. If the vehicle completes the entire training road and the maximum value of the trajectory tracking error does not exceed 0.5m, the entire training process ends.

[0173] In summary, this embodiment obtains the standard action parameters and standard state parameters of the target user driving the target distributed electric vehicle on the target road, and constitutes a pre-trained data set; the original deep deterministic policy gradient model is initialized and trained according to the pre-trained data set to obtain an initial deep deterministic policy gradient model; according to the experience pool containing multiple state data training samples, the initial deep deterministic policy gradient model and the preset target vehicle environment model, the experience pool is updated, and the initial deep deterministic policy gradient model is trained until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking; the initial deep deterministic policy gradient model that meets the preset requirements of trajectory tracking is used as a target trajectory tracking model suitable for the target distributed electric vehicle. It can be seen that this embodiment trains a trajectory tracking model for an electric vehicle with front-wheel steering, four-wheel independent drive and distributed stability torque algorithm, and obtains a target trajectory tracking model suitable for the target distributed electric vehicle, so as to achieve trajectory tracking of the target distributed electric vehicle.

[0174] This embodiment applies reinforcement learning to the trajectory tracking control field of distributed electric vehicles with front-wheel steering and four-wheel independent drive, and uses the powerful learning ability of reinforcement learning to complete the trajectory tracking task of distributed electric vehicles, while ensuring the vehicle's maximum trajectory tracking speed and best stability.

[0175] This embodiment models a target vehicle environment model of the characteristics of a distributed electric vehicle with front-wheel steering and four-wheel independent drive, which interacts with a deep deterministic policy gradient model to guide the deep deterministic policy gradient model training process.

[0176] This embodiment uses interpolation fitting to encode the interpolation parameters as decomposed trajectory characteristic information into the target vehicle environment model, and innovates the representation method of measuring relative position, that is, the nearest interpolation point returns to the positive vector, which can better measure the effect of vehicle tracking trajectory.

[0177] This embodiment realizes the continuous characterization of the actions applicable to distributed electric vehicles and avoids control jitter during vehicle trajectory tracking.

[0178] This embodiment proposes to let the driver drive the vehicle that implements distributed stability torque control to complete data collection during the pre-training stage. The pre-training database constructed in this way can reflect the advantages of distributed control for vehicle motion stability in terms of control quantity.

[0179] Based on the same inventive concept, this embodiment provides Figure 5 A trajectory tracking model training device is shown, the device comprising:

[0180] The pre-training set acquisition module 51 is used to acquire standard action parameters and standard state parameters of a target user driving a target distributed electric vehicle on a target road, and form a pre-training data set; the target distributed electric vehicle refers to an electric vehicle with front-wheel steering, four-wheel independent drive and distributed stability torque algorithm;

[0181] The model initial training module 52 is used to initialize the training of the original deep deterministic policy gradient model according to the pre-training data set to obtain an initial deep deterministic policy gradient model;

[0182] A model update training module 53 is used to update the experience pool according to the experience pool containing multiple state data training samples, the initial deep deterministic policy gradient model and the preset target vehicle environment model, and to train the initial deep deterministic policy gradient model until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking;

[0183] The target model determination module 54 is used to use the initial deep deterministic policy gradient model that meets the preset requirements of trajectory tracking as a target trajectory tracking model suitable for the target distributed electric vehicle.

[0184] Further, the model update training module 53 is used to:

[0185] According to the initial deep deterministic policy gradient model and the preset target vehicle environment model, the experience pool containing multiple state data training samples is updated;

[0186] Every time the experience pool is updated, a target training sample is randomly selected from the experience pool containing multiple state data training samples, and the initial deep deterministic policy gradient model is trained once according to the target training sample;

[0187] Determine whether the initial deep deterministic policy gradient model after training meets the preset requirements of trajectory tracking;

[0188] When the trained initial deep deterministic policy gradient model does not meet the preset requirements of trajectory tracking, based on the trained initial deep deterministic policy gradient model, return to execute the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking.

[0189] Further, the model update training module 53 is used to:

[0190] t starts from 1 and traverses to N in sequence, where N is the number of path segments of the preset road, and both N and t are positive integers, 0<t≤N;

[0191] Input the t-th state parameter of the target vehicle environment model in the t-th path into the online actor network of the initial deep deterministic policy gradient model, and obtain the action probability of taking the t-th action parameter under the t-th state parameter in the t-th path; t is a positive integer;

[0192] Input the t-th action parameter into the preset target vehicle environment model to obtain the t-th reward corresponding to the t-th action parameter and the t+1-th state parameter corresponding to the t+1-th path segment;

[0193] The tth state parameter, tth action parameter, tth reward corresponding to the tth path segment and the t+1th state parameter corresponding to the t+1th path segment are taken as a state data training sample and put into the experience pool to update the experience pool containing multiple state data training samples.

[0194] Further, the model update training module 53 is used to:

[0195] Input the target state and target action in the target training sample into the online critic network in the initial deep deterministic policy gradient model to obtain the actual probability of selecting the target action under the target state;

[0196] Adjusting the internal parameters of the online critic of the online critic network according to the difference between the actual probability and the preset target probability;

[0197] adjusting the target critic internal parameters of the target critic network according to the target critic internal parameters of the target critic network in the initial deep deterministic policy gradient model and the adjusted online critic internal parameters of the online critic network;

[0198] Determine the optimal action corresponding to the target state based on the target state in the target training sample and the adjusted online critic network;

[0199] Input the target state into the online actor network in the initial deep deterministic policy gradient model to obtain the actual action corresponding to the target state;

[0200] Adjusting the internal parameters of the online actor of the online actor network according to the difference between the actual action and the optimal action;

[0201] The target actor internal parameters of the target actor network are adjusted according to the target actor internal parameters of the target actor network in the initial deep deterministic policy gradient model and the online actor internal parameters adjusted by the online actor network.

[0202] Further, the model update training module 53 is used to:

[0203] After training the initial deep deterministic policy gradient model once according to the target training sample, before judging whether the trained initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking:

[0204] Determine whether the trajectory tracking error corresponding to the M consecutive state data training samples most recently updated in the experience pool exceeds a first preset distance; M is a positive integer;

[0205] When the trajectory tracking error corresponding to the most recently updated M consecutive state data training samples in the experience pool does not exceed the first preset distance, for the next preset road, return to execute the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model;

[0206] Determine whether the distance between the actual position of the vehicle corresponding to the latest state data training sample and the target end position of the current preset road is less than or equal to a second preset distance;

[0207] When the distance between the actual vehicle position corresponding to the latest state data training sample and the target terminal position of the current preset road is less than or equal to the second preset distance, for the next preset road, return to execute the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model.

[0208] Further, the model update training module 53 is used to:

[0209] Determine whether the initial deep deterministic policy gradient model has been trained for a preset total distance;

[0210] After the initial deep deterministic policy gradient model has been trained for a preset total distance, controlling the initial deep deterministic policy gradient model to perform path tracking on a target road having a third preset distance;

[0211] After the initial deep deterministic policy gradient model completes the path tracking of the target road, it is determined whether a maximum trajectory tracking error of the initial deep deterministic policy gradient model during the path tracking process is less than or equal to a fourth preset distance.

[0212] Furthermore, the standard action parameters include the front wheel turning angle change rate of the target distributed electric vehicle during driving and the output torque change rate corresponding to the driving motor of each tire;

[0213] The standard state parameters include the trajectory characteristics of the target road and the return vector, longitudinal velocity, lateral velocity and yaw angular velocity of the target distributed electric vehicle during driving.

[0214] Based on the same inventive concept, this embodiment provides Figure 6 An electronic device as shown includes:

[0215] Processor 61;

[0216] A memory 62 for storing instructions executable by the processor 61;

[0217] The processor 61 is configured to execute to implement a trajectory tracking model training method as provided above.

[0218] Based on the same inventive concept, this embodiment provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the processor 61 of the electronic device, the electronic device can execute a trajectory tracking model training method as provided above.

[0219] Since the electronic device introduced in this embodiment is an electronic device used to implement the information processing method in the embodiment of the present application, based on the information processing method introduced in the embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the electronic device used by the information processing method in the embodiment of the present application, it belongs to the scope of protection of this application.

[0220] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0221] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0222] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0223] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0224] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0225] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A trajectory tracking model training method, characterized in that: The method comprises: Acquire standard action parameters and standard state parameters of a target user driving a target distributed electric vehicle on a target road, and form a pre-training data set; the target distributed electric vehicle refers to an electric vehicle with front-wheel steering, four-wheel independent drive, and a distributed stability torque algorithm; Initializing the original deep deterministic policy gradient model according to the pre-trained data set to obtain an initial deep deterministic policy gradient model; According to an experience pool containing a plurality of state data training samples, the initial deep deterministic policy gradient model and a preset target vehicle environment model, the experience pool is updated, and the initial deep deterministic policy gradient model is trained until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking; The initial deep deterministic policy gradient model that meets the preset requirements of the trajectory tracking is used as a target trajectory tracking model suitable for the target distributed electric vehicle.

2. The method according to claim 1, characterized in that The updating of the experience pool according to the experience pool containing a plurality of state data training samples, the initial deep deterministic policy gradient model and the preset target vehicle environment model, and the training of the initial deep deterministic policy gradient model until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking, includes: According to the initial deep deterministic policy gradient model and the preset target vehicle environment model, updating the experience pool containing multiple state data training samples; Whenever the experience pool is updated, a target training sample is randomly selected from the experience pool containing a plurality of state data training samples, and the initial deep deterministic policy gradient model is trained once according to the target training sample; Determining whether the trained initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking; When the trained initial deep deterministic policy gradient model does not meet the preset requirements of the trajectory tracking, based on the trained initial deep deterministic policy gradient model, return to execute the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model until the initial deep deterministic policy gradient model meets the preset requirements of the trajectory tracking.

3. The method according to claim 2, characterized in that The updating of the experience pool containing a plurality of state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model includes: t starts from 1 and traverses to N in sequence, where N is the number of path segments of the preset road, and both N and t are positive integers, 0<t≤N; Inputting the t-th state parameter of the target vehicle environment model at the t-th path into the online actor network of the initial deep deterministic policy gradient model, and obtaining the action probability of the t-th path taking the t-th action parameter under the t-th state parameter; t is a positive integer; Inputting the t-th action parameter into the preset target vehicle environment model to obtain the t-th reward corresponding to taking the t-th action parameter and the t+1-th state parameter corresponding to the t+1-th path segment; The tth state parameter, the tth action parameter, the tth reward corresponding to the tth path segment, and the t+1th state parameter corresponding to the t+1th path segment are taken as a state data training sample and put into the experience pool to update the experience pool containing multiple state data training samples.

4. The method according to claim 2, characterized in that The training of the initial deep deterministic policy gradient model once according to the target training sample comprises: Inputting the target state and target action in the target training sample into the online critic network in the initial deep deterministic policy gradient model to obtain an actual probability of selecting the target action under the target state; Adjusting the online critic internal parameters of the online critic network according to the difference between the actual probability and the preset target probability; Adjusting the target critic internal parameters of the target critic network according to the target critic internal parameters of the target critic network in the initial deep deterministic policy gradient model and the online critic internal parameters adjusted by the online critic network; Determining an optimal action corresponding to the target state according to the target state in the target training sample and the adjusted online critic network; Inputting the target state into an online actor network in the initial deep deterministic policy gradient model to obtain an actual action corresponding to the target state; adjusting an internal parameter of an online actor of the online actor network according to a difference between the actual action and the optimal action; The target actor internal parameters of the target actor network are adjusted according to the target actor internal parameters of the target actor network in the initial deep deterministic policy gradient model and the online actor internal parameters adjusted by the online actor network.

5. The method according to claim 2, characterized in that After training the initial deep deterministic policy gradient model once according to the target training sample, before determining whether the trained initial deep deterministic policy gradient model meets the preset requirement of trajectory tracking, the method further includes: Determine whether the trajectory tracking errors corresponding to the M consecutive state data training samples most recently updated in the experience pool exceed a first preset distance; M is a positive integer; When the trajectory tracking errors corresponding to the most recently updated M consecutive state data training samples in the experience pool do not exceed the first preset distance, for the next preset road, returning to the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model; Determine whether the distance between the actual position of the vehicle corresponding to the latest state data training sample and the target end position of the current preset road is less than or equal to a second preset distance; When the distance between the actual vehicle position corresponding to the latest state data training sample and the target terminal position of the current preset road is less than or equal to the second preset distance, for the next preset road, return to the step of updating the experience pool containing multiple state data training samples according to the initial deep deterministic policy gradient model and the preset target vehicle environment model.

6. The method according to claim 2, characterized in that The determining whether the trained initial deep deterministic policy gradient model meets the preset requirement of trajectory tracking includes: Determining whether the initial deep deterministic policy gradient model has undergone a training process up to a preset total distance; After the initial deep deterministic policy gradient model has been trained for the preset total distance, controlling the initial deep deterministic policy gradient model to perform path tracking on a target road with a third preset distance; After the initial deep deterministic policy gradient model completes the path tracking of the target road, it is determined whether a maximum trajectory tracking error of the initial deep deterministic policy gradient model during the path tracking process is less than or equal to a fourth preset distance.

7. The method according to claim 1, characterized in that The standard action parameters include the front wheel turning angle change rate of the target distributed electric vehicle during driving and the output torque change rate corresponding to the driving motor of each tire; The standard state parameters include the trajectory characteristics of the target road and the return vector, longitudinal speed, lateral speed and yaw angular velocity of the target distributed electric vehicle during driving.

8. A trajectory tracking model training device, characterized in that: The device comprises: A pre-training set acquisition module is used to acquire standard action parameters and standard state parameters of a target user driving a target distributed electric vehicle on a target road, and form a pre-training data set; the target distributed electric vehicle refers to an electric vehicle with front-wheel steering, four-wheel independent drive and distributed stability torque algorithm; A model initial training module, used to initialize the training of the original deep deterministic policy gradient model according to the pre-trained data set to obtain an initial deep deterministic policy gradient model; A model update training module, used to update the experience pool according to an experience pool containing multiple state data training samples, the initial deep deterministic policy gradient model and a preset target vehicle environment model, and train the initial deep deterministic policy gradient model until the initial deep deterministic policy gradient model meets the preset requirements of trajectory tracking; The target model determination module is used to use the initial deep deterministic policy gradient model that meets the preset requirements of the trajectory tracking as a target trajectory tracking model suitable for the target distributed electric vehicle.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute to implement a trajectory tracking model training method as described in any one of claims 1 to 7.

10. A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to implement a trajectory tracking model training method as described in any one of claims 1 to 7.

Images