An end-to-end autonomous driving model training method and device

Abstract

The application provides an end-to-end automatic driving model training method and device, which comprises collecting operation parameters of a vehicle, establishing an automatic driving model based on an "actor-critic" framework, initializing parameters of critic network and actor network in a DDPG algorithm by using the DDPG algorithm fused with adaptive parameter space noise; selecting an Adam optimizer and initializing an experience replay pool, storing experience of interaction between an agent and an environment, setting an initial learning rate and determining a batch size; updating parameters of the critic network and the actor network by using experience in the experience replay pool; and adjusting performance of the automatic driving vehicle. The end-to-end automatic driving model training method and device provided by the application accelerate the convergence speed of the model by adjusting the learning rate and the batch size to appropriate values, and adjust the learning rate and the batch size in the hyperparameters, so that the model is more stable during the training process.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and more particularly to an end-to-end autonomous driving model training method and apparatus. Background Technology

[0002] Autonomous driving, also known as driverless driving, refers to a vehicle's ability to perceive and navigate its environment and reach its destination smoothly without a driver. It uses its own equipment to perceive and understand the surrounding environment, and then automatically plans the driving route and performs motion control according to the user's needs to achieve a safe, comfortable and efficient riding experience.

[0003] Autonomous vehicles typically possess a variety of functions, such as autonomous driving, automatic transmission, automatic braking, automatic monitoring of the surrounding environment, automatic lane changing, automatic steering, automatic signal warning, and connected autonomous driving assistance functions. Autonomous vehicles utilize onboard sensors to perceive their surroundings, including information such as roads, vehicle position, and obstacles, and then control the vehicle's steering and speed based on this information, enabling the vehicle to drive safely and reliably on the road.

[0004] Existing end-to-end autonomous driving model training methods and devices employ an "actor-critic" framework. The actor network in the autonomous driving model is responsible for generating driving action strategies. Based on the current environmental state, such as vehicle position, surrounding traffic conditions, and its own network parameters, it calculates the appropriate driving actions, such as steering, accelerating, or braking. The actor network is a deep neural network; its input is the environmental state, and its output is the driving action. During training, the actor network continuously adjusts its parameters based on feedback, i.e., the evaluation results of the critic network, to optimize the driving strategy.

[0005] In the autonomous driving model, the critic network is responsible for evaluating the driving actions generated by the actor network. Based on the current environmental state, the driving actions generated by the actor network, and subsequent reward or penalty signals, it calculates the value or quality of the action. The critic network's input includes the environmental state, the driving action, and subsequent state or reward signals; its output is the value assessment of the action. During training, the critic network continuously adjusts its parameters based on real reward signals and its own prediction errors to improve the accuracy of its value assessment.

[0006] The existing "actor-critic" framework is usually combined with the DDPG algorithm, and the parameters of the neural network strategy change during the training process, which causes the sensitivity of the strategy weight to change, making it difficult to predict the next action of the strategy, that is, the parameter sensitivity problem of the "actor-critic" framework, and when combined with the DDPG algorithm, there are multiple hyperparameters such as learning rate and batch size, and each hyperparameter interacts with each other, and it is difficult to adjust multiple hyperparameters, which causes the model to be unstable during the training process, that is, the model has problems such as gradient explosion, gradient disappearance or training process shock during the training process.

[0007] Therefore, it is necessary to provide a new end-to-end autonomous driving model training method and device to solve the above technical problems. SUMMARY

[0008] To solve the above technical problems, the present application provides an end-to-end autonomous driving model training method and device.

[0009] The end-to-end autonomous driving model training method provided by the present application comprises the following steps:

[0010] Step one, collect the running parameters of the vehicle, including the vehicle position, the surrounding traffic conditions and the network parameters of the vehicle itself, establish an autonomous driving model based on the "actor-critic" framework, use the DDPG algorithm fused with adaptive parameter space noise to initialize the parameters of the critic network and the actor network in the DDPG algorithm. Wherein, the network parameters of the vehicle itself include the steering, acceleration and braking parameters of the vehicle during driving.

[0011] Step two, according to the initialized parameters of the critic network and the actor network, select the Adam optimizer and initialize the experience replay pool, store the experience of the interaction between the agent and the environment, set the initial learning rate and determine the batch size; wherein, the agent includes the autonomous driving vehicle; the environment includes the road and the traffic.

[0012] Step three, randomly sample a batch of experiences with a number of BS from the experience replay pool for training, and use the batch of experiences and the current learning rate lr to update the parameters of the critic network and the actor network.

[0013] Step four, according to the updated network parameters, adjust the performance of the autonomous driving vehicle, including updating the perception, decision and planning of the vehicle to the environment.

[0014] Step five, according to the required adjusted performance, set the decay rate DR and the decay period DS, at the end of each training period or after reaching a certain number of training steps, calculate a new learning rate according to the preset exponential decay strategy.

[0015] Step six, update the global variable lr to nlr according to the new learning rate after calculation.

[0016] Step seven, continue the next training cycle or step training using the updated learning rate nlr and network parameters, and regularly evaluate the trained model, adjust the learning rate, batch size and network structure according to the evaluation results, and realize the optimization of the performance of automatic driving.

[0017] Preferably, the DDPG algorithm fusing adaptive parameter space noise comprises the following steps:

[0018] S1, initialization and preparation: including random initialization of the weights θ of the critic network and the actor network Q and θ π , initializing the critic target network and the actor target network using the weights θ Q and the weights θ π respectively, obtaining and At this time, initialize the parameter noise and the experience replay pool R, and jitter the policy network;

[0019] S2, training cycle and execution: according to the parameters after initialization and preparation, set the initial episode = 1, start the global training cycle, and at the beginning of each episode, receive the initial observation state S t ;

[0020] S3, policy selection and execution: in the process of training cycle, sample noise variable ψ from noise distribution, add noise variable ψ to actor network: θ Q = θ Q + ψ, After adding, the record jitter network is: At this time, select action a t , execute according to the selected action a t , observe the next state S t+1 , obtain the reward r t , and whether the termination state is reached;

[0021] S4, according to the data in the process of policy selection and execution, record (S t , a t , r t , S t+1 ), and store the experience (S t , a t , r t , S t+1 ) into the experience replay pool R, and randomly sample a batch of experience from the experience replay pool R for training;

[0022] S5, network update: according to the experience after random sampling, the following operations are performed:

[0023] Operation 1, calculate the target value using the critic target network:

[0024] y i = r i + γ * Q 1 (S i+1 , π 1 (S i+1 ))

[0025] Where r i is the immediate reward, γ is the discount factor, Q 1 (S i+1 , π 1 (S i+1 ) is the Q value predicted by the target network;

[0026] Operation 2, update the parameters of the critic network using gradient descent

[0027] Operation 3, update the parameters θ of the actor network using gradient update strategy;

[0028] Operation 4, update the parameters of the target network using soft update strategy, so that:

[0029] τθ+(1-τ)θ 1 →θ 1

[0030]

[0031] Where τ is 0.001;

[0032] The specific operation mode is: copy the parameters of the critic network and the actor network to the corresponding target network to realize stable update of the target network;

[0033] S6, cycle and termination: according to the data obtained by network update, it is judged whether the termination condition is reached, if the termination condition is not reached, the episode number is updated, and the training is continued back to S2, if the termination condition is reached, the training is ended, and the training result is output.

[0034] Preferably, in the step one, the parameters of the critic and actor network optimizers are initialized, a batch size variable BS and a learning rate variable lr are initialized, the batch size variable BS is set to 64, the initial value of the learning rate is lr=0.001, and the current learning rate is used when updating the parameters of the critic network and the actor network each time.

[0035] Preferably, in the step three, the number of random sampling is the same as the batch size variable BS, both of which are 64.

[0036] Preferably, in the step S3, normalization processing is further included for each layer of the neural network to ensure consistent influence of noise on each layer.

[0037] Preferably, the gradient descent method is used to update the parameters of the critic network comprising the following steps:

[0038] Step1, loss calculation: using mean square error (MSE) loss function to calculate the difference L between the Q value (Q(S i , a i )) predicted by the current network and the target value (y i );

[0039] Step2, calculate the gradient of the loss function L with respect to the parameters of the critic network, denoted as

[0040] Step3, use the gradient descent method to update the parameters of the critic network where α is the learning rate.

[0041] Preferably, the gradient update strategy is used to update the parameters θ of the actor network, including the following operations:

[0042] Step1, design the objective function: J(θ) = E[Q(S, π(S))];

[0043] Step2, use the policy gradient method to calculate the gradient of the actor network parameters θ, denoted as

[0044] Step3, use the chain rule to expand the policy gradient;

[0045] Step4, use the gradient ascent method to update the parameters θ of the actor network:

[0046]

[0047] where α is the learning rate.

[0048] Preferably, it includes an initialization unit, an experience collection and storage unit, a network training and updating unit, a learning rate adjustment unit, and a model evaluation and optimization unit, wherein:

[0049] The initialization unit is responsible for setting the initial parameters of the critic network and the actor network, as well as configuring the optimizer, experience replay pool, learning rate and batch size;

[0050] An experience collection and storage unit interacts with the autonomous vehicle and environment settings provided by the initialization unit, collects experiences and stores them into an experience replay pool, and continuously provides new experience data for network training.

[0051] A network training and updating unit is configured to update the parameters of the critic network and the actor network, and to periodically run the autonomous vehicle to adjust its performance after each update.

[0052] A learning rate adjustment unit is configured to update the global learning rate variable to control the subsequent network training process.

[0053] A model evaluation and optimization unit is configured to periodically evaluate the trained model, and to adjust the learning rate, batch size, and network structure based on the evaluation results.

[0054] Preferably, the network training and updating unit further comprises randomly sampling a batch of experiences from the experience replay pool, updating the parameters of the critic network and the actor network using these experiences and the current learning rate, and adjusting the performance of the autonomous vehicle based on the new parameters after the update.

[0055] Preferably, the model evaluation and optimization unit is configured to periodically run, and the running period is longer than that of the network training and updating unit.

[0056] Compared with the related art, the end-to-end autonomous driving model training method and device provided by the present application has the following advantages:

[0057] 1. The method can adaptively adjust the parameter space noise, which helps to alleviate the parameter sensitivity problem of the "actor-critic" framework, making the model more stable during training, reducing the occurrence of problems such as gradient explosion, gradient disappearance, or training process shock, and by setting the decay rate DR and decay period DS and using an exponential decay strategy to calculate the new learning rate, the learning rate can be dynamically adjusted according to the training progress and model performance, which helps the model to better converge during training and improves the training efficiency.

[0058] 2. The method uses the DDPG algorithm that integrates adaptive parameter space noise, which can fully utilize multi-source information, i.e., using vehicle position and surrounding traffic conditions for decision-making and planning, which helps to improve the model's generalization ability and make it better adapt to different roads and traffic scenarios.

[0059] 3、The method can adjust the learning rate and batch size in the multi-parameter, adopt an exponential decay strategy, reduce the learning rate according to a fixed decay rate DR, and gradually adjust the initial learning rate and decay rate according to the convergence speed and stability of the model, that is, the learning rate lr controls the step size of parameter update, a smaller learning rate can reduce the oscillation during parameter update, improve the stability of the DDPG algorithm when used, and by adjusting to a proper learning rate, the convergence speed of the model can be accelerated, and by adjusting to a proper batch size, the accuracy and efficiency of each gradient update can be improved, so that the model is more stable during training. BRIEF DESCRIPTION OF DRAWINGS

[0060] Fig. 1 The flowchart of the whole process of the end-to-end automatic driving model training method and device provided by the application is shown in the figure.

[0061] Fig. 2 The flowchart of the DDPG algorithm fused with adaptive parameter space noise is shown in the figure.

[0062] Fig. 3 The flowchart of the end-to-end automatic driving model training device is shown in the figure. DETAILED DESCRIPTION

[0063] The application will be further described below in combination with the drawings and embodiments.

[0064] Please refer to Figs. 1-3 , wherein Fig. 1 The flowchart of the whole process of the end-to-end automatic driving model training method and device provided by the application is shown in the figure. Fig. 2 The flowchart of the DDPG algorithm fused with adaptive parameter space noise is shown in the figure. Fig. 3 The flowchart of the end-to-end automatic driving model training device is shown in the figure.

[0065] In the specific implementation process, as shown in the figure, Figs. 1-3 The steps include the following steps:

[0066] Step 1, collect the running parameters of the vehicle, including the vehicle position, the surrounding traffic condition and the network parameters of the vehicle itself, establish an automatic driving model based on the "actor-critic" framework, adopt the DDPG algorithm fused with adaptive parameter space noise, and initialize the parameters of the critic network and the actor network in the DDPG algorithm; wherein the network parameters of the vehicle itself include the steering, acceleration and braking parameters of the vehicle during driving.

[0067] It should be noted that in the initialization phase, adaptive parameter space noise is introduced to provide sufficient exploration space for the actor network at the beginning of training, avoiding falling into local optimal solution. As the training progresses, the noise will gradually decrease to ensure that the model can converge to a better strategy;

[0068] It should be noted that when initializing parameters, a reasonable parameter initialization method such as Xavier initialization is used to ensure that the network weights maintain appropriate distribution during training, avoiding the problem of gradient vanishing or explosion.

[0069] Step two, according to the initialized parameters of the critic network and the actor network, select the Adam optimizer and initialize the experience replay pool, store the experience of the agent interacting with the environment, set the initial learning rate and determine the batch size; Wherein, the agent includes an autonomous vehicle; The environment includes roads and traffic;

[0070] It should be noted that the Adam optimizer combines the advantages of the momentum method and the RMSprop optimization algorithm, which can adaptively adjust the learning rate to improve training efficiency and model performance;

[0071] It should be noted that in the experience replay pool management, priority sampling is used, historical experience data is used, and the learning rate and batch size are dynamically adjusted according to the actual situation during training to balance training speed and model performance.

[0072] Step three, randomly sample a batch of experiences with a size of BS from the experience replay pool for training, and use this batch of experiences and the current learning rate lr to update the parameters of the critic network and the actor network;

[0073] It should be noted that after sampling, the data can be preprocessed, including normalization, to improve training efficiency and model performance.

[0074] Step four, according to the updated network parameters, adjust the performance of the autonomous vehicle, including updating the vehicle's perception of the environment, decision-making and planning;

[0075] It should be noted that when adjusting the performance of the autonomous vehicle, a feedback mechanism can be introduced to further adjust and optimize the model based on the evaluation results; For example: when the model performs poorly in terms of safety, more training and data collection in related areas can be strengthened.

[0076] Step five, according to the required adjusted performance, set the decay rate DR and the decay period DS, at the end of each training period or after reaching a certain number of training steps, calculate the new learning rate according to the preset exponential decay strategy;

[0077] It should be noted that when using the learning rate decay strategy, cosine annealing decay, segmented decay, etc. can also be used to observe the influence on the performance of the model.

[0078] Among them, the cosine annealing decay strategy is to reduce the learning rate according to the law of the cosine function after each epoch or step. This strategy can maintain a larger learning rate in the early stage of training, and gradually reduce the learning rate in the later stage of training, achieving a smooth decay effect.

[0079] Segmented decay is to decay the learning rate by a fixed ratio after each fixed epoch or step interval.

[0080] It should be noted that according to the required adjusted performance, the decay rate DR and the decay period DS are set, and the decay rate DR and the decay period DS are dynamically adjusted according to the actual situation in the training process, so as to more flexibly control the decay process of the learning rate.

[0081] Step six, according to the new learning rate calculated, update the global variable lr to nlr;

[0082] Step seven, using the updated learning rate nlr and network parameters to continue the next training period or step training, and regularly evaluating the trained model, adjusting the learning rate, batch size and network structure according to the evaluation result, realizing the optimization of the performance of automatic driving.

[0083] It should be noted that when evaluating the model, multiple model evaluation methods can be used, such as offline testing, online testing and cross-validation, to comprehensively evaluate the performance of the model.

[0084] It should be further explained that through the experience replay mechanism, historical experience data can be fully utilized for training, improving data utilization and training efficiency, and at the same time, the random sampling method can break the time correlation between data, which is helpful to the stability of training;

[0085] It should be further explained that by continuously updating network parameters and adjusting learning rate and other parameters, the performance of automatic driving vehicles is gradually optimized, including perception, decision-making and planning, which helps to improve the safety and comfort of automatic driving.

[0086] As shown in Figs. 1-2 , the DDPG algorithm combined with adaptive parameter space noise includes the following steps:

[0087] S1, initialization and preparation: including randomly initializing the weights θ Q and θ π of the critic network and the actor network, using the weights θ Q and the weights θ πInitialize critic target network and actor target network respectively, and obtain and At this time, initialize the parameter noise and the experience replay pool R, and jitter the policy network.

[0088] S2, training cycle and execution: according to the initialized and prepared parameters, set the initial episode = 1, start the global training cycle, and at the beginning of each episode, receive the initial observation state S t .

[0089] S3, policy selection and execution: in the process of training cycle, sample noise variable ψ from noise distribution, and add noise variable ψ to actor network: θ Q = θ Q + ψ, After adding, the record jitter network is: At this time, select action a t , according to the selected action a t , execute, observe the next state S t+1 , obtain the reward r t , and whether the termination state is reached.

[0090] S4, according to the data in policy selection and execution, record (S t , a t , r t , S t+1 ), and store the experience (S t , a t , r t , S t+1 ) in the experience replay pool R, and randomly sample a batch of experiences from the experience replay pool R for training.

[0091] S5, network update: according to the experience obtained after random sampling, the following operations are performed:

[0092] Operation 1: calculate the target value using the critic target network:

[0093] y i = r i + γ * Q 1 (S i+1 , π 1 (S i+1 ))

[0094] Wherein, r i is the immediate reward, γ is the discount factor, Q 1 (S i+1 , π 1 (S i+1 ) is the Q value predicted by the target network.

[0095] Operation II, updating the parameters of the critic network using gradient descent method

[0096] Operation III, updating the parameters of the actor network using gradient update policy

[0097] Operation IV, updating the parameters of the target network using soft update policy, so that:

[0098] τθ+(1-τ)θ 1 →θ 1

[0099]

[0100] Wherein, τ is 0.001;

[0101] The specific operation mode is: copying the parameters of the critic network and the actor network into the corresponding target network to realize stable updating of the target network;

[0102] S6, cycle and termination: according to the data obtained by network updating, it is judged whether the termination condition is reached, if the termination condition is not reached, the episode number is updated, and the training is continued back to S2, if the termination condition is reached, the training is ended, and the training result is output.

[0103] It should be noted that in the step S1, the parameter noise is initialized For increasing the exploratory of the policy in the training process; initialize the experience replay pool R for storing the experience data in the training process; shake the policy network to improve the stability and efficiency of the training.

[0104] It should be noted that in the step S5, the purpose of updating the parameters of the target network using soft update policy is to maintain the stability of the target network.

[0105] It should be noted that in the step S6, according to the data obtained by network updating, the training round number, performance index and the like can be used to judge whether the termination condition is reached. For example: using the training round number: when the preset maximum training round number is reached, the training will stop when the training round number is set to 1000;

[0106] Using performance index: the performance index of the model reaches or exceeds a certain preset threshold. For example: in the task of automatic driving, if the average reward of the model reaches a higher level, or the loss value decreases to a lower range, it can be considered that the model has been trained well enough, and the training will stop.

[0107] In step one, the parameters of the critic and actor network optimizers are initialized, a batch size variable BS and a learning rate variable lr are initialized, and the batch size variable BS is set to 64, the initial value of the learning rate is lr = 0.001, and the current learning rate is used to update the parameters of the critic network and the actor network each time.

[0108] In step three, the number of randomly sampled samples is the same as the number of batch size variables BS, both of which are 64.

[0109] In the S3 step, normalization processing is also performed on each layer of the neural network to ensure that the influence of noise on each layer is consistent.

[0110] The parameters of the critic network are updated using the gradient descent method The steps include:

[0111] Step 1, calculate the loss: use the mean square error (MSE) loss function to calculate the difference L between the Q value (Q(S i , a i )) predicted by the current network and the target value (y i );

[0112] The calculation formula is:

[0113] Step 2, calculate the gradient of the loss function L with respect to the parameters of the critic network , denoted as

[0114] where

[0115] Step 3, update the parameters of the critic network using the gradient descent method where α is the learning rate.

[0116] The parameters θ of the actor network are updated using the gradient update strategy, including the following operations:

[0117] Step 1, design the target function: J(θ) = E[Q(S, π(S))];

[0118] Step 2, calculate the gradient of the actor network parameters θ using the policy gradient method, denoted as

[0119] Step 3, use the chain rule to expand the policy gradient:

[0120]

[0121] Step 4, update the parameters θ of the actor network using the gradient ascent method:

[0122]

[0123] wherein a is a learning rate.

[0124] Referring to Figs. 1-3 As shown, it includes an initialization unit, an experience collection and storage unit, a network training and updating unit, a learning rate adjustment unit, and a model evaluation and optimization unit, wherein:

[0125] The initialization unit is responsible for setting the initial parameters of the critic network and the actor network, and configuring the optimizer, the experience replay pool, the learning rate, and the batch size.

[0126] It should be noted that in the initialization unit, the Adam optimizer is configured for subsequent network parameter updates, and the learning rate and batch size are set to the initial value of the learning rate lr = 0.001 and the batch size BS value is 64.

[0127] The experience collection and storage unit interacts with the autonomous vehicle and environment settings provided by the initialization unit, collects experiences and stores them in the experience replay pool, and continuously provides new experience data for network training.

[0128] It should be noted that the interaction with the autonomous vehicle and environment settings provided by the initialization unit simulates real driving scenarios.

[0129] It should be noted that the experience data generated during the interaction between the vehicle and the environment, such as vehicle state, environmental information, actions taken, and rewards obtained, is collected.

[0130] The network training and updating unit is used to update the parameters of the critic network and the actor network, so that it is periodically run, and the performance of the autonomous vehicle is adjusted after each update.

[0131] The learning rate adjustment unit is used to update the global learning rate variable to control the subsequent network training process.

[0132] It should be noted that the adjustment of the learning rate helps the model to quickly converge in the early stage of training and fine-tune in the later stage to avoid overfitting or underfitting.

[0133] The model evaluation and optimization unit is responsible for regularly evaluating the trained model and adjusting the learning rate, batch size, and network structure based on the evaluation results.

[0134] The network training and updating unit also includes randomly sampling a batch of experiences from the experience replay pool, updating the parameters of the critic network and the actor network using these experiences and the current learning rate, and adjusting the performance of the autonomous vehicle based on the new parameters after updating.

[0135] The model evaluation and optimization unit is periodically run, and the running period is longer than that of the network training and updating unit.

[0136] It should be noted that the historical experience data is effectively utilized through the experience replay pool and the random sampling strategy, and the training efficiency is improved.

[0137] It should be noted that the learning rate adjustment unit and the model evaluation and optimization unit can dynamically adjust according to the actual situation in the training process, ensuring that the model always maintains the best training state; the network training and updating unit can continuously learn and optimize the performance of the autonomous vehicle, making it more adaptable to complex and variable driving environments; the model evaluation and optimization unit is periodically run, ensuring that the model is comprehensively evaluated and optimized after a certain number of training cycles, further improving the stability and reliability of the model.

[0138] The circuit and control involved in the application are prior art, and will not be described in detail here.

[0139] The above is only an embodiment of the application, and does not limit the patent scope of the application, and any equivalent structure or equivalent process transformation using the content of the application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the application.

Claims

1. A method for training an end-to-end autonomous driving model, the method comprising: Comprise the following steps: Step one, collect the running parameters of the vehicle, including vehicle position, surrounding traffic conditions and its own network parameters, based on the "actor-critic" framework to establish an autonomous driving model, using the DDPG algorithm of fusion adaptive parameter space noise, the parameters of the critic network and the actor network in the DDPG algorithm are initialized; The DDPG algorithm of fusion adaptive parameter space noise comprises the following steps: S1, initialization and preparation: including randomly initializing the weights of the critic network and the actor network and , initializing the critic target network and the actor target network using the weights and the weights respectively, obtaining and respectively, at this time, initializing the parameter noise and the experience replay pool R, and jittering the policy network; S2, training cycle and execution: according to the initialized and prepared parameters, set the initial episode = 1, start the global training cycle, and at the beginning of each episode, receive the initial observation state ; S3, policy selection and execution: in the process of training cycle, sample noise variable ψ from noise distribution, add noise variable ψ to actor network: , after adding, the post-jitter network is: , at this time, select action , execute according to the selected action , observe the next state , the reward obtained and whether the termination state is reached; In S3, each layer of the neural network is also normalized to ensure that the noise has a consistent impact on each layer; S4. Based on the data from strategy selection and execution, record (…). ), and will experience ( The experience is stored in the experience replay pool R, and a batch of experience is randomly sampled from the experience replay pool R for training. S5, network update: according to the experience obtained after random sampling, the following operations are performed: Operation one, calculate the target value using the critic target network: ） wherein, is an instant reward, is a discount factor, is a target network predicted Q-value; Operation II, updating the parameters of the critic network using gradient descent ; Operation three, updating the parameters of the actor network using a gradient update strategy ; comprising the following operations: Step 1, design objective function: J( ) = E[Q(S, ) ]; Step 2, calculate the gradient of the actor network parameters using the policy gradient method , denoted as J( ). Step3, use the chain rule to expand the policy gradient; Step 4, updating the parameters of the actor network using the gradient ascent method : wherein, is the learning rate; Operation four, update the parameters of the target network using the soft update policy, so that: wherein is 0.001 ; The specific operation mode is to copy the parameters of the critic network and the actor network to the corresponding target network to realize stable update of the target network; S6, cycle and termination: according to the data obtained by network updating, it is judged whether the termination condition is reached, if the termination condition is not reached, the episode number is updated, and the training is continued back to S2, if the termination condition is reached, the training is ended, and the training result is outputted; Initialize the parameters of the critic and actor network optimizer, initialize a batch size variable BS and a learning rate variable lr, set the batch size variable BS value to 64, the initial value of the learning rate is lr=0.001, and use the current learning rate to update the parameters of the critic network and the actor network each time; Step two, according to the initialized parameters of the critic network and the actor network, select the Adam optimizer and initialize the experience replay pool, store the experience of the autonomous driving vehicle interacting with the environment, set the initial learning rate and determine the batch size, wherein the environment includes road environment and traffic environment; Step three, randomly sample a batch of experiences with a number of BS from the experience replay pool for training, and update the parameters of the critic network and the actor network using the batch of experiences and the current learning rate lr; Step four, adjust the performance of the autonomous driving vehicle according to the updated network parameters, including updating the perception, decision and planning of the vehicle to the environment; Step five, according to the required adjusted performance, set the decay rate DR and the decay period DS, at the end of each training period or after reaching a certain number of training steps, calculate a new learning rate according to the preset exponential decay strategy; Step six, update the global variable lr to nlr according to the calculated new learning rate nlr; Step seven, continue the next training period or step using the updated learning rate nlr and network parameters, and periodically evaluate the trained model, and adjust the learning rate, batch size and network structure according to the evaluation result. 2.The end-to-end autonomous driving model training method of claim 1, wherein, In the step three, the number of random sampling and the number of batch size variable BS are the same, both are 64. 3.The end-to-end autonomous driving model training method of claim 1, wherein, updating the parameters of the critic network using a gradient descent method comprising the steps of: Step1, calculate the loss: use the mean square error loss function to calculate the difference between the Q value predicted by the current network and the target value; Step 2, compute the gradient of the loss function L with respect to the critic network parameters ; Step 3, using gradient descent method to update the parameters of the critic network.

4. A device suitable for the end-to-end autonomous driving model training method according to any one of claims 1-3, characterized in that, The device includes an initialization unit responsible for setting the initial parameters of the critic network and the actor network, as well as configuring the optimizer, experience replay pool, learning rate and batch size. Experience collection and storage unit, using the automatic driving vehicle and environment provided by the initialization unit to interact, collect experience and store it in the experience replay pool, continuously provide new experience data for network training; Network training and updating unit, used to update the parameters of the critic network and the actor network, so that it runs periodically, and adjusts the performance of the autonomous driving vehicle after each update; Learning rate adjustment unit, used to update the global learning rate variable to control the subsequent network training process; Model evaluation and optimization unit, responsible for regularly evaluating the trained model and adjusting the learning rate, batch size and network structure according to the evaluation results.

5. The apparatus of claim 4, wherein, The network training and updating unit also includes randomly sampling a batch of experiences from the experience replay pool, using these experiences and the current learning rate to update the parameters of the critic network and the actor network, and adjusting the performance of the autonomous driving vehicle according to the new parameters after updating.

6. The apparatus of claim 4, wherein, The model evaluation and optimization unit is periodically run, and the running period is longer than that of the network training and updating unit.

Images