End-to-end automatic driving model training method and device

Abstract

The invention provides an end-to-end automatic driving model training method and device, and the method comprises the steps: collecting the operation parameters of a vehicle, building an automatic driving model based on an actor-commentator frame, employing a DDPG algorithm fusing adaptive parameter space noise, and carrying out the training of an end-to-end automatic driving model. Initializing parameters of a commentator network and an actor network in the DDPG algorithm; an Adam optimizer is selected, an experience playback pool is initialized, interaction experience of the intelligent agent and the environment is stored, an initial learning rate is set, and the size of a batch is determined; updating parameters of the commentator network and the actor network by using the experience in the experience playback pool; according to the end-to-end automatic driving model training method and device provided by the invention, the learning rate and the batch size are adjusted to be proper, the convergence speed of the model is accelerated, and the learning rate and the batch size in hyper-parameters are adjusted, so that the model is more stable in the training process.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to an end-to-end autonomous driving model training method and device. Background Art

[0002] Autonomous driving, also known as unmanned driving, means that without the control of the driver, the vehicle can complete environmental perception and navigation and smoothly reach the destination. It can perceive and understand the surrounding environment through the vehicle's own equipment, and then automatically plan the driving route according to the user's needs and perform motion control to achieve a safe, comfortable and efficient riding experience.

[0003] Self-driving cars usually have multiple functions, such as automatic driving, automatic speed change, automatic braking, automatic monitoring of the surrounding environment, automatic lane change, automatic steering, automatic signal reminders, and networked automatic driving assistance functions. Self-driving cars use on-board sensors to perceive the vehicle's surrounding environment, including information such as roads, vehicle positions and obstacles, and then control the vehicle's steering and speed based on this information, allowing the vehicle to travel safely and reliably on the road.

[0004] The existing end-to-end autonomous driving model training method and device adopt the "actor-critic" framework. The actor network is responsible for generating driving action strategies in the autonomous driving model. It calculates the driving actions that should be taken based on the current environmental state, such as the vehicle's position, surrounding traffic conditions, and its own network parameters. For example, the vehicle should turn, accelerate, or brake at this time. The actor network is a deep neural network whose input is the environmental state and output is the driving action. During the training process, the actor network will continuously adjust its own parameters based on the feedback, that is, the evaluation results of the critic network, to optimize the driving strategy;

[0005] The critic network is responsible for evaluating the driving actions generated by the actor network in the autonomous driving model. It calculates the value or quality of the action based on the current environmental state, the driving action generated by the actor network, and the subsequent reward or penalty signal. The input of the critic network includes the environmental state, the driving action, and the subsequent state or reward signal, and the output is the value evaluation of the action. During the training process, the critic network will continuously adjust its own parameters according to the real reward signal and its own prediction error to improve the accuracy of the value evaluation.

[0006] The existing "actor-critic" framework is usually combined with the DDPG algorithm. When used, the parameters of the neural network strategy will change during the training process, causing the sensitivity of the strategy weights to change accordingly, making it difficult to predict the next action of the strategy, that is, the parameter sensitivity problem of the "actor-critic" framework. In addition, when combined with the DDPG algorithm, there are multiple hyperparameters, such as learning rate and batch size. There is mutual influence between the hyperparameters, and multiple hyperparameters are difficult to adjust, resulting in instability in the model during training, that is, the model has problems such as gradient explosion, gradient disappearance, or training process oscillation during training.

[0007] Therefore, it is necessary to provide a new end-to-end autonomous driving model training method and device to solve the above technical problems. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides an end-to-end autonomous driving model training method and device.

[0009] The end-to-end autonomous driving model training method provided by the present invention comprises the following steps:

[0010] Step 1: Collect the vehicle's operating parameters, including the vehicle's location, surrounding traffic conditions, and its own network parameters, establish an autonomous driving model based on the "actor-critic" framework, and use the DDPG algorithm with adaptive parameter space noise to initialize the parameters of the critic network and actor network in the DDPG algorithm. Among them, the network parameters include the steering, acceleration, and braking parameters of the vehicle during driving.

[0011] Step 2: Based on the parameters of the initialized critic network and actor network, select the Adam optimizer and initialize the experience replay pool to store the experience of the agent's interaction with the environment, set the initial learning rate, and determine the batch size; the agent includes an autonomous driving vehicle; the environment includes roads and traffic.

[0012] Step 3: Randomly sample a batch of experiences of size BS from the experience replay pool for training, and use this batch of experiences and the current learning rate lr to update the parameters of the critic network and the actor network.

[0013] Step 4: Adjust the performance of the autonomous vehicle based on the updated network parameters, including updating the vehicle's perception, decision-making, and planning of the environment.

[0014] Step 5: Set the decay rate DR and decay period DS according to the performance after adjustment. After each training cycle ends or a certain number of training steps are reached, calculate the new learning rate according to the preset exponential decay strategy.

[0015] Step 6. Update the global variable lr to nlr based on the calculated new learning rate.

[0016] Step 7. Use the updated learning rate nlr and network parameters to continue training for the next training cycle or step, and regularly evaluate the trained model. Adjust the learning rate, batch size, and network structure based on the evaluation results to optimize the performance of autonomous driving.

[0017] Preferably, the DDPG algorithm integrating adaptive parameter space noise comprises the following steps:

[0018] S1. Initialization and preparation: including randomly initializing the weights θ of the critic network and the actor network Q and θ π , using weight θ Q and weight θ π Initialize the critic target network and the actor target network respectively, and get as well as At this time, the initialization parameter noise And the experience replay pool R, and jitter strategy network;

[0019] S2, training loop and execution: according to the parameters after initialization and preparation, set the initial episode = 1, start the global training loop, and receive the initial observation state S at the beginning of each episode t ;

[0020] S3, strategy selection and execution: During the training cycle, sample the noise variable ψ from the noise distribution and add the noise variable ψ to the actor network: θ Q =θ Q +ψ, After adding the jitter network: Now select action a t , according to the selected action a t Execute and observe the next state S after execution t+1 , the reward r t and whether the termination state has been reached;

[0021] S4. According to the data in the strategy selection and execution, record (S t , a t , r t , S t+1 ), and the experience (S t , a t , r t , S t+1 ) is stored in the experience replay pool R, and a batch of experiences is randomly sampled from the experience replay pool R for training;

[0022] S5. Network update: Based on the experience gained from random sampling, the following operations are performed:

[0023] Operation 1: Calculate the target value using the critic target network:

[0024] y i =r i +γ*Q 1 (S i+1 ,π 1 (S i+1 ))

[0025] Among them, r i is the immediate reward, γ is the discount factor, Q 1 (S i+1 ,π 1 (S i+1 ) is the Q value predicted by the target network;

[0026] Operation 2: Update the parameters of the critic network using gradient descent

[0027] Operation 3: Update the parameters θ of the actor network using the gradient update strategy;

[0028] Operation 4: Use the soft update strategy to update the parameters of the target network so that:

[0029] τθ+(1-τ)θ 1 →θ 1

[0030]

[0031] Among them, τ is 0.001;

[0032] The specific operation method is: copy the parameters of the critic network and the actor network to the corresponding target network to achieve stable update of the target network;

[0033] S6, loop and termination: Based on the data obtained from the network update, determine whether the termination condition is met. If the termination condition is not met, update the number of episodes and return to S2 to continue training. If the termination condition is met, end the training and output the training results.

[0034] Preferably, in the step 1, the parameters of the critic and actor network optimizers are initialized, a batch size variable BS and a learning rate variable lr are initialized, and the batch size variable BS value is set to 64, the initial value of the learning rate is lr=0.001, and the current learning rate is used each time the parameters of the critic network and the actor network are updated.

[0035] Preferably, in step three, the number of random samples is the same as the number of batch size variables BS, both of which are 64.

[0036] Preferably, in the step S3, each layer of the neural network is also normalized to ensure that the impact of noise on each layer is consistent.

[0037] Preferably, the parameters of the critic network are updated using the gradient descent method. The following steps are involved:

[0038] Step 1. Calculate the loss: Use the mean square error (MSE) loss function to calculate the Q value (Q(S i , a i )) and the target value (y i ) between the difference L;

[0039] Step 2. Calculate the loss function L for the critic network parameters The gradient of

[0040] Step 3: Update the parameters of the critic network using gradient descent Where α is the learning rate.

[0041] Preferably, the updating of the parameter θ of the actor network using the gradient updating strategy comprises the following operations:

[0042] Step 1. Design objective function: J(θ)=E[Q(S,π(S))];

[0043] Step 2: Use the policy gradient method to calculate the gradient of the actor network parameter θ, denoted as

[0044] Step 3. Use the chain rule to expand the policy gradient;

[0045] Step 4. Update the parameters θ of the actor network using the gradient ascent method:

[0046]

[0047] Among them, α is the learning rate.

[0048] Preferably, it includes an initialization unit, an experience collection and storage unit, a network training and updating unit, a learning rate adjustment unit, and a model evaluation and optimization unit, wherein:

[0049] The initialization unit is responsible for setting the initial parameters of the critic network and the actor network, as well as configuring the optimizer, experience replay pool, learning rate, and batch size;

[0050] The experience collection and storage unit interacts with the autonomous driving vehicle and environment settings provided by the initialization unit, collects experience and stores it in the experience playback pool, and continuously provides new experience data for network training;

[0051] A network training and updating unit, which is used to update the parameters of the critic network and the actor network so that they can run periodically and adjust the performance of the autonomous vehicle after each update;

[0052] The learning rate adjustment unit is used to update the global learning rate variable and control the subsequent network training process;

[0053] The model evaluation and optimization unit is responsible for regularly evaluating the trained model and adjusting the learning rate, batch size, and network structure based on the evaluation results.

[0054] Preferably, the network training and updating unit further comprises randomly sampling a batch of experiences from the experience replay pool, using these experiences and the current learning rate to update the parameters of the critic network and the actor network, and adjusting the performance of the autonomous driving vehicle according to the updated new parameters.

[0055] Preferably, the model evaluation and optimization unit operates periodically, and the operating period is longer than that of the network training and updating unit.

[0056] Compared with related technologies, the end-to-end autonomous driving model training method and device provided by the present invention have the following beneficial effects:

[0057] 1. This method can adaptively adjust parameter space noise, which helps alleviate the parameter sensitivity problem of the "actor-critic" framework, making the model more stable during training and reducing the occurrence of problems such as gradient explosion, gradient disappearance, or oscillation during training. In addition, by setting the decay rate DR and decay period DS, and using an exponential decay strategy to calculate a new learning rate, the learning rate can be dynamically adjusted according to the training progress and model performance, which helps the model converge better during training and improves training efficiency.

[0058] 2. This method uses the DDPG algorithm with adaptive parameter space noise, which can make full use of multi-source information, that is, use the vehicle position and surrounding traffic conditions for decision-making and planning, which helps to improve the generalization ability of the model and enable it to better adapt to different roads and traffic scenarios;

[0059] 3. This method can adjust the learning rate and batch size in multiple parameters, adopt an exponential decay strategy, reduce the learning rate according to a fixed decay rate DR, and gradually adjust the initial learning rate and decay rate according to the convergence speed and stability of the model, that is, the learning rate lr controls the step size of the parameter update. A smaller learning rate can reduce the oscillation during parameter update, improve the stability of the DDPG algorithm when in use, and by adjusting to an appropriate learning rate, the convergence speed of the model can be accelerated, and by adjusting to an appropriate batch size, the accuracy and efficiency of each gradient update can be improved. The learning rate and batch size in the hyperparameters are adjusted to make the model more stable during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flowchart of the end-to-end autonomous driving model training method and device provided by the present invention;

[0061] Figure 2 Flowchart of the DDPG algorithm for integrating adaptive parameter space noise;

[0062] Figure 3 The figure shows the flow chart of the end-to-end autonomous driving model training device. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0064] Please refer to Figures 1 to 3 ,in, Figure 1 A flowchart of the end-to-end autonomous driving model training method and device provided by the present invention; Figure 2 Flowchart of the DDPG algorithm for integrating adaptive parameter space noise; Figure 3 The figure shows the flow chart of the end-to-end autonomous driving model training device.

[0065] In the specific implementation process, Figures 1 to 3 As shown, the following steps are included:

[0066] Step 1: Collect the vehicle's operating parameters, including the vehicle's position, surrounding traffic conditions, and its own network parameters, establish an autonomous driving model based on the "actor-critic" framework, and use the DDPG algorithm that integrates adaptive parameter space noise to initialize the parameters of the critic network and the actor network in the DDPG algorithm; among them, its own network parameters include the steering, acceleration, and braking parameters of the vehicle during driving.

[0067] It should be noted that in the initialization phase, adaptive parameter space noise is introduced to provide the actor network with sufficient exploration space in the early stage of training to avoid falling into the local optimal solution. As the training progresses, the noise will gradually decrease to ensure that the model can converge to a better strategy;

[0068] It should be noted that when initializing the parameters, a reasonable parameter initialization method, such as Xavier initialization, is used to ensure that the weights of the network can maintain a proper distribution during the training process to avoid the problem of gradient disappearance or explosion.

[0069] Step 2: According to the parameters of the initialized critic network and actor network, select the Adam optimizer and initialize the experience replay pool to store the experience of the agent's interaction with the environment, set the initial learning rate, and determine the batch size; the agent includes the autonomous driving vehicle; the environment includes roads and traffic;

[0070] It should be noted that the Adam optimizer combines the advantages of the momentum method and the RMSprop optimization algorithm, and can adaptively adjust the learning rate to improve training efficiency and model performance;

[0071] It should be noted that in the experience replay pool management, priority sampling is adopted, historical experience data is utilized, and the learning rate and batch size are dynamically adjusted according to the actual situation during training to balance the training speed and model performance.

[0072] Step 3: Randomly sample a batch of BS experiences from the experience replay pool for training, and use this batch of experiences and the current learning rate lr to update the parameters of the critic network and the actor network;

[0073] It should be noted that after sampling, the data can be preprocessed, including normalization, to improve training efficiency and model performance.

[0074] Step 4: Adjust the performance of the autonomous vehicle based on the updated network parameters, including updating the vehicle's perception, decision-making, and planning of the environment;

[0075] It should be noted that when adjusting the performance of autonomous vehicles, a feedback mechanism can be introduced to further adjust and optimize the model based on the evaluation results; for example, when the model performs poorly in terms of safety, relevant training and data collection can be strengthened.

[0076] Step 5: Set the decay rate DR and decay period DS according to the performance after adjustment. After each training cycle ends or a certain number of training steps are reached, calculate the new learning rate according to the preset exponential decay strategy.

[0077] It should be noted that when using the learning rate decay strategy, you can also use cosine annealing decay, segmented decay, etc. to observe the impact on model performance;

[0078] Among them, the cosine annealing decay strategy is to reduce the learning rate according to the law of the cosine function after each epoch or step. This strategy can maintain a large learning rate in the early stage of training, and gradually reduce the learning rate in the later stage of training to achieve a smooth decay effect;

[0079] The segmented decay is to decay the learning rate at a fixed ratio after each fixed epoch or step interval.

[0080] It should be noted that the decay rate DR and the decay period DS are set according to the performance after adjustment, and the decay rate DR and the decay period DS are dynamically adjusted according to the actual situation during the training process to more flexibly control the decay process of the learning rate.

[0081] Step 6: Update the global variable lr to nlr according to the calculated new learning rate;

[0082] Step 7. Use the updated learning rate nlr and network parameters to continue training for the next training cycle or step, and regularly evaluate the trained model. Adjust the learning rate, batch size, and network structure based on the evaluation results to optimize the performance of autonomous driving.

[0083] It should be noted that when evaluating the model, multiple model evaluation methods can be used, such as offline testing, online testing, and cross-validation, to comprehensively evaluate the performance of the model.

[0084] It should be further explained that the experience replay mechanism can make full use of historical experience data for training, improve data utilization and training efficiency. At the same time, the random sampling method can break the time correlation between data, which is conducive to the stability of training.

[0085] It should be further explained that by continuously updating network parameters and adjusting parameters such as learning rate, the performance of autonomous driving vehicles, including perception, decision-making and planning, can be gradually optimized, which will help improve the safety and comfort of autonomous driving.

[0086] References Figure 1 to Figure 2 As shown in Figure 1, the DDPG algorithm integrating adaptive parameter space noise includes the following steps:

[0087] S1. Initialization and preparation: including randomly initializing the weights θ of the critic network and the actor network Q and θ π , using weight θ Q and weight θ πInitialize the critic target network and the actor target network respectively, and get as well as At this time, the initialization parameter noise And the experience replay pool R, and jitter strategy network;

[0088] S2, training loop and execution: according to the parameters after initialization and preparation, set the initial episode = 1, start the global training loop, and receive the initial observation state S at the beginning of each episode t ;

[0089] S3, strategy selection and execution: During the training cycle, sample the noise variable ψ from the noise distribution and add the noise variable ψ to the actor network: θ Q =θ Q +ψ, After adding the jitter network: Now select action a t , according to the selected action a t Execute and observe the next state S after execution t+1 , the reward r t and whether the termination state has been reached;

[0090] S4. According to the data in the strategy selection and execution, record (S t , a t , r t , S t+1 ), and the experience (S t , a t , r t , S t+1 ) is stored in the experience replay pool R, and a batch of experiences is randomly sampled from the experience replay pool R for training;

[0091] S5. Network update: Based on the experience gained from random sampling, the following operations are performed:

[0092] Operation 1: Calculate the target value using the critic target network:

[0093] y i =r i +γ*Q 1 (S i+1 ,π 1 (S i+1 ))

[0094] Among them, r i is the immediate reward, γ is the discount factor, Q 1 (S i+1 ,π 1 (S i+1 ) is the Q value predicted by the target network;

[0095] Operation 2: Update the parameters of the critic network using gradient descent

[0096] Operation 3: Update the parameters θ of the actor network using the gradient update strategy;

[0097] Operation 4: Use the soft update strategy to update the parameters of the target network so that:

[0098] τθ+(1-τ)θ 1 →θ 1

[0099]

[0100] Among them, τ is 0.001;

[0101] The specific operation method is: copy the parameters of the critic network and the actor network to the corresponding target network to achieve stable update of the target network;

[0102] S6, loop and termination: Based on the data obtained from the network update, determine whether the termination condition is met. If the termination condition is not met, update the number of episodes and return to S2 to continue training. If the termination condition is met, end the training and output the training results.

[0103] It should be noted that in step S1, the initialization parameter noise Used to increase the exploratory nature of the strategy during training; initialize the experience replay pool R to store the experience data during training; jitter the strategy network to improve the stability and efficiency of training.

[0104] It should be noted that, in step S5, the purpose of using the soft update strategy to update the parameters of the target network is to maintain the stability of the target network.

[0105] It should be noted that in step S6, based on the data obtained from the network update, the number of training rounds, performance indicators, etc. can be used to determine whether the termination condition is reached. For example, the number of training rounds is used: the preset maximum number of training rounds is reached. When the training is set to 1000 rounds, the training will stop when this number of rounds is reached;

[0106] Using performance indicators: The performance indicators of the model reach or exceed a preset threshold. For example, in an autonomous driving task, if the average reward of the model reaches a high level, or the loss value drops to a low range, then the model can be considered to have been trained well enough and training will stop.

[0107] In step 1, initialize the parameters of the critic and actor network optimizers, initialize a batch size variable BS and a learning rate variable lr, and set the batch size variable BS value to 64, the initial value of the learning rate lr = 0.001, and use the current learning rate each time the parameters of the critic network and the actor network are updated.

[0108] In step 3, the number of random samples is the same as the number of batch size variables BS, which is 64.

[0109] In step S3, each layer of the neural network is also normalized to ensure that the impact of noise on each layer is consistent.

[0110] Update the parameters of the critic network using gradient descent The following steps are involved:

[0111] Step 1. Calculate the loss: Use the mean square error (MSE) loss function to calculate the Q value (Q(S i , a i )) and the target value (y i ) between the difference L;

[0112] The calculation formula is:

[0113] Step 2. Calculate the loss function L for the critic network parameters The gradient of

[0114] in,

[0115] Step 3: Update the parameters of the critic network using gradient descent Where α is the learning rate.

[0116] Update the parameters θ of the actor network using the gradient update strategy, which includes the following operations:

[0117] Step 1. Design objective function: J(θ)=E[Q(S,π(S))];

[0118] Step 2: Use the policy gradient method to calculate the gradient of the actor network parameter θ, denoted as

[0119] Step 3. Use the chain rule to expand the policy gradient:

[0120]

[0121] Step 4. Update the parameters θ of the actor network using the gradient ascent method:

[0122]

[0123] Among them, α is the learning rate.

[0124] refer to Figures 1 to 3 As shown, it includes an initialization unit, an experience acquisition and storage unit, a network training and update unit, a learning rate adjustment unit, and a model evaluation and optimization unit, wherein:

[0125] The initialization unit is responsible for setting the initial parameters of the critic network and the actor network, as well as configuring the optimizer, experience replay pool, learning rate, and batch size;

[0126] It should be noted that in the initialization unit, the Adam optimizer is configured for subsequent network parameter updates, and the learning rate and batch size are set to have an initial value of lr=0.001 and a batch size BS value of 64.

[0127] The experience collection and storage unit interacts with the autonomous driving vehicle and environment settings provided by the initialization unit, collects experience and stores it in the experience playback pool, and continuously provides new experience data for network training;

[0128] It should be noted that the autonomous driving vehicle and environment settings provided by the initialization unit are used to interact to simulate real driving scenarios.

[0129] It should be noted that the experience data generated by the vehicle in the process of interacting with the environment is collected, such as vehicle status, environmental information, actions taken and rewards obtained.

[0130] A network training and updating unit, which is used to update the parameters of the critic network and the actor network so that they can run periodically and adjust the performance of the autonomous vehicle after each update;

[0131] The learning rate adjustment unit is used to update the global learning rate variable and control the subsequent network training process;

[0132] It should be noted that the adjustment of the learning rate helps the model converge quickly in the early stage of training and fine-tune it in the later stage to avoid overfitting or underfitting.

[0133] The model evaluation and optimization unit is responsible for regularly evaluating the trained model and adjusting the learning rate, batch size, and network structure based on the evaluation results.

[0134] The network training and updating unit also includes randomly sampling a batch of experiences from the experience replay pool, using these experiences and the current learning rate to update the parameters of the critic network and the actor network, and adjusting the performance of the autonomous driving vehicle according to the updated new parameters.

[0135] The model evaluation and optimization unit runs periodically, and its running cycle is longer than that of the network training and update unit.

[0136] It should be noted that the experience replay pool and random sampling strategy effectively utilize historical experience data and improve training efficiency.

[0137] It should be noted that the learning rate adjustment unit and the model evaluation and optimization unit can be dynamically adjusted according to the actual situation during the training process to ensure that the model always remains in the best training state; the network training and update unit can continuously learn and optimize the performance of the autonomous driving vehicle to make it more adaptable to complex and changeable driving environments; the model evaluation and optimization unit ensures that the model is comprehensively evaluated and optimized after a certain number of training cycles through periodic operation, further improving the stability and reliability of the model.

[0138] The circuits and controls involved in the present invention are all prior art and will not be described in detail here.

[0139] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An end-to-end autonomous driving model training method, characterized in that: The steps include: Step 1: Collect the vehicle's operating parameters, including the vehicle's location, surrounding traffic conditions, and its own network parameters, establish an autonomous driving model based on the "actor-critic" framework, and use the DDPG algorithm with adaptive parameter space noise to initialize the parameters of the critic network and actor network in the DDPG algorithm; Step 2: According to the parameters of the initialized critic network and actor network, select the Adam optimizer and initialize the experience replay pool to store the experience of the autonomous driving vehicle interacting with the environment, set the initial learning rate and determine the batch size, where the environment includes the road environment and the traffic environment; Step 3: Randomly sample a batch of BS experiences from the experience replay pool for training, and use this batch of experiences and the current learning rate lr to update the parameters of the critic network and the actor network; Step 4: Adjust the performance of the autonomous vehicle based on the updated network parameters, including updating the vehicle's perception, decision-making, and planning of the environment; Step 5: Set the decay rate DR and decay period DS according to the performance after adjustment. After each training cycle ends or a certain number of training steps are reached, calculate the new learning rate according to the preset exponential decay strategy. Step 6: Update the global variable lr to nlr according to the calculated new learning rate; Step 7. Use the updated learning rate nlr and network parameters to continue training for the next training cycle or step, and regularly evaluate the trained model, and adjust the learning rate, batch size, and network structure based on the evaluation results.

2. The end-to-end autonomous driving model training method according to claim 1, characterized in that: The DDPG algorithm integrating adaptive parameter space noise comprises the following steps: S1. Initialization and preparation: including randomly initializing the weights θ of the critic network and the actor network Q and θ π , using weight θ Q and weight θ π Initialize the critic target network and the actor target network respectively, and get as well as At this time, the initialization parameter noise And the experience replay pool R, and jitter strategy network; S2, training loop and execution: according to the parameters after initialization and preparation, set the initial episode = 1, start the global training loop, and receive the initial observation state S at the beginning of each episode t ; S3, strategy selection and execution: During the training cycle, sample the noise variable ψ from the noise distribution and add the noise variable ψ to the actor network: θ Q =θ Q +ψ, After adding the jitter network: Now select action a t , according to the selected action a t Execute and observe the next state S after execution t+1 , the reward r t and whether the termination state has been reached; S4. According to the data in the strategy selection and execution, record (S t , a t , r t , S t+1 ), and the experience (S t , a t , r t , S t+1 ) is stored in the experience replay pool R, and a batch of experiences is randomly sampled from the experience replay pool R for training; S5. Network update: Based on the experience gained from random sampling, the following operations are performed: Operation 1: Calculate the target value using the critic target network: y i =r i +γ*Q 1 (S i+1 ,p 1 (S i+1 )) Among them, r i is the immediate reward, γ is the discount factor, Q 1 (S i+1 ,π 1 (S i+1 ) is the Q value predicted by the target network; Operation 2: Update the parameters of the critic network using gradient descent Operation 3: Update the parameters θ of the actor network using the gradient update strategy; Operation 4: Use the soft update strategy to update the parameters of the target network so that: τθ+(1-τ)θ 1 →θ 1 Among them, τ is 0.001; The specific operation method is: copy the parameters of the critic network and the actor network to the corresponding target network to achieve stable update of the target network; S6, loop and termination: Based on the data obtained from the network update, determine whether the termination condition is met. If the termination condition is not met, update the number of episodes and return to S2 to continue training. If the termination condition is met, end the training and output the training results.

3. The end-to-end autonomous driving model training method according to claim 1, characterized in that: In the step 1, the parameters of the critic and actor network optimizers are initialized, a batch size variable BS and a learning rate variable lr are initialized, and the batch size variable BS value is set to 64, the initial value of the learning rate is lr=0.001, and the current learning rate is used each time the parameters of the critic network and the actor network are updated. The network parameters include the steering, acceleration and braking parameters of the vehicle during driving.

4. The end-to-end autonomous driving model training method according to claim 1, characterized in that: In step 3, the number of random samples is the same as the number of batch size variables BS, both of which are 64.

5. The end-to-end autonomous driving model training method according to claim 2, characterized in that: In the step S3, each layer of the neural network is also normalized to ensure that the impact of noise on each layer is consistent.

6. The end-to-end autonomous driving model training method according to claim 2, characterized in that: The parameters of the critic network are updated using the gradient descent method. The following steps are involved: Step 1. Calculate the loss: Use the mean square error loss function to calculate the difference between the Q value predicted by the current network and the target value; Step 2. Calculate the loss function L for the critic network parameters The gradient of Step 3. Update the parameters of the critic network using gradient descent.

7. The end-to-end autonomous driving model training method and device according to claim 2, characterized in that: The method of updating the parameters θ of the actor network using the gradient update strategy includes the following operations: Step 1. Design objective function: J(θ)=E[Q(S,π(S))]; Step 2: Use the policy gradient method to calculate the gradient of the actor network parameter θ, denoted as Step 3. Use the chain rule to expand the policy gradient; Step 4. Update the parameters θ of the actor network using the gradient ascent method: Among them, α is the learning rate.

8. A device for the end-to-end autonomous driving model training method according to any one of claims 1 to 7, characterized in that: The apparatus includes an initialization unit responsible for setting initial parameters of the critic network and the actor network, and configuring the optimizer, the experience replay pool, the learning rate, and the batch size; The experience collection and storage unit interacts with the autonomous driving vehicle and environment settings provided by the initialization unit, collects experience and stores it in the experience playback pool, and continuously provides new experience data for network training; A network training and updating unit, which is used to update the parameters of the critic network and the actor network so that they can run periodically and adjust the performance of the autonomous vehicle after each update; The learning rate adjustment unit is used to update the global learning rate variable and control the subsequent network training process; The model evaluation and optimization unit is responsible for regularly evaluating the trained model and adjusting the learning rate, batch size, and network structure based on the evaluation results.

9. The device according to claim 8, characterized in that The network training and updating unit also includes randomly sampling a batch of experiences from the experience replay pool, using these experiences and the current learning rate to update the parameters of the critic network and the actor network, and adjusting the performance of the autonomous driving vehicle according to the updated new parameters.

10. The device according to claim 8, characterized in that The model evaluation and optimization unit operates periodically, and the operating cycle is longer than that of the network training and updating unit.

Images