A federated reinforcement learning method for autonomous driving based on meta-learning and experience vector

Abstract

The present invention discloses a federated reinforcement learning method for autonomous driving based on meta-learning and experience vectors, comprising: constructing a heterogeneous environment for autonomous driving; constructing an Actor network, and updating the Actor network parameters through deterministic policy gradients; constructing a Critic network, and updating the Critic network parameters through a meta-learning algorithm and an error back-propagation algorithm. During the training process, experience vectors are added to integrate and update the learning experience of each autonomous driving vehicle to optimize the performance of the global strategy; during the training process, all autonomous driving vehicles regularly send local Critic network parameters, Actor network parameters, and local experience vectors to a central server for aggregation. The present invention solves the problems of slow convergence and decreased strategy generalization ability in existing federated reinforcement learning due to dynamic changes in the environment and fluctuations in strategy performance by improving the adaptability of autonomous driving vehicles to environmental heterogeneity, thereby improving training efficiency and strategy robustness.

Description

Technical Field

[0001] The present invention relates to an autonomous driving federated reinforcement learning method based on meta-learning and experience vectors, and belongs to the technical field of autonomous driving. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, reinforcement learning, a technique that learns optimal decision-making strategies through interaction with the environment, has been widely used in fields such as robotic control, autonomous driving, and intelligent recommendation systems. However, in the complex and dynamic application scenario of autonomous driving, existing systems face several severe challenges. First, autonomous vehicles must make real-time decisions in diverse traffic environments, including coping with complex traffic conditions, pedestrian behavior, weather changes, and varying road conditions. This environmental heterogeneity makes it difficult to generalize a unified decision-making strategy across all scenarios, impacting the safety and reliability of autonomous driving systems. Furthermore, data privacy issues are becoming increasingly prominent. When collecting and processing large amounts of real-time data, autonomous driving systems must ensure that user privacy is protected, especially in multi-vehicle connected environments. This further complicates data sharing and model training.

[0003] Existing reinforcement learning techniques have achieved some success in some tasks, but they still have significant shortcomings in autonomous driving scenarios. Traditional centralized reinforcement learning methods face the problem of environmental heterogeneity when dealing with distributed systems and multi-agent tasks. The differences in the environments of different agents lead to inconsistent data distribution, which in turn affects the ability of the shared global model to adapt to all local environments. In addition, these methods often rely on large amounts of historical data for training, while the difficulty of acquiring and processing data in real time in autonomous driving is relatively high, thus limiting the application of the algorithms. Existing algorithms perform inconsistently in different environmental states and are affected by environmental changes, resulting in unstable decision-making and slow convergence. At the same time, data collection and model training in centralized training frameworks are time-consuming and labor-intensive, making it difficult to meet the real-time requirements of autonomous driving systems.

[0004] Federated learning, an emerging distributed learning paradigm, effectively protects data privacy and reduces communication overhead by allowing multiple agents to train models locally and share only model updates, not raw data. Federated reinforcement learning (FRL) introduces the concept of federated learning to the field of reinforcement learning, allowing multiple distributed agents to collaboratively train a global policy. However, FRL still faces several challenges in its application, particularly environmental heterogeneity, where different agents operate in environments that differ significantly, making it difficult to generalize the global policy across all environments. Environmental heterogeneity is a major challenge in federated reinforcement learning. Because the data collected by agents in different environments is inconsistently distributed, the shared global model may not be adaptable to all local environments. This heterogeneity manifests itself in differences in state space, action space, reward function, and state transition probabilities. This heterogeneity makes it difficult for agents to learn a unified global policy, severely impacting the convergence speed and generalization ability of federated reinforcement learning.

[0005] Meta-learning, a technique that "learns how to learn," can train models on multiple tasks to rapidly adapt to new tasks. By learning commonalities across multiple tasks, meta-learning enables models to rapidly adapt to new tasks. Therefore, it is used in federated learning to train personalized models and reduce performance variations across different clients. This combined approach of federated learning and meta-learning is called federated meta-learning. While meta-learning has been widely studied for addressing data heterogeneity in federated learning, directly applying its approach to federated reinforcement learning still faces several challenges. First, the dynamic and uncertain environment in federated reinforcement learning (FRL) not only affects the data distribution but also directly influences the agent's decision-making and learning outcomes. This requires meta-learning methods to possess stronger dynamic adaptability. Traditional meta-learning, which primarily addresses static data heterogeneity, struggles to cope with this complexity. Second, the performance of policies in FRL often fluctuates with environmental changes. Even the same policy may perform differently under different environmental conditions. This requires meta-learning methods to handle performance fluctuations caused by policy changes and ensure model stability. Furthermore, data generation in FRL depends on the current policy, a typical exploration-exploitation problem, resulting in dynamic adjustments in the data distribution as the agent's policy changes. The static data processing model of traditional federated meta-learning methods is unable to quickly adapt to these dynamic changes. Therefore, a federated reinforcement learning method is needed that can effectively cope with the heterogeneity of autonomous driving environments, improve the generalization ability of autonomous driving policies and improve the robustness of models, and meet the needs of data privacy protection and distributed collaborative learning in practical applications. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an autonomous driving federated reinforcement learning method based on meta-learning and experience vectors, which solves the strategy generalization and adaptability problems caused by the heterogeneity of the participating node environments in the federated learning process, and improves the reliability and robustness of the autonomous driving strategy.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] A federated reinforcement learning method for autonomous driving based on meta-learning and experience vectors includes the following steps:

[0009] Step 1: Build a heterogeneous virtual simulation environment for each autonomous vehicle and initialize the global Actor parameter φ 0 , global critic parameter w 0 and the global experience vector and sent to each autonomous vehicle;

[0010] Step 2: The local Actor network of each autonomous vehicle loads the global Actor parameter φ 0 , while φ 0 Copy to the target Actor network; the local Critic network of each autonomous vehicle loads the global Critic parameter w 0 , and w 0 Copy to the target critic network; initialize the experience replay pool of each autonomous driving vehicle and start local training;

[0011] Step 3: For time step t, each autonomous vehicle interacts with its environment and obtains feedback from the environment, that is, according to the current state s t Select action a t , get the execution action a t The rewards brought t and the next state s after executing the action t+1 , will (s t ,a t ,r t ,s t+1 ) is stored as an experience tuple in the experience replay pool;

[0012] Step 4: In each time step, each autonomous vehicle uses the model-independent meta-learning method Sharp-MAML to update the local critic network parameters and local experience vector based on the loss function and correlation perturbations. When the time step is an integer multiple of the local critic network federation frequency, the local critic network parameters and local experience vectors of all autonomous vehicles are aggregated respectively. The aggregated global critic parameters and global experience vectors are then sent to each autonomous vehicle to continue local training.

[0013] Step 5: Using the delay strategy, when the time step is an integer multiple of the delay strategy update frequency, the local Actor network parameters of each autonomous driving vehicle are updated through deterministic policy gradients, and the number of updates of the local Actor network parameters is recorded. When the number of updates is an integer multiple of the local Actor network federation frequency, the local Actor network parameters of all autonomous driving vehicles are aggregated, and the aggregated global Actor parameters are sent to each autonomous driving vehicle to continue local training.

[0014] Step 6: During the training process, when the time step is an integer multiple of the delay strategy update frequency, the target critic network and target actor network parameters of each autonomous driving vehicle are updated using the soft update strategy;

[0015] Step 7: When the time step reaches the preset number of training steps for federated reinforcement learning, the training is completed; each vehicle is controlled according to the global Actor network at the completion of training to achieve autonomous driving.

[0016] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0017] 1. This invention introduces meta-learning technology to provide shared initialization parameters for autonomous driving vehicles, enabling them to quickly adapt to different environments, thereby improving the generalization ability of the model and its response speed to environmental changes.

[0018] 2. The present invention designs an experience vector mechanism to integrate and update the learning experience of each autonomous driving vehicle during the training process to optimize the performance of the global strategy.

[0019] 3. The present invention solves the problems of slow convergence and reduced strategy generalization ability caused by dynamic changes in the environment and fluctuations in strategy performance in the existing technology by improving the adaptability of autonomous driving vehicles to environmental heterogeneity, thereby improving the overall training efficiency and robustness of the strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of an autonomous driving federated reinforcement learning method based on meta-learning and experience vectors of the present invention. DETAILED DESCRIPTION

[0021] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be interpreted as limiting the present invention.

[0022] Assume that there is a federated reinforcement learning system, which includes a central server and N clients. In each round of training, the central server sends the current global policy π t Broadcast to all or some of the agents, which will perform multiple local training iterations. In each iteration, the agent interacts with its environment to collect T time steps of data and optimizes its local policy using the gradient ascent method based on its local target. At the end of each round, the training results are uploaded to the server for aggregation. In a heterogeneous environment, each client agent interacts with its environment independently. These environments have the same state space S, the same action space A, the same (or different) reward function R, and different state transition functions. For example, for agent i and agent j, there exists P i (s,a)≠P j (s,a). The local training problem of each agent is modeled as a Markov decision process (MDP). Therefore, the federated reinforcement learning system can be defined as a collection of distributed MDPs. S represents a finite set of states, A represents a finite set of actions, and R i (s,a) represents the immediate reward obtained after taking action a in state s in the i-th MDP, P n (s′|s,a) represents the probability that the i-th MDP transitions from state s to s′ after taking action a, which characterizes the environment dynamics of the i-th MDP. γ∈(0,1) represents the reward discount factor.

[0023] The goal of federated reinforcement learning is to enable n agents to jointly learn a well-performing policy function or value function in n environments, thereby maximizing their expected discounted rewards in the long term. Due to privacy constraints, the n agents cannot share their collected experiences, observations, and states. Similar to federated learning, the goal of federated reinforcement learning is to find the best value estimation function Q on the state-action pairs extracted from the agent's local experience replay area. w* (s t ,a t ), local function Q w* (s t ,a t ) can be defined by the optimal Bellman equation:

[0024]

[0025] Among them, r t is the agent in the current state s t Take action t The immediate reward received after . Random variable S t+1 From the state transition probability distribution function p(s t+1 ∣s t ,at From the perspective of federated learning, federated reinforcement learning can be formulated as the following optimization problem:

[0026]

[0027] Where k represents the agent number. K represents the number of agents in the federated system, F k represents the local loss function of the kth agent. k represents the number of samples collected by the k-th agent in its local environment, l(w; x kj ) represents the loss of the local model of agent k in its local environment, and kj represents the jth sample collected by the kth agent.

[0028] Therefore, the final goal can be rewritten as:

[0029]

[0030] Among them, D + =D1∪…∪D K , represents the union of all agent data.

[0031] In the context of federated reinforcement learning, the Federated Experience Vector (FEV) extracts each agent's experience into a fixed-size experience vector. In each round of federated learning, the local experience vectors of each agent are aggregated to generate a new federated (global) experience vector. Multiple agents share processed experience representations (such as state, action, and reward) to improve collaborative learning efficiency without compromising privacy. This distributed experience sharing can accelerate model convergence and enhance overall performance.

[0032] The core idea of ​​meta-learning is to improve generalization ability by training a model on a set of tasks and enabling it to quickly adapt to new tasks. Our goal is to design a personalized variant of the FRL problem using the basic idea behind the model-agnostic meta-learning (MAML) framework. The core of this method is to find a shared initial model that can adapt to its specific local environment after a small number of gradient descent steps on the client. However, MAML currently has problems such as unstable training gradients and easy to fall into local optimal solutions. To address the above problems, a model-agnostic meta-learning method called Sharp-MAML is adopted, which optimizes model-agnostic meta-learning (MAML) by utilizing the sharpness-aware minimization (SAM) algorithm. Assuming that the computational budget for updating the model after the arrival of a new task is limited, and in this new setting, finding an initialization that performs well after being updated for this new task, may be through one or a few steps of gradient descent. If it is assumed that each user obtains an initial point and updates it using a one-step gradient descent relative to its own loss function, the federated reinforcement learning optimization problem becomes:

[0033]

[0034] according to Figure 1 , the method proposed in the present invention is elaborated in detail.

[0035] Step 1: Construct a heterogeneous environment by adding Gaussian noise to the environmental parameters of each agent's environment, so that the training environment of each agent is heterogeneous.

[0036] Step 2: Initialize the global Q network parameters w 0 In order to facilitate subsequent calculations, the two Q networks are merged into a large network, and the output contains the two Q networks' estimates of the value of the current given state and action. TD3 introduces two Critic networks to alleviate the problem of overestimation of Q values. Initialize the global Actor network parameter φ 0 , used to calculate the deterministic action a generated in a given state s. Initialize the Critic and Actor Initialize the target network (i.e. w′, φ′) and set the parameters of these target networks to be the same as the original network. The target network is used for soft update to make the training process more stable. Initialize the global federated experience vector After initialization, the central server sets the global Q network parameter w 0 , global Actor network parameter φ 0 and the federated experience vector Sent to all agents.

[0037] Step 3: All agents receive the initialized global Q network parameters w from the central server 0, initialize the global Actor network parameters φ 0 and the federated experience vector After that, the model will be trained for its own environment. The local model of each agent will load the global critic network parameters w 0 , global Actor model φ 0 , and copies the same parameters to the critic target network w′ and the actor target network φ′. Each agent also initializes its own experience replay pool, which stores its experience with the environment (i.e., state, action, reward, and next state) for subsequent sampling and training. After client initialization is complete, local training begins, and at regular intervals, the local model parameters are sent to the central server for model aggregation.

[0038] Step 4: During the local training process, that is, at time step t, the agent will perform the following operations. First, the agent will perform the following operations for the current state s. t Select an action t This action is achieved through the behavior network π φ (s), and to increase exploratory power, a noise term ∈ that follows a normal distribution N(0,σ) is added to the action, where σ is the variance of the normal distribution. The purpose of adding noise is to introduce more exploration during training, thereby helping the agent discover potentially better strategies rather than relying solely on the currently estimated optimal action.

[0039] Step 5: When the agent selects and executes the action a t After that, it will interact with the environment and obtain feedback information from the environment: including the reward r brought by the current action t and the next state s after executing the action t+1 This information will be packaged into a state transition tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool B. The experience replay pool is used to store the large amount of experience collected during the interaction between the agent and the environment. It can then be used to sample small batches of data for network training. This method helps to break the correlation between samples and improve the stability and convergence speed of training.

[0040] Step 6: The agent collects a small batch of experience tuples D of size N from its local experience replay pool t =(s j ,a j ,r j ,s j+1), in order to calculate the target Q value, the agent needs to generate an action corresponding to the next state s′. This process is first performed by the target behavior network π φ′ Execution. The target behavior network is similar to the main behavior network, but its parameters are updated more slowly and it is used to calculate a smoother and more stable target value. The agent inputs the next state s′ to the target behavior network π φ′ , the network outputs a preliminary action π φ′ (s′). In order to enhance the exploration and robustness of the strategy and avoid the behavior network from being too deterministically dependent on specific actions, we generate the action π φ′ (s′) adds an additional noise term ∈, where ∈ obeys a normal distribution N(0,σ). The introduction of noise is intended to simulate the uncertainty in the real world, helping the intelligent agent to better cope with complex environmental changes, while also preventing the strategy from falling into local optimality too early. In order to ensure that the generated actions are still reasonable and within the action space allowed by the system, we clip the noise. Specifically, the noise value is limited to a predefined range [-c,c] to avoid excessive noise causing the action to get out of control. Finally, the action after noise addition and clipping It will be used as the target action corresponding to the next state s′ for subsequent calculation of the target Q value.

[0041] Step 7: At each time step t, the agent needs to update the critic network to accurately evaluate the value of the actions taken by the current policy. To perform this update, it first needs to calculate the target Q value y, which is determined based on the current reward r and the Q value of the next state s'. The target Q value is calculated as follows: Where r is the reward obtained by the agent in the current time step, and γ is the discount factor, which is used to measure the influence of future rewards on the current decision. Its value is usually between 0 and 1. A larger γ means that the agent pays more attention to future rewards, while a smaller γ means that the agent pays more attention to immediate rewards.

[0042] Step 8: To update the parameters of the Critic network The agent first calculates the target Q value y (completed in step 7). Next, the critic network will adjust its prediction based on this target Q value to narrow the gap between its output Q value and the target Q value. The update goal of the critic network is to minimize the mean squared error (MSE) of the Q value in the batch data, that is, the difference between the target Q value y and the Q value currently output by the critic network. For each data sample in a batch, the loss function of the Critic network is defined as follows: Where J represents the batch size sampled from the experience replay pool, (sj ,a j ) is the sampled state-action pair, y j is the target Q value corresponding to the state-action pair, and its gradient is calculated according to the loss function of the Critic network That is to say

[0043] Step 9: Next, calculate a quadratic penalty function It is a regularization term between the current Critic network parameters of the agent and the experience vector, which aggregates the experience learned by other agents in their respective environments. Therefore, when the model is updated, the agent can not only adapt to the current environment, but also learn some knowledge from other environments. represents the local critic model parameters of the current agent at step t, represents the local experience vector of the current agent at step t, and v represents the weight coefficient of the regularization term, which is usually set to 0.01. Its goal is to help the model retain its memory of past tasks while adapting to the current task by limiting the range of variation of model parameters. By adjusting the value of v, the strength of regularization can be controlled, affecting the degree to which the model remembers past tasks.

[0044] Step 10: Calculate the correlation perturbation Indicates that the k-th agent uses a dataset randomly sampled from the experience replay pool in the t-th step of the local training process The gradient calculated on its loss function.

[0045] Step 11: Calculate the fine-tuned parameters Randomly collect a batch of data from the experience replay pool Calculate the gradient according to step 8 based on this batch of data That is to say

[0046] Step 12: Randomly collect a batch of data from the experience replay pool Used for meta-update, updating the Critic network parameters in The fine-tuned parameters use randomly collected data The second-order gradient calculated on its loss function usually takes a lot of time. To improve efficiency, the approximate calculation method of Hessian vector product is used, that is, the calculation of the second-order gradient is replaced by the gradient difference: This method can significantly improve computational efficiency and avoid the high cost of directly calculating the Hessian matrix.

[0047] Step 13: After completing a round of local critic network parameter updates, the agent will update the local experience vector and add the historical gradient information of the training to the experience vector, so that the local experience vector has partial knowledge of the environment. According to the formula Complete the update of the agent's local experience vector.

[0048] Step 14: Determine t mod N = 0, where N is the federal frequency of the Critic model. If the condition is met, the Critic model trained locally by the agent is used. and the local experience vector The data is then sent to the central server for aggregation. For simplicity, the model aggregation function uses a naive weighted aggregation method. After receiving the local critic models from each agent, the central server begins aggregation. After aggregation is complete, the aggregated global critic model parameters and global experience vector are sent to each agent to continue the next round of federated learning training.

[0049] Step 15: Determine t mod M = 0, where M is the frequency of delayed policy updates. Postponing the update of the Actor network can prevent frequent updates from affecting training stability. If the conditions are met, the agent will update the Actor network parameter φ through deterministic policy gradients. The gradient of the Actor network is calculated by the formula Using the gradient above, we use the gradient ascent method to update the parameters of the Actor network. The update formula for the Actor network parameters is: β π is the learning rate of the Actor network, which determines the size of the parameter adjustment step at each update.

[0050] Step 16: Use a variable count to record the number of times the Actor network is updated, and determine count mod L==0, where L is the federation frequency of the Actor model. If the condition is met, the Actor model trained locally by the agent is used. The model is then sent to the central server for aggregation. For simplicity, the model aggregation function uses a naive weighted aggregation method. After receiving the local actor models from each agent, the central server begins aggregation. Once aggregation is complete, the aggregated global actor model parameters are sent to each agent to continue the next round of federated learning training.

[0051] Step 17: If the condition t mod M == 0 is met, the target network is updated using a soft update strategy. The target network is updated by smoothly integrating the latest parameters of the main network and the existing parameters of the target network at a ratio τ. For the critic target network parameters, the formula w′ is used. i ←τw i+(1-τ)w′ i To update, w′ i is the parameter of the target Critic network (i=1,2 means two target Critic networks), w i τ is the parameter of the main critic network, and τ is the soft update coefficient, which is usually between 0 and 1 (for example, τ = 0.005). For the parameters of the actor target network, the formula φ′←τφ+(1-τ)φ′ is used to update them, where φ′ is the parameter of the target actor network and φ is the parameter of the main actor network. If the current number of training steps has reached the training step requirement preset by federated reinforcement learning, training is stopped and the process ends.

[0052] Example

[0053] This paper designs a federated reinforcement learning method based on meta-learning and experience vectors. This method can be applied to federated reinforcement learning tasks in autonomous driving systems to enhance the collaborative learning capabilities of multiple autonomous driving vehicles in heterogeneous environments and improve the robustness and generalization capabilities of the system.

[0054] For example, in an autonomous driving system, consider a federated reinforcement learning system with five client nodes and a central server node. Each client node corresponds to an independent autonomous vehicle, and the central server is used to aggregate model updates from each vehicle. The environment in which each autonomous vehicle operates may have significant differences, such as road type, traffic conditions, weather changes, etc. This environmental heterogeneity will lead to different experiences learned by different vehicles in their respective environments. The following is a detailed implementation:

[0055] Constructing a heterogeneous environment: The training environment for each autonomous vehicle (agent) is heterogeneous, particularly in terms of weather conditions, road conditions, and traffic density. For example, vehicle 1 might be driving on a sunny city road, while vehicle 2 might be driving on a rainy rural road. These environmental differences result in different data distributions for the data collected by each vehicle's sensors (such as LIDAR, cameras, and radar).

[0056] Initialize the global model: The central server initializes the global Actor network parameters φ 0 , Critic network parameters w 0 and the global federated experience vector Each autonomous vehicle loads these global parameters and initializes its local target network and experience replay pool for subsequent learning.

[0057] Local training: Each autonomous vehicle interacts with its environment, performing various driving actions (such as acceleration, steering, and braking) to obtain feedback from the environment. This feedback includes reward values ​​(e.g., whether a collision was avoided or whether driving was smooth) and the next state (e.g., vehicle acceleration, speed, and location). Each vehicle packages its local environment state, action, reward, and next state into an experience tuple and stores it in a local experience replay pool.

[0058] Policy Exploration and Update: During local training, each autonomous vehicle generates an action based on its current state using its behavior network. To enhance the policy's exploratory nature, a noise term ∈ (N(0,σ)) with a normal distribution N(0,σ) is added to the generated action. This noise simulates the uncertainties inherent in real-world driving, such as vehicle vibration and ambient noise. To prevent excessive exploration from leading to uncontrolled vehicle behavior, the range of actions is limited to a certain safety threshold.

[0059] Critic network update and meta-learning application:

[0060] (1) Collecting local experience tuples: At each time step t, the autonomous vehicle samples batch data from the local experience replay pool, uses two critic networks to calculate the target Q value of the current strategy, and selects the smaller Q value of the two critic networks as the target value, thereby reducing the over-estimation problem of the Q value.

[0061] (2) Calculate the target Q value and update the critic network parameters: The critic network is updated by minimizing the mean square error between the target Q value and the estimated Q value, ensuring that the critic network is locally updated to adapt to the current task.

[0062] (3) Calculate the quadratic penalty function: It is the regularization term between the current critic network parameters of the autonomous driving vehicle and the experience vector. The experience vector aggregates the experience learned by other agents in their respective environments. Therefore, when the model is updated, the agent can not only adapt to the current environment, but also learn some knowledge from other environments.

[0063] (4) Calculation of correlation perturbation: In order to deal with the instability that may occur in meta-learning when the environment heterogeneity is too large, the system introduces a correlation perturbation term. Calculate the correlation perturbation term This perturbation term is used to limit the update amplitude of the model parameters, ensuring that the model is not overly biased towards a specific environment, but can better adapt to the global task.

[0064] (5) Fine-tuning the critic network parameters: After calculating the correlation perturbation, the agent fine-tunes the critic network based on the perturbation term.

[0065] (6) Meta-update phase of meta-learning: After fine-tuning the critic network, the autonomous vehicle performs a meta-update based on the updated results of the local task. During the meta-update phase, the agent samples new data from the experience replay pool and optimizes the critic network parameters using second-order gradients, further improving the model's ability to quickly adapt to new tasks.

[0066] (7) Experience vector update: After each update of the critic network, the autonomous vehicle not only updates its model parameters, but also updates the local experience vector. The experience vector records the knowledge learned by each autonomous vehicle in different environments.

[0067] Step 7: Federated Model Aggregation: At regular intervals (e.g., every 200 time steps), each vehicle sends its locally trained critic network parameters and local experience vectors to the central server. The central server aggregates the local models uploaded by all autonomous vehicles to form a global critic model parameter set, which it then distributes back to each vehicle. Through this federated aggregation process, the central server can synthesize the driving experience of different vehicles in heterogeneous environments, ensuring that the global model is adaptable to a variety of driving scenarios.

[0068] Step 8 Delayed update of the Actor network: In the present invention, the delayed update of the Actor network is based on the characteristics of the TD3 (TwinDelayed Deep Deterministic Policy Gradient) algorithm. The TD3 algorithm aims to reduce the impact of the policy update frequency on the stability of the model by introducing delayed updates of the Actor network, thereby avoiding the policy network from prematurely fitting inaccurate Critic value estimates. For example, every 100 steps, the vehicle updates the Actor network through deterministic policy gradients to ensure that the policy update is based on a fully trained Critic network. This delayed update mechanism enables the Actor network to avoid the instability caused by frequent updates, thereby improving the convergence and robustness of the policy.

[0069] Step 9: Soft Update Strategy: Every certain number of steps, the system soft-updates the target network parameters to ensure smooth training. Both the critic and actor target network parameters are updated at a rate of τ to avoid drastic changes in the policy and value functions.

[0070] Step 10: Termination Condition: When the training reaches the set number of steps, the system stops training and tests the performance of the final model. During the testing process, the central server evaluates the performance of the global policy in different vehicle environments to ensure the generalization and robustness of the model.

[0071] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps of the aforementioned autonomous driving federated reinforcement learning method based on meta-learning and experience vectors are implemented.

[0072] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the aforementioned autonomous driving federated reinforcement learning method based on meta-learning and experience vectors.

[0073] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0077] The above embodiments are only for illustrating the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A federated reinforcement learning method for autonomous driving based on meta-learning and experience vectors, characterized in that: The steps include: Step 1: Build a heterogeneous virtual simulation environment for each autonomous vehicle and initialize the global Actor parameter φ 0 , global critic parameter w 0 and the global experience vector and sent to each autonomous vehicle; Step 2: The local Actor network of each autonomous vehicle loads the global Actor parameter φ 0 , while φ 0 Copy to the target Actor network; the local Critic network of each autonomous vehicle loads the global Critic parameter w 0 , and w 0 Copy to the target critic network; initialize the experience replay pool of each autonomous driving vehicle and start local training; Step 3: For time step t, each autonomous vehicle interacts with its environment and obtains feedback from the environment, that is, according to the current state s t Select action a t , get the execution action a t The rewards brought t and the next state s after executing the action t+1 , will (s t , a t , r t , s t+1 ) is stored as an experience tuple in the experience replay pool; Step 4: In each time step, each autonomous vehicle uses the model-independent meta-learning method Sharp-MAML to update the local critic network parameters and local experience vector based on the loss function and correlation perturbations. When the time step is an integer multiple of the local critic network federation frequency, the local critic network parameters and local experience vectors of all autonomous vehicles are aggregated respectively. The aggregated global critic parameters and global experience vectors are then sent to each autonomous vehicle to continue local training. Step 5: Using the delay strategy, when the time step is an integer multiple of the delay strategy update frequency, the local Actor network parameters of each autonomous driving vehicle are updated through deterministic policy gradients, and the number of updates of the local Actor network parameters is recorded. When the number of updates is an integer multiple of the local Actor network federation frequency, the local Actor network parameters of all autonomous driving vehicles are aggregated, and the aggregated global Actor parameters are sent to each autonomous driving vehicle to continue local training. Step 6: During the training process, when the time step is an integer multiple of the delay strategy update frequency, the target critic network and target actor network parameters of each autonomous driving vehicle are updated using the soft update strategy; Step 7: When the time step reaches the preset number of training steps for federated reinforcement learning, the training is completed; each vehicle is controlled according to the global Actor network at the completion of training to achieve autonomous driving.

2. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 1, a virtual simulation environment identical to the real environment is constructed for each autonomous driving vehicle, and the virtual simulation environment in which each autonomous driving vehicle is located is heterogeneous, wherein the real environment includes urban roads on sunny days, rural roads on sunny days, urban roads on rainy days, and rural roads on rainy days.

3. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 3, for time step t, the autonomous driving vehicle is in state s t Select action a t , a t Generated by the local Actor network corresponding to time step t, the state s t Input the local Actor network, output the preliminary action, add an additional noise term ∈ to the preliminary action, and get the action a t ,∈ obeys the normal distribution N(0,σ), σ is the variance of the normal distribution.

4. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 4, the specific update process of the local critic network parameters and the local experience vector is as follows: 1) At time step t, the target Q value y is calculated based on the reward obtained by the autonomous vehicle at the current time step and the next state: Where r is the reward obtained by the autonomous vehicle in the current time step, γ is the discount factor, s′ is the next state, is the target action corresponding to the next state s′, is the target Q function, and the target Critic network is regarded as the merger of two small target Critic networks, w′ i is the parameter of the i-th small critic target network; Generated by the target Actor network corresponding to time step t, that is, the next state s′ is input to the target Actor network π φ′ , output preliminary action π φ′ (s′), for the initial action π φ′ (s′) adds an additional noise term ∈, ∈ obeys the normal distribution N(0, σ), and cuts the action after adding the noise term to obtain the action 2) Randomly sample a batch of experience tuples with a size of J from the experience replay pool Calculate the loss function of the local Critic network based on the data set: in, is the loss function of the local Critic network, is the local critic network parameter corresponding to the kth autonomous vehicle at time step t, J represents the batch size sampled from the experience replay pool, (s j , a j ) is the sampled state-action pair, y j is the target Q value corresponding to the state-action pair, is the Q function, and the local critic network is regarded as the merger of two small local critic networks, w i is the parameter of the i-th small local critic network; Calculate its gradient according to the loss function Right now represents the dataset randomly sampled from the experience replay pool by the k-th autonomous vehicle at time step t; 3) Calculate the quadratic penalty function in, represents the local experience vector of the kth autonomous vehicle at time step t, ν represents the weight coefficient of the regularization term; its gradient is calculated according to the quadratic penalty function 4) According to the gradient Calculating correlation perturbations And calculate the gradient according to the same method as 2) 5) Calculate the fine-tuned parameters Where α represents the learning rate in the meta-adaptation phase, α∈(0.001,0.01); Randomly sample a batch of size J from the experience replay pool According to the data set Calculate the gradient in the same way as 2) Right now 6) Randomly sample a dataset of batch size J from the experience replay pool Update local critic network parameters: in, is the local critic network parameter corresponding to the k-th autonomous vehicle at time step t+1, β represents the learning rate in the meta-update phase, β∈(0.001,0.01), I is the identity matrix, The fine-tuned parameters use randomly collected data The second-order gradient calculated on its loss function; 7) Update local experience vector: in, are the local experience vectors corresponding to the k-th autonomous driving vehicle at time steps t+1 and t, respectively.

5. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 5, when time step t mod M == 0, M is the delay policy update frequency, and the local Actor network parameters are updated by deterministic policy gradient; The gradient of the local Actor network is: The update formula for local Actor network parameters is: in, is the gradient of the local Actor network, J is the size of a batch of data sampled by the updated Actor network, is the gradient of the policy function, is the gradient of the first local Critic network, φ is the local Actor network parameter, β π is the learning rate of the local Actor network.

6. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 6, when the time step t mod M == 0, M is the delay strategy update frequency, and the target critic network and target actor network parameters of each autonomous driving vehicle are updated using a soft update strategy; The target critic network parameters are updated as follows: w′ i ←τw i +(1-τ)w′ i The target Actor network parameters are updated as follows: φ′←τφ+(1-τ)φ′ Where τ is the soft update coefficient, w i is the local critic network parameter corresponding to time step t, w′ i is the target Critic network parameter corresponding to time step t, φ is the local Actor network parameter corresponding to time step t, and φ′ is the target Actor network parameter corresponding to time step t.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the autonomous driving federated reinforcement learning method based on meta-learning and experience vectors as described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the autonomous driving federated reinforcement learning method based on meta-learning and experience vectors as described in any one of claims 1 to 6 are implemented.

Images