Automatic driving federal reinforcement learning method based on meta learning and empirical vector

Abstract

The invention discloses an automatic driving federal reinforcement learning method based on meta learning and empirical vectors. The method comprises the following steps: constructing an automatic driving heterogeneous environment; constructing an Actor network, and updating Actor network parameters through a deterministic strategy gradient; constructing a Critic network, updating Critic network parameters through a meta learning algorithm and an error back propagation algorithm, adding experience vector integration in a training process, and updating learning experience of each automatic driving vehicle so as to optimize performance of a global strategy; in the training process, all automatic driving vehicles regularly send local Critic network parameters, Actor network parameters and local experience vectors to the central server for aggregation. According to the method, the adaptive capacity of the automatic driving vehicle to the environmental heterogeneity is improved, the problems that in existing federal reinforcement learning, the convergence speed is reduced and the strategy generalization capacity is reduced due to the dynamic change of the environment and the fluctuation of the strategy performance are solved, and therefore the training efficiency and the strategy robustness are improved.

Description

Technical Field

[0001] The present invention relates to an autonomous driving federated reinforcement learning method based on meta-learning and experience vectors, and belongs to the technical field of autonomous driving. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, reinforcement learning, as a technology that learns optimal decision-making strategies by interacting with the environment, has been widely used in robot control, autonomous driving, intelligent recommendation systems and other fields. However, in the complex and dynamic application scenario of autonomous driving, existing systems face several severe challenges. First, autonomous vehicles need to make real-time decisions in different traffic environments, including dealing with complex traffic conditions, pedestrian behavior, weather changes, and differences in road conditions. This environmental heterogeneity makes it difficult to generalize a unified decision-making strategy in all scenarios, affecting the safety and reliability of the autonomous driving system. In addition, data privacy issues are becoming more and more prominent. When collecting and processing large amounts of real-time data, autonomous driving systems must ensure that user privacy is protected, especially in a multi-vehicle network environment, which makes the process of data sharing and model training more complicated.

[0003] Existing reinforcement learning technologies have achieved certain success in some tasks, but they still have obvious shortcomings in autonomous driving scenarios. Traditional centralized reinforcement learning methods face the problem of environmental heterogeneity when facing distributed systems and multi-agent tasks. The differences in the environments of different agents make the data distribution inconsistent, which in turn affects the ability of the shared global model to adapt to all local environments. In addition, these methods usually rely on a large amount of historical data for training, while it is difficult to obtain and process data in real time in autonomous driving, which limits the application of the algorithm. Existing algorithms perform inconsistently under different environmental conditions and are affected by environmental changes, resulting in instability of decision-making and reduced convergence speed. At the same time, data collection and model training in centralized training frameworks are time-consuming and labor-intensive, and it is difficult to meet the real-time requirements of autonomous driving systems.

[0004] As an emerging distributed learning paradigm, federated learning effectively protects data privacy and reduces communication overhead by allowing multiple agents to train models locally and only share model updates rather than raw data. Federated reinforcement learning (FRL) introduces the concept of federated learning into the field of reinforcement learning, allowing multiple distributed agents to collaboratively train a global strategy. However, FRL still faces several challenges in application, especially the problem of environmental heterogeneity, that is, the environments in which different agents are located have significant differences, making it difficult to generalize the global strategy in all environments. Environmental heterogeneity is a major challenge in federated reinforcement learning. Due to the inconsistent distribution of data collected by each agent in different environments, the shared global model may not be able to adapt to all local environments. This heterogeneity is manifested in differences in state space, action space, reward function, and state transition probability. This heterogeneity makes it difficult for agents to learn a unified global strategy, which seriously affects the convergence speed of federated reinforcement learning and the generalization ability of the strategy.

[0005] Meta-learning, as a "learning how to learn" technology, can learn the ability to quickly adapt to new tasks by training models on multiple tasks. Meta-learning enables models to quickly adapt to new tasks by learning the commonalities between multiple tasks. Therefore, it is used in federated learning to train personalized models and reduce the performance differences of models on different clients. This method that combines federated learning and meta-learning is called federated meta-learning. Although meta-learning has been widely studied in solving data heterogeneity problems in federated learning, there are still a series of challenges in directly copying its methods to federated reinforcement learning. First, the environment in FRL is dynamic and full of uncertainty, which not only affects the data distribution, but also directly affects the decision-making and learning effects of the agent, making the meta-learning method need to have stronger dynamic adaptability. Traditional meta-learning mainly deals with static data heterogeneity and is difficult to cope with this complexity. Secondly, the performance of strategies in FRL often fluctuates with changes in the environment. Even the same strategy may perform inconsistently in different environmental states. This requires the meta-learning method to be able to handle performance fluctuations caused by strategy changes and ensure the stability of the model. In addition, data generation in FRL depends on the current strategy, which is a typical exploration and utilization problem, resulting in the data distribution being dynamically adjusted as the agent strategy changes. The static data processing mode of traditional federated meta-learning methods is difficult to quickly adapt to these dynamic changes. Therefore, a federated reinforcement learning method is needed that can effectively cope with the environmental heterogeneity of autonomous driving, improve the generalization ability of autonomous driving strategies and the robustness of models, so as to meet the needs of data privacy protection and distributed collaborative learning in practical applications. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an autonomous driving federated reinforcement learning method based on meta-learning and experience vectors, which solves the strategy generalization and adaptability problems caused by the heterogeneity of the participating node environments in the federated learning process, and improves the reliability and robustness of the autonomous driving strategy.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] A federated reinforcement learning method for autonomous driving based on meta-learning and experience vectors includes the following steps:

[0009] Step 1: Build a heterogeneous virtual simulation environment for each autonomous driving vehicle and initialize the global Actor parameter φ 0 , global critic parameter w 0 And the global experience vector and sent to each autonomous vehicle;

[0010] Step 2: Each autonomous vehicle’s local Actor network loads the global Actor parameter φ 0 , and at the same time 0 Copy to the target Actor network; each autonomous vehicle’s local Critic network loads the global Critic parameters w 0 , and w 0 Copy to the target Critic network; initialize the experience replay pool of each autonomous driving vehicle and start local training;

[0011] Step 3: For time step t, each autonomous driving vehicle interacts with its environment and obtains feedback from the environment, that is, according to the current state s t Select action a t , get the execution action a t The rewards brought t and the next state s after executing the action t+1 , will (s t ,a t ,r t ,s t+1 ) is stored as an experience tuple in the experience replay pool;

[0012] Step 4: In each time step, each autonomous driving vehicle updates the local critic network parameters and local experience vectors based on the loss function and correlation perturbations using the model-independent meta-learning method Sharp-MAML. When the time step is an integer multiple of the local critic network federation frequency, the local critic network parameters and local experience vectors of all autonomous driving vehicles are aggregated, and the aggregated global critic parameters and global experience vectors are sent to each autonomous driving vehicle to continue local training.

[0013] Step 5: When the time step is an integer multiple of the update frequency of the delay strategy, the local Actor network parameters of each autonomous driving vehicle are updated through the deterministic policy gradient, and the number of updates of the local Actor network parameters is recorded. When the number of updates is an integer multiple of the local Actor network federation frequency, the local Actor network parameters of all autonomous driving vehicles are aggregated, and the aggregated global Actor parameters are sent to each autonomous driving vehicle to continue local training.

[0014] Step 6: During the training process, when the time step is an integer multiple of the delay strategy update frequency, the target critic network and target actor network parameters of each autonomous driving vehicle are updated using a soft update strategy;

[0015] Step 7: When the time step reaches the preset number of training steps for federated reinforcement learning, the training is completed; each vehicle is controlled according to the global Actor network at the completion of the training to achieve autonomous driving.

[0016] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0017] 1. The present invention provides shared initialization parameters for autonomous driving vehicles by introducing meta-learning technology, enabling them to quickly adapt to different environments, thereby improving the generalization ability of the model and the response speed to environmental changes.

[0018] 2. The present invention designs an experience vector mechanism to integrate and update the learning experience of each autonomous driving vehicle during the training process to optimize the performance of the global strategy.

[0019] 3. The present invention solves the problems of slow convergence and reduced strategy generalization ability caused by dynamic changes in the environment and fluctuations in strategy performance in the prior art by improving the adaptability of autonomous driving vehicles to environmental heterogeneity, thereby improving the overall training efficiency and the robustness of the strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of an autonomous driving federated reinforcement learning method based on meta-learning and experience vectors of the present invention. DETAILED DESCRIPTION

[0021] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0022] Assume that there is a federated reinforcement learning system, which includes a central server and N clients. In each round of training, the central server sends the current global strategy π t Broadcast to all or some of the agents, which will perform multiple local training iterations. In each iteration, the agent interacts with its environment to collect T time steps of data and optimizes its local policy using a gradient ascent method based on its local objective. At the end of each round, the training results are uploaded to the server for aggregation. In a heterogeneous environment, each client agent interacts with its environment independently. These environments have the same state space S, the same action space A, the same (or different) reward function R, and different state transition functions For example, for agent i and agent j, there exists P i (s,a)≠P j (s,a). The local training problem of each agent is modeled as a Markov decision process (MDP). Therefore, the federated reinforcement learning system can be defined as a collection of distributed MDPs. S represents a finite set of states, A represents a finite set of actions, and R i (s,a) represents the immediate reward obtained after taking action a in state s in the i-th MDP, P n (s′|s,a) represents the probability that the i-th MDP transitions from state s to s′ after taking action a, which characterizes the environment dynamics of the i-th MDP. γ∈(0,1) represents the reward discount factor.

[0023] The goal of federated reinforcement learning is to allow n agents to jointly learn a well-behaved policy function or value function in n environments to maximize their expected discounted rewards in the long run. Due to privacy constraints, the n agents cannot share their collected experiences, observations, and states. Similar to federated learning, the goal of federated reinforcement learning is to find the best value estimation function Q on the state-action pairs extracted from the agent's local experience replay area. w* (s t ,a t ), local function Q w* (s t ,a t ) can be defined by the optimal Bellman equation:

[0024]

[0025] Among them, r t is the agent in the current state s t Take action t The immediate reward received after. Random variable S t+1 From the state transition probability distribution function p(s t+1 ∣s t ,at ). From the perspective of federated learning, federated reinforcement learning can be formulated as the following optimization problem:

[0026]

[0027] Where k is the agent number. K is the number of agents in the federated system. k represents the local loss function of the kth agent. k represents the number of samples collected by the kth agent in its local environment, l(w; x kj ) represents the loss of the local model of agent k in its local environment, and kj represents the jth sample collected by the kth agent.

[0028] Therefore, the final goal can be rewritten as:

[0029]

[0030] Among them, D + =D1∪…∪D K , represents the union of all agent data.

[0031] In the context of federated reinforcement learning, the Federated Experience Vector (FEV) extracts the experience of each agent into an experience vector of fixed size, and generates a new federated (global) experience vector by aggregating the local experience vectors of each agent in each round of federated learning. Multiple agents share processed experience representations (such as state, action, reward, etc.) to improve the efficiency of collaborative learning without leaking privacy. It can accelerate model convergence and improve overall performance through distributed experience sharing.

[0032] The core idea of ​​meta-learning is to improve the generalization ability by training the model on a set of tasks and enabling it to quickly adapt to new tasks. Our goal is to design a personalized variant of the FRL problem using the basic idea behind the model-agnostic meta-learning (MAML) framework. The core of this method is to find a shared initial model that can adapt to its specific local environment after a small number of gradient descent steps on the client. However, MAML currently has problems such as unstable training gradients and easy to fall into local optimal solutions. To address the above problems, a model-agnostic meta-learning method called Sharp-MAML is adopted, which optimizes model-agnostic meta-learning (MAML) by utilizing the sharpness-aware minimization (SAM) algorithm. Assume that the computational budget for updating the model after the arrival of a new task is limited, and in this new setting, find an initialization that performs well after being updated for this new task, perhaps through one or a few steps of gradient descent. If it is assumed that each user takes the initial point and updates it using a one-step gradient descent relative to its own loss function, the federated reinforcement learning optimization problem becomes:

[0033]

[0034] according to Figure 1 , the method proposed in the present invention is elaborated in detail.

[0035] Step 1: Construct a heterogeneous environment by adding Gaussian noise after the environmental parameters of each agent’s environment, so that the training environment of each agent is heterogeneous.

[0036] Step 2: Initialize the global Q network parameters w 0 For the convenience of subsequent calculations, the two Q networks are merged into a large network, and the output contains the two Q networks' estimates of the value of the current given state and action. TD3 introduces two Critic networks to alleviate the problem of overestimation of Q values. Initialize the global Actor network parameter φ 0 , which is used to calculate the deterministic action a generated in a given state s. Initialize Critic and Actor Initialize the target network (i.e. w′, φ′), and set the parameters of these target networks to be the same as the original network. The target network is used for soft update, which can make the training process more stable. Initialize the global federated experience vector After initialization, the central server will set the global Q network parameter w 0 、Global Actor Network Parameters φ 0 and the federated experience vector Send to all agents.

[0037] Step 3: All agents receive the initialized global Q network parameters w sent by the central server 0, Initialize the global Actor network parameter φ 0 and the federated experience vector After that, the model will be trained for its own environment. The local model of each agent will load the global Critic network parameters w 0 、Global Actor Model φ 0 , and copies the same parameters to the Critic target network w′ and the Actor target network φ′. At the same time, each agent initializes its own experience replay pool to store the experience of the agent's interaction with the environment (i.e., state, action, reward, next state) for subsequent sampling for training. After the client is initialized, local training begins, and the local model parameters are sent to the central server for model aggregation at intervals.

[0038] Step 4: During the local training process, that is, at time step t, the agent will perform the following operations: first, the agent will perform the following operations for the current state s t Select an action t This action is done through the behavior network π φ (s), and in order to increase the exploratory nature, a noise term ∈ that follows the normal distribution N(0,σ) is added to the action, where σ is the variance of the normal distribution. The purpose of adding noise is to introduce more exploration during the training process, thereby helping the agent to discover potentially better strategies rather than just relying on the currently estimated optimal action.

[0039] Step 5: When the agent selects and executes the action a t After that, it will interact with the environment and obtain feedback from the environment: including the reward r brought by the current action t and the next state s after executing the action t+1 This information will be packaged into a state transition tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool B. The experience replay pool is used to store a large amount of experience collected during the interaction between the agent and the environment, from which small batches of data can be sampled for network training. This method helps to break the correlation between samples and improve the stability and convergence speed of training.

[0040] Step 6: The agent collects a small batch of experience tuples D of size N from its local experience replay pool t =(s j ,a j ,r j ,s j+1), in order to calculate the target Q value, the agent needs to generate an action corresponding to the next state s′. This process is first performed by the target behavior network π φ′ Execute. The target behavior network is similar to the main behavior network, its parameters are updated slowly, and it is used to calculate a smoother and more stable target value. The agent inputs the next state s′ to the target behavior network π φ′ , the network outputs a preliminary action π φ′ (s′). In order to enhance the exploration and robustness of the strategy and avoid the behavior network from being too deterministically dependent on specific actions, we generate actions π φ′ (s′) adds an additional noise term ∈, where ∈ obeys a normal distribution N(0,σ). The introduction of noise is intended to simulate the uncertainty in the real world, helping the intelligent agent to better cope with complex environmental changes, while also preventing the strategy from falling into local optimality too early. In order to ensure that the generated actions are still reasonable and within the action space allowed by the system, we clip the noise. Specifically, the noise value is limited to a predefined range [-c,c] to prevent excessive noise from causing the action to get out of control. Finally, the action after noise addition and clipping It will be used as the target action corresponding to the next state s′ for subsequent calculation of the target Q value.

[0041] Step 7: At each time step t, the agent needs to update the Critic network in order to accurately evaluate the value of the actions taken by the current strategy. In order to perform this update, the target Q value y needs to be calculated first. This target Q value is determined based on the current reward r and the Q value of the next state s′. The formula for calculating the target Q value is: Where r is the reward obtained by the agent in the current time step, and γ is the discount factor, which is used to measure the influence of future returns on the current decision. The value is usually between 0 and 1. A larger γ means that the agent pays more attention to future returns, while a smaller γ means that the agent pays more attention to immediate returns.

[0042] Step 8: To update the parameters of the Critic network The agent first calculates the target Q value y (completed in step 7). Next, the Critic network will adjust its prediction based on this target Q value to narrow the gap between its output Q value and the target Q value. The update goal of the Critic network is to minimize the mean squared error (MSE) of the Q value in the batch data, that is, the difference between the target Q value y and the Q value currently output by the Critic network. For each batch of data samples, the loss function of the Critic network is defined as follows: Where J is the batch size sampled from the experience replay pool, (sj ,a j ) is the sampled state-action pair, y j is the target Q value corresponding to the state-action pair, and its gradient is calculated according to the loss function of the Critic network That is to say

[0043] Step 9: Next, calculate a quadratic penalty function It is the regularization term between the current Critic network parameters of the agent and the experience vector, which aggregates the experience learned by other agents in their respective environments. Therefore, when the model is updated, the agent can not only adapt to the current environment, but also learn some knowledge from other environments. represents the local critic model parameters of the current agent at step t, represents the local experience vector of the current agent at step t, and v represents the weight coefficient of the regularization term, which is usually set to 0.01. Its goal is to help the model retain its memory of past tasks while adapting to the current task by limiting the range of variation of model parameters. By adjusting the value of v, the strength of regularization can be controlled, affecting the degree of memory of the model for historical tasks.

[0044] Step 10: Calculate the correlation perturbation Indicates that the k-th agent uses a dataset randomly sampled from the experience replay pool in the t-th step of the local training process The gradient calculated on its loss function.

[0045] Step 11: Calculate the fine-tuned parameters Randomly collect a batch of data from the experience replay pool Calculate the gradient according to step 8 based on this batch of data That is to say

[0046] Step 12: Randomly collect a batch of data from the experience replay pool Used for meta-update, update the Critic network parameters in is the fine-tuned parameter using randomly collected data The second-order gradient calculated on its loss function usually takes a lot of time. In order to improve efficiency, the approximate calculation method of Hessian vector product is adopted, that is, the calculation of the second-order gradient is replaced by the gradient difference: This method can significantly improve computational efficiency and avoid the high cost of directly calculating the Hessian matrix.

[0047] Step 13: After completing a round of local critic network parameter updates, the agent will update the local experience vector and add the historical gradient information of the training to the experience vector, so that the local experience vector has partial knowledge of the environment. According to the formula Complete the update of the agent's local experience vector.

[0048] Step 14: Determine t mod N = = 0, where N is the federation frequency of the Critic model. If the condition is met, the Critic model trained locally by the agent is used. and the local experience vector The data is sent to the central server for aggregation. For simplicity, the model aggregation function uses a naive weighted aggregation method. After receiving the local critic model from each agent, the central server starts aggregation. After the aggregation is completed, the aggregated global critic model parameters and global experience vector are sent to each agent to continue the next round of federated learning training.

[0049] Step 15: Determine t mod M = = 0, where M is the delay policy update frequency. Delaying the update of the Actor network can prevent frequent updates from affecting training stability. If the conditions are met, the agent will update the Actor network parameter φ through a deterministic policy gradient. The gradient of the Actor network is calculated by the formula Through the above gradient, the gradient ascent method is used to update the parameters of the Actor network. The update formula of the Actor network parameters is: β π is the learning rate of the Actor network, which determines the size of the parameter adjustment step at each update.

[0050] Step 16: Use a variable count to record the number of times the Actor network is updated, and determine count mod L==0, where L is the federation frequency of the Actor model. If the condition is met, the Actor model trained locally by the agent is used. The model is sent to the central server for aggregation. For simplicity, the model aggregation function uses a naive weighted aggregation method. After receiving the local Actor model from each agent, the central server starts aggregation. After the aggregation is completed, the aggregated global Actor model parameters are sent to each agent to continue the next round of federated learning training.

[0051] Step 17: If the condition t mod M == 0 is met, the target network is updated using a soft update strategy. The target network is updated by smoothly integrating the latest parameters of the main network and the existing parameters of the target network at a ratio of τ. For the Critic target network parameters, the formula w′ is used. i ←τw i+(1-τ)w′ i To update, w′ i is the parameter of the target Critic network (i=1,2 means two target Critic networks), w i is the parameter of the main Critic network, τ is the soft update coefficient, usually between 0 and 1 (for example, τ = 0.005). For the Actor target network parameters, the formula φ′←τφ+(1-τ)φ′ is used to update, φ′ is the parameter of the target Actor network, and φ is the parameter of the main Actor network. If the current number of training steps has reached the number of training steps preset by the federated reinforcement learning, the training is stopped and the process ends.

[0052] Example

[0053] The present invention designs a federated reinforcement learning method based on meta-learning and experience vectors, which can be applied to the federated reinforcement learning tasks in autonomous driving systems to enhance the collaborative learning ability of multiple autonomous driving vehicles in heterogeneous environments and improve the robustness and generalization ability of the system.

[0054] For example, in an autonomous driving system, a federated reinforcement learning system with five client nodes and one central server node can be considered. Each client node corresponds to an independent autonomous driving vehicle, and the central server is used to aggregate the model updates of each vehicle. The environment in which each autonomous driving vehicle is located may be significantly different, such as road type, traffic conditions, weather changes, etc. This environmental heterogeneity will cause different vehicles to learn different experiences in their respective environments. The following is a detailed implementation method:

[0055] Constructing heterogeneous environments: The training environment of each autonomous driving vehicle (agent) is heterogeneous, which is reflected in weather conditions, road conditions, traffic density, etc. For example, the scene faced by vehicle 1 may be a city road on a sunny day, while vehicle 2 may be driving on a rural road on a rainy day. These environmental differences result in different data distributions of the data collected by each vehicle's sensors (such as LIDAR, cameras, radars, etc.).

[0056] Initialize the global model: The central server initializes the global Actor network parameters φ 0 、Critic network parameters w 0 and the global federated experience vector And send it to all autonomous vehicles. Each autonomous vehicle loads these global parameters and initializes its local target network and experience replay pool for subsequent learning.

[0057] Local training: Each autonomous vehicle interacts with its environment and obtains feedback from the environment by performing various driving behaviors (such as acceleration, steering, braking, etc.). This feedback information includes reward values ​​(such as whether a collision is avoided, whether the driving is smooth) and the next state (such as the vehicle's acceleration, speed, location information, etc.). Each vehicle will package the state, action, reward, and next state in its local environment into experience tuples and store them in the local experience playback pool.

[0058] Exploration and update of strategies: During local training, each autonomous vehicle generates an action using its behavior network based on its current state. To increase the exploratory nature of the strategy, a noise term ∈ that follows a normal distribution N(0,σ) is added to the generated action. This noise simulates the uncertainty of driving in the real world, such as vehicle jitter and environmental noise that may occur during driving. To avoid excessive exploration leading to uncontrolled vehicle behavior, the range of actions is limited to a certain safety threshold.

[0059] Critic network update and application of meta-learning:

[0060] (1) Collecting local experience tuples: At each time step t, the autonomous vehicle samples batch data from the local experience replay pool, uses two critic networks to calculate the target Q value of the current strategy, and selects the smaller Q value of the two critic networks as the target value, thereby reducing the over-estimation problem of the Q value.

[0061] (2) Calculate the target Q value and update the critic network parameters: The critic network is updated by minimizing the mean square error between the target Q value and the estimated Q value, ensuring that the critic network is locally updated to adapt to the current task.

[0062] (3) Calculate the quadratic penalty function: It is the regularization term between the current critic network parameters of the autonomous driving vehicle and the experience vector, which aggregates the experience learned by other agents in their respective environments. Therefore, when the model is updated, the agent can not only adapt to the current environment, but also learn some knowledge from other environments.

[0063] (4) Calculation of correlation perturbation: In order to deal with the instability that may occur in meta-learning when the environment heterogeneity is too large, the system introduces a correlation perturbation term. Calculate the correlation perturbation term This perturbation term is used to limit the update amplitude of the model parameters, ensuring that the model is not overly biased towards a specific environment, but can better adapt to the global task.

[0064] (5) Fine-tuning the critic network parameters: After calculating the correlation perturbation, the agent fine-tunes the critic network based on the perturbation term.

[0065] (6) Meta-update phase of meta-learning: After completing fine-tuning of the critic network, the autonomous vehicle will perform meta-update based on the update results of the local task. In the meta-update phase, the agent samples new data from the experience replay pool and optimizes the critic network parameters through second-order gradients to further improve the model's ability to quickly adapt to new tasks.

[0066] (7) Update of experience vector: After each update of the critic network, the autonomous vehicle not only updates its model parameters, but also updates the local experience vector. The experience vector records the knowledge learned by each autonomous vehicle in different environments.

[0067] Step 7: Aggregation of the federated model: At regular intervals (e.g., 200 time steps), each vehicle sends the locally trained critic network parameters and local experience vectors to the central server. The central server aggregates the local models uploaded by all autonomous vehicles to form a global critic model parameter and distributes it back to each vehicle. Through this federated aggregation process, the central server can integrate the driving experience of different vehicles in heterogeneous environments to ensure that the global model can adapt to a variety of driving scenarios.

[0068] Step 8 Delayed update of the Actor network: In the present invention, the delayed update of the Actor network is based on the characteristics of the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm. The TD3 algorithm introduces delayed updates of the Actor network to reduce the impact of the policy update frequency on the stability of the model, thereby avoiding the policy network from fitting inaccurate Critic value estimates too early. For example, every 100 steps, the vehicle updates the Actor network through deterministic policy gradients to ensure that the policy update is based on a fully trained Critic network. This delayed update mechanism enables the Actor network to avoid the instability caused by frequent updates and improves the convergence and robustness of the policy.

[0069] Step 9 Soft Update Strategy: Every certain number of steps, the system will soft update the target network parameters to ensure the smoothness of the training process. The parameters of the Critic target network and the Actor target network will be updated according to the ratio τ to avoid drastic changes in the strategy and value function.

[0070] Step 10 Termination condition: When the training reaches the set number of steps, the system will stop training and test the performance of the final model. During the test, the central server will evaluate the performance of the global strategy in different vehicle environments to ensure the generalization and robustness of the model.

[0071] Based on the same inventive concept, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the aforementioned meta-learning and experience vector-based federated reinforcement learning method for autonomous driving are implemented.

[0072] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the aforementioned autonomous driving federated reinforcement learning method based on meta-learning and experience vectors.

[0073] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0077] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A federated reinforcement learning method for autonomous driving based on meta-learning and experience vector, characterized in that: The steps include: Step 1: Build a heterogeneous virtual simulation environment for each autonomous driving vehicle and initialize the global Actor parameter φ 0 , global critic parameter w 0 And the global experience vector and sent to each autonomous vehicle; Step 2: Each autonomous vehicle’s local Actor network loads the global Actor parameter φ 0 , and at the same time 0 Copy to the target Actor network; each autonomous vehicle’s local Critic network loads the global Critic parameters w 0 , and w 0 Copy to the target Critic network; initialize the experience replay pool of each autonomous driving vehicle and start local training; Step 3: For time step t, each autonomous driving vehicle interacts with its environment and obtains feedback from the environment, that is, according to the current state s t Select action a t , get the execution action a t The rewards brought t and the next state s after executing the action t+1 , will (s t , a t , r t ,s t+1 ) is stored as an experience tuple in the experience replay pool; Step 4: In each time step, each autonomous driving vehicle updates the local critic network parameters and local experience vectors based on the loss function and correlation perturbations using the model-independent meta-learning method Sharp-MAML. When the time step is an integer multiple of the local critic network federation frequency, the local critic network parameters and local experience vectors of all autonomous driving vehicles are aggregated, and the aggregated global critic parameters and global experience vectors are sent to each autonomous driving vehicle to continue local training. Step 5: When the time step is an integer multiple of the update frequency of the delay strategy, the local Actor network parameters of each autonomous driving vehicle are updated through the deterministic policy gradient, and the number of updates of the local Actor network parameters is recorded. When the number of updates is an integer multiple of the local Actor network federation frequency, the local Actor network parameters of all autonomous driving vehicles are aggregated, and the aggregated global Actor parameters are sent to each autonomous driving vehicle to continue local training. Step 6: During the training process, when the time step is an integer multiple of the delay strategy update frequency, the target critic network and target actor network parameters of each autonomous driving vehicle are updated using a soft update strategy; Step 7: When the time step reaches the preset number of training steps for federated reinforcement learning, the training is completed; each vehicle is controlled according to the global Actor network at the completion of the training to achieve autonomous driving.

2. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 1, a virtual simulation environment identical to the real environment is constructed for each autonomous driving vehicle, and the virtual simulation environment in which each autonomous driving vehicle is located is heterogeneous, wherein the real environment includes urban roads on sunny days, rural roads on sunny days, urban roads on rainy days, and rural roads on rainy days.

3. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 3, for time step t, the autonomous driving vehicle is in state s t Select action a t , a t Generated by the local Actor network corresponding to time step t, the state s t Input the local Actor network, output the preliminary action, add an additional noise term ∈ to the preliminary action, and get the action a t ,∈ obeys the normal distribution N(0,σ), σ is the variance of the normal distribution.

4. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 4, the specific update process of the local critic network parameters and the local experience vector is as follows: 1) At time step t, based on the reward and next state obtained by the autonomous driving vehicle at the current time step, calculate the target Q value y: Where r is the reward obtained by the autonomous vehicle in the current time step, γ is the discount factor, s′ is the next state, is the target action corresponding to the next state s′, is the target Q function, and the target Critic network is regarded as the merger of two small target Critic networks, w′ i is the parameter of the i-th small critic target network; Generated by the target Actor network corresponding to time step t, that is, the next state s′ is input to the target Actor network π φ′ , output preliminary action π φ′ (s′), for the initial action π φ′ (s′) adds an additional noise term ∈, ∈ obeys the normal distribution N(0, σ), and cuts the action after adding the noise term to get the action 2) Randomly sample a batch of experience tuples with a size of J from the experience replay pool Calculate the loss function of the local Critic network based on the data set: in, is the loss function of the local Critic network, is the local critic network parameter corresponding to the kth autonomous vehicle at time step t, J represents the batch size sampled from the experience replay pool, (s j , a j ) is the sampled state-action pair, y j is the target Q value corresponding to the state-action pair, is the Q function, which regards the local critic network as the merger of two small local critic networks, w i is the parameter of the i-th small local Critic network; Calculate its gradient according to the loss function Right now represents the dataset randomly sampled from the experience replay pool by the k-th autonomous vehicle at time step t; 3) Calculate the quadratic penalty function in, represents the local experience vector of the kth autonomous driving vehicle at time step t, ν represents the weight coefficient of the regularization term; its gradient is calculated according to the quadratic penalty function 4) According to the gradient Calculate correlation perturbations And calculate the gradient using the same method as 2) 5) Calculate the fine-tuned parameters Among them, α represents the learning rate in the meta-adaptation phase, α∈(0.001,0.01); Randomly sample a batch of size J from the experience replay pool According to the data set Calculate the gradient in the same way as 2) Right now 6) Randomly sample a batch size J of data from the experience replay pool Update local Critic network parameters: in, is the local critic network parameter corresponding to the k-th autonomous vehicle at time step t+1, β represents the learning rate in the meta-update phase, β∈(0.001,0.01), I is the identity matrix, is the fine-tuned parameter using randomly collected data The second-order gradient calculated on its loss function; 7) Update local experience vector: in, are the local experience vectors corresponding to the kth autonomous driving vehicle at time steps t+1 and t respectively.

5. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 5, when the time step t mod M == 0, M is the delay policy update frequency, and the local Actor network parameters are updated by deterministic policy gradients; The gradient of the local Actor network is: The update formula for the local Actor network parameters is: in, is the gradient of the local Actor network, J is the size of a batch of data sampled by the updated Actor network, is the gradient of the policy function, is the gradient of the first local Critic network, φ is the local Actor network parameter, β π is the learning rate of the local Actor network.

6. The autonomous driving federated reinforcement learning method based on meta-learning and experience vector according to claim 1, characterized in that: In step 6, when the time step t mod M==0, M is the delay strategy update frequency, and the target Critic network and target Actor network parameters of each autonomous driving vehicle are updated using a soft update strategy; The target Critic network parameters are updated as follows: w′ i ←τw i +(1-τ)w′ i The target Actor network parameters are updated as follows: φ′←τφ+(1-τ)φ′ Where τ is the soft update coefficient, w i is the local critic network parameter corresponding to time step t, w′ i are the target Critic network parameters corresponding to time step t, φ are the local Actor network parameters corresponding to time step t, and φ′ are the target Actor network parameters corresponding to time step t.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the steps of the autonomous driving federated reinforcement learning method based on meta-learning and experience vectors as described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the autonomous driving federated reinforcement learning method based on meta-learning and experience vectors as described in any one of claims 1 to 6 are implemented.

Images