Vehicle-road cooperative control system and method based on multi-agent federated reinforcement learning at complex intersections

Abstract

The present invention discloses a vehicle-road cooperative control system and method based on multi-agent federated reinforcement learning at complex intersections, proposes a vehicle-road cooperative control framework based on a road-side static processing module and a vehicle-side dynamic processing module, and uses the advantages of the road side to complete the road history information; proposes a federated reinforcement learning algorithm FTD3, which is used to connect the reinforcement learning module and the federated learning module. The algorithm only transmits neural network parameters instead of vehicle-side data to protect privacy. The algorithm only selects part of the neural network for aggregation to reduce communication overhead, selects networks that produce smaller Q values ​​for aggregation to prevent overfitting, and realizes a deep combination of federated learning and reinforcement learning: the RSU neural network participates in aggregation but does not participate in training, and only uses the shared model after aggregation to update instead of the experience generated by the vehicle side. Protect the privacy of the vehicle side and slow down the convergence of neural networks; only select part of the neural network to participate in aggregation to reduce the network aggregation cost.

Description

Technical Field

[0001] The present invention belongs to the field of transportation and relates to a vehicle-road collaborative control system and method based on multi-agent federated reinforcement learning at complex intersections. Background Art

[0002] In recent years, research on autonomous driving has surged. However, the intelligence of a single vehicle has significant limitations. Its limited perception range and computing power may affect decision-making in complex traffic scenarios. Simply increasing costs to enhance the performance of a single vehicle is not a foolproof solution. In contrast, collaborative perception and shifting the computational burden are more realistic. Vehicle-infrastructure collaboration technology, in addition to vehicle intelligence, installs perception sensors on the roadside. After the roadside unit completes the calculation, the data is provided to the vehicle, which reduces the burden on the individual vehicle to support autonomous driving. However, in current vehicle-infrastructure collaboration technology, complex traffic scenarios and redundant traffic information directly lead to difficulties in extracting effective information, extremely high communication overhead, and poor control results. Furthermore, information asymmetry caused by privacy awareness has gradually become a major bottleneck for vehicle-infrastructure collaboration.

[0003] Federated learning is a distributed collaborative approach that allows multiple partners to independently train data and build shared models. It protects vehicle-side privacy through a unique learning architecture, training methods, and transmission principles, providing a safer learning environment and collaborative process. Reinforcement learning, on the other hand, optimizes vehicle control strategies in complex driving environments by setting complex reward functions and using trial-and-error training methods, while embodying altruism while ensuring safety. Federated reinforcement learning is a combination of federated learning and reinforcement learning. It leverages the distributed multi-agent training framework of federated learning for collaborative training. It protects privacy and significantly reduces communication overhead by transmitting network parameters rather than training data. Combined with reinforcement learning's training method of continuously improving strategies through trial and error, it demonstrates great potential in the field of autonomous driving. However, existing federated reinforcement learning algorithms have problems. Federated reinforcement learning has stringent requirements for network aggregation settings, and the two exhibit incompatibility in multi-network algorithms, resulting in unstable network convergence, poor training results, and significant network overhead. Summary of the Invention

[0004] To address the aforementioned technical issues, the present invention provides a vehicle-infrastructure cooperative control system and method based on multi-agent federated reinforcement learning for complex intersections. By leveraging road-side vantage points to guide training, this system achieves vehicle-infrastructure cooperative perception, training, and evaluation, truly enabling vehicle-infrastructure cooperative control. Furthermore, the proposed FTD3 algorithm improves upon multiple perspectives, combining federated learning with reinforcement learning. This algorithm accelerates convergence, improves convergence, and reduces communication costs while protecting vehicle-side privacy.

[0005] The technical solution of the vehicle-road cooperative control system based on multi-agent federated reinforcement learning of the present invention includes two main contents: a vehicle-road cooperative framework including a road-side static processing module, a simulation environment and sensors, and a vehicle-side dynamic processing module; and an FTD3 algorithm including a reinforcement learning module and a federated learning module.

[0006] The main purpose of the vehicle-road collaboration framework is to synthesize collaborative state variables for training. The road-side static processing module is used to obtain static road information and separate the lane centerline information from it as a static matrix to be transmitted to the vehicle-side dynamic processing module.

[0007] The simulation environment Carla is used for the interaction between the agent and the environment, and the sensors are used to obtain the dynamic state of the vehicle. The collision sensor and the line-crossing detection sensor can detect and record both collision and line-crossing events. The navigation satellite sensor can obtain the vehicle's position information, and the speed information can also be obtained through the position of two frames. The inertial sensor can obtain the vehicle's acceleration information and orientation. The specific interaction process is to use the sensor to capture the state of the agent, and then the neural network outputs the control quantity based on the state quantity, and finally the control quantity is handed over to the simulation environment Carla for execution, and this cycle continues;

[0008] The vehicle-side dynamic processing module is used to synthesize the collaborative state matrix information. It clips the static matrix obtained by the road-side static processing module according to the vehicle's position information to a 56×56 matrix centered on the center of gravity of the smart vehicle. It then stacks the matrices and sensor information of two consecutive frames to synthesize the collaborative state quantity and transmits it to the reinforcement learning module.

[0009] The FTD3 algorithm's primary goal is to output control variables based on the collaborative state matrix. The reinforcement learning module, described in this paper, is used to output a control strategy described by a Markov decision process. In a Markov decision process, the next state is dependent only on the current state and not on the previous state. This state sequence, formed under this premise, forms the basis of the reinforcement learning module of the present invention. The reinforcement learning module includes three submodules: a neural network module, a reward function module, and a network training module.

[0010] The neural network module is used to extract the features of the input collaborative state matrix and output control quantities based on the features, which are then executed by the simulation environment. In addition to the performance network and two critic networks of the traditional TD3 algorithm, each agent in FTD3 also has its own target network. The six neural network structures are exactly the same except for the output layer, using one convolutional layer and four fully connected layers to extract and integrate features. For the performance network, the output layer is mapped to [-1, 1] after the tanh activation function. Figure 1 As shown, the neural network outputs a t1Represents the steering wheel control amount in the CARLA simulator, a t2 The value is split into [-1, 0] and [0, 1], representing the brake and throttle control values, respectively. For the critic network, the output layer does not use an activation function and directly outputs the evaluation value.

[0011] The reward function module judges the quality of the output value of the neural network module based on the new state achieved after executing the action, and guides the network training module to learn. lateral and the vertical reward function r longgitudinal Two aspects are considered:

[0012] r=r lateral +r longitudinal

[0013] The first is the horizontal reward function setting:

[0014] r1 lateral =-log 1.1 (|d0|+1)

[0015] r2 lateral =-10*|sin(radians(θ))|

[0016] r lateral =r1 lateral +r2 lateral

[0017] Among them, r1 lateral is the lateral error related reward function, r2 lateral is the reward function related to heading angle deviation. The second is the longitudinal reward function setting:

[0018]

[0019]

[0020] r2 longitudinal =-|v ego -9|

[0021] r longitudinal =r1 longitudinal +r2 longitudinal

[0022] Among them, r1 longitudinal is the distance-related reward function, r2 longitudinal is the longitudinal velocity-related reward function. d0 represents the minimum distance from the ego vehicle to the lane centerline, x represents the minimum collision time, θ represents the ego vehicle’s heading angle deviation, and d min represents the minimum distance from the vehicle to the other vehicle, v ego Indicates the current speed of the vehicle.min Calculated by the Euclidean distance of the elements in the matrix:

[0023] d0=min(||a 28，28 -b center line || 2 )

[0024] d min =min(||a 28，28 -b x，y || 2 )

[0025] Among them, a 28，28 Indicates the position of the vehicle's center of gravity in the matrix, b center line represents the position of the lane centerline in the collaborative perception matrix, b x，y Indicates the position of the other vehicle’s center of gravity in the collaborative perception matrix.

[0026] The network training module is mainly used to train the neural network in the neural network module according to the set method. Under the guidance of the reward function module, the performance network and the critic network update parameters through backpropagation, and all target networks update parameters through soft updates, thereby achieving the training goal and finding the optimal solution that maximizes the cumulative benefit under specific conditions. After sampling in small batches from the experience pool, the objective function y is calculated:

[0027]

[0028]

[0029] in represents the target network strategy of the performance network, represents the normally distributed noise between constants -c, c, represents the action output after noise. r represents the immediate reward, γ represents the discount factor, Indicates that the state s′ takes the action of the dual-objective network μ′(s′|θμ′) of the performance network The smaller value, θμ′, obtained represents the parameter, θ′, of the target network of the performance network. l represents the parameters of the critic network target network. Then the critic network is updated by minimizing the loss loss:

[0030]

[0031] Where N represents the number of small batch samples, y represents the objective function, represents the value of taking action a in state s under strategy π, θ l Denotes the parameters of the critic network. After a certain delay, the performance network is updated using policy gradient descent:

[0032]

[0033] Where N represents the number of small batch samples, express Partial division of action a, express θ μ The partial points, represents the performance network, θ μ Represents the parameters of the performance network. Finally, use soft update to update the target network:

[0034] θ′ i ←τθ l +(1-τ)θ′ l

[0035] θ μ′ ←τθ μ +(1-τ)θ μ′

[0036] Where τ is the soft update parameter.

[0037] The federated learning module is mainly used to obtain the neural network parameters trained by the training module, aggregate the shared model parameters, and send the shared model parameters to the agent for local update. The federated learning module includes two small modules: the network parameter module and the aggregation module:

[0038] The network parameter module is used to obtain the parameters of each neural network before aggregation begins, and upload the parameters to the aggregation module for aggregating shared model parameters; after aggregation is completed, it is used to obtain the shared model parameters and send the parameters to each intelligent agent for local update.

[0039] The aggregation module aggregates the shared model parameters by averaging the parameters of each neural network uploaded by the network parameter module according to the aggregation interval:

[0040]

[0041] Among them, θ i is the neural network of agent i, n is the number of neural networks, θ * are the shared model parameters after aggregation.

[0042] In general, the FTD3 algorithm connects the reinforcement learning module and the federated learning module. It transmits only neural network parameters, not vehicle-side data, to protect privacy. It also selects only a subset of neural networks for aggregation, reducing communication overhead. It also selects networks with smaller Q values ​​for aggregation to prevent overfitting.

[0043] The technical solution of the vehicle-road cooperative control method based on multi-agent federated reinforcement learning of the present invention includes the following steps:

[0044] Step 1: Build the vehicle-road collaboration framework in a simulation environment. Use the roadside static processing module and the vehicle-side dynamic processing module to synthesize collaborative state variables for reinforcement learning. The roadside static processing module divides the roadside unit (RSU) bird's-eye view information into static (road, lanes, and lane centerlines) and dynamic (intelligent connected vehicles). Lane centerlines extracted from the static information serve as the basis for collaborative state variables for reinforcement learning, while dynamic information serves as the basis for state trimming. The vehicle-side dynamic processing module trims the static matrix obtained by the roadside static processing module based on vehicle position information. The trimmed 56×56 matrix serves as the perception range of a single vehicle, covering a physical space of approximately 14m×14m. To obtain more comprehensive dynamic information, the dynamic information is stacked using two consecutive frames. The dynamic processing module superimposes the trimmed static matrix and the stacked dynamic information to synthesize the collaborative state variables for FTD3.

[0045] Step 2: Formulate the control method as a Markov decision problem. The Markov decision process is described by the tuple (S, A, P, R, γ), where:

[0046] S represents the state set, which in this invention corresponds to the collaborative state quantity output by the vehicle-road collaborative framework. It consists of two matrices. The first is the collaborative perception matrix. This matrix, obtained through the proposed vehicle-side dynamic processing module, contains static road information, dynamic vehicle speed and position information, and implicit information such as the vehicle's acceleration distance from the lane centerline, direction of travel, and heading angle deviation. This feature is integrated through convolutional and fully connected layers. The second is the sensor information matrix at the current moment, which includes speed, orientation, and acceleration information obtained and calculated by the vehicle-side sensors.

[0047] A represents an action set, which corresponds to the vehicle-side throttle and steering wheel control values ​​in the present invention;

[0048] P represents the state transition equation p: S×A→P(S). For each state-action pair (s, a)∈S×A, there is a probability distribution p(·|s, a) that represents the probability of entering a new state after taking action a in state s.

[0049] R represents the reward function R: S×S×A→R, R(s t+1 , s t , a t ) indicates that the original state s t Entering a new state t+1 The reward obtained after the action is executed is defined by the reward function in this invention.

[0050] γ represents the discount factor, γ∈[0,1], which is used to calculate the cumulative return

[0051] The solution to the Markov decision problem is to find a strategy π:S→A that maximizes the cumulative reward π * :=argmax θ η(π θ In the present invention, based on the cooperative state quantity output by the vehicle-road cooperative framework, the optimal control strategy corresponding to the cooperative state matrix is ​​output through the FTD3 algorithm.

[0052] Step 3: Build the FTD3 algorithm, which consists of two main components: a reinforcement learning module and a federated learning module. The reinforcement learning module is constructed using the elements (S, A, P, R, γ) in the Markov problem, while the federated learning module is constructed using the network parameter module and the aggregation module. Each agent has its own target network in addition to the performance network and two critic networks, for a total of six neural networks.

[0053] Step 4: Interactive training is conducted in a simulation environment. The training process consists of two phases: free exploration and sampling learning. During the free exploration phase, policy noise is added to the algorithm, causing it to generate random actions. Throughout the training process, the vehicle-infrastructure collaboration framework captures and synthesizes collaborative state variables. The FTD3 algorithm then uses these states as input and outputs noisy actions. After the action is executed, the vehicle-infrastructure collaboration framework captures the new state variables, and the reward function module finally determines the quality of the action. This tuple consisting of the state variable, action, next state variable, and reward function is the experience. Randomly generated experience samples are stored in an experience pool. Once the number of experiences reaches 3,000 or more, training enters the sampling learning phase. Samples are drawn from the experience pool in small batches and learned according to the training method of the FTD3 network training module. Policy noise is reduced as the learning progresses.

[0054] Step 5: Obtain the parameters of each neural network through the network parameter module in federated learning and upload the parameters to the aggregation module of the roadside unit (RSU). The aggregation module aggregates the parameters of each neural network uploaded by the network parameter module into shared model parameters using parameter averaging according to the aggregation interval.

[0055] Step 6: Use the network parameter module in federated learning to send the aggregated shared model to the vehicle side for model update, and repeat the cycle until the network converges.

[0056] Preferably, in step 2, the size of the collaborative state quantity is a collaborative state matrix of (56*56*1) and a sensor information matrix of (3*1).

[0057] Preferably, in step 3, the neural network model structure used in the performance network in the FTD3 algorithm consists of one convolutional layer and four fully connected layers. Except for the last layer, which uses a tanh activation function to map the output to the interval [-1, 1], all other layers use a ReLU activation function. The critic network also uses one convolutional layer and four fully connected layers. Except for the last layer, which does not use an activation function and directly outputs a Q value for evaluation, all other layers use a ReLU activation function.

[0058] Preferably, in step 4, during the network training process, the learning rates selected for the Actor and Critic networks are both 0.0001; the policy noise is 0.2; the delay update parameter is 2; the discount factor γ is 0.95; and the target network update weight tau is 0.995.

[0059] Preferably, in step 4, the maximum capacity of the experience pool is selected to be 10,000; and the number of minibatches extracted from the experience pool is 128.

[0060] Preferably, in step 5, the neural network used by the roadside end-unit (RSU) participates in aggregation but not in training; only some neural networks (the performance network, the performance network's target network, and the critic target network that produces a higher number of smaller Q values) are selected for aggregation. Regarding the selection of the critic target network, for example, when the sample extraction minibatch is 128, the two critic target networks each score 128 samples. Samples with more than 64 smaller Q values ​​are selected for aggregation.

[0061] Beneficial effects of the present invention:

[0062] (1) The present invention uses a vehicle-road collaborative control framework based on a road-side static processing module and a vehicle-side dynamic processing module. To address the difficulty of feature extraction, innovative collaborative state quantities are constructed by leveraging the advantages of the road side to reduce the difficulty of training. This framework implements vehicle-side and road-side collaborative perception, collaborative training, and collaborative evaluation, truly realizing vehicle-road collaborative control and providing new ideas for vehicle-road collaboration.

[0063] (2) The present invention uses the proposed FTD3 algorithm to address existing technical issues and improves them from multiple aspects. To address user privacy issues, FTD3 only transmits neural network parameters rather than vehicle-side samples to protect privacy. To address the problem of huge communication overhead, FTD3 only selects part of the network for aggregation to reduce communication costs. To address the problem of overfitting, FTD3 uses a filter to aggregate only neural networks with smaller Q values. Unlike the previous hard connection between federated learning and reinforcement learning, this achieves a deep combination of the two. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 The vehicle-road collaboration framework proposed in this invention;

[0065] Figure 2 Schematic diagram of collaborative perception set by the present invention;

[0066] Figure 3 The neural network structure used in the present invention;

[0067] Figure 4 The framework of the FTD3 algorithm proposed in this invention. DETAILED DESCRIPTION

[0068] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the content of the present invention is not limited thereto.

[0069] The present invention provides a vehicle-road cooperative control framework and FTD3 algorithm based on federated reinforcement learning, which can realize multi-vehicle control in roundabout conditions. Specifically, it includes the following steps:

[0070] (1) Build a vehicle-road cooperative control framework in the CARLA simulator, such as Figure 1 As shown in the figure, it includes a smart car with a camera RSU and multiple sensors, and initializes the corresponding road-side static processing module and vehicle-side dynamic processing module to build collaborative perception, such as Figure 2 As shown in the figure, various sensors are used to obtain the vehicle's dynamic state information. The collision sensor and the wire crossing detection sensor can detect and record collision and wire crossing events. The navigation satellite sensor can obtain the vehicle's position information, and speed information can also be obtained from the position of the two frames. The inertial sensor can obtain the vehicle's acceleration information and heading.

[0071] (2) Construct the FTD3 algorithm and assign a neural network to the agent, such as Figure 3 As shown in the figure. The network input, output, and reward function are determined. The input is the collaborative state quantity, which consists of two matrices. The first is the collaborative perception matrix. Through the proposed vehicle-side dynamic processing module, the collaborative perception matrix obtained contains static road information, dynamic vehicle speed, position information, and implicit information such as the distance of the vehicle acceleration from the lane centerline, travel direction, and heading angle deviation. The second is the sensor information matrix at the current moment, which includes speed information, orientation, and acceleration information obtained and calculated by the vehicle-side sensors. The two matrices are respectively extracted and integrated through the corresponding convolutional layer and fully connected layer.

[0072] The output is combined with the vehicle control method in the Carla simulator. The output layer of the neural network module is mapped to [-1, 1] after the tanh activation function, such as Figure 1 As shown, a t1 Represents the steering wheel control amount in the CARLA simulator, a t2It is split into [-1, 0] and [0, 1], representing the brake and throttle control amounts respectively.

[0073] The reward function is set up from both horizontal and vertical perspectives. The reward function will judge the quality of the intelligent car's actions and guide training:

[0074] r=r lateral +r longitudinal

[0075] The first is the horizontal reward function setting:

[0076] r1 lateral =-log 1.1 (|d0|+1)

[0077] r2 lateral =-10*|sin(radians(θ))|

[0078] r lateral =r1 lateral +r2 lateral

[0079] The second is the vertical reward function setting:

[0080]

[0081]

[0082] r2 longitudinal =-|v ego -9|

[0083] r longitudinal =r1 longitudinal +r2 longitudinal

[0084] Where d0 represents the minimum distance from the vehicle to the center line of the lane, θ represents the heading angle deviation of the vehicle, and d min represents the minimum distance from the vehicle to the other vehicle, v ego Indicates the current speed of the vehicle. min Calculated by the Euclidean distance of the elements in the matrix:

[0085] d0=min(||a 28，28 -b center line || 2 )

[0086] d min =min(||a 28，28 -b x，y || 2 )

[0087] where bcenter line represents the position of the lane centerline in the collaborative perception matrix, b x，y Indicates the position of the other vehicle’s center of gravity in the collaborative perception matrix.

[0088] (4) Based on the OpenDD real driving dataset, random positions and initial velocities are obtained and combined with random noise to enable the reinforcement learning agent to generate experience in the interaction with the simulation environment and store it in a pre-set experience pool.

[0089] (5) When the experience pool is full, the system extracts minibatches from the experience pool and trains the network using the gradient descent method. The parameters used in the training are: the learning rate selected for both the Actor and Critic networks is 0.0001; the policy noise is 0.2; the delay update parameter is 2; the discount factor γ is 0.95; the target network update weight tau is 0.995; the maximum capacity of the experience pool is selected as 10,000, and the minibatches extracted from the experience pool are 128. Specific algorithm flow: After sampling from the experience pool in small batches, calculate the objective function y:

[0090]

[0091]

[0092] Where r represents the immediate return, γ represents the discount factor, Represents the dual-objective network μ′(s′|θ) of the state s′ taking the performance network μ′ ) strategy The smaller value obtained, θ μ′ The parameters of the target network representing the performance network, θ′ l represents the parameters of the critic network target network. Then the critic network is updated by minimizing the loss loss:

[0093]

[0094] Where N represents the number of small batch samples, y i represents the objective function, represents the value of taking action a in state s under strategy π, θ l Denotes the parameters of the critic network. After a certain delay, the performance network is updated using policy gradient descent:

[0095]

[0096] Where N represents the number of small batch samples, express Partial division of action a, express θ μThe partial of μ(s|θ μ ) represents the performance network, θ μ Represents the parameters of the performance network. Finally, use soft update to update the target network:

[0097] θ′ l ←τθ l +(1-τ)θ′ l

[0098] θ μ′ ←τθ μ +(1-τ)θ μ′

[0099] Where τ represents the soft update parameter. At a certain aggregation interval, the network parameter module selects the parameters of some networks (the performance network, the target network of the performance network, and the critic target network that produces smaller Q values) and sends them to the aggregation module for aggregation to generate a shared model, such as Figure 4 As shown. Then send the aggregated shared model to the vehicle side for model update. The specific algorithm flow is as follows:

[0100]

[0101]

[0102] For the initialization process, Q1(s,a|θ) i ,Q2(s,a|θ) i ,μ(s|θ) i are two critic networks and one performance network for the i-th agent, θ 1,i ,θ 2,i , Its network weight. i ,Q2′ i ,μ′ i is the target network of the i-th agent, θ 1,i ′,θ 2,i ′, Its network weight, R i is the experience pool of the i-th agent. is the collaborative state of the ith agent, where is the collaborative state matrix of the ith agent, is the static information obtained by the road-side static processing module of the i-th agent, The dynamic information obtained by the vehicle-side dynamic processing module of the i-th intelligent agent, is the sensor information, including heading angle yaw, velocity v, acceleration a. For action output, represents the target network strategy of the performance network of the i-th agent, represents the normally distributed noise between constants -c, c, Represents the action output after noise. For the objective function calculation, y represents the objective function, r represents the immediate return, γ represents the discount factor, Indicates that the i-th agent is in state s T+1 Take targeted network actions that perform network For the critic network update, N represents the number of mini-batch samples, Indicates state s T Take action a under policy π t For performance network updates, represents the gradient, express For action a t The partial express right For soft update, τ is the soft update parameter.

[0103] The specific process is described as follows: The agent's neural network and experience pool are randomly initialized. When the experience pool contains fewer than 3000 samples, the agent enters a random exploration phase. Vehicle dynamic information is obtained from the vehicle's sensors, while the roadside static module obtains static road information. The vehicle-side dynamic module then crops the road information into a 56×56 matrix centered on the vehicle's center of gravity. The matrix and sensor information from two consecutive frames are then stacked to generate the collaborative state. Based on the state, the neural network module outputs steering and throttle control variables with normally distributed noise, which are then executed in the simulation environment. Vehicle dynamic information is again obtained from the vehicle's sensors, while the roadside static module obtains static road information. The vehicle-side dynamic module then crops the road information into a 56×56 matrix centered on the vehicle's center of gravity. The matrix and sensor information from two consecutive frames are then stacked to generate the collaborative state for the next moment. The reward function module then obtains a specific reward based on the new state. The collaborative state, control variable, reward, and the next collaborative state are stored as tuples in the experience pool. When the experience pool contains 3000 or more experiences, the normally distributed noise begins to decay, and the training phase begins. Learning is performed by sampling from the experience pool in mini-batches. The performance network and critic network are trained using gradient descent, while the other target networks are trained using soft updates. At the aggregation interval, the network parameter module obtains the parameters of the performance network, the target network of the performance network, and the critic target network that produces the most low Q values ​​before aggregation begins. These parameters are then uploaded to the aggregation module for aggregation of shared model parameters. After aggregation is complete, the network parameter module obtains the shared model parameters and distributes them to each agent for local update. This cycle continues until the network converges.

[0104] (6) Feasibility analysis: The proposed control method based on federated reinforcement learning can still perform well even in a communication environment with delays. This is mainly due to the algorithm characteristics of only transmitting neural network parameters and the algorithm setting of only selecting individual networks to participate in aggregation. These advantages make its communication requirements less demanding and can work in existing Wi-Fi and 4G environments, with a wider range of application scenarios.

[0105] In summary, the vehicle-road collaborative control framework proposed in the present invention, which is based on the road-side static processing module and the vehicle-side dynamic processing module, constructs innovative collaborative state quantities and reward functions through the advantages of the road side, realizes vehicle-side and road-side collaborative perception, collaborative training, and collaborative evaluation, and truly realizes vehicle-road collaborative control. In addition, a federated reinforcement learning algorithm FTD3 is proposed to improve the algorithm performance from three aspects, and realize the deep integration of federated learning and reinforcement learning: the RSU neural network participates in aggregation but not in training, and only uses the aggregated shared model update instead of the experience generated by the vehicle side. Protect the privacy of the vehicle side and slow down the convergence of neural networks; only select some neural networks to participate in aggregation to reduce the network aggregation cost; select target networks that generate more smaller Q values ​​for aggregation to further prevent over-estimation. The proposed FTD3 algorithm is different from the hard connection of federated learning and reinforcement learning, and realizes the deep integration of the two.

[0106] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent methods or changes that do not deviate from the technology of the present invention should be included in the scope of protection of the present invention.

Claims

1. A vehicle-road cooperative control system based on multi-agent federated reinforcement learning at complex intersections, characterized by: It includes a vehicle-road collaborative framework part and an FTD3 algorithm part; the vehicle-road collaborative framework part includes a road-side static processing module, a sensor module, and a vehicle-side dynamic processing module, which are used to synthesize collaborative state quantities, wherein the road-side static processing module is used to obtain static road information, and separately separate the lane centerline information from it as a static matrix and transmit it to the vehicle-side dynamic processing module; the sensor is used to obtain the dynamic state quantity of the vehicle; the vehicle-side dynamic processing module is used to synthesize collaborative state matrix information, and the static matrix obtained by the road-side static processing module is cropped according to the vehicle's position information, and then the matrices and sensor information of two consecutive frames are stacked to synthesize the collaborative state quantity, and transmit it to the FTD3 algorithm part; the FTD3 algorithm part outputs the control quantity according to the collaborative state matrix, and includes a reinforcement learning module and a federated learning module, wherein the reinforcement learning module is used to output the control strategy, adopting a Markov decision process, and the federated learning module is used to obtain the neural network parameters trained by the reinforcement learning module, aggregate the shared model parameters, and send the shared model parameters to the intelligent agent for local update; The reinforcement learning module includes: a neural network module, a reward function module, and a network training module; The neural network module is used to extract the features of the collaborative state matrix and output the control amount based on the features. In addition to the performance network and two critic networks, the individual agents in FTD3 also have their own target networks. The six neural network structures are exactly the same except for the output layer. One convolutional layer and four fully connected layers are used to extract and integrate features. For the performance network, the output layer is mapped to [-1, 1] after the tanh activation function. The neural network outputs a t1 Represents the steering wheel control amount in the CARLA simulator, a t2 The value is split into [-1, 0] and [0, 1], representing the brake and throttle control values ​​respectively. For the critic network, the output layer does not use an activation function and directly outputs the evaluation value. The reward function module judges the quality of the output value of the neural network module according to the new state achieved after executing the action, and guides the network training module to learn, including the horizontal reward function r lateral and the vertical reward function r longitudinal : r＝r lateral +r longitudinal The lateral reward function: r1 lateral =-log 1.1 (|d0|+1) r2 lateral =-10*|sin(radians(θ))| r lateral ＝r1 lateral +r2 lateral Among them, r1 lateral is the lateral error related reward function, r2 lateral is the reward function related to heading angle deviation; the longitudinal reward function: r2 longitudinal ＝-|v ego -9| r longitudinal ＝r1 longitudinal +r2 longitudinal Among them, r1 longitudinal is the distance-related reward function, r2 longitudinal is the longitudinal velocity-related reward function, where d0 represents the minimum distance from the vehicle to the center line of the lane, θ represents the heading angle deviation of the vehicle, and d min represents the minimum distance from the vehicle to the other vehicle, v ego Indicates the current speed of the vehicle, d0, d min Calculated by the Euclidean distance of the elements in the matrix: d0=min(||a 28,28 -b center line || 2 ) d min =min(||a 28,28 -b x,y || 2 ) Among them, a 28,28 represents the center of gravity of the vehicle, b center line represents the position of the lane centerline in the collaborative perception matrix, b x,y Indicates the position of the other vehicle's center of gravity in the collaborative perception matrix; The network training module is used to train the neural network in the neural network module according to the set method. Under the guidance of the reward function module, the performance network and the critic network update parameters through backpropagation, and all target networks update parameters through soft update, so as to achieve the training purpose and find the optimal solution that maximizes the cumulative benefit; sampling from the experience pool in small batches, calculate the objective function y: in represents the target network strategy of the performance network, represents the normally distributed noise between constants -c, c, represents the action output after noise, where r represents the immediate reward, γ represents the discount factor, Represents the state s' taking the dual-objective network μ'(s'|θ μ' ) action The smaller value obtained, θ μ' The parameters of the target network representing the performance network, θ' l Represents the target network parameters of the critic network, and then updates the critic network by minimizing the loss loss: Where N represents the number of small batch samples, y i represents the objective function, represents the value of taking action a in state s under strategy π, θ l Represents the parameters of the critic network, and updates the performance network using policy gradient descent: Where N represents the number of small batch samples, express Partial division of action a, express θ μ The partial points, represents the performance network, θ μ Represents the parameters of the performance network, using soft update to update the target network:

2. The vehicle-road cooperative control system based on multi-agent federated reinforcement learning at complex intersections according to claim 1 is characterized in that: The sensor module includes a collision sensor, a line pressure sensor, a navigation satellite sensor, and an inertial sensor. The collision sensor and the line pressure detection sensor detect and record collision and line pressure events respectively. The navigation satellite sensor can obtain the vehicle's position information and speed information, and the inertial sensor can obtain the vehicle's acceleration information and facing direction.

3. The vehicle-road cooperative control system based on multi-agent federated reinforcement learning at complex intersections according to claim 1 is characterized in that: The federated learning module includes a network parameter module and an aggregation module; The network parameter module is used to obtain the parameters of each neural network before the aggregation begins, and upload the parameters to the aggregation module for aggregating the shared model parameters; after the aggregation is completed, it is used to obtain the shared model parameters and send the shared model parameters to each agent for local update; The aggregation module aggregates the shared model parameters by averaging the parameters of each neural network according to the aggregation interval: Among them, θ i is the neural network of agent i, n is the number of neural networks, θ * are the shared model parameters after aggregation.

4. The vehicle-road cooperative control system based on multi-agent federated reinforcement learning at complex intersections according to any one of claims 1 to 3, characterized in that: It also includes a simulation module, which is used for intelligent agent interaction.

5. The control method of the vehicle-road cooperative control system based on multi-agent federated reinforcement learning at a complex intersection according to claim 1 is characterized in that: The steps include: Step 1: Build a vehicle-road collaboration framework in a simulation environment. Use the roadside static processing module and the vehicle-side dynamic processing module to synthesize collaborative state quantities for reinforcement learning. The roadside static processing module divides the roadside unit (RSU) bird's-eye view information into static and dynamic types. The lane centerline extracted from the static information will serve as the basis for the reinforcement learning collaborative state quantity, while the dynamic information will be used as the basis for state quantity cropping. The vehicle-side dynamic processing module will crop the static matrix obtained by the roadside static processing module based on the vehicle's position information and coordinate transformation. The cropped 56×56 matrix serves as the perception range of a single vehicle, covering a physical space of 14m×14m. To obtain more comprehensive dynamic information, the dynamic information is stacked using two consecutive frames. The dynamic processing module superimposes the cropped static matrix and the stacked dynamic information to synthesize the collaborative state quantity for FTD3. Step 2: Model the control process as a Markov decision process, which is described by the tuple (S, A, P, R, γ), where: S represents the state set, corresponding to the collaborative state quantity output by the vehicle-road collaborative framework. It consists of two matrices: the collaborative perception matrix. The collaborative perception matrix obtained through the proposed vehicle-side dynamic processing module contains static road information, dynamic vehicle speed and position information, and implicit information such as the distance from the lane centerline, travel direction, and heading angle deviation of the vehicle acceleration. These features are integrated through convolutional and fully connected layers. The second matrix is ​​the sensor information matrix at the current moment, which includes speed information, orientation, and acceleration information obtained and calculated by the vehicle-side sensors. A represents the action set, corresponding to the vehicle-side throttle and steering wheel control quantities; P represents the state transition equation p:S×A→P(S). For each state-action pair (s,a)∈S×A, there is a probability distribution p(·|s,a) that represents the probability of entering a new state after taking action a in state s. R represents the reward function R: S×S×A→R, R(s t+1 ,s t ,a t ) indicates that the original state s t Entering a new state t+1 The reward function is used to define the quality of the action. γ represents the discount factor, γ∈[0,1], which is used to calculate the cumulative return The solution to the Markov decision problem is to find a strategy π:S→A that maximizes the cumulative reward π * :=argmax θ η(π θ ), which is based on the cooperative state quantity output by the vehicle-road cooperative framework, and then outputs the optimal control strategy corresponding to the cooperative state matrix through the FTD3 algorithm; Step 3: Design the FTD3 algorithm, which includes a reinforcement learning module and a federated learning module. The reinforcement learning module is constructed by the elements (S, A, P, R, γ) in the Markov problem, and the federated learning module is constructed by the network parameter module and the aggregation module. Step 4: Conduct interactive training in a simulation environment. The training process includes two phases: free exploration and sampling learning. In the free exploration phase, the algorithm's policy noise is increased to cause it to generate random actions. Throughout the training process, the vehicle-road collaboration framework captures and synthesizes the collaborative state. The FTD3 algorithm then uses the collaborative state as input and outputs a noisy action. After the action is executed, the vehicle-road collaboration framework captures the new state. Finally, the reward function module determines the quality of the action. The tuple consisting of the state, action, next state, and reward function is the experience. The randomly generated experience samples are stored in the experience pool. When the amount of experience meets certain conditions, the training enters the sampling learning phase. Samples are extracted from the experience pool in small batches and learned according to the training method of the FTD3 network training module. The policy noise will decay with the increase of learning level. Step 5: Obtain the parameters of each neural network through the network parameter module in federated learning, and upload the parameters to the aggregation module. The aggregation module aggregates the parameters of each neural network uploaded by the network parameter module into shared model parameters by parameter averaging according to the aggregation interval; Step 6: Use the network parameter module in federated learning to send the aggregated shared model parameters to the vehicle side for model update, and repeat the cycle until the network converges.

6. The control method of the vehicle-road cooperative control system based on multi-agent federated reinforcement learning at complex intersections according to claim 5 is characterized in that: In step 2, the size of the collaborative state matrix is ​​a (56*56*1) collaborative state matrix and a (3*1) sensor information matrix.

7. The control method of the vehicle-road cooperative control system based on multi-agent federated reinforcement learning at complex intersections according to claim 5 is characterized in that: In step 3, the neural network model structure used by the performance network in the reinforcement learning module of the FTD3 algorithm includes 1 convolutional layer and 4 fully connected layers. Except for the last layer of the network which uses the tanh activation function to map the output to the [-1, 1] interval, the other layers use the relu activation function. The critic network also uses 1 convolutional layer and 4 fully connected layers. Except for the last layer of the network which does not use the activation function to directly output the Q value for evaluation, the other layers use the relu activation function.

8. The control method of the vehicle-road cooperative control system based on multi-agent federated reinforcement learning at complex intersections according to claim 5 is characterized in that: In step 4, during the network training process, the learning rate selected for the performance network and the critic network is 0.0001; the policy noise is 0.2; the delay update parameter is 2; the discount factor γ is 0.95; the target network update weight tau is 0.995; the maximum capacity of the experience pool is selected to be 10,000; and the minibatch extracted from the experience pool is 128.

9. The control method of the vehicle-road cooperative control system based on multi-agent federated reinforcement learning at complex intersections according to claim 5 is characterized in that: In step 5, the six neural networks used by the intelligent agent RSU participate in aggregation but do not participate in training; only some neural networks are selected to participate in aggregation, and the target network that produces more smaller Q values ​​is selected for aggregation.

Images