Federated reinforcement learning end-to-end autonomous driving control system and method based on complex network cognition, and vehicle equipment

Abstract

The present invention discloses a federated reinforcement learning end-to-end autonomous driving control system and method and vehicle equipment based on complex network cognition, and provides a federated reinforcement learning end-to-end autonomous driving algorithm framework FLDPPO based on complex network cognition under dense urban traffic. By designing a loss function, the combination of rule-based complex network cognition and end-to-end federated reinforcement learning is realized. FLDPPO uses dynamic driving suggestions to guide the intelligent agent to learn rules so that the intelligent agent can cope with complex urban driving environments and dense traffic scenarios. In addition, the proposed framework uses a multi-intelligent federated reinforcement learning architecture to train the model by parameter aggregation. On the basis of protecting the privacy of the vehicle side, it accelerates network convergence, reduces communication consumption, and achieves a balance between sample efficiency and high robustness of the model.

Description

Technical Field

[0001] The present invention belongs to the field of transportation and autonomous driving, and relates to a federated reinforcement learning end-to-end autonomous driving control system and method based on complex network cognition under dense urban traffic, and vehicle equipment. Background Art

[0002] Urban driving involves extensive interactions between driving agents, traffic participants, and road infrastructure, and is an open problem in the field of transportation. It is difficult to model the environment and agent cognition in dynamic interactions that present high dimensionality and diversity. Although traditional rule-based methods are good at handling simple driving scenarios, rule-based solutions are difficult to apply when faced with complex traffic situations.

[0003] On the other hand, end-to-end methods do not rely on hand-crafted rules, but instead achieve competitive results through data-driven training methods. Deep reinforcement learning is a typical data-driven algorithm that generates samples by interacting with the environment. The generated samples are stored in the experience pool and used for model training through micro-batch sampling. The characteristics of deep reinforcement learning lead to its shortcomings in the application process. First, deep reinforcement learning requires a large number of samples, which makes it more difficult to converge when training large networks compared to supervised learning. If the samples are insufficient, the robustness of the model will be limited. Secondly, the use of the experience pool imposes restrictions on the network input, such as the size of the image input. The computational load and communication consumption of the deep reinforcement learning algorithm with image input will increase linearly with training. In addition, end-to-end deep reinforcement learning, as a black box algorithm, makes it difficult for us to understand the motivation of the agent to make decisions and lacks interpretability. Due to the above limitations, deep reinforcement learning was mainly applied to simpler driving scenarios in the early stages.

[0004] Although complex research scenarios can be divided into multiple single objectives to deconstruct complex environments, the single-objective research method is insufficient in understanding scenarios where multiple elements interact with each other. Although deconstructing complex environments has a positive impact, simple splitting limits the development of autonomous driving applications in complex environments. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a federated reinforcement learning end-to-end autonomous driving algorithm framework FLDPPO based on complex network cognition under dense urban traffic. By designing the loss function, it realizes the combination of rule-based complex network cognition and end-to-end federated reinforcement learning. FLDPPO uses dynamic driving suggestions to guide the agent to learn rules so that the agent can cope with complex urban driving environments and dense traffic scenes. In addition, the proposed framework uses a multi-intelligent federated reinforcement learning architecture to train the model by parameter aggregation. On the basis of protecting the privacy of the vehicle side, it accelerates network convergence, reduces communication consumption, and achieves a balance between sample efficiency and high robustness of the model.

[0006] The technical solution of the federated reinforcement learning end-to-end autonomous driving control system based on complex network cognition of the present invention includes five parts: a state quantity encoder, an image encoder, a complex network cognition module, a reinforcement learning module and a federated learning module.

[0007] The state quantity encoder is mainly used to obtain the state quantities required by the complex network cognitive module and the reinforcement learning module. The state quantities required by the complex network cognitive module include the x-coordinate, y-coordinate, heading angle change and speed of the driving subject, which are handed over to the complex network cognitive module as input. The state quantities required by the reinforcement learning module include the steering wheel angle, throttle, brake, gear, lateral speed and longitudinal speed, which are handed over to the reinforcement learning module as partial input after the features are extracted by the two-layer fully connected network;

[0008] The image encoder is mainly used to obtain the implicit state quantity of the image required by the reinforcement learning module. The image is a 15-channel semantic bird's-eye view, i RL ∈[0,1] 192*192*15 . 192 units are pixels, and the bird's-eye view used is 5px / m. The 15 channels include the drivable domain, the expected path, the road sidelines, 4 frames of other vehicles, 4 frames of pedestrians, and 4 frames of traffic signs. The expected path is calculated using the A* algorithm. The semantic bird's-eye view extracts implicit features through multiple convolutional layers and is then handed over to the reinforcement learning module as another input;

[0009] The complex network cognitive module is mainly used to model the driving situation of the driving subject, and obtain the maximum risk value of the driving subject in the current driving situation according to the state quantity provided by the state quantity encoder, and finally output dynamic driving suggestions based on the risk value through the activation function.

[0010] The modeling process of the complex network cognitive module takes traffic participants and road facilities as nodes to construct a dynamic complex network model:

[0011] G t =(P,E,W,Θ) t

[0012] Among them, G t represents the dynamic complex network at time t, P = {p1, p2, ..., p N} is a set of nodes, the number of nodes is N; E = {e 1,2 ,e 1,3 ,…,e i,j} is a set of edges, the number of edges is e i,j Represents node p i and p j The connection between them; W = {w 1,2 ,w 1,3 ,…,w i,j} is the weight set of the edge, w i,j Represents node p i and p j The coupling strength between them; Θ is the active area of ​​the node, which represents the dynamic constraints on the nodes in the network. Θ is modeled as a smooth bounded surface:

[0013] F Θ (x,y,z)=0,st

[0014] Where Ω is the boundary of the sliding surface. Consider a continuous-time dynamic network with N nodes on Θ, whose node state equation is:

[0015]

[0016] Among them, X i ∈R m Represents node p i The state vector, R m represents the vector space composed of m-dimensional real numbers R, U i ∈R q is the input vector, R q represents the vector space composed of q-dimensional real numbers R, A i represents the dynamic matrix, B i Represents the input matrix. Based on the node state equation, we can get node p i The output vector is:

[0017] Y i =f i (X i )

[0018] f i represents the output function of the node. Then the node p i and p j The weight function between is:

[0019] w ij=F(Y i ,Y j )

[0020] F represents the weight function between nodes. The present invention uses Gaussian function to reveal the static characteristics between nodes:

[0021]

[0022] Among them, S sta represents the static field strength, C a represents the field strength coefficient, x0 and y0 represent the coordinates of the risk center O(x0,y0), a x and b y Represents the vehicle appearance coefficient, specifically the length and width. The safety field has the characteristic of shape anisotropy:

[0023]

[0024] φ=a x / b y = l v / w v

[0025] Where φ is the aspect ratio, l v Indicates vehicle length, w v Indicates the width of the vehicle.

[0026] The safety field is delineated by multiple equal field lines, and its top view projection is the area covered by a series of ellipses. The area covered by the smallest central ellipse is the core domain, the area between the smallest central ellipse and the second ellipse is the restricted domain, and the area between the second ellipse and the largest ellipse is the extended domain. The size and shape of the area are determined by the equal field lines, which are related to the shape and motion state of the vehicle and are described based on the Gaussian function. The direction of the safety field is consistent with the direction of vehicle motion.

[0027] When the vehicle is in motion, the risk center O(x0, y0) of the safety field will be transferred to the new risk center O′(x′0, y′0) as the vehicle moves:

[0028]

[0029] Among them, k v represents the adjustment factor, and Its positive or negative value is related to the direction of movement, and β represents the transfer vector The angle between the coordinate axis and the Cartesian coordinate system. Under the action of risk center transfer, a virtual vehicle is formed with a length of l′ v , width is w′ v , the dynamic safety field after the change:

[0030]

[0031] Among them, S dyn represents the dynamic field strength, and the new aspect ratio is expressed as φ′=a′ x / b′ y = l′ v / w′ v As the vehicle's motion state changes, the shape of the Gaussian safety field will change, thereby changing the three major fields covered by the safety field: the core field, the restricted field, and the extended field.

[0032] According to different levels of human driving reaction time, the present invention divides risk cognition into three layers within a plane range: a first cognitive domain, a second cognitive domain and an extra-domain space.

[0033] First cognitive domain:

[0034] a′ x ≤s th1

[0035] s th1 =t c1 ·v e

[0036] Second cognitive domain:

[0037] s th1 <a′ x ≤s th2

[0038] s th2 =t c2 ·v e

[0039] Extraterritorial space:

[0040] s th2 <a′ x

[0041] Among them, s th1 represents the first cognitive domain threshold, which is the first reaction time t of human driving c1 (usually 3s) and the maximum approaching speed v of other nodes relative to the ego vehicle e Get.s th2 represents the second cognitive domain threshold, and the second reaction time t of human driving c2 (usually 6s) and the maximum approaching speed v of other nodes relative to the ego vehicle e get.

[0042] Under the variable safety field model, the risk perception function between nodes is established:

[0043]

[0044] in, Represents node p i At node p j The field strength at k c represents the risk-adjusted perception coefficient, Represents node p j The scalar velocity, θ i,j Represents node p j The velocity vector With the field strength vector The angle between the two (clockwise is positive). The risk value Risk(p i ,p j ), which indicates the coupling strength between nodes. The larger the risk value, the greater the coupling strength, which means the greater the correlation between nodes.

[0045] The activation function is mainly used to map the risk value. Activate(Risk) uses different activation functions according to different driving suggestions. The mapped risk value will serve as the basis for guiding the output strategy of the reinforcement learning module:

[0046]

[0047]

[0048] Among them, Activate go (Risk) indicates the activation function when the suggestion is forward. top (Risk) indicates the activation function when the recommendation is to stop, and Risk indicates the current risk value of the vehicle. risk :

[0049] B risk =B(Activate go (Risk),β go ),go

[0050] B risk =B(α stop ,Activate stop (Risk)),stop

[0051] Among them, B represents beta distribution, α stop =β go =1.

[0052] For the reinforcement learning module, it is mainly used to integrate the state quantities output by the state quantity encoder and the image encoder, output the corresponding strategy according to the integrated network input, and interact with the environment to generate experience stored in the local experience pool in the federated learning module. When the number of samples reaches a certain threshold, samples are taken from the local experience pool for training, and finally the trained neural network parameters are uploaded to the federated learning module.

[0053] The interactive environment is the Carla simulator, which realizes vehicle control by inputting the control amount of the steering wheel, accelerator and brake, where the steering wheel ∈ [-1, 1], the accelerator ∈ [0, 1], and the brake ∈ [0, 1]. Based on the control method of Carla, the action space of reinforcement learning ∈ [-1, 1] 2 , where the actions are classified into two types: steering wheel angle and accelerator brake. When outputting accelerator brake, [-1,0] represents brake, and [0,1] represents accelerator. The present invention uses beta distribution to output two parameters of distribution through reinforcement learning, and then obtains the strategy action through sampling:

[0054] Beta=B(α,β),α,β>0

[0055] Compared to the Gaussian distribution commonly used in model-free reinforcement learning, the Beta distribution has boundaries and does not require mandatory constraints.

[0056] The experience generated by the interaction is a tuple, including the state quantity at the previous moment, action, reward function, state quantity at the next moment and dynamic driving suggestions. Among them, the reward function uses the mapped risk value as the weight of the termination reward to weight the reward function.

[0057] r=r speed +r position +r action +Activate(Risk)*r terminal

[0058]

[0059] r position =-0.5*Δd-Δθ

[0060] Among them, r speed represents the speed-related reward function, r position represents the position-dependent reward function, r action represents the action-related reward function, r terminal represents the terminal state related reward function, v represents the vehicle speed, v desire represents the expected speed, v max =6m / s represents the maximum speed. Δd represents the lateral distance between the vehicle and the desired path, and Δθ represents the angle between the vehicle's driving direction and the tangent of the desired path. Table 1 describes r in detailaction and r terminal The value of , where Δsteering represents the change in the steering wheel angle between two frames.

[0061] Table 1

[0062]

[0063] The training process of the reinforcement learning module updates parameters through the following loss function:

[0064]

[0065]

[0066]

[0067]

[0068] in, represents the policy gradient clipping loss estimated using the generalized advantage estimate. represents the maximum entropy loss, H(π θ (·∣i RL ,m RL )) represents the strategy π θ In the image input i RL and state input m RL The entropy under represents uniform distribution. It can make the action distribution close to uniform distribution to encourage the agent to explore, exp Represents the weight of the maximum entropy loss. represents the loss based on the dynamic risk advice, It means that by calculating the driving subject N before the termination state z = KL divergence of the strategy and dynamic driving suggestions outputted in 100 steps to guide the agent, λ risk Represents the weight of the dynamic proposal loss.

[0069] The federated learning module is mainly used to receive the neural network parameters uploaded by the reinforcement learning modules of each agent, aggregate the global parameters based on multiple neural network parameters, and finally send the global parameters to each agent until the network converges. The global parameter aggregation is performed by the following equation:

[0070]

[0071] in, represents the global parameters at time m, N represents the number of agents, Represents the neural network parameters of the nth agent at time m.

[0072] In general, the FLDPPO algorithm combines rule-based complex network cognition with end-to-end federated reinforcement learning by designing a loss function. In addition, the multi-intelligent federated reinforcement learning architecture is used to train the model by parameter aggregation, which accelerates network convergence, reduces communication consumption, and achieves a balance between sample efficiency and high robustness of the model while protecting the privacy of the vehicle.

[0073] The technical solution of the federated reinforcement learning end-to-end autonomous driving control method based on complex network cognition of the present invention includes the following steps:

[0074] Step 1: Build a dense urban traffic simulation environment in the Carla simulator. The simulation environment includes driving subjects, traffic participants, and traffic infrastructure.

[0075] The driving subject is a plurality of intelligent agents, which are modeled as Markov decision processes respectively, and a reinforcement learning module is used to control the steering wheel, throttle and brake. The Markov decision process is described by a tuple (S, A, P, R, γ), where S represents the state set, which corresponds to the state quantity obtained by the state quantity encoder and the image encoder in the present invention, including the steering wheel angle, throttle, brake, gear, lateral speed and longitudinal speed, and a 15-channel semantic bird's-eye view; A represents the action set, which corresponds to the steering wheel, throttle and brake control quantity of the driving subject in the present invention; P represents the state transfer equation p: S × A → P (S), and for each state-action pair (s, a) ∈ S × A, there is a probability distribution p (·|s, a) representing the possibility of entering a new state after taking action a in state s; R represents the reward function R: S × S × A → R, R (s t+1 ,s t ,a t ) indicates that from the original state s t Entering a new state t+1 The reward obtained after the action is executed is defined by the reward function in this invention; γ represents the discount factor, γ∈[0,1], which is used to calculate the cumulative reward Where T represents the current time, γ i represents the discount factor at time i, r i represents the instantaneous reward at time i. The solution to the Markov decision problem is to find a strategy π:S→A that maximizes the cumulative reward π * :=argmax θ η(π θ ). In the present invention, the reinforcement learning module integrates the implicit state quantities output by the state quantity encoder and the image encoder, and outputs the corresponding optimal control strategy;

[0076] Step 2: Build a complex network cognitive module, model the environment in which the driving subject is located, establish a complex network model, and output dynamic driving suggestions based on the state quantity provided by the state quantity encoder through the activation function. The complex network model represents the dynamic relationship between nodes within the field range through a variable Gaussian safety field based on risk center transfer. The nodes include driving subjects, traffic participants and traffic infrastructure.

[0077] Step 3: Build an end-to-end neural network, including 2 fully connected layers used by the state encoder, 6 convolutional layers used by the image encoder, and 6 fully connected layers used by the reinforcement learning module. The neural network has two output heads: action and value. The action output head outputs the two parameters of the beta distribution, and the value output head outputs the value of the action.

[0078] Step 4: The driving subject interacts with the simulation environment Carla, and the generated experience is stored in each local experience pool. When the number of samples reaches a certain threshold, they are sampled from their local experience pools according to the minimum batch, and then the neural network parameters are updated according to the designed loss function.

[0079] Step 5: Upload the neural network parameters corresponding to each driving subject to the federated learning module. The module aggregates the global parameters according to the aggregation interval based on multiple neural network parameters and sends them to each intelligent agent until the network converges.

[0080] Preferably, in step 1, the traffic participants include other vehicles and pedestrians, the number of other vehicles is 100, the number of pedestrians is 250, and Carla's roaming model is used for control;

[0081] Preferably, in step 1, the traffic infrastructure includes traffic lights and traffic signs (stop), which are represented by a 4-frame semantic bird's-eye view in the image encoder.

[0082] Preferably, in step 2, the complex network cognitive module adopts different activation functions according to different driving suggestions:

[0083]

[0084]

[0085] Preferably, in step 2, the dynamic driving suggestion is represented by beta distribution:

[0086] B risk =B(Activate go (Risk),β),go

[0087] B risk =B(α,Activate stop(Risk)),stop

[0088] Preferably, in step 3, the two fully connected layers of the neural network of the state encoder use the Relu activation function, the neural network of the image encoder uses the Relu activation function except the last convolution layer which flattens the state and does not use the activation function, the other five convolution layers use the Relu activation function, the last layer of the action output head of the reinforcement learning module neural network uses the Softplus activation function, the last layer of the value output head does not use the activation function, and the other fully connected layers use the Relu activation function.

[0089] Preferably, in step 4, the parameters used in the training process are: learning rate is 0.00001; total step size is 12288; minimum batch sampling is 256; loss function weight λ ppo ,λ exp ,λ risk are 0.5, 0.01 and 0.05 respectively; the PPO clipping range is 0.2; the parameters γ of the generalized advantage estimate are 0.99 and λ are 0.9.

[0090] Preferably, in step 4, the loss function used for training adopts the loss based on dynamic risk suggestion By calculating the driving subject's N z = The KL divergence of the strategy and dynamic driving suggestions outputted in 100 steps is used to guide the intelligent agent.

[0091] Preferably, in step 5, the federated learning module is a multi-agent framework, and a local experience pool architecture is used between the agent and the experience pool.

[0092] Preferably, in step 5, a parameter averaging aggregation method is used in the aggregation process, and the aggregation interval is 256.

[0093] The present invention also proposes a vehicle device, which is capable of executing the content of the federated reinforcement learning end-to-end autonomous driving control system based on complex network cognition, or the federated reinforcement learning end-to-end autonomous driving control method based on complex network cognition.

[0094] Beneficial effects of the present invention:

[0095] (1) The present invention uses the federated reinforcement learning end-to-end autonomous driving algorithm framework FLDPPO based on complex network cognition, and realizes the combination of rule-based complex network cognition and end-to-end federated reinforcement learning by designing the loss function. FLDPPO uses dynamic driving suggestions to guide the intelligent agent to learn rules, so that the intelligent agent can cope with complex urban driving environments and dense traffic scenes.

[0096] (2) The framework proposed in the present invention uses a multi-intelligent federated reinforcement learning architecture to train the model by parameter aggregation. On the basis of protecting the privacy of the vehicle, it accelerates network convergence, reduces communication consumption, and achieves a balance between sample efficiency and high robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 The federated reinforcement learning framework based on complex network cognition proposed in this invention;

[0098] Figure 2 The risk recognition schematic diagram based on the dynamic safety field proposed by the present invention;

[0099] Figure 3 Schematic diagram of the reward function used in the present invention;

[0100] Figure 4 Schematic diagram of the neural network used in the present invention.

[0101] Figure 5 A diagram of the federated learning architecture used in the present invention.

[0102] Figure 6 The FLDPPO framework diagram proposed in this invention. DETAILED DESCRIPTION

[0103] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings, but the content of the present invention is not limited thereto.

[0104] The present invention provides an end-to-end algorithm framework of federated reinforcement learning based on complex network cognition, which can realize autonomous driving under dense urban traffic, and specifically includes the following steps:

[0105] (1) Build a federated reinforcement learning algorithm framework based on complex network cognition in the Carla simulator, such as Figure 1 , 6 As shown in the figure, it includes a state encoder, an image encoder, a complex network cognitive module, a reinforcement learning module, and a federated learning module. The state encoder obtains the x-coordinate, y-coordinate, heading angle change, and speed of the driving subject, and passes them to the complex network cognitive module as input. The steering wheel angle, throttle, brake, gear, lateral speed, and longitudinal speed are obtained, and after the features are extracted by two layers of fully connected networks, they are passed to the reinforcement learning module as part of the input; the image encoder uses a 15-channel semantic bird's-eye view, i RL ∈[0,1] 192*192*15. 192 units are pixels, and the bird's-eye view used is 5px / m. The 15 channels include the drivable domain, the expected path, the road sidelines, 4 frames of other vehicles, 4 frames of pedestrians, and 4 frames of traffic signs. The expected path is calculated using the A* algorithm. The semantic bird's-eye view extracts implicit features through multiple convolutional layers and is then handed over to the reinforcement learning module as another input;

[0106] (2) Model the driving situation of the driving subject, such as Figure 2 As shown in the figure, a dynamic complex network model is constructed with traffic participants and road facilities as nodes:

[0107] G t =(P,E,W,Θ) t

[0108] Among them, G t represents the dynamic complex network at time t, P = {p1, p2, ..., p N} is a set of nodes, the number of nodes is N; E = {e 1,2 ,e 1,3 ,…,e i,j} is a set of edges, the number of edges is e i,j Represents node p i and p j The connection between them; W = {w 1,2 ,w 1,3 ,…,w i,j} is the weight set of the edge, w i,j Represents node p i and p j The coupling strength between them; Θ is the active area of ​​the node, which represents the dynamic constraints on the nodes in the network. Θ is modeled as a smooth bounded surface:

[0109] F Θ (x,y,z)=0,st

[0110] Where Ω is the boundary of the sliding surface. Consider a continuous-time dynamic network with N nodes on Θ, whose node state equation is:

[0111]

[0112] Among them, X i ∈R m Represents node p i The state vector, R m represents the vector space composed of m-dimensional real numbers R, U i ∈R q is the input vector, R q represents the vector space composed of q-dimensional real numbers R, Ai represents the dynamic matrix, B i Represents the input matrix. Based on the node state equation, we can get node p i The output vector is:

[0113] Y i =f i (X i )

[0114] f i represents the output function of the node. Then the node p i and p j The weight function between is:

[0115] w ij =F(Y i ,Y j )

[0116] F represents the weight function between nodes. The present invention uses Gaussian function to reveal the static characteristics between nodes:

[0117]

[0118] Among them, S sta represents the static field strength, C a represents the field strength coefficient, x0 and y0 represent the coordinates of the risk center O(x0,y0), a x and b y Represents the vehicle appearance coefficient. The safety field has the characteristics of shape anisotropy:

[0119]

[0120] φ=a x / b y = l v / w v

[0121] Where φ is the aspect ratio, l v Indicates vehicle length, w v Indicates the width of the vehicle.

[0122] The safety field is delineated by multiple equal field lines, and its top view projection is as follows: Figure 2 The area covered by the series of ellipses is shown. The area covered by the smallest central ellipse is the core domain, the area between the smallest central ellipse and the second ellipse is the restricted domain, and the area between the second ellipse and the largest ellipse is the extended domain. The size and shape of the area are determined by the isofield lines, which are related to the shape and motion state of the vehicle and are described based on the Gaussian function. The direction of the safety field is consistent with the direction of vehicle motion.

[0123] When the vehicle is in motion, the risk center O(x0, y0) of the safety field will be transferred to the new risk center O′(x′0, y′0) as the vehicle moves:

[0124]

[0125] Among them, k v represents the adjustment factor, and Its positive or negative value is related to the direction of movement, and β represents the transfer vector The angle between the coordinate axis and the Cartesian coordinate system. Under the action of risk center transfer, a virtual vehicle is formed with a length of l′ v , width is w′ v , the dynamic safety field after the change:

[0126]

[0127] Among them, S dyn represents the dynamic field strength, and the new aspect ratio is expressed as φ′=a′ x / b′ y = l′ v / w′ v As the vehicle's motion state changes, the shape of the Gaussian safety field will change, thereby changing the three major fields covered by the safety field: the core field, the restricted field, and the extended field.

[0128] According to different levels of human driving reaction time, the present invention divides risk cognition into three layers within a plane range: a first cognitive domain, a second cognitive domain and an extra-domain space.

[0129] First cognitive domain:

[0130] a′ x ≤s th1

[0131] s th1 =t c1 ·v e

[0132] Second cognitive domain:

[0133] s th1 <a′ x ≤s th2

[0134] s th2 =t c2 ·v e

[0135] Extraterritorial space:

[0136] s th2 <a′ x

[0137] Among them, s th1 represents the first cognitive domain threshold, which is the first reaction time t of human driving c1 and the maximum approaching speed v of other nodes relative to the ego vehicle e Get.s th2 represents the second cognitive domain threshold, and the second reaction time t of human driving c2 and the maximum approaching speed v of other nodes relative to the ego vehicle e get.

[0138] Under the variable safety field model, the risk perception function between nodes is established:

[0139]

[0140] in, Represents node p i At node p j The field strength at k c represents the risk-adjusted perception coefficient, Represents node p j The scalar velocity, θ i,j Represents node p j The velocity vector With the field strength vector The angle between the two (clockwise is positive). The risk value Risk(p i ,p j ), which indicates the coupling strength between nodes. The larger the risk value, the greater the coupling strength, which means the greater the correlation between nodes.

[0141] The activation function is mainly used to map the risk value. Activate(Risk) uses different activation functions according to different driving suggestions. The mapped risk value will serve as the basis for guiding the output strategy of the reinforcement learning module:

[0142]

[0143] Among them, Activate go (Risk) indicates the activation function when the suggestion is forward. stop (Risk) indicates the activation function when the recommendation is to stop, and Risk indicates the current risk value of the vehicle. risk :

[0144] B risk =B(Activate go (Risk),β go ),go

[0145] B risk=B(α stop ,Activate stop (Risk)),stop

[0146] Among them, α stop =β go =1;

[0147] (3) Construct a reinforcement learning model for the driving subject. Based on Carla’s control method, the action space of reinforcement learning is ∈[-1,1] 2 , where the actions are classified into two types: steering wheel angle and accelerator brake. When outputting accelerator brake, [-1,0] represents brake, and [0,1] represents accelerator. The present invention uses beta distribution to output two parameters of distribution through reinforcement learning, and then obtains the strategy action through sampling:

[0148] Beta=B(α,β),α,β>0

[0149] Compared to the Gaussian distribution commonly used in model-free reinforcement learning, the Beta distribution has boundaries and does not require mandatory constraints.

[0150] The reward function is set as Figure 3 As shown in the figure, considering the four aspects of speed, coordinates, action and terminal state, the mapped risk value is used as the weight of the terminal reward to weight the reward function:

[0151] r=r speed +r position +r action +Activate(Risk)*r terminal

[0152]

[0153] r position =-0.5*Δd-Δθ

[0154] Among them, r speed represents the speed-related reward function, r position represents the position-dependent reward function, r action represents the action-related reward function, r terminal represents the terminal state related reward function, v represents the vehicle speed, v desire represents the expected speed, v max =6m / s represents the maximum speed. Δd represents the lateral distance between the vehicle and the desired path, and Δθ represents the angle between the vehicle's driving direction and the tangent of the desired path. Table 1 describes r in detail action and r terminal The value of , where Δsteering represents the change in the steering wheel angle between two frames.

[0155] Build an end-to-end neural network, such as Figure 4 As shown in the figure, it includes 2 fully connected layers used by the state encoder, 6 convolutional layers used by the image encoder, and 6 fully connected layers used by the reinforcement learning module. The neural network has two output heads: action and value. The action output head outputs two parameters of the beta distribution, and the value output head outputs the value of the action.

[0156] (4) The driving subject interacts with the simulation environment Carla, and the generated experience is stored in their respective local experience pools. Figure 5 As shown in the figure, when the number of samples reaches a certain threshold, they are sampled from their respective local experience pools according to the minimum batch, and then the neural network parameters are updated according to the designed loss function:

[0157]

[0158]

[0159]

[0160]

[0161] in, represents the policy gradient clipping loss estimated using the generalized advantage estimate. represents the maximum entropy loss, H(π θ (·∣i RL ,m RL )) represents the strategy π θ In the image input i RL and state input m RL The entropy under represents uniform distribution. It can make the action distribution close to uniform distribution to encourage the agent to explore, exp Represents the weight of the maximum entropy loss. represents the loss based on the dynamic risk advice, It means that by calculating the driving subject N before the termination state z = KL divergence of the strategy and dynamic driving suggestions outputted in 100 steps to guide the agent, λ risk Represents the weight of the dynamic proposal loss.

[0162] (5) Using the federated learning module, the neural network parameters uploaded by the reinforcement learning modules of each agent are received, and the global parameters are aggregated according to the multiple neural network parameters, and finally the global parameters are sent to each agent until the network converges. The global parameter aggregation is performed by the following equation:

[0163]

[0164] in, represents the global parameters at time m, N represents the number of agents, Represents the neural network parameters of the nth agent at time m.

[0165] In summary, the present invention proposes a federated reinforcement learning algorithm framework FLDPPO based on complex network cognition in dense traffic scenarios of urban autonomous driving. The framework realizes the combination of rule-based complex network cognition and end-to-end federated reinforcement learning by designing loss functions; through dynamic driving suggestions, it guides the agent to learn rules so that the agent can cope with complex urban driving environments and dense traffic scenarios; introduces federated learning, trains the model by parameter aggregation, accelerates network convergence and reduces communication consumption on the basis of protecting vehicle-side privacy; the multi-agent architecture in the algorithm framework not only improves sample efficiency, but also the trained model shows high robustness and generalization.

[0166] An embodiment of the present invention also proposes a vehicle device, such as a vehicle controller, etc., which is capable of executing the content of the federated reinforcement learning end-to-end autonomous driving control system based on complex network cognition, or the federated reinforcement learning end-to-end autonomous driving control method based on complex network cognition.

[0167] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. Federated reinforcement learning end-to-end autonomous driving control system based on complex network cognition, characterized by: include: State encoder, image encoder, complex network cognition module, reinforcement learning module and federated learning module; The state quantity encoder is used to obtain the state quantities required by the complex network cognitive module and the reinforcement learning module, wherein the state quantities required by the complex network cognitive module include the x-coordinate, y-coordinate, heading angle change and speed of the driving subject, which are used as inputs to the complex network cognitive module; the state quantities required by the reinforcement learning module include the steering wheel angle, throttle, brake, gear, lateral speed and longitudinal speed, which are used as part of the input of the reinforcement learning module after extracting features through a two-layer fully connected network; The image encoder is used to obtain the image implicit state quantity required by the reinforcement learning module, wherein the image is a 15-channel semantic bird's-eye view, i RL ∈[0,1] 192*192*15 , where 192 units are pixels, the bird's-eye view used is 5px / m, and the 15 channels include the drivable domain, the expected path, the road sideline, 4 frames of other vehicles, 4 frames of pedestrians, and 4 frames of traffic signs. The expected path is calculated using the A* algorithm. The semantic bird's-eye view is used as another part of the input of the reinforcement learning module after extracting implicit features through multiple convolutional layers; The complex network cognitive module is used to model the driving situation of the driving subject, and obtain the maximum risk value of the driving subject in the current driving situation according to the state quantity provided by the state quantity encoder, and finally output dynamic driving suggestions according to the risk value through the activation function; The reinforcement learning module is used to integrate the state quantities output by the state quantity encoder and the image encoder, output the corresponding strategy according to the integrated network input, and interact with the environment to generate experience and store it in the local experience pool in the federated learning module. When the number of samples reaches a certain threshold, samples are taken from the local experience pool for training, and finally the trained neural network parameters are uploaded to the federated learning module; The federated learning module is used to receive the neural network parameters uploaded by the reinforcement learning modules of each agent, aggregate the global parameters according to the multiple neural network parameters, and finally send the global parameters to each agent until the network converges; the global parameter aggregation is performed by the following equation: in, represents the global parameters at time m, N represents the number of agents, Represents the neural network parameters of the nth agent at time m.

2. The complex network cognition-based federated reinforcement learning end-to-end autonomous driving control system according to claim 1, characterized in that: The modeling process of the complex network cognitive module takes traffic participants and road facilities as nodes to construct a dynamic complex network model: G t =(P, E, W, Θ) t Among them, G t represents the dynamic complex network at time t, P = {p1, p2, ..., p N } is the set of nodes, E = {e 1，2 , e 1，3 ,…,e i，j } is a set of edges, the number of edges is e i，j Represents node p i and p j The connection between them; W = {w 1，2 , w 1，3 ,…,w i，j } is the weight set of the edge, w i，j Represents node p i and p j The coupling strength between them; Θ is the active area of ​​the node, which represents the dynamic constraints on the nodes in the network, and Θ is modeled as a smooth bounded surface: Where Ω represents the boundary of the sliding surface. Consider a continuous-time dynamic network with N agent nodes on Θ, and its node state equation is: Among them, X i ∈R m Represents node p i The state vector, R m represents the vector space composed of m-dimensional real numbers R, U i ∈R q is the input vector, R q represents the vector space composed of q-dimensional real numbers R, A i represents the dynamic matrix, B i represents the input matrix; based on the node state equation, we can get node p i The output vector is: Y i =f i (X i ) f i represents the output function of the node, then the node p i and p j The weight function between is: w ij =F(Y i ,Y j ) F represents the weight function between nodes.

3. The complex network cognition-based federated reinforcement learning end-to-end autonomous driving control system according to claim 2, characterized in that: The complex network uses Gaussian functions to reveal the static characteristics between nodes: Among them, S sta represents the static field strength, C a represents the field strength coefficient, x0 and y0 represent the coordinates of the risk center O(x0, y0), a x and b y Represents the vehicle appearance coefficient; the safety field has the characteristics of shape anisotropy: φ=a x / b y =l v / w v Where φ is the aspect ratio, l v Indicates vehicle length, w v Indicates the width of the vehicle.

4. The federated reinforcement learning end-to-end autonomous driving control system based on complex network cognition according to claim 3 is characterized in that: The safety field is delineated by a plurality of equal field lines, and its top view projection is the area covered by a series of ellipses, the area covered by the smallest central ellipse is the core area, the area between the smallest central ellipse and the second ellipse is the restricted area, and the area between the second ellipse and the largest ellipse is the extended area; the size and shape of the area are determined by the equal field lines, which are related to the shape and motion state of the vehicle. Based on the description of the Gaussian function, the direction of the safety field is consistent with the direction of motion of the vehicle; When the vehicle is in motion, the risk center O(x0, y0) of the safety field will be transferred to the new risk center O′(x′0, y′0) as the vehicle moves: Among them, k v represents the adjustment factor, and Its positive or negative value is related to the direction of movement, and β represents the transfer vector The angle between the coordinate axis and the Cartesian coordinate system forms a virtual vehicle with a length of l′ under the action of risk center transfer. v , width is w′ v , the dynamic safety field after the change: Among them, S dyn represents the dynamic field strength, and the new aspect ratio is expressed as φ′=a′ x / b′ y = l′ v / w′ v ,As the motion state of the vehicle changes, the shape of the Gaussian safety field will change to change the three ,domains covered by the safety field: the core domain, the restricted domain and the extended domain.

5. The complex network cognition-based federated reinforcement learning end-to-end autonomous driving control system according to claim 4, characterized in that: According to different levels of human driving reaction time, risk cognition is divided into three levels within the plane: the first cognitive domain, the second cognitive domain and the extra-domain space; First cognitive domain: a′ x ≤s th1 s th1 =t c1 ·v e Second cognitive domain: s th1 <a′ x ≤s th2 s th2 =t c2 ·v e Extraterritorial space: s th2 <a′ x Among them, s th1 represents the first cognitive domain threshold, which is the first reaction time t of human driving c1 and the maximum approaching speed v of other nodes relative to the ego vehicle e Get;s th2 represents the second cognitive domain threshold, and the second reaction time t of human driving c2 and the maximum closing speed v of other nodes relative to the maximum closing speed of the ego vehicle e get; Under the variable safety field model, the risk perception function between nodes is established: in, Represents node p i At node p j The field strength at k c represents the risk-adjusted perception coefficient, Represents node p j The scalar velocity, θ i,j Represents node p j The velocity vector With the field strength vector The angle between the two (clockwise is positive), the risk value Risk(p i , p j ), which indicates the coupling strength between nodes. The larger the risk value, the greater the coupling strength, which means the greater the correlation between nodes.

6. The federated reinforcement learning end-to-end autonomous driving control system based on complex network cognition according to claim 1 or 5, characterized in that: The activation function is mainly used to map the risk value. Activate(Risk) uses different activation functions according to different driving suggestions. The mapped risk value will serve as the basis for guiding the output strategy of the reinforcement learning module: Among them, Activate go (Risk) indicates the activation function when the suggestion is forward. stop (Risk) indicates the activation function when the recommendation is to stop, Risk indicates the current risk value of the vehicle, and the dynamic risk recommendation B risk : B risk =B(Activate go (Risk),β go ),go B risk =B(α stop ,Activate stop (Risk)),stop Among them, B represents beta distribution, α stop =β go =1.

7. The complex network cognition-based federated reinforcement learning end-to-end autonomous driving control system according to claim 1, characterized in that: The simulation environment used by the federated reinforcement learning end-to-end autonomous driving control system based on complex network cognition is the Carla simulator, which realizes vehicle control by inputting the control amount of the steering wheel, throttle and brake, where the steering wheel ∈ [-1, 1], the throttle ∈ [0, 1], and the brake ∈ [0, 1]; based on the Carla control method, the action space of reinforcement learning ∈ [-1, 1] 2 , where the actions are classified into two types: steering wheel angle and accelerator brake. When outputting accelerator brake, [-1,0] represents brake, and [0,1] represents accelerator; With the help of beta distribution, two parameters of the output distribution are learned through reinforcement learning, and then the strategy action is obtained through sampling: Beta=B(α,β), α,β>0 The experience generated by the interaction is a tuple, which includes the state quantity, action, reward function, state quantity and dynamic driving suggestion at the previous moment. The reward function uses the mapped risk value as the weight of the termination reward to weight the reward function. r=r speed +r position +r action +Activate(Risk)*r terminal r position =-0.5*Δd-Δθ Among them, r speed represents the speed-related reward function, r position represents the position-dependent reward function, r action represents the action-related reward function, r terminal represents the terminal state related reward function, v represents the vehicle speed, v desire represents the expected speed, v max =6m / s represents the maximum speed, Δd represents the lateral distance between the vehicle and the desired path, and Δθ represents the angle between the vehicle's driving direction and the tangent of the desired path.

8. The complex network cognition-based federated reinforcement learning end-to-end autonomous driving control system according to claim 1 or 7, characterized in that: The training process of the reinforcement learning module updates parameters through the following loss function: in, represents the policy gradient clipping loss estimated using the generalized advantage estimate, represents the maximum entropy loss, H(π θ (·∣i RL ,m RL )) represents the strategy π θ In the image input i RL and state input m RL The entropy under represents uniform distribution, which can make the action distribution close to uniform distribution to encourage the agent to explore, λ exp represents the weight of maximum entropy loss, represents the loss based on the dynamic risk advice, It means that by calculating the driving subject N before the termination state z = KL divergence of the strategy and dynamic driving suggestions outputted in 100 steps to guide the agent, λ risk Represents the weight of the dynamic proposal loss.

9. The federated reinforcement learning end-to-end autonomous driving control method based on complex network cognition is characterized by: The steps include: Step 1: Build a dense urban traffic simulation environment, which includes driving subjects, traffic participants and traffic infrastructure; The driving subject is a plurality of intelligent agents, each of which is modeled as a Markov decision process, and a reinforcement learning module is used to control the steering wheel, throttle and brake. The Markov decision process is described by a tuple (S, A, P, R, γ), where S represents a state set, corresponding to the state quantity obtained by the state quantity encoder and the image encoder, including the steering wheel angle, throttle, brake, gear, lateral speed and longitudinal speed, and a 15-channel semantic bird's-eye view; A represents an action set, corresponding to the steering wheel, throttle and brake control quantity of the driving subject; P represents the state transfer equation p:S×A→P(S), for each state-action pair (s,a)∈S×A there is a probability distribution p(·|s,a) which represents the possibility of entering a new state after taking action a in state s; R represents the reward function R:S×S×A→R, R(s t+1 ,s t ,a t ) indicates that from the original state s t Entering a new state t+1 The reward function is used to define the quality of the action. γ represents the discount factor, γ∈[0,1], which is used to calculate the cumulative reward. Where T represents the current time, γ i represents the discount factor at time i, r i represents the immediate reward at time i; the solution to the Markov decision problem is to find a strategy π:S→A that maximizes the cumulative return π * :=argmax θ η(π θ ), the reinforcement learning module integrates the implicit state quantities output by the state quantity encoder and the image encoder, and outputs the corresponding optimal control strategy; Step 2: Build a complex network cognitive module, model the environment in which the driver is located, establish a complex network model, and output dynamic driving suggestions based on the state quantity provided by the state quantity encoder through an activation function. The complex network model represents the dynamic relationship between nodes within the field range through a variable Gaussian safety field based on risk center transfer. The nodes include the driver, traffic participants, and traffic infrastructure. Step 3: Build an end-to-end neural network, including 2 fully connected layers used by the state encoder, 6 convolutional layers used by the image encoder, and 6 fully connected layers used by the reinforcement learning module. The neural network has two output heads: action and value. The action output head outputs two parameters of the beta distribution, and the value output head outputs the value of the action. Step 4: The driving subject interacts with the simulation environment, and the generated experience is stored in each local experience pool. When the number of samples reaches a certain threshold, they are sampled from their local experience pools according to the minimum batch, and then the neural network parameters are updated according to the designed loss function. Step 5: Upload the neural network parameters corresponding to each driving subject to the federated learning module, aggregate the global parameters according to the aggregation interval based on multiple neural network parameters, and send them to each intelligent agent until the network converges; In step 1, the traffic participants include other vehicles and pedestrians; the traffic infrastructure includes traffic lights and traffic signs, which are represented by a 4-frame semantic bird's-eye view in the image encoder; In step 2, the complex network cognitive module adopts different activation functions according to different driving suggestions: In step 2, the dynamic driving suggestion is represented by beta distribution: B risk =B(Activate go (Risk),β),go B risk =B(α,Activate stop (Risk)),stop In step 3, the two fully connected layers of the neural network of the state encoder use the Relu activation function. The neural network of the image encoder uses the Relu activation function except for the last convolutional layer that flattens the state and does not use the activation function. The last layer of the action output head of the reinforcement learning module neural network uses the Softplus activation function, the last layer of the value output head does not use the activation function, and the other fully connected layers use the Relu activation function. In step 4, the training parameters used are: learning rate 0.00001, total step size 12288, minimum batch size 256, loss function weight λ ppo , exp , risk are 0.5, 0.01, and 0.05 respectively; the ppo clipping range is 0.2; the parameters γ and λ of the generalized advantage estimate are 0.99 and 0.9 respectively; In step 4, the loss function used for training adopts the loss based on dynamic risk recommendation By calculating the driving subject's N z = KL divergence of the strategy and dynamic driving suggestions outputted in 100 steps to guide the agent; In step 5, the federated learning module is a multi-agent framework, and the local experience pool architecture is used between the agent and the experience pool; In step 5, the parameter averaging aggregation method is used in the aggregation process, and the aggregation interval is 256.

10. A vehicle device, characterized in that: The vehicle-use equipment is capable of executing the content of the federated reinforcement learning end-to-end autonomous driving control system based on complex network cognition as described in any one of claims 1-8, or the federated reinforcement learning end-to-end autonomous driving control method based on complex network cognition as described in claim 9.

Images