End-to-end autonomous driving control system and method using federated reinforcement learning based on complex network cognition, and vehicle equipment

Abstract

This invention discloses an end-to-end autonomous driving control system and method using federated reinforcement learning based on complex network cognition, as well as a vehicle device. It provides an algorithm framework called FLDPPO, which uses federated reinforcement learning based on complex network cognition for end-to-end autonomous driving in high-density urban traffic. By designing a loss function, it combines rule-based complex network cognition and end-to-end federated reinforcement learning. FLDPPO provides dynamic driving advice to guide agents in learning rules, enabling them to cope with complex urban driving environments and dense traffic scenes. The proposed framework also uses a multi-agent federated reinforcement learning mechanism to train models through parameter aggregation, preserving vehicle privacy while accelerating network convergence, reducing communication consumption, and achieving a balance between sample efficiency and high model robustness.

Description

[Technical Field]

[0001] The present invention belongs to the field of traffic transportation and autonomous driving, and relates to an end-to-end autonomous driving control system and method using federated reinforcement learning based on complex network cognition in high-density urban traffic, and to vehicle equipment. [Background technology]

[0002] Urban driving involves extensive interactions between drivers, traffic participants, and road infrastructure, making it an open-ended problem in the field of transportation. Recognizing and modeling the dynamic interaction process between the environment and the participants is challenging, as it exhibits high dimensionality and diversity. While traditional rule-based methods are adept at handling simple driving scenarios, applying rule-based methods to complex traffic scenarios is difficult.

[0003] On the other hand, end-to-end methods do not rely on manually defined rules but instead use data-driven training methods to achieve competitive results. Deep reinforcement learning (DRL) is a typical data-driven algorithm that generates samples through interactions with the environment. The generated samples are stored in an experience pool and then sampled in microbatches for model training. Due to the characteristics of DRL, there are drawbacks to its application. First, DRL requires a large number of samples, which makes convergence more difficult when training large networks compared to supervised learning. Insufficient samples limits the robustness of the model. Second, the use of an experience pool limits the network input, such as the image input size. DRL algorithms that use image inputs experience linear increases in computational load and communication consumption with training. Furthermore, end-to-end DRL is a black-box algorithm, making it difficult to understand the agent's decision-making motivations and lacking interpretability. Due to these limitations, DRL is primarily used in early stages for simpler driving scenarios.

[0004] Complex research scenes can be divided into multiple single objects to analyze complex environments. However, single-object research methods have shortcomings in situations where it is necessary to understand the interactions of multiple elements. While analyzing complex environments can have positive effects, simple divisions limit the advancement of autonomous driving to be applied to complex environments. Summary of the Invention [Problem to be solved by the invention]

[0005] To solve the above technical problems, the present invention provides an end-to-end autonomous driving algorithm framework for urban high-density traffic using federated reinforcement learning based on complex network perception, FLDPPO, which combines rule-based complex network perception and end-to-end federated reinforcement learning by designing a loss function. FLDPPO guides agents to learn rules through dynamic driving advice, enabling them to cope with complex urban driving environments and dense traffic scenes. Furthermore, the framework uses a multi-agent federated reinforcement learning mechanism and parameter aggregation to train the model, preserving vehicle privacy while accelerating network convergence, reducing communication consumption, and achieving a balance between sample efficiency and high model robustness. [Means for solving the problem]

[0006] The technical means of the end-to-end autonomous driving control system using associative reinforcement learning based on complex network cognition according to the present invention includes five parts: a state encoder, an image encoder, a complex network cognition module, a reinforcement learning module, and an associative learning module.

[0007] The state encoder is mainly used to obtain the state variables required for the complex network perception module and the reinforcement learning module. The state variables required for the complex network perception module include the x-coordinate, y-coordinate, heading angle change, and speed of the driver, which are passed to the complex network perception module as input. The state variables required for the reinforcement learning module include the steering, accelerator, brake, gear, lateral speed, and longitudinal speed, which are passed to the reinforcement learning module as part of the input after features are extracted through a two-layer fully connected network.

[0008] The image encoder is mainly used to obtain the implicit state of the image required for the reinforcement learning module. The image is a 15-channel semantic bird's-eye view, i RL ∈[0,1] 192*192*15 Here, 192 units are pixels, and the bird's-eye view used is 5px / m. The 15 channels include the drivable area, the intended route, road sidelines, 4 frames of other vehicles, 4 frames of pedestrians, and 4 frames of traffic signs, of which the intended route is calculated using the A* algorithm. The semantic bird's-eye view passes through multiple convolutional layers to extract implicit features, which are then passed to the reinforcement learning module as input for the other parts.

[0009] The complex network cognitive module is mainly used to model the driving scenario in which the driver is present, obtain the maximum risk value of the driver in the current driving scenario based on the state quantity provided by the state quantity encoder, and finally output dynamic driving advice via the activation function based on the risk value.

[0010] The modeling process of the complex network recognition module involves constructing a dynamic complex network model with traffic participants and road facilities as nodes; G t =(P,E,W,Θ) t where G t represents the dynamic complex network at time t, and P={p1,p2,…,p N} is the set of nodes, the number of nodes is N, and E={e 1,2’ e 1,3’ …,e i,j} is a set of edges, and the number of edges is

number

number

number

number

[0011] The safety field is defined by a number of iso-field lines, the top view projection of which is the area covered by a series of ellipses, where the area covered by the smallest central ellipse is the core area, the area between the smallest central ellipse and the second ellipse is the restricted area, and the area between the second ellipse and the largest ellipse is the extended area. The size and shape of the area are determined by the iso-field lines, which are related to the shape and motion state of the vehicle, and according to the Gaussian function description, the direction of the safety field coincides with the direction of vehicle motion.

[0012] When the vehicle is in motion, the risk center 0 (x0, y0) of the safety field transitions to a new risk center 0' (x'0, y'0) as the vehicle moves.

number

number

number

[0013] In the present invention, according to different levels of human driving reaction time, risk perception is divided into three levels within the plane range, namely, the first perception area, the second perception area and the outside space; The first cognitive domain is as follows:

number

number

number

number

number

number

number

number

[0014] The activation function is mainly used to map risk values, and Activate(Risk) adopts different activation functions according to different driving advice, and the risk values ​​after mapping serve as the basis for guiding the reinforcement learning module to output strategies;

number

[0015] The reinforcement learning module is mainly used to align the state variables output from the state encoder and the image encoder, output a corresponding strategy based on the aligned network input, and interact with the environment to generate experience and store it in a local experience pool in the associative learning module. When the number of samples reaches a predetermined threshold, it samples from the local experience pool for training, and finally uploads the trained neural network parameters to the associative learning module.

[0016] The interactive environment is the Carla simulator, and vehicle control is realized by inputting the control variables of the steering wheel, accelerator, and brake. The steering wheel is ∈ [-1, 1], the accelerator is ∈ [0, 1], and the brake is ∈ [0, 1]. According to the control method of Carla, the reinforcement learning action space is ∈ [-1, 1] 2 The behavior is divided into two types: steering and accelerator braking. When the accelerator braking is output, [-1,0] represents the brake and [0,1] represents the accelerator. In the present invention, the two distributed parameters are output through reinforcement learning via the beta distribution, and the policy behavior is obtained by further sampling. Beta=B(α,β),α,β>0 Compared to the Gaussian distribution commonly used in model-free reinforcement learning, the Beta distribution is bounded and does not require any enforced constraints. The experience generated by the interaction is a tuple, which includes a state quantity at the previous time, an action, a reward function, a state quantity at the next time, and dynamic driving advice, where the reward function is weighted by the risk value after mapping as the reward termination weight.

number

[0017] The training procedure of the reinforcement learning module updates parameters using the following loss function:

number

number

number

number

number

number

number

[0018] The federated learning module is mainly used to receive the neural network parameters uploaded by the reinforcement learning module of each agent, aggregate a global parameter based on multiple neural network parameters, and finally send the global parameter to each agent until the network converges. The aggregation of the global parameter is performed according to the following equation:

number

[0019] In summary, the FLDPPO algorithm combines rule-based complex network perception and end-to-end federated reinforcement learning by designing a loss function, and uses a multi-agent federated reinforcement learning mechanism to train the model through parameter aggregation, preserving vehicle privacy while accelerating network convergence, reducing communication consumption, and achieving a balance between sample efficiency and high model robustness.

[0020] In the present invention, the technical means of the end-to-end autonomous driving control method using associative reinforcement learning based on complex network cognition includes the following steps 1 to 5. Step 1: An urban high-density traffic simulation environment is constructed in the Carla simulator, and the simulation environment includes driving subjects, traffic participants, and traffic infrastructure. The driving subjects are multiple agents, each modeled as a Markov decision process, and a reinforcement learning module is used to control the steering wheel, accelerator, and brake. The Markov decision process is described by a tuple (S, A, P, R, γ), where S represents a state set, which corresponds to the state variables acquired by the state encoder and image encoder in this invention, including steering, accelerator, brake, gear, lateral speed, longitudinal speed, and 15 channels of semantic bird's-eye view. A represents an action set, which corresponds to the control variables of the steering wheel, accelerator, and brake by the driving subject in this invention. P represents a state transition equation p: S×A→P(S), and for each state-action pair (s, a)∈S×A, there is a probability distribution p(·|s, a), which represents the probability of entering a new state after taking action a in state s. R represents a reward function RS×S×A→R, where R(s t+1 ,s t ,a t ) is the initial state s t From new state s t+1 represents the reward obtained after reaching the target value. In the present invention, the reward function defines whether the execution behavior is good or bad. γ represents the discount coefficient, γ∈[0,1]. The cumulative reward

number

number

[0021] Preferably, in step 1, the traffic participants include other vehicles and pedestrians, the number of other vehicles is 100, the number of pedestrians is 250, and Carla's roaming model is used for control.

[0022] Preferably, in step 1, the traffic infrastructure includes traffic lights and traffic signs (stop), and is represented in an image encoder by a four-frame semantic bird's-eye view.

[0023] Preferably, in step 2, the complex network cognitive module adopts different activation functions according to different driving advices;

number

[0024] Preferably, in step 3, the neural network of the state encoder uses the Relu activation function in two fully connected layers, the neural network of the image encoder uses the Relu activation function in the other five convolutional layers except for the last convolutional layer that flattens the state and does not use an activation function, and the neural network of the reinforcement learning module uses the Softplus activation function in the last layer of the action output head, does not use an activation function in the last layer of the value output head, and uses the Relu activation function in the other fully connected layers.

[0025] Preferably, in step 4, the parameters used in the training procedure are a learning rate of 0.00001, a total step size of 12288, a minimum batch sampling of 256, and a loss function weight λ ppo , λ e×p , λ riskare 0.5, 0.01, and 0.05, respectively, the ppo clipping range is 0.2, the generalized advantage estimation parameters γ and λ are 0.99 and 0.9, respectively.

[0026] Preferably, in step 4, the loss function used for training is a loss function based on dynamic risk advice.

number

[0027] Preferably, in step 5, the federated learning module is a multi-agent framework, and a local experience pool mechanism is used between the agents and the experience pool.

[0028] Preferably, in step 5, the aggregation procedure uses a parameter averaging aggregation method, and the aggregation interval is 256.

[0029] The present invention further proposes a vehicle device capable of executing the contents of the end-to-end autonomous driving control system using associative reinforcement learning based on complex network cognition or the end-to-end autonomous driving control method using associative reinforcement learning based on complex network cognition. [Effects of the Invention]

[0030] The beneficial effects of the present invention are as follows: (1) This invention combines rule-based complex network perception and end-to-end federated reinforcement learning (FLDPPO), an end-to-end autonomous driving algorithm framework based on complex network perception and federated reinforcement learning, by designing a loss function. FLDPPO guides the agent to learn rules through dynamic driving advice, enabling the agent to cope with complex urban driving environments and dense traffic scenes. (2) The framework proposed in this invention uses a multi-agent federated reinforcement learning mechanism and trains the model through a parameter aggregation means, while preserving vehicle-side privacy, accelerating network convergence, reducing communication consumption, and balancing sample efficiency with high model robustness. [Brief explanation of the drawings]

[0031] [Figure 1] 1 is a framework for associative reinforcement learning based on complex network cognition according to the present invention. [Figure 2] FIG. 1 is a schematic diagram of risk perception based on dynamic safety field according to the present invention. [Figure 3] FIG. 1 is a schematic diagram of a reward function used in the present invention. [Figure 4] FIG. 1 is a schematic diagram of a neural network used in the present invention. [Figure 5] FIG. 1 is a diagram of a federated learning mechanism used in the present invention. [Figure 6] FIG. 1 is a diagram of the FLDPPO framework according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] Next, the technical means of the present invention will be described in detail with reference to the drawings, but the contents of the present invention are not limited thereto.

[0033] The present invention provides an end-to-end algorithm framework using federated reinforcement learning based on complex network cognition, which can realize autonomous driving in high-density urban traffic, and specifically includes the following steps (1) to (5):

[0034] (1) A federated reinforcement learning algorithm framework based on complex network perception is constructed in the Carla simulator. As shown in Figures 1 and 6, it includes a state encoder, an image encoder, a complex network perception module, a reinforcement learning module, and a federated learning module. The state encoder acquires the x- and y-coordinates, heading angle change, and speed of the driver, and passes them to the complex network perception module as input. The steering, accelerator, brake, gear, lateral speed, and longitudinal speed are acquired, and features are extracted through a two-layer fully connected network, and then passed to the reinforcement learning module as part of the input. The image encoder uses a 15-channel semantic bird's-eye view and i RL ∈[0,1] 192*192*15 Here, 192 units are pixels, and the bird's-eye view used is 5px / m. 15 channels include the drivable area, the intended route, road sidelines, 4 frames of other vehicles, 4 frames of pedestrians, and 4 frames of traffic signs. The intended route is calculated using the A* algorithm. The semantic bird's-eye view is then passed through multiple convolutional layers to extract implicit features, which are then passed to the reinforcement learning module as input for the other parts.

[0035] (2) A driving scenario in which the driver is present is modeled, and a dynamic complex network model is constructed with traffic participants and road facilities as nodes, as shown in Figure 2. t =(P,E,W,Θ) t where G t represents the dynamic complex network at time t, and P={p1,p2,…,p N} is the set of nodes, the number of nodes is N, and E={e 1,2’ e 1,3’ …,e i,j} is a set of edges, and the number of edges is

number

number

number

number

[0036] The safety field is defined by multiple iso-field lines, and its top view projection is the area covered by a series of ellipses as shown in Figure 2. The area covered by the smallest central ellipse is the core area, the area between the smallest central ellipse and the second ellipse is the restricted area, and the area between the second ellipse and the largest ellipse is the extended area. The size and shape of the area are determined by the iso-field lines, which are related to the shape and motion state of the vehicle. According to the Gaussian function description, the direction of the safety field coincides with the direction of vehicle motion.

[0037] When the vehicle is in motion, the risk center 0 (x0, y0) of the safety field transitions to a new risk center 0' (x'0, y'0) as the vehicle moves.

number

number

number

[0038] In the present invention, according to different levels of human driving reaction time, risk perception is divided into three levels within the plane range, namely, the first perception area, the second perception area and the out-of-area space. The first cognitive domain is as follows:

number

number

number

number

number

number

number

number

[0039] The activation function is mainly used to map risk values, and Activate(Risk) adopts different activation functions according to different driving advice, and the risk values ​​after mapping serve as the basis for guiding the reinforcement learning module to output strategies;

number

[0040] (3) Build a driving-based reinforcement learning model, and according to Carla's control method, the reinforcement learning action space ∈ [-1, 1] 2 The behavior is divided into two types: steering and accelerator braking. When the accelerator braking is output, [-1,0] represents the brake and [0,1] represents the accelerator. In the present invention, the two distributed parameters are output through reinforcement learning via the beta distribution, and the policy behavior is obtained by further sampling. Beta=B(α,β),α,β>0 Compared to the Gaussian distribution commonly used in model-free reinforcement learning, the Beta distribution is bounded and does not require any enforced constraints.

[0041] The reward function is set, and as shown in Figure 3, the risk value after mapping is weighted as the reward end weight, taking into account four aspects: speed, coordinates, action, and final state.

number

[0042] An end-to-end neural network is constructed, which includes two fully connected layers used for the state encoder, six convolutional layers used for the image encoder, and six fully connected layers used for the reinforcement learning module, as shown in Figure 4. The neural network has two output heads, one for action and one for value. The action output head outputs two parameters according to the beta distribution, and the value output head outputs the value of the action.

[0043] (4) The driving subject interacts with the simulation environment Carla and stores the generated experience in their local experience pool. As shown in Figure 5, when the number of samples reaches a predetermined threshold, they are sampled from their local experience pool in minimum batches, and the neural network parameters are updated by the designed loss function.

number

number

number

number

number

number

number

[0044] (5) Using a federated learning module, receive the neural network parameters uploaded by the reinforcement learning module of each agent, aggregate global parameters based on multiple neural network parameters, and finally send the global parameters to each agent until the network converges. The global parameter aggregation is performed according to the following equation:

number

[0045] In summary, this invention proposes a federated reinforcement learning algorithm framework (FLDPPO) based on complex network perception for dense traffic scenes in urban autonomous driving. The framework combines rule-based complex network perception and end-to-end federated reinforcement learning by designing a loss function. Dynamic driving advice guides agents to learn rules, enabling them to cope with complex urban driving environments and dense traffic scenes. Federated learning is introduced and a parameter aggregation method is used to train the model, accelerating network convergence and reducing communication consumption while preserving vehicle privacy. The multi-agent mechanism in the algorithm framework improves sampling efficiency and ensures that the trained model exhibits high robustness and generalizability.

[0046] An embodiment of the present invention further provides a vehicle device, such as a vehicle controller, which can execute the content of the end-to-end autonomous driving control system using associative reinforcement learning based on complex network cognition or the end-to-end autonomous driving control method using associative reinforcement learning based on complex network cognition.

[0047] The above detailed description is merely a specific description of the possible embodiments of the present invention, and is not intended to limit the scope of protection of the present invention. Any equivalent forms or modifications made without departing from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. The system includes a state encoder, an image encoder, a complex network recognition module, a reinforcement learning module, and a federated learning module; The state encoder is used to obtain state quantities required for the complex network recognition module and the reinforcement learning module, and the state quantities required for the complex network recognition module include the x-coordinate, y-coordinate, course angle change amount, and speed of the driver as inputs to the complex network recognition module, and the state quantities required for the reinforcement learning module include steering, accelerator, brake, gear, lateral speed, and longitudinal speed, and after features are extracted through a two-layer fully connected network, they become part of the inputs to the reinforcement learning module; The image encoder is used to obtain the implicit state of the image required for the reinforcement learning module, and the image is a 15-channel semantic bird's-eye view. ＲＬ ∈[0, 1] １９２＊１９２＊１５ where 192 units are pixels, the bird's-eye view used is 5px / m, 15 channels include the drivable area, the intended route, road sidelines, 4 frames of other vehicles, 4 frames of pedestrians, and 4 frames of traffic signs, the intended route is calculated using the A* algorithm, and the semantic bird's-eye view is used to extract implicit features through multiple convolutional layers and then becomes the input of another part of the reinforcement learning module, The complex network recognition module is used to model a driving scenario in which a driver exists, obtain a maximum risk value of the driver in the current driving scenario based on the state quantity provided by the state quantity encoder, and finally output dynamic driving advice through an activation function based on the risk value; The reinforcement learning module is used to align the state variables output from the state encoder and the image encoder, output a corresponding strategy according to the aligned network input, and interact with the environment to generate experiences and store them in a local experience pool in the associative learning module. When the number of samples reaches a predetermined threshold, the reinforcement learning module samples from the local experience pool for training, and finally uploads the trained neural network parameters to the associative learning module; The federated learning module is used to receive the neural network parameters uploaded by the reinforcement learning module of each agent, aggregate a global parameter based on multiple neural network parameters, and finally send the global parameter to each agent until the network converges, and the aggregation of the global parameter is performed according to the following equation: [Number 55] where φ ＊ ｍ represents the global parameters at time m, N represents the number of agents, and φ ｎ ｍ represents the neural network parameters of the nth agent at time m.

2. The modeling process of the complex network recognition module involves constructing a dynamic complex network model with traffic participants and road facilities as nodes; G ｔ =(P, E, W, Θ) ｔ Here, G ｔ represents the dynamic complex network at time t, and P = {p １ , p ２ , ..., p Ｎ } is a set of nodes, the number of nodes is N, and E={e １，２’ e １，３’ ..., e ｉ，ｊ } is a set of edges, and the number of edges is [Number 56] and e ｉ，ｊ is node p ｉ and p ｊ and W = {w １，２’ W １，３’ …, lol ｉ，ｊ } is the set of edge weights, and w ｉ，ｊ is node p ｉ and p ｊ represents the connection strength between,Θ,and,Θ,is the action domain of the node, which represents the dynamic constraints on the node in the network.,Θ,is modeled as a smooth bounded surface, FΘ(x,y,z)=0, s. t. ∀(x,y)∈Ω where Ω represents the boundary of the sliding surface, and consider a continuous-time dynamical network with N nodes in Θ, whose nodal state equations are: [Number 57] Here, X ｉ ∈R ｍ is node p ｉ represents the state vector of ｍ represents a vector space consisting of m-dimensional real numbers R, and U ｉ ∈R ｑ is the input vector, and R ｑ represents a vector space consisting of q-dimensional real numbers R, and A ｉ represents the dynamic matrix, and B ｉ represents the input matrix, and based on the node state equation, ｉ The output vector is obtained as Y ｉ =f ｉ (X ｉ ) f ｉ represents the output function of the node, and thus the node p ｉ and p ｊ The weight function between w ｉｊ =F(Y ｉ ,Y ｊ ) The end-to-end autonomous driving control system using associative reinforcement learning based on complex network cognition as described in claim 1, wherein F represents a weight function between nodes.

3. The complex network represents static characteristics between nodes using Gaussian functions, [Number 58] Here, S ｓｔａ represents the static field strength, and C ａ represents the field strength coefficient, and x ０ and y ０ is the risk center 0 (x ０ , y ０ ) coordinates, and a ｘ and b ｙ represents the vehicle appearance coefficient, the safety field has shape anisotropy characteristics, [Number 59] where φ is the aspect ratio and l ｖ represents the commander, and w ｖ The end-to-end autonomous driving control system using associative reinforcement learning based on complex network cognition according to claim 2, characterized in that represents the vehicle width.

4. The safety field is defined by a plurality of iso-field lines, and its top view projection is the area covered by a series of ellipses, the area covered by the smallest central ellipse is the core area, the area between the smallest central ellipse and a second ellipse is the restricted area, and the area between the second ellipse and the largest ellipse is the extended area, the size and shape of the area are determined by the iso-field lines, and the iso-field lines are related to the shape and motion state of the vehicle, and according to the Gaussian function description, the direction of the safety field is consistent with the motion direction of the vehicle; When the vehicle is in motion, the risk center 0 (x ０ , y ０ ) is the new risk center 0'(x') as the vehicle moves. ０ , y′ ０ ) and [Number 60] Here, k ｖ represents the adjustment factor, and k ｖ ∋{(-1,0)∪(0,1)}, and its sign is related to the direction of motion, and β is the transition vector [Number 61] and represent the included angle of the coordinate axis in the Cartesian coordinate system. A virtual vehicle is formed by the risk center transition action, and its length is l'. ｖ and the width is w' ｖ and the dynamic safety field after the change is as follows: [Number 62] Here, S ｄｙｎ represents the dynamic field strength, and the new aspect ratio is φ' = a' ｘ / b' ｙ =l' ｖ / w' ｖ and as the vehicle's motion state changes, the shape of the Gaussian safety field changes to change the three regions covered by the safety field, namely, the core region, the restricted region, and the extended region.

5. According to different levels of human driving reaction time, risk perception is divided into three levels within the plane range: the first perception area, the second perception area, and the out-of-area space; The first cognitive domain is as follows: [Number 63] The second cognitive domain is as follows: [Number 64] The outer space is as follows: [Number 65] Here, s ｔｈ１ represents the first cognitive domain threshold, and the first reaction time t ｃ１ and the maximum approach speed v of other nodes to the own vehicle. ｅ is obtained from s ｔｈ２ represents the second cognitive domain threshold, and the second human driving reaction time t ｃ２ and the maximum approach speed v of other nodes to the own vehicle. ｅ is obtained from In the variable safety field model, we establish the risk perception function between nodes; [Number 66] where: [Number 67] is node p ｊ Node p at ｉ represents the field strength of k ｃ represents the risk-adjusted perception coefficient, [Number 68] is node p ｊ represents the scalar velocity of θ ｉ，ｊ is node p ｊ The velocity vector v ｊ and the field strength vector [Number 69] represents the included angle (clockwise is positive) of ｉ , p ｊ ) represents the connection strength between nodes, and the larger the risk value, the larger the connection strength, meaning that the correlation between nodes is greater.

6. The activation function is mainly used to map risk values, and Activate(Risk) adopts different activation functions according to different driving advice, and the risk values ​​after mapping serve as the basis for guiding the reinforcement learning module to output strategies; [Number 70] Here, Activate ｇｏ (Risk) represents the activation function when advising to move forward, and Activate ｓｔｏｐ (Risk) represents the activation function when advising to stop, Risk represents the current risk value of the vehicle, and Dynamic Risk Advice B ｒｉｓｋ is as follows: B ｒｉｓｋ =B(Activate ｇｏ (Risk),β ｇｏ ),go B ｒｉｓｋ =B(α ｓｔｏｐ ,Activate ｓｔｏｐ (Risk)),stop where B represents the beta distribution and α ｓｔｏｐ = β ｇｏ The end-to-end autonomous driving control system using associative reinforcement learning based on complex network cognition according to claim 1 or 5, wherein:

7. The Carla simulator is used as the interaction environment, and vehicle control is realized by inputting the control amounts of the steering wheel, accelerator, and brake. The steering wheel is ∈ [−1, 1], the accelerator is ∈ [0, 1], and the brake is ∈ [0, 1]. According to the Carla control method, the reinforcement learning action space is ∈ [−1, 1]. ２ The behavior is divided into two types: steering and accelerator braking. When the accelerator braking is output, [-1, 0] represents the brake and [0, 1] represents the accelerator. Through the beta distribution, the two distributed parameters are output by reinforcement learning, and the policy behavior is obtained by further sampling. Beta=B (α, β), α, β>0 The experience generated by the interaction is a tuple, which includes the state quantity of the previous time, the action, the reward function, the state quantity of the next time, and the dynamic driving advice. The reward function is weighted by the risk value after mapping as the reward termination weight. [Number 71] Here, r ｓｐｅｅｄ represents the speed-related reward function, and r ｐｏｓｉｔｉｏｎ represents the location-related reward function, and r ａｃｔｉｏｎ represents the behavior-related reward function, and r ｔｅｒｍｉｎａｌ represents the end-state related reward function, v represents the vehicle speed, and v ｄｅｓｉｒｅ represents the desired velocity, and v ｍａｘ = 6 m / s represents the maximum speed, Δd represents the lateral distance between the vehicle and the intended path, Δθ represents the included angle between the vehicle running direction and the tangent to the intended path, and r ａｃｔｉｏｎ and ｔｅｒｍｉｎａｌ The values ​​of are detailed, where Δsteering represents the change in steering between two frames, 【number】 The end-to-end autonomous driving control system using associative reinforcement learning based on complex network cognition according to claim 1, characterized in that

8. The training procedure of the reinforcement learning module updates parameters using the following loss function: [Number 72] where: [Number 73] represents the policy gradient clipping loss estimated by generalized advantage estimation, [Number 74] denotes the maximum entropy loss, [Number 75] is the image input i ＲＬ and state input m ＲＬ Strategy π θ represents the entropy of [Number 76] represents a uniform distribution, and by making the behavior distribution close to a uniform distribution, we can give the agent an incentive to explore, and λ ｅ×ｐ represents the weight of the maximum entropy loss, [Number 77] represents the loss based on dynamic risk advice, [Number 78] The driving subject is N from the end state Ｚ λ represents the realization of guidance for the agent by calculating the KL divergence of the policy output before 100 steps and the dynamic driving advice. ｒｉｓｋ The end-to-end autonomous driving control system using associative reinforcement learning based on complex network cognition according to claim 1 or 7, wherein represents the weight of the dynamic advice loss.

9. The following steps 1 to 5 are included: Step 1: Build a high-density urban traffic simulation environment that includes drivers, traffic participants, and traffic infrastructure. The driving subjects are multiple agents, each modeled as a Markov decision process, and a reinforcement learning module is used to control the steering wheel, accelerator, and brake. The Markov decision process is described by a tuple (S, A, P, R, γ), where S represents a state set, corresponding to the state quantities acquired by the state quantity encoder and image encoder, including the steering, accelerator, brake, gear, lateral velocity, longitudinal velocity, and 15 channels of semantic bird's-eye view. A represents an action set, corresponding to the control quantities of the steering wheel, accelerator, and brake of the driving subject. P represents a state transition equation p: S × A → P(S). For each state-action pair (s, a) ∈ S × A, there is a probability distribution p(·|s, a), which represents the probability of entering a new state after taking action a in state s. R represents a reward function R: S × S × A → R, where R(S ｔ＋１ , S ｔ , a ｔ ) is the initial state S ｔ to a new state S ｔ＋１ The reward function defines the quality of the executed action, γ represents the discount coefficient, γ∈[0,1], and the cumulative reward [Number 79] where T represents the current time and γ ｉ represents the discount coefficient at time i, and r ｉ represents the immediate reward at time i, and the solution to the Markov decision problem is the solution that maximizes the cumulative reward. [Number 80] The purpose of the present invention is to find a strategy π:S→A to obtain the optimal control strategy. The reinforcement learning module aligns the implicit state variables output from the state encoder and the image encoder, and outputs the corresponding optimal control strategy. Step 2: Construct a complex network perception module to model the environment in which the driving subject exists, establish a complex network model, and output dynamic driving advice through an activation function according to the state quantity provided by the state quantity encoder, where the complex network model represents the dynamic relationship between nodes within the domain range by a variable Gaussian safety field based on risk center transition, where the nodes include the driving subject, traffic participants and traffic infrastructure; Step 3: Construct an end-to-end neural network, which includes two fully connected layers used for the state encoder, six convolutional layers used for the image encoder, and six fully connected layers used for the reinforcement learning module. The neural network has two output heads, one for action and one for value. The action output head outputs two parameters according to the beta distribution, and the value output head outputs the value of the action. Step 4: The driving subject interacts with the simulation environment, and stores the generated experiences in their respective local experience pools. When the number of samples reaches a predetermined threshold, the driving subject samples from their respective local experience pools in minimum batches, and then updates the neural network parameters according to the designed loss function. Step 5: Upload the neural network parameters corresponding to each driving subject to the federated learning module, and aggregate the global parameters at aggregation intervals according to multiple neural network parameters until the network converges, and send the aggregated global parameters to each agent; In step 1, the traffic participants include other vehicles and pedestrians, and the traffic infrastructure includes traffic lights and traffic signs, and are represented in an image encoder by a four-frame semantic bird's-eye view; In step 2, the complex network cognitive module adopts different activation functions according to different driving advices; [Number 81] In step 2, the dynamic driving advice is expressed as a beta distribution; B ｒｉｓｋ =B(Activate ｇｏ (Risk),β),go B ｒｉｓｋ =B(α,Activate ｓｔｏｐ (Risk)),stop In step 3, the neural network of the state encoder uses the Relu activation function in two fully connected layers; the neural network of the image encoder uses the Relu activation function in the other five convolutional layers except for the last convolutional layer that flattens the state and does not use an activation function; the neural network of the reinforcement learning module uses the Softplus activation function in the last layer of the action output head, does not use an activation function in the last layer of the value output head, and uses the Relu activation function in the other fully connected layers; In step 4, the parameters used in the training procedure are: learning rate 0.00001, total step size 12288, minimum batch sampling 256, and loss function weight λ ｐｐｏ , λ ｅ×ｐ , λ ｒｉｓｋ are 0.5, 0.01, and 0.05, respectively, the ppo clipping range is 0.2, the generalized advantage estimation parameter γ is 0.99, and λ is 0.9; In step 4, the loss function used for training is the loss function based on dynamic risk advice. [Number 82] The driving subject is N from the end state. Ｚ = 100 steps ago, calculate the KL divergence of the policy and dynamic driving advice, and realize guidance for the agent. In step 5, the federated learning module is a multi-agent framework, and a local experience pool mechanism is used between the agents and the experience pool; In step 5, the end-to-end autonomous driving control method using associative reinforcement learning based on complex network cognition is characterized in that a parameter averaging aggregation method is used in the aggregation procedure and the aggregation interval is 256.

10. A vehicle device capable of executing the contents of the end-to-end autonomous driving control system using associative reinforcement learning based on complex network cognition described in any one of claims 1 to 8, or the end-to-end autonomous driving control method using associative reinforcement learning based on complex network cognition described in claim 9.

Images