A Multi-Agent Vehicle-Road-Cloud Integrated Collaborative Decision-Making Architecture and Method Based on Federated Reinforcement Learning

Abstract

The present invention discloses a multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system and method based on federated reinforcement learning. A multi-agent federated reinforcement learning decision-making and control framework with embedded vehicle dynamics characteristics is adopted to solve the problem of deep integration of intelligent transportation systems and intelligent vehicles, and realize autonomous driving with deep decision-making and control coordination between vehicles and traffic; a semantic matrix is ​​generated on the road side as the input of vehicle-side reinforcement learning to construct global and local trajectory planning for the vehicle side guided by the road side; a fusion reward function of vehicle-side reinforcement learning is designed based on the driving safety field constructed on the road side to realize comprehensive consideration of vehicle-side safety and comfort; based on road-side federated learning, the vehicle-side neural network parameters are uploaded through V2I communication to solve the problem of vehicle-road information asymmetry caused by privacy awareness; for the distribution of samples in different environments, the local optimal strategy for the current environment is selected through neural network screening, and a shared model that benefits from different environments is synthesized to achieve a balance between sample efficiency and model robustness.

Description

Technical Field

[0001] The present invention belongs to the field of transportation and relates to a multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system and method based on federated reinforcement learning. Background Art

[0002] Due to the limited perception range and computing power, it is difficult for a single vehicle to cope with complex traffic conditions. The intelligent transportation system based on vehicle-road cooperative technology can achieve vehicle-road cooperative perception and vehicle-side computing load transfer through information exchange between vehicles and road infrastructure, providing a new solution for autonomous driving under complex traffic conditions.

[0003] The intelligent transportation system based on vehicle-road cooperative technology relies on V2X communication technology, which includes V2V vehicle-to-vehicle communication and V2I vehicle-to-infrastructure communication. Through V2X communication, the intelligent transportation system can obtain all-round traffic information, including information about traffic participants and road obstacles. Vehicle-road cooperative technology has many advantages and characteristics. First, through vehicle-road cooperation, the vision of intelligent connected vehicles is expanded from the driver's perspective to the higher and wider BEV bird's-eye view; at the same time, through V2I vehicle-to-infrastructure communication, the deployment of partial autonomous driving algorithms for infrastructure can be realized, thereby reducing the computing burden of intelligent connected vehicles; and the fixed sensor system of the roadside unit RSU can access road history information, making it easier to detect random obstacles that appear on the road. Based on vehicle-road cooperative technology, the intelligent transportation system makes full use of the advantages of the road side, thereby solving the problems of limited perception range of a single vehicle, limited computing power of a single vehicle, and difficulty in detecting random obstacles.

[0004] The existing intelligent transportation system architecture can provide drivers with additional traffic information and driving assistance. However, in the field of autonomous driving, problems such as redundant traffic information, difficulty in extracting key information, and huge communication overhead directly affect the decision-making efficiency and control effect of vehicle-road collaboration, resulting in the immaturity of research on integrated collaborative architecture serving autonomous driving and the lack of practical application technology. In addition, at this stage, vehicle-road collaborative technology usually defaults to connecting connected cars to the collaborative network, and the vehicle-road information asymmetry caused by privacy awareness further restricts the development of integrated collaborative technology. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture based on federated reinforcement learning. It adopts a multi-agent federated reinforcement learning decision-making and control framework with embedded vehicle dynamics characteristics, solves the deep information fusion problem of intelligent transportation system ITS and intelligent vehicle IV, and realizes autonomous driving based on deep decision-making and control coordination between vehicles and traffic. Based on the road-side perspective, a semantic matrix is ​​generated as the input of vehicle-side reinforcement learning, and the vehicle-side global and local trajectory planning under the guidance of the road-side is constructed; the driving safety field modeling is realized by taking advantage of the road-side, and the fusion reward function used in vehicle-side reinforcement learning is constructed, realizing the comprehensive consideration of vehicle-side safety and comfort; based on the road-side federated learning architecture, the vehicle-side neural network parameters are transmitted through V2I communication, solving the problem of asymmetric information between vehicles and roads caused by privacy awareness; for the distribution of samples in different environments, the local optimal strategy for the current environment is selected based on the neural network screening process, and a global shared model that benefits from different environments is synthesized to achieve a balance between sample efficiency and model robustness.

[0006] The technical solution of the multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system based on federated reinforcement learning in the present invention includes three main contents: cloud side, road side and vehicle side.

[0007] The cloud is mainly used for uploading and sending information to realize different levels of cloud control applications. The cloud consists of a cloud control platform and related support platforms, and realizes efficient information exchange through V2N vehicle-to-cloud communication combined with optical bearer network communication.

[0008] The cloud control platform can be divided into three cloud control applications according to the application level: connected car empowerment, traffic management and control, and traffic data empowerment, corresponding to the three cloud control levels of edge cloud, regional cloud, and central cloud in the platform.

[0009] The supporting platform includes but is not limited to a traffic management platform that provides a complete traffic network and real-time traffic information, a logistics platform that provides logistics supervision information, a map platform that provides high-precision maps, a positioning platform that provides high-precision positioning and navigation, a meteorological platform that provides real-time weather information, etc.

[0010] Based on the real-time traffic dynamics collected by the edge cloud, the cloud control platform provides enabling services such as meeting the driving safety of connected vehicles and improving traffic efficiency. Through the edge cloud combined with the support platform to provide information, the regional cloud provides services such as traffic situation perception and assessment, traffic planning, order management, and transportation management, achieving gain effects such as reducing traffic accidents, alleviating road congestion, and improving road network efficiency. Furthermore, through the regional cloud combined with the support platform to provide information, the central cloud provides big data analysis services and business support for colleges and universities, scientific research institutions, travel service providers, automobile manufacturers, etc. based on traffic big data empowerment.

[0011] For the road side, it is mainly used to coordinate the vehicle side group according to the different levels of cloud control applications in the cloud. The road side is composed of multiple road side units, where a single road side unit contains an information processing module and a group coordination decision module.

[0012] The information processing module processes the image input from the bird's-eye view into a semantic bird's-eye view, and uses the road-side advantage to generate a Gaussian safety field. The semantic bird's-eye view is used as the input of the vehicle-side perception processing module, and the Gaussian safety field participates in generating the fusion reward function in the reinforcement learning module. The Gaussian safety field is established by the following equation:

[0013]

[0014]

[0015] φ=b x / c y = l v / w v

[0016] Among them, S sta Represents the static safety field strength, C a represents the static safety field strength coefficient, x0 and y0 represent the coordinates of the static risk center O(x0, y0), ε represents the safety field shape coefficient, b x and c y represents the appearance coefficient of the intelligent networked vehicle, φ is the length-to-width ratio of the intelligent networked vehicle, l v Indicates vehicle length, w v Indicates the width of the vehicle.

[0017] When the intelligent connected car moves, the risk center O(x0, y0) of the Gaussian safety field will be transferred to the new risk center O′(x′0, y′0) as the vehicle moves:

[0018]

[0019]

[0020] Among them, k v represents the mobile adjustment factor, and The sign is related to the direction of movement, β represents the transfer vector of the connected car The angle with the coordinate axis in the Cartesian coordinate system, Represents the speed vector of the connected car. A virtual vehicle is formed under the action of risk center transfer in the dynamic safety field, with a length of l′ v , width is w′ v , S dyn represents the dynamic safety field strength, and the new aspect ratio is expressed as φ′=b′ x / c′ y = l′v / w′ v .

[0021] The group coordination decision module obtains the vehicle-side neural network parameters of the intelligent connected vehicles in the vehicle-side group through V2I vehicle-road communication, and screens and aggregates the local vehicle-side group neural network parameters and shared neural network parameters based on federated learning to achieve multi-agent group optimization. The group coordination decision process only transmits network parameters instead of training samples, ensuring the privacy of agents while reducing communication and computing overhead. The neural network for uploading and downloading parameters is a reinforcement learning actor-critic Actor-Critic networks neural network, in which the actor network outputs the regulation and control strategy and implements the strategy evaluation through the critic network Critic network. For the distribution of samples in different environments, the local optimal strategy for the current environment is selected based on the neural network screening process, and a global shared model that benefits from different environments is synthesized to achieve a balance between sample efficiency and model robustness.

[0022] In the neural network screening process, the present invention uses driving safety and driving aggressiveness based on Gaussian safety field as network screening indicators. The rule-based method is used as the screening basis to achieve the fusion drive of rules and data, and enhance the interpretability of the algorithm framework through interpretable rules. The actor network adopts the following aggregation process:

[0023]

[0024] in, Represents the intelligent connected vehicle's own parameters θ μ，i and another intelligent connected car parameter θ μ，i′ The new network parameters obtained by aggregation. Safety represents the safety-related reward function of the intelligent connected car, which is calculated from the road-side driving Gaussian safety field from the aspects of driving safety and driving aggressiveness:

[0025]

[0026]

[0027]

[0028]

[0029] r Safety =r Risk +r Agg

[0030] Among them, R i，j(t) represents the driving risk caused by intelligent connected vehicle j to intelligent connected vehicle i, represents the field strength of intelligent connected car j to intelligent connected car i, k c represents the risk perception coefficient, represents the speed of smart connected vehicle j at time t, θ i，j (t) represents the driving angle between intelligent connected vehicles i and j at time t, r Risk represents the driving risk-related reward function, f Risk (ξ) represents the driving risk score, R thr represents the risk threshold, τ rc represents the duration of exceeding the risk threshold, R j，i (t′) represents the driving risk caused by intelligent connected vehicle i to intelligent connected vehicle j, represents the field strength of intelligent connected vehicle i to intelligent connected vehicle j, represents the field strength between intelligent connected car j and intelligent connected car i, represents the speed of intelligent connected vehicle i at time t′, θ j，i (t′) represents the driving angle between intelligent connected vehicle j and intelligent connected vehicle i at time t′, r Agg represents the driving aggressiveness-related reward function, f Agg (ξ) represents the driving aggressiveness score.

[0031] The critic network adopts the following aggregation process:

[0032]

[0033] in, Represents the intelligent connected vehicle's own parameters θ l，i and another intelligent connected car parameter θ l，i′ The new network parameters obtained by aggregation, Represents the value output by the critic network at the state-action pair (s, a).

[0034] For the vehicle side, it is mainly used to plan the control quantity according to the input from the road side combined with the information output from the sensors of the intelligent networked vehicles. A single roadside unit contains multiple vehicle side groups, each of which consists of N intelligent networked vehicles, and information sharing is achieved through V2V vehicle-to-vehicle communication. The intelligent networked vehicle consists of a perception processing module, a trajectory prediction module, a reinforcement learning module, and an intelligent chassis coupling system.

[0035] The perception processing module matches and segments the semantic bird's-eye view provided by the road side according to the vehicle side positioning, and fuses the vehicle dynamic information (sensor data) and the road side static information into a state matrix as the input of the reinforcement learning module neural network. The dynamic information includes the vehicle speed and position information, the road side static information includes the road, expected path, and lane information, and the state matrix i RL ∈[0, 1] W×H×C , where W×H=56 represents the matrix size, which actually covers a physical range of about 14m×14m, and C=4 represents the number of channels.

[0036] The trajectory prediction module realizes multi-time-step trajectory prediction through the semantic bird's-eye view provided by the road end, and the prediction result will be used as part of the input of the reinforcement learning module.

[0037] The reinforcement learning module generates the planning control output in the CARLA simulator through the input of the perception processing module and the input of the trajectory prediction module, combined with the fusion reward function generated by the roadside safety field. In the interaction process, the reinforcement learning actor network generates the planning control output according to the fusion state quantity. t1 with a t2 , the output is mapped to the range [-1, 1] using an activation function, where a t1 Indicates the steering wheel control amount, a t2 It is divided into two parts [-1, 0] and [0, 1], representing the brake and throttle control amounts respectively.

[0038] The intelligent chassis coupling system is mainly used for a single intelligent connected vehicle to output a secondary planning control quantity based on the reinforcement learning regulation output and the expected path. The corresponding chassis intelligent subsystem is controlled by a distributed controller, wherein the intelligent subsystem includes but is not limited to the steering subsystem, the DYC subsystem, the suspension subsystem and the drive subsystem, and there is state coupling and cost coupling between the intelligent subsystems. The steering subsystem and the DYC subsystem calculate the lateral control quantity of the intelligent connected vehicle, the suspension subsystem calculates the vertical control quantity of the intelligent connected vehicle, and the drive subsystem calculates the longitudinal control quantity of the intelligent connected vehicle. The steering subsystem and the DYC subsystem calculate the yaw torque of the intelligent connected vehicle based on the steering wheel control quantity output by the reinforcement learning module, the preview error between the actual path and the expected path. The drive subsystem obtains the expected target speed and calculates the four-wheel drive\braking torque of the intelligent connected vehicle based on the throttle control quantity and the current speed of the intelligent connected vehicle. First, the driving force equation of the intelligent connected vehicle is constructed according to PID:

[0039]

[0040] Among them, F x(t) represents the driving force of the intelligent connected vehicle at time t, K p represents the proportional gain, e(t) represents the speed error at time t, K i Indicates the integral gain, K d Represents the differential gain.

[0041] Then consider the following optimization problem:

[0042]

[0043] where F xij represents the longitudinal force of the four wheels of the intelligent networked vehicle, F yij represents the lateral force of the four wheels of the intelligent connected vehicle, F zfl 、F zfr 、F zrl 、F zrr represents the vertical force of the four wheels of the intelligent networked vehicle, fl represents the left front wheel, fr represents the right front wheel, rl represents the left rear wheel, rr represents the right rear wheel, and μ represents the adjustment factor. Since the steering subsystem and DYC subsystem have calculated the four-wheel turning angle, that is, F yij , ij=fl, fr, rl, rr are constants, so the above formula is simplified to:

[0044]

[0045] subject to

[0046]

[0047]

[0048]

[0049] Among them, m represents the quality of intelligent connected vehicles, a x represents the driving acceleration, d represents the wheelbase of the intelligent networked vehicle, M z represents the additional yaw moment, R w represents the wheel radius of the smart connected car, T min Indicates the minimum driving torque or braking torque, T max The above three formulas respectively indicate that the total driving force satisfies the driving subsystem constraints, the additional yaw moment satisfies the steering subsystem and DYC subsystem constraints, and the driving force satisfies the actuator constraints.

[0050] Finally, by solving the optimization problem, the four-wheel drive or braking torque is obtained:

[0051] T ij =F xij R w , i=fl,fr,rl,rr

[0052] After the intelligent chassis coupling system outputs the quadratic planning control quantity, it controls the intelligent networked vehicles to interact in the CARLA simulator to generate experience samples, and trains the neural network parameters of the reinforcement learning module based on the experience samples. The group coordination decision module on the road side then obtains the vehicle-side neural network parameters of the intelligent networked vehicles in the vehicle-side group through V2I vehicle-road communication, and based on federated learning, screens and aggregates the local vehicle-side group neural network parameters and shares the neural network parameters to achieve multi-agent group optimization.

[0053] The technical solution of the multi-agent vehicle-road-cloud integrated collaborative decision-making and control method based on federated reinforcement learning of the present invention includes the following steps:

[0054] Step 1: Build a cloud platform, and provide three cloud control applications through the three cloud control levels of the cloud control platform: connected car empowerment, traffic management and control, and traffic data empowerment. Obtain complete traffic road network and real-time traffic information, logistics supervision information, high-precision map information, high-precision positioning and navigation information, real-time weather information, etc. through the relevant support platform on the cloud. Provide high-level information such as traffic dispatch and management for the road side through the cloud control platform combined with relevant support platforms.

[0055] Step 2: Build a road-side platform, process the bird's-eye view image input into a semantic bird's-eye view through the road-side information processing module, and use the road-side advantage to generate a Gaussian safety field. The semantic bird's-eye view is used as the input of the vehicle-side perception processing module, and the Gaussian safety field participates in generating the fusion reward function in the reinforcement learning module. The vehicle-side neural network parameters of the intelligent connected vehicles in the vehicle-side group are obtained through the road-side group coordination decision module, and the local vehicle-side group neural network parameters and shared neural network parameters are screened and aggregated based on federated learning to build a multi-agent group optimization architecture.

[0056] Step 3: Build a vehicle-side platform. A single roadside unit contains multiple vehicle-side groups. Each vehicle-side group consists of N intelligent connected vehicles, and information sharing is achieved through V2V vehicle-to-vehicle communication. Intelligent connected vehicles are composed of perception processing modules, trajectory prediction modules, reinforcement learning modules, and intelligent chassis coupling systems. The perception processing module matches and segments the semantic bird's-eye view provided by the road side according to the vehicle-side positioning, and fuses the state matrix of the vehicle dynamic information (sensor data) and the road side static information as part of the input of the reinforcement learning module. The trajectory prediction module realizes multi-time-step trajectory prediction based on the semantic bird's-eye view provided by the road side, and the prediction result will be used as part of the input of the reinforcement learning module. The planning and control process is modeled as a Markov decision process through the reinforcement learning module, and the output of the perception processing module is used as the reinforcement learning input. Combined with the fusion reward function generated by the road side safety field, the control output is interactively generated in the CARLA simulator. Then, according to the reinforcement learning control output and the expected path, the secondary planning control quantity of a single intelligent connected vehicle is output through the intelligent chassis coupling system. Finally, the corresponding chassis intelligent subsystem is controlled by the distributed controller.

[0057] Step 4: After the intelligent chassis coupling system of a single intelligent connected vehicle on the vehicle side outputs the quadratic planning control quantity, it interacts in the CARLA simulator to generate experience samples, and trains the neural network parameters of the reinforcement learning module based on the experience samples.

[0058] Step 5: The road-side group coordination decision module obtains the vehicle-side neural network parameters of the intelligent connected vehicles in the vehicle-side group through V2I vehicle-road communication, and screens and aggregates the local vehicle-side group neural network parameters and shared neural network parameters based on federated learning to achieve multi-agent group optimization.

[0059] Preferably, in step 1, the three cloud control levels are edge cloud, regional cloud, and central cloud, respectively. Based on the real-time traffic dynamics collected by the edge cloud, the cloud control platform provides enabling services such as meeting the driving safety of connected vehicles and improving traffic efficiency. The edge cloud is combined with the support platform to provide information, and the regional cloud provides services such as traffic situation awareness and assessment, traffic planning, order management, and transportation management to achieve gain effects such as reducing traffic accidents, alleviating road congestion, and improving road network efficiency. Furthermore, by providing information through the regional cloud combined with the support platform, the central cloud provides big data analysis services and business support for colleges and universities, scientific research institutions, travel service providers, automobile manufacturers, etc. based on traffic big data empowerment.

[0060] Preferably, in step 2, the Gaussian safety field is established by the following equation:

[0061]

[0062]

[0063] φ=b x / c y = l v / w v

[0064] Among them, S sta Represents the static safety field strength, C a represents the static safety field strength coefficient, x0 and y0 represent the coordinates of the static risk center O(x0, y0), b x and c y represents the appearance coefficient of the intelligent networked vehicle, φ is the length-to-width ratio of the intelligent networked vehicle, l v Indicates vehicle length, w v Indicates the width of the vehicle.

[0065] When the intelligent connected car moves, the risk center O(x0, y0) of the Gaussian safety field will be transferred to the new risk center O′(x′0, y′0) as the vehicle moves:

[0066]

[0067]

[0068] Among them, k v represents the mobile adjustment factor, and The sign is related to the direction of movement, β represents the transfer vector of the connected car The angle between the coordinate axis and the Cartesian coordinate system. A virtual vehicle is formed under the action of risk center transfer in the dynamic safety field, with a length of l′ v , width is w′ v , S dyn represents the dynamic safety field strength, and the new aspect ratio is expressed as φ′=b′ x / c′ y = l′ v / w′ v .

[0069] Preferably, in step 2, the neural network screening process uses driving safety and driving aggressiveness based on Gaussian safety field as network screening indicators. The rule-based method is used as the screening basis to achieve the fusion drive of rules and data, and the interpretability of the algorithm framework is enhanced through interpretable rules.

[0070] The actor network adopts the following aggregation process:

[0071]

[0072] in, Represents the intelligent connected vehicle's own parameters θ μ，iand another intelligent connected car parameter θ μ，i′ The new network parameters obtained by aggregation. Safety represents the safety-related reward function of the intelligent connected car, which is calculated by the roadside driving safety field from the aspects of driving safety and driving aggressiveness:

[0073]

[0074]

[0075]

[0076]

[0077] r Safety =r Risk +r Agg

[0078] Among them, R i，j (t) represents the driving risk caused by intelligent connected vehicle j to intelligent connected vehicle i, represents the field strength of intelligent connected car j to intelligent connected car i, k c represents the risk perception coefficient, represents the speed of smart connected vehicle j at time t, θ i，j (t) represents the driving angle between intelligent connected vehicles i and j at time t, r Risk represents the driving risk-related reward function, f Risk (ξ) represents the driving risk score, R thr represents the risk threshold, τ rc represents the duration of exceeding the risk threshold, R j，i (t′) represents the driving risk caused by intelligent connected vehicle i to intelligent connected vehicle j, represents the field strength of intelligent connected vehicle i to intelligent connected vehicle j, represents the speed of intelligent connected vehicle i at time t′, θ j，i (t′) represents the driving angle between intelligent connected vehicle j and intelligent connected vehicle i at time t′, r Agg represents the driving aggressiveness-related reward function, f Agg (ξ) represents the driving aggressiveness score.

[0079] The critic network adopts the following aggregation process:

[0080]

[0081] in, Represents the intelligent connected vehicle's own parameters θ l，iand another intelligent connected car parameter θ l，i′ The new network parameters obtained by aggregation, The value output by the critic network at a state-action pair (s, a).

[0082] Preferably, in step 3, the intelligent subsystem includes but is not limited to a steering subsystem and a DYC subsystem for calculating the lateral control amount of the intelligent connected vehicle, a suspension subsystem for calculating the vertical control amount of the intelligent connected vehicle, and a driving subsystem for calculating the longitudinal control amount of the intelligent connected vehicle. The steering subsystem and the DYC subsystem calculate the yaw torque of the intelligent connected vehicle based on the steering wheel control amount output by the reinforcement learning module and the preview error between the actual path and the expected path. The driving subsystem obtains the expected target speed and calculates the four-wheel drive\braking torque of the intelligent connected vehicle based on the throttle control amount and the current speed of the intelligent connected vehicle.

[0083] Preferably, in step 4, the experience sample is composed of a tuple (s t , a t , r t ,s t+1 ) description, where s t Corresponding state matrix i RL ∈[0, 1] W×H×C , where W×H=56 represents the matrix size, which actually covers a physical range of about 14m×14m, and C=4 represents the number of channels. t The corresponding reinforcement learning actor network Actor network generates a control output according to the state quantity t1 with a t2 , the output is mapped to the range [-1, 1] using an activation function, where a t1 Indicates the steering wheel control amount, a t2 It is divided into two parts: [-1, 0] and [0, 1], which represent the brake and throttle control values ​​respectively. t Corresponding to the expected path-related reward function and the safety-related reward function, where s t+1 Represents the state matrix of the next frame.

[0084] Preferably, in step 5, the vehicle-road communication only transmits network parameters instead of training samples, thereby ensuring the privacy of the agent while reducing communication and computing overhead. The neural network for parameter uploading and downloading is an actor-critic neural network of reinforcement learning, in which the actor network outputs the control strategy and implements the strategy evaluation through the critic network.

[0085] Beneficial effects of the present invention:

[0086] (1) The present invention provides a multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture based on federated reinforcement learning. It adopts a multi-agent federated reinforcement learning decision-making and control framework with embedded vehicle dynamics characteristics, solves the problem of deep information fusion between intelligent transportation system ITS and intelligent vehicle IV, and realizes autonomous driving based on deep decision-making and control collaboration between vehicles and traffic.

[0087] (2) A semantic matrix is ​​generated from the road-side perspective as the input for vehicle-side reinforcement learning, and the vehicle-side global and local trajectory planning under the guidance of the road-side is constructed. The advantages of the road-side are used to realize driving safety scene modeling, and a fusion reward function used in vehicle-side reinforcement learning is constructed to achieve comprehensive consideration of vehicle-side safety and comfort. Based on the road-side federated learning architecture, the vehicle-side neural network parameters are transmitted through V2I communication, which solves the problem of information asymmetry between the vehicle and the road caused by privacy awareness.

[0088] (3) According to the sample distribution of different environments, the local optimal strategy for the current environment is selected based on the neural network screening process, and a global shared model that benefits from different environments is synthesized to achieve a balance between sample efficiency and model robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 Schematic diagram of the vehicle-road-cloud integrated collaborative decision-making and control architecture proposed in the present invention;

[0090] Figure 2 Schematic diagram of vehicle-road-cloud communication proposed by the present invention;

[0091] Figure 3 Schematic diagram of group coordination decision-making proposed by the present invention;

[0092] Figure 4 Schematic diagram of neural network screening used in the present invention; DETAILED DESCRIPTION

[0093] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings, but the content of the present invention is not limited thereto.

[0094] The present invention provides a multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture based on federated reinforcement learning, such as Figure 1 As shown in Figure 1, multi-agent autonomous driving based on vehicle-road-cloud integrated collaborative decision-making control in complex driving scenarios can be realized. The vehicle-road-cloud communication diagram is shown in Figure 1. Figure 2 As shown, the specific steps include:

[0095] (1) Build a cloud platform, and provide three cloud control applications through the three cloud control levels of the cloud control platform: connected vehicle empowerment, traffic management and control, and traffic data empowerment. Obtain complete traffic network and real-time traffic information, logistics supervision information, high-precision map information, high-precision positioning and navigation information, real-time weather information, etc. through the relevant support platform on the cloud. Provide high-level information such as traffic dispatch and management to the road side through the cloud control platform combined with relevant support platforms.

[0096] (2) Build a road-side platform, process the bird's-eye view image input into a semantic bird's-eye view through the road-side information processing module, and use the road-side advantage to generate a Gaussian safety field. The semantic bird's-eye view is used as the input of the vehicle-side perception processing module, and the Gaussian safety field is used to generate the fusion reward function in the vehicle-side reinforcement learning module. The vehicle-side neural network parameters of the intelligent connected vehicle in the vehicle-side group are obtained through the road-side group coordination decision module, such as Figure 3 As shown, based on federated learning, the local vehicle-side group neural network parameters are screened and aggregated, as well as the shared neural network parameters, to build a multi-agent group optimization architecture. The Gaussian safety field is established through the following equation:

[0097]

[0098]

[0099] φ=b x / c y = l v / w v

[0100] Among them, S sta Represents the static safety field strength, C a represents the static safety field strength coefficient, x0 and y0 represent the coordinates of the static risk center O(x0, y0), b x and c y represents the appearance coefficient of the intelligent networked vehicle, φ is the length-to-width ratio of the intelligent networked vehicle, l v Indicates vehicle length, w v Indicates the width of the vehicle.

[0101] When the intelligent connected car moves, the risk center O(x0, y0) of the Gaussian safety field will be transferred to the new risk center O′(x′0, y′0) as the vehicle moves:

[0102]

[0103]

[0104] Among them, k v represents the mobile adjustment factor, and The sign is related to the direction of movement, β represents the transfer vector of the connected car The angle between the coordinate axis and the Cartesian coordinate system. A virtual vehicle is formed under the action of risk center transfer in the dynamic safety field, with a length of l′ v , width is w′ v , S dyn represents the dynamic safety field strength, and the new aspect ratio is expressed as φ′=b′ x / c′ y = l′ v / w′ v .

[0105] (3) Build a vehicle-side platform. A single roadside unit contains multiple vehicle-side groups. Each vehicle-side group consists of N intelligent connected vehicles, and information sharing is achieved through V2V vehicle-to-vehicle communication. The intelligent connected vehicle consists of a perception processing module, a trajectory prediction module, a reinforcement learning module, and an intelligent chassis coupling system. The perception processing module matches and segments the semantic bird's-eye view provided by the road side according to the vehicle-side positioning, and fuses the dynamic information of the vehicle (sensor data) and the static information of the road side into a state matrix as part of the reinforcement learning input. The trajectory prediction module realizes multi-time-step trajectory prediction based on the semantic bird's-eye view provided by the road side, and the prediction result will be used as part of the reinforcement learning module input. The planning and control process is modeled as a Markov decision process through the reinforcement learning module, and the output of the perception processing module is used as the reinforcement learning input. The fusion reward function is generated in combination with the road side Gaussian safety field, and the control output is interactively generated in the CARLA simulator. Then, according to the reinforcement learning control output and the expected path, the secondary planning control quantity of a single intelligent connected vehicle is output through the intelligent chassis coupling system. Finally, the corresponding chassis intelligent subsystem is controlled by the distributed controller. The intelligent subsystems include but are not limited to the steering subsystem, DYC subsystem, suspension subsystem and drive subsystem, and there is state coupling and cost coupling between the intelligent subsystems. The steering subsystem and DYC subsystem calculate the lateral control amount of the intelligent connected vehicle, the suspension subsystem calculates the vertical control amount of the intelligent connected vehicle, and the drive subsystem calculates the longitudinal control amount of the intelligent connected vehicle. The steering subsystem and the DYC subsystem calculate the yaw torque of the intelligent connected vehicle based on the steering wheel control amount output by the reinforcement learning module and the preview error between the actual path and the expected path. The drive subsystem obtains the expected target speed and calculates the four-wheel drive\braking torque of the intelligent connected vehicle based on the throttle control amount and the current speed of the intelligent connected vehicle. First, the driving force equation of the intelligent connected vehicle is constructed based on PID:

[0106]

[0107] Among them, F x (t) represents the driving force of the intelligent connected vehicle at time t, K p represents the proportional gain, e(t) represents the speed error at time t, Ki Indicates the integral gain, K d Denotes the differential gain. Then consider the following optimization problem:

[0108]

[0109] Among them, F xij represents the longitudinal force of the four wheels of the intelligent connected vehicle, F yij represents the lateral force of the four wheels of the intelligent connected vehicle, F zfl 、F zfr 、F zrl 、F zrr represents the vertical force of the four wheels of the intelligent networked vehicle, fl represents the left front wheel, fr represents the right front wheel, rl represents the left rear wheel, rr represents the right rear wheel, and μ represents the adjustment factor. Since the steering subsystem and DYC subsystem have calculated the four-wheel turning angle, that is, F yij , ij=fl, fr, rl, rr are constants, so the above formula is simplified to:

[0110]

[0111] subject to

[0112]

[0113]

[0114]

[0115] Among them, m represents the quality of intelligent connected vehicles, a x represents the driving acceleration, d represents the wheelbase of the intelligent networked vehicle, M z represents the additional yaw moment, R w represents the wheel radius of the smart connected car, T min Indicates the minimum driving torque\braking torque, T max Represents the maximum driving torque. The above three equations respectively indicate that the total driving force satisfies the driving subsystem constraints, the additional yaw torque satisfies the steering subsystem and DYC subsystem constraints, and the driving force satisfies the actuator constraints. Finally, by solving the optimization problem, the four-wheel drive\braking torque is obtained:

[0116] T ij =F xij R w , i=fl,fr,rl,rr

[0117] (4) After the intelligent chassis coupling system of a single intelligent connected vehicle on the vehicle side outputs the quadratic programming control quantity, it interacts in the CARLA simulator to generate experience samples, and trains the neural network parameters of the reinforcement learning module based on the experience samples. The experience samples are composed of tuples (st , a t , r t ,s t+1 ) description, where s t Corresponding state matrix i RL ∈[0, 1] W×H×C , where W×H=56 represents the matrix size, which actually covers a physical range of about 14m×14m, and C=4 represents the number of channels. t The corresponding reinforcement learning actor network Actor network generates a control output according to the state quantity t1 with a t2 , the output is mapped to the range [-1, 1] using an activation function, where a t1 Indicates the steering wheel control amount, a t2 It is divided into two parts: [-1, 0] and [0, 1], which represent the brake and throttle control values ​​respectively. t Corresponding to the expected path-related reward function and the safety-related reward function, where s t+1 Represents the state matrix of the next frame.

[0118] (5) The road-side group coordination decision module obtains the vehicle-side neural network parameters of the intelligent connected vehicles in the vehicle-side group through V2I vehicle-road communication, and screens and aggregates the local vehicle-side group neural network parameters and shared neural network parameters based on federated learning. The neural network screening process uses driving safety and driving aggressiveness based on Gaussian risk field as network screening indicators, such as Figure 4 As shown. The rule-based method is used as the screening basis to achieve the fusion drive of rules and data, and the interpretability of the algorithm framework is enhanced through interpretable rules. The actor network adopts the following aggregation process:

[0119]

[0120] in, Represents the intelligent connected vehicle's own parameters θ μ，i and another intelligent connected car parameter θ μ，i′ The new network parameters obtained by aggregation. Safety represents the safety-related reward function of the intelligent connected car, which is calculated from the road-side driving Gaussian safety field from the aspects of driving safety and driving aggressiveness:

[0121]

[0122]

[0123]

[0124]

[0125] r Safety =r Risk +r Agg

[0126] Among them, R i，j (t) represents the driving risk caused by intelligent connected vehicle j to intelligent connected vehicle i, represents the field strength of intelligent connected car j to intelligent connected car i, k c represents the risk perception coefficient, represents the speed of smart connected vehicle j at time t, θ i，j (t) represents the driving angle between intelligent connected vehicles i and j at time t, r Risk represents the driving risk-related reward function, f Risk (ξ) represents the driving risk score, R thr represents the risk threshold, τ rc represents the duration of exceeding the risk threshold, R j，i (t′) represents the driving risk caused by intelligent connected vehicle i to intelligent connected vehicle j, represents the field strength of intelligent connected vehicle i to intelligent connected vehicle j, represents the speed of intelligent connected vehicle i at time t′, θ j，i (t′) represents the driving angle between intelligent connected vehicle j and intelligent connected vehicle i at time t′, r Agg represents the driving aggressiveness-related reward function, f Agg (ξ) represents the driving aggressiveness score. The critic network adopts the following aggregation process:

[0127]

[0128] in, Represents the intelligent connected vehicle's own parameters θ l,i and another intelligent connected car parameter θ l,i′ The new network parameters obtained by aggregation, The value output by the critic network at the state-action pair (s, a).

[0129] In summary, the present invention proposes a multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture based on federated reinforcement learning. It adopts a multi-agent federated reinforcement learning decision-making and control framework with embedded vehicle dynamics characteristics, solves the problem of deep information fusion between intelligent transportation system ITS and intelligent vehicle IV, and realizes autonomous driving based on deep decision-making and control collaboration between vehicles and traffic. Based on the road-side perspective, a semantic matrix is ​​generated as the input of vehicle-side reinforcement learning, and the vehicle-side global and local trajectory planning under the guidance of the road-side is constructed; the driving safety field modeling is realized by taking advantage of the road-side, and the fusion reward function used in vehicle-side reinforcement learning is constructed, realizing the comprehensive consideration of vehicle-side safety and comfort; based on the road-side federated learning architecture, the vehicle-side neural network parameters are transmitted through V2I communication, solving the problem of asymmetric information between vehicles and roads caused by privacy awareness; for the distribution of samples in different environments, the local optimal strategy for the current environment is selected based on the neural network screening process, and a global shared model that benefits from different environments is synthesized to achieve a balance between sample efficiency and model robustness.

[0130] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system based on federated reinforcement learning, characterized by: include: Cloud, road and vehicle; The cloud includes a cloud control platform and a support platform. The cloud control platform includes three cloud control applications: connected vehicle empowerment, traffic management and control, and traffic data empowerment; the support platform includes a traffic management platform that provides a complete traffic network and real-time traffic information, a logistics platform that provides logistics supervision information, a map platform that provides high-precision maps, a positioning platform that provides high-precision positioning and navigation, and a meteorological platform that provides real-time weather information; The roadside is used to coordinate the vehicle-side group according to different cloud control applications in the cloud. The roadside includes multiple roadside units, and a single roadside unit includes an information processing module and a group coordination decision module; the information processing module processes the image input from a bird's-eye view into a semantic bird's-eye view, and generates a Gaussian safety field. The semantic bird's-eye view is used as the input of the vehicle-side perception processing module, and the Gaussian safety field is used to generate a fusion reward function in the vehicle-side reinforcement learning module; the group coordination decision module obtains the neural network parameters of the vehicle-side reinforcement learning module of the intelligent networked vehicle in the vehicle-side group through V2I vehicle-road communication, and screens and aggregates the local vehicle-side group neural network parameters and shared neural network parameters based on federated learning to achieve multi-agent group optimization; The vehicle side is used to output the planned control quantity according to the input of the road side combined with the sensor information of the intelligent networked vehicle; a single roadside unit contains multiple vehicle side groups, each vehicle side group is composed of N intelligent networked vehicles, and information sharing is realized through V2V vehicle-to-vehicle communication; the vehicle side includes a perception processing module, a trajectory prediction module, a reinforcement learning module and an intelligent chassis coupling system, the perception processing module matches and segments the semantic bird's-eye view provided by the road side according to the vehicle side positioning, and fuses the dynamic information of the vehicle and the static information of the road side to obtain a state quantity matrix as the input of the neural network of the reinforcement learning module, the trajectory prediction module realizes multi-time-step trajectory prediction through the semantic bird's-eye view provided by the road side, and the prediction result will be used as part of the input of the reinforcement learning module, the reinforcement learning module generates a fusion reward function through the input of the perception processing module, the input of the trajectory prediction module and the Gaussian safety field of the road side, and interactively generates the planned control quantity output, the intelligent chassis coupling system is based on a single intelligent networked vehicle, and outputs a secondary planned control quantity according to the planned control quantity output by the reinforcement learning module and the expected path, and then controls the corresponding chassis intelligent subsystem through a distributed controller; The cloud control platform includes edge cloud, regional cloud and central cloud; The edge cloud collects real-time traffic dynamics and provides services that meet the driving safety of connected vehicles and improve traffic efficiency; the regional cloud provides traffic situation awareness and assessment, traffic planning, order management, and transportation management services; the central cloud provides big data analysis services and business support based on traffic big data empowerment; The Gaussian safety field is established by the following equation: φ=b x / c y =l v / w v Among them, S sta Represents the static safety field strength, C a represents the static safety field strength coefficient, x0 and y0 represent the coordinates of the static risk center O(x0,y0), b x and c y represents the appearance coefficient of the intelligent networked vehicle, φ is the length-to-width ratio of the intelligent networked vehicle, l v Indicates vehicle length, w v Indicates vehicle width; When the intelligent connected vehicle moves, the static risk center O(x0,y0) of the Gaussian safety field will be transferred to the new static risk center O(x0,y0) as the vehicle moves. ′ (x ′ 0,y0 ′ ): Among them, k v represents the mobile adjustment factor, and The sign is related to the direction of movement, β represents the transfer vector of the connected car The angle between the coordinate axis and the Cartesian coordinate system, the dynamic safety field forms a virtual vehicle under the action of the static risk center transfer, with a length of l′ v , width is w′ v , S dyn represents the dynamic safety field strength, and the new aspect ratio is expressed as φ′=b′ x / c′ y = l′ v / w′ v .

2. The multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system based on federated reinforcement learning according to claim 1 is characterized in that: The neural network on the vehicle side is an Actor-Critic network neural network of reinforcement learning, in which the Actor network outputs the control strategy and implements the strategy evaluation through the Critic network; The actor network adopts the following aggregation process: in, Represents the intelligent connected vehicle's own parameters θ μ,i and another intelligent connected car parameter θ μ,i′ The new network parameters obtained by aggregation, r Safety represents the safety-related reward function of the intelligent connected car, which is calculated from the road-side driving Gaussian safety field from the aspects of driving safety and driving aggressiveness: r safety ＝r Risk +r Agg Among them, R i,j (t) represents the driving risk caused by intelligent connected vehicle j to intelligent connected vehicle i, represents the field strength of intelligent connected car j to intelligent connected car i, k c represents the risk perception coefficient, represents the speed of smart connected vehicle j at time t, θ i,j (t) represents the driving angle between intelligent connected vehicles i and j at time t, r Risk represents the driving risk-related reward function, f Risk (ξ) represents the driving risk score, R thr represents the risk threshold, τ rc represents the duration of exceeding the risk threshold, R j ,i(t ′ ) represents the driving risk caused by intelligent connected vehicle i to intelligent connected vehicle j, represents the field strength of intelligent connected vehicle i to intelligent connected vehicle j, represents the speed of intelligent connected vehicle i at time t′, θ j,i (t′) represents the driving angle between intelligent connected vehicle j and intelligent connected vehicle i at time t′, r Agg represents the driving aggressiveness-related reward function, f Agg (ξ) represents the driving aggressiveness score; The critic network adopts the following aggregation process: in, Represents the intelligent connected vehicle's own parameters θ l,i and another intelligent connected car parameter θ l,i′ The new network parameters obtained by aggregation, Represents the value output by the critic network at the state-action pair (s, a).

3. The multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system based on federated reinforcement learning according to claim 1 is characterized in that: The dynamic information includes the vehicle speed and position information, the road-end static information includes the road, expected path, and lane information, and the state matrix i RL ∈[0,1] W×H×C , where W×H=56 represents the matrix size, which actually covers a physical range of about 14m×14m, and C=4 represents the number of channels.

4. The multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system based on federated reinforcement learning according to claim 1 is characterized in that: The interaction of the reinforcement learning module generates the planning control output, which is generated by the reinforcement learning actor network according to the fusion state matrix. t1 with a t2 , the output is mapped to the range [-1, 1] using an activation function, where a t1 Indicates the steering wheel control amount, a t2 It is divided into two parts: [-1, 0] and [0, 1], which represent the brake and throttle control values ​​respectively.

5. The multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system based on federated reinforcement learning according to claim 1 is characterized in that: The intelligent subsystem includes a steering subsystem, a DYC subsystem, a suspension subsystem and a drive subsystem, wherein the steering subsystem and the DYC subsystem calculate the lateral control amount of the intelligent connected vehicle, the suspension subsystem calculates the vertical control amount of the intelligent connected vehicle, and the drive subsystem calculates the longitudinal control amount of the intelligent connected vehicle; the steering subsystem and the DYC subsystem calculate the yaw moment of the intelligent connected vehicle according to the steering wheel control amount output by the reinforcement learning module and the preview error between the actual path and the expected path; The driving subsystem obtains the desired target speed and calculates the four-wheel drive or braking torque of the intelligent connected vehicle according to the throttle control amount and the current speed of the intelligent connected vehicle.

6. The multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system based on federated reinforcement learning according to claim 1 is characterized in that: After the intelligent chassis coupling system outputs the quadratic planning control quantity, it also includes using the quadratic planning control quantity to generate experience samples and training the neural network parameters of the reinforcement learning module based on the experience samples; the group coordination decision-making module on the road side then obtains the vehicle-side neural network parameters of the intelligent connected vehicles in the vehicle-side group through V2I vehicle-road communication, and based on federated learning, screens and aggregates the local vehicle-side group neural network parameters and shared neural network parameters to achieve multi-agent group optimization.

7. The method of the multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system based on federated reinforcement learning as claimed in claim 1, characterized in that: These include: Step 1: Build a cloud platform, and provide three cloud control applications through the three cloud control levels of the cloud control platform on the cloud: connected car empowerment, traffic management and control, and traffic data empowerment; obtain complete traffic road network and real-time traffic information, logistics supervision information, high-precision map information, high-precision positioning and navigation information, and real-time weather information through the cloud support platform, and provide traffic dispatch and management information for the road side through the cloud control platform combined with the support platform; Step 2: Build a road-side platform, process the bird's-eye view image input into a semantic bird's-eye view through the road-side information processing module, and generate a Gaussian safety field. The semantic bird's-eye view is used as the input of the vehicle-side perception processing module, and the Gaussian safety field is used to generate the fusion reward function in the reinforcement learning module. Step 3: Build a vehicle-side platform. A single roadside unit contains multiple vehicle-side groups. Each vehicle-side group consists of N intelligent connected vehicles, and information sharing is achieved through V2V vehicle-to-vehicle communication. The intelligent connected vehicle includes a perception processing module, a trajectory prediction module, a reinforcement learning module, and an intelligent chassis coupling system. The perception processing module matches and segments the semantic bird's-eye view provided by the road side according to the vehicle-side positioning, and fuses the dynamic information of the vehicle and the static information of the road side into a state matrix as part of the input of the reinforcement learning module. The trajectory prediction module realizes multi-time-step trajectory prediction based on the semantic bird's-eye view provided by the road side, and the prediction result will be used as part of the input of the reinforcement learning module. The planning and control process is modeled as a Markov decision process through the reinforcement learning module, and the output of the perception processing module is used as the reinforcement learning input. The fusion reward function is generated in combination with the road-side Gaussian safety field, and the control output is interactively generated in the CARLA simulator. According to the reinforcement learning control output and the expected path, the secondary planning control quantity of a single intelligent connected vehicle is output through the intelligent chassis coupling system, and the corresponding chassis intelligent subsystem is controlled through the distributed controller. Step 4: After the intelligent chassis coupling system of a single intelligent connected vehicle on the vehicle side outputs the quadratic programming control quantity, it interacts in the CARLA simulator to generate experience samples, and trains the neural network parameters of the reinforcement learning module based on the experience samples; Step 5: The group coordination decision module on the road side obtains the neural network parameters of the vehicle-side reinforcement learning module of the intelligent connected vehicle in the vehicle-side group through V2I vehicle-road communication, and selects and aggregates the neural network parameters of the local vehicle-side group reinforcement learning module and the neural network parameters of the shared reinforcement learning module based on federated learning to achieve multi-agent group optimization; Among them: in step 1, the three cloud control levels correspond to edge cloud, regional cloud, and central cloud respectively. Based on the real-time traffic dynamics collected by the edge cloud, the cloud control platform provides driving safety of connected vehicles and improves traffic efficiency. Through the edge cloud combined with the information provided by the support platform, the regional cloud provides traffic situation awareness and evaluation, traffic planning, order management, and transportation management services. Through the regional cloud combined with the information provided by the support platform, the central cloud provides big data analysis services and business support based on traffic big data empowerment; In step 2, the Gaussian safety field is established by the following equation: φ=b x / c y =l v / w v Among them, S sta Represents the static safety field strength, C a represents the static safety field strength coefficient, x0 and y0 represent the coordinates of the static risk center O(x0,y0), b x and c y represents the appearance coefficient of the intelligent networked vehicle, φ is the length-to-width ratio of the intelligent networked vehicle, l v Indicates vehicle length, w v Indicates vehicle width; When the intelligent connected vehicle moves, the risk center O(x0,y0) of the Gaussian safety field will shift to a new risk center O as the vehicle moves. ′ (x ′ 0,y0 ′ ): Among them, k v represents the mobile adjustment factor, and The sign is related to the direction of movement, β represents the transfer vector of the connected car The angle between the coordinate axis and the Cartesian coordinate system forms a virtual vehicle with a length of l′ under the action of the risk center transfer in the dynamic safety field. v , width is w′ v , S dyn represents the dynamic safety field strength, and the new aspect ratio is expressed as φ′=b′ x / c′ y = l′ v / w′ v ; In step 2, the neural network screening process of the vehicle-side reinforcement learning module uses driving safety and driving aggressiveness based on the Gaussian safety field as network screening indicators, and uses a rule-based method as the screening basis to achieve the fusion drive of rules and data, and enhance the interpretability of the algorithm framework through interpretable rules; The actor network adopts the following aggregation process: in, Represents the intelligent connected vehicle's own parameters θ μ,i and another intelligent connected car parameter θ μ,i′ The new network parameters obtained by aggregation, r Safety represents the safety-related reward function of the intelligent connected car, which is calculated by the roadside driving safety field from the aspects of driving safety and driving aggressiveness: r safety ＝r Risk +r Agg Among them, R i,j (t) represents the driving risk caused by intelligent connected vehicle j to intelligent connected vehicle i, represents the field strength of intelligent connected car j to intelligent connected car i, k c represents the risk perception coefficient, represents the speed of smart connected vehicle j at time t, θ i,j (t) represents the driving angle between intelligent connected vehicles i and j at time t, r Risk represents the driving risk-related reward function, f Risk (ξ) represents the driving risk score, R thr represents the risk threshold, τ rc represents the duration of exceeding the risk threshold, R j ,i(t ′ ) represents the driving risk caused by intelligent connected vehicle i to intelligent connected vehicle j, represents the field strength of intelligent connected vehicle i to intelligent connected vehicle j, represents the intelligent connected vehicle i at time t ′ The speed, θ j,i (t ′ ) represents the intelligent networked car j and the intelligent networked car i at time t ′ The driving angle, r Agg represents the driving aggressiveness-related reward function, f Agg (ξ) represents the driving aggressiveness score; The critic network adopts the following aggregation process: in, Represents the intelligent connected vehicle's own parameters θ l,i and another intelligent connected car parameter θ l,i′ The new network parameters obtained by aggregation, The value output by the critic network at the state-action pair (s, a); In step 3, the intelligent subsystem includes a steering subsystem and a DYC subsystem for calculating the lateral control amount of the intelligent connected vehicle, a suspension subsystem for calculating the vertical control amount of the intelligent connected vehicle, and a driving subsystem for calculating the longitudinal control amount of the intelligent connected vehicle; the steering subsystem and the DYC subsystem calculate the yaw torque of the intelligent connected vehicle according to the steering wheel control amount output by the reinforcement learning module and the preview error between the actual path and the expected path; the driving subsystem obtains the expected target speed and calculates the four-wheel drive or braking torque of the intelligent connected vehicle according to the throttle control amount and the current speed of the intelligent connected vehicle; In step 4, the experience sample is composed of tuples (s t ,a t ,r t ,s t+1 ) description, where s t Corresponding state matrix i RL ∈[0,1] W×H×C , where W×H=56 represents the matrix size, which actually covers a physical range of about 14m×14m, C=4 represents the number of channels, a t The corresponding reinforcement learning actor network Actor network generates a control output according to the state quantity t1 with a t2 , the output is mapped to the range [-1, 1] using an activation function, where a t1 Indicates the steering wheel control amount, a t2 It is divided into two parts: [-1, 0] and [0, 1], which represent the brake and throttle control values, respectively. t Corresponding to the expected path-related reward function and the safety-related reward function, where s t+1 Represents the state matrix of the next frame; In step 5, the V2I vehicle-road communication only transmits the neural network parameters of the vehicle-side reinforcement learning module of the intelligent connected vehicle in the vehicle-side group, and the neural network for uploading and downloading parameters is the reinforcement learning actor-critic neural network Actor-Critic networks, where the actor network Actor network outputs the regulation and control strategy and realizes the strategy evaluation through the critic network Criticnetwork.

Images