A deep reinforcement learning based cAV speed guidance system and method

Abstract

The application discloses a CAV speed guiding system based on deep reinforcement learning, which comprises the following steps: a vehicle-mounted system detects vehicle driving data and vehicle state information data; a roadside monitoring system detects driving environment, vehicle information data on different roads and pedestrian information data, matches the pedestrian and vehicle information data with a lane-level road link to obtain a data packet; a vehicle-road cooperation cloud computing center processes and stores the received data, and sends the processed data to a mobile edge computing center; the mobile edge computing center comprehensively evaluates road accident risks, constructs a Markov decision process and a speed guiding model of a multi-target additional reward function, calculates an optimal guiding speed of a CAV by using the multi-target additional reward function, and sends the optimal guiding speed of the CAV to the vehicle-mounted system. The communication delay problem is solved; a multi-target environment reward mechanism is comprehensively integrated, and the guiding speed of the CAV is improved in terms of traffic safety, stability and passing efficiency.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a CAV speed guidance system and method based on deep reinforcement learning. Background Technology

[0002] Traffic congestion has become a major bottleneck restricting urban development, making the vigorous development of intelligent transportation and congestion mitigation technologies imperative. Speed ​​guidance for Connected Autonomous Vehicles (CAVs) on urban roads is not only an essential component of intelligent transportation information service systems but also a crucial support for intelligent traffic control and management systems. The average speed of CAVs on urban roads is a vital parameter for evaluating traffic operation status and assessing traffic accidents.

[0003] In terms of CAV speed guidance system frameworks, cloud architecture and edge computing are effective solutions. Cloud computing can process large amounts of data collected from various devices through distributed computing and highly scalable computing resources. However, when it comes to deploying a real system in a V2X communication environment, designing and building such a cloud-based speed guidance system framework is very challenging. In terms of algorithms, traditional deep learning-based methods do not consider the actual time latency between server infrastructure and CAVs or mobile edge computing, and the CAV speed cannot be precisely matched with the system's expected guidance speed. Furthermore, they do not fully integrate multi-objective environmental reward mechanisms and lack a comprehensive consideration of the impact of guidance speed on traffic safety, stability, and traffic efficiency. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a CAV speed guidance system and method based on deep reinforcement learning. It adopts a mobile edge and vehicle-road cooperative cloud framework to reduce the actual communication latency between server infrastructure and CAV. By constructing a CAV speed guidance model with a deep near-end policy gradient algorithm, the model calculates the optimal guidance speed for CAV through a Markov decision process with a multi-objective additional reward function, thereby comprehensively improving the impact of CAV guidance speed on traffic safety, stability, and traffic efficiency.

[0005] In a first aspect, the present invention provides a CAV speed guidance system based on deep reinforcement learning, comprising: an in-vehicle system, a roadside monitoring system, a vehicle-road cooperative cloud, and a mobile edge computing center, wherein,

[0006] The in-vehicle system is used to detect vehicle driving data and vehicle status information data, and transmit the detected data to the vehicle-road cooperative cloud computing center.

[0007] The roadside monitoring system detects the driving environment, vehicle information data and pedestrian information data on different roads, matches the pedestrian and vehicle information data with the lane-level road links defined in the high-definition map, obtains pedestrian parameter and road link data packets and vehicle parameter and road link data packets, and sends the data packets to the vehicle-road cooperative cloud computing center.

[0008] The vehicle-road cooperative cloud computing center is used to receive detection data transmitted by the vehicle system, data packets sent by the roadside monitoring system, and real-time traffic data obtained from the intelligent transportation system cloud management platform. The vehicle-road cooperative cloud computing center processes and stores the received data and sends the processed data to the mobile edge computing center.

[0009] The mobile edge computing center is used to receive data sent by the vehicle-road cooperative cloud computing center, obtain traffic condition information and CAV vehicle information, screen vehicles with accident risks in road segments based on vehicle speed dispersion, conduct a comprehensive risk assessment of road accident risks, construct a Markov decision process with a multi-objective additional reward function and a speed guidance model based on a deep proximal policy gradient algorithm, construct a multi-objective additional reward function to calculate the optimal guidance speed for CAV, and send the optimal guidance speed to the vehicle system.

[0010] Secondly, this invention provides a CAV speed guidance method based on deep reinforcement learning, applicable to mobile edge computing centers, comprising:

[0011] Receive vehicle driving data and vehicle status information data sent by the vehicle-road cooperative cloud computing center;

[0012] Obtain traffic condition information and CAV vehicle information, screen vehicles with accident risk in road segments based on vehicle speed dispersion, and conduct a comprehensive risk assessment of road accident risk.

[0013] A Markov decision process with a multi-objective additional reward function and a speed guidance model based on a deep proximal policy gradient algorithm are constructed. The multi-objective additional reward function is used to calculate the optimal guidance speed for CAV and the optimal guidance speed is sent to the vehicle system.

[0014] The beneficial effects of this invention are:

[0015] This invention provides a CAV speed guidance system based on deep reinforcement learning. The system is designed based on a mobile edge and vehicle-road cooperative cloud architecture. It includes a traffic accident risk calculation module based on mobile edge computing. The system can process large amounts of data collected from various devices through distributed computing and highly scalable computing resources, thereby reducing the actual communication latency between the server infrastructure and the CAV.

[0016] A CAV speed guidance model based on the DPPO algorithm is constructed. The model utilizes a Markov decision process with a multi-objective additional reward function to calculate the optimal guidance speed for the CAV. This achieves precise matching between the CAV speed and the system's desired guidance speed, comprehensively integrating multi-objective environmental reward mechanisms to enhance the impact of CAV guidance speed on traffic safety, stability, and traffic efficiency.

[0017] This invention provides a CAV speed guidance method based on deep reinforcement learning. It constructs a CAV speed guidance model using the DPPO algorithm. The model builds a Markov decision process with a multi-objective additional reward function, designing an additional reward function that considers multiple objectives to calculate the optimal guidance speed for the CAV. This achieves precise matching between the CAV speed and the system's desired guidance speed, comprehensively integrating multi-objective environmental reward mechanisms, and comprehensively improving the impact of CAV guidance speed on traffic safety, stability, and traffic efficiency. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0019] Figure 1 The diagram shows a structural block diagram of a CAV speed guidance system based on deep reinforcement learning provided in the first embodiment of the present invention;

[0020] Figure 2 The diagram shows a system architecture of a CAV speed guidance system based on deep reinforcement learning provided in the first embodiment of the present invention.

[0021] Figure 3 A flowchart of a CAB speed guidance method based on deep reinforcement learning, provided by another embodiment of the present invention, is shown. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0024] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0027] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0028] Please refer to Figure 1 , 2The first embodiment of this invention provides a CAV speed guidance system based on deep reinforcement learning, comprising: an in-vehicle system, a roadside monitoring system, a vehicle-road cooperative cloud, and a mobile edge computing center. The in-vehicle system detects vehicle driving data and vehicle status information data, and transmits the detected data to the vehicle-road cooperative cloud computing center. The roadside monitoring system detects the driving environment, vehicle information data on different roads, and pedestrian information data. It matches the pedestrian and vehicle information data with lane-level road links defined in a high-definition map to obtain pedestrian parameter and road link data packets and vehicle parameter and road link data packets, and sends these data packets to the vehicle-road cooperative cloud computing center. The vehicle-road cooperative cloud computing center receives the detection data transmitted by the in-vehicle system, the data packets sent by the roadside monitoring system, and real-time traffic data obtained from the intelligent transportation system cloud management platform. The vehicle-road cooperative cloud computing center processes and stores the received data and sends the processed data to the mobile edge computing center. The mobile edge computing center receives data sent by the vehicle-road cooperative cloud computing center, obtains traffic condition information and CAV vehicle information, screens vehicles with accident risks in road segments based on vehicle speed dispersion, conducts a comprehensive risk assessment of road accident risks, constructs a Markov decision process with a multi-objective additional reward function and a speed guidance model based on a deep proximal policy gradient algorithm, and constructs a multi-objective additional reward function to calculate the optimal guidance speed for CAVs.

[0029] Specifically, the vehicle-mounted system includes an onboard unit (OBU), a V2X node, a time transceiver center, and a CAV control center connected in sequence. The vehicle-mounted system detects vehicle driving data and vehicle status information data, and transmits the detected data to the vehicle-road cooperative cloud computing center.

[0030] Roadside monitoring systems detect the driving environment and acquire information data on vehicles and pedestrians on different road sections. The system uses sensors (including radar, lidar, and vision sensors) and edge processors to collect and detect pedestrian and vehicle information. The information output by the sensors and edge controllers includes pedestrian detection data and vehicle detection data. Pedestrian detection data is... in, The current pedestrian status, These represent the pedestrian's x-coordinate, y-coordinate, and speed, respectively, with f being the number of detected pedestrians and vehicle detection data. in, Current vehicle status These represent the vehicle's x-coordinate, y-coordinate, and speed, respectively, with m representing the number of detected vehicles. The detected pedestrian and vehicle information data on different road segments is matched with the lane-level road links defined in the high-definition map, outputting pedestrian parameters and road link data packets. Vehicle parameters and road link data packets Among them, l tLINK To detect the current road link location of pedestrians and vehicles, the pedestrian parameter and road link data packet consists of the detected pedestrian objects and lane-level road link information from a high-definition map (HDMAP). Similarly, the vehicle parameter and road link data packet consists of the detected vehicle objects (containing information about the detected vehicles, such as location and speed) and the corresponding lane-level road link information from a high-definition map (HDMAP). This data is transmitted to the vehicle-road cooperative cloud computing center via a wireless communication module.

[0031] The vehicle-road cooperative cloud computing center's multi-source data storage and processing module stores and processes three types of data. The first type is CAV driving and detection data, such as information detected by onboard sensors, sensor status information, and vehicle-related dynamic parameters, such as vehicle acceleration, speed, and position. The second type is data obtained from the intelligent transportation system cloud management platform, mainly including real-time traffic flow density and speed information, used to improve CAV driving efficiency to cope with various traffic conditions downstream. The third type is traffic condition information collected by the roadside monitoring system on different road segments, pedestrian parameters, and road link data packets. Vehicle parameters and road link data packets

[0032] The mobile edge computing center includes an accident risk calculation module and a speed guidance module. The accident risk calculation module is used to calculate the vehicle speed dispersion based on the speed of the detected vehicles, and to filter out vehicles with accident risks in the road segment based on the vehicle speed dispersion. It also performs a comprehensive risk assessment on the behavior of vehicles with accident risks. The speed guidance module is used to build a speed guidance model based on a deep near-end policy gradient algorithm. It uses simulation training in different scenarios to obtain the guidance speed for different road segments as the upper limit of the longitudinal speed of the corresponding road segment. It calculates the optimal guidance speed for CAV by constructing a Markov decision process with a multi-objective additional reward function.

[0033] Using information output from the multi-source data storage and processing module, driving risk is calculated based on the driving stability of each road segment. This calculated driving risk is then used as the basis for calculating the optimal guidance speed to minimize the driving risks associated with the limited driving capabilities of the CAV (Continuously Operated Vehicle). Vehicle speed dispersion is used as a basis for assessing traffic accident risk; a higher vehicle speed dispersion indicates a higher overtaking frequency on that road segment, and a higher risk of accidents. The accident risk calculation module includes a vehicle speed dispersion analysis unit, which calculates the vehicle speed dispersion based on the detected vehicle's speed and average speed. (The text then abruptly shifts to a different topic: vehicle speed dispersion.) Represented as:

[0034]

[0035] In the formula, To detect the vehicle's current speed, To detect the average vehicle speed, n represents the number of times the vehicle speed is detected.

[0036] The accident risk calculation module also includes a vehicle speed risk discrimination unit, which compares the vehicle speed dispersion with a preset threshold. If the dispersion is greater than or equal to a preset threshold, the vehicle is determined to have an accident risk; otherwise, it is determined not to have an accident risk. The module also includes an abnormal vehicle speed screening unit, which identifies vehicles with accident risks and counts the number of such vehicles. Finally, the module includes a risk assessment unit, which assigns different risk parameter values ​​to different abnormal vehicle behaviors and assesses the overall risk based on the risk parameter values, the number of risky vehicles, the length of the detected road segment, and real-time traffic flow density.

[0037] Specifically, when the speed dispersion of vehicles on the detected road segment exceeds the boundary value, the vehicle speed risk assessment unit determines that the vehicle poses an accident risk and filters out high-risk vehicles:

[0038]

[0039] Vehicles with varying degrees of risk have different impacts on traffic. Certain abnormal behaviors, such as sudden acceleration and deceleration (Level 1), are more likely to cause traffic accidents than sudden lane changes. When multiple abnormal behaviors are present in a single vehicle, that vehicle is considered high-risk. Therefore, different parameter values ​​are assigned to different abnormal behaviors, and these values ​​are used for subsequent road accident risk assessments. The comprehensive risk parameter is calculated as follows:

[0040]

[0041] In the formula, λ1 is the speed abnormality vehicle risk parameter, which refers to the abnormal value assigned to a vehicle with a large speed deviation, such as during emergency acceleration / deceleration; λ2 is the emergency lane change abnormality vehicle risk parameter; and λ3 is the mixed abnormality vehicle risk parameter, which refers to the abnormal value assigned to a vehicle that combines the first two types of abnormalities occurring simultaneously. 风险 The number of risky vehicles selected, ρ is the traffic flow density obtained from the intelligent transportation system cloud management platform, and l LINK To detect the length of the road segment.

[0042] The mobile edge computing center calculates the optimal guidance speed based on traffic condition information and CAV vehicle information collected from the roadside monitoring system and a CAV speed guidance model based on deep reinforcement learning, generating the optimal guidance speed for different road sections.

[0043] A speed guidance model based on the Deep Proximal Policy Gradient (DPPO) algorithm was constructed and trained using simulations in different scenarios to generate various training scenarios based on three types of collected data. A segmented speed guidance generation module, using data trained through deep reinforcement learning, generates the optimal guidance speed for different road segments as the upper limit of the longitudinal speed for each segment. This model aims to improve the overall performance of the CAV (Carrier Aided Vehicle), such as driving safety, passenger comfort, and energy efficiency, by providing the optimal guidance speed for each road segment. The CAV updates the maximum speed for each road segment based on the guidance speed and travels at that speed. When generating the optimal guidance speed for each road segment, driving safety is prioritized, while passenger comfort and energy efficiency are considered only under safe conditions. On the other hand, if any dangerous situation is detected solely based on information from onboard sensors, the CAV will reduce its speed below the guidance speed to ensure driving safety.

[0044] The velocity guidance module includes a Markov decision-making unit (MDI) for constructing a Markov decision process with a multi-objective additional reward function to determine the optimal guidance velocity generation conditions. Specifically, the construction of the multi-objective additional reward function Markov decision process considers the CAV continuous state and action transitions as Markov decision processes. To account for the optimal guidance velocity generation conditions, this study constructs a Markov decision process with a multi-objective additional reward function. A Markov process tuple with an additional reward function is represented as: (S t ,a t ,P t ,r t ,γ), where S t a represents all states in which the vehicle interacts with its driving environment. t For the actions performed by the vehicle, P t Let r be the transition probability function from the current state to the next state. t Let t be the environmental feedback reward at time t. Here, the reward function is a multi-objective additional reward function, and γ is a discount factor.

[0045] The velocity guidance module also includes a CAV velocity guidance model building unit. This unit is used to build a velocity guidance model based on the deep proximal policy gradient algorithm. The model is evaluated and updated using a state-value function and an operation-value function to determine the CAV guidance velocity. Specifically, a velocity guidance model based on the DPPO algorithm is constructed. DPPO is a model-free method that includes an Actor-Critic and a policy gradient, updating the policy by maximizing the reward value. DPPO simplifies the computation of the proximal gradient using first-order optimization and mainly includes two sub-modules: the Actor module generates the policy (or policy distribution) based on the current state, and the Critic module calculates the expected value of the current state. These are based on the state-value function V... sand operation value function Q(a t |s t The model is then evaluated and updated. The advantage function A is then... t =Q(a t |s t )-V s Update PPO using the following objective function:

[0046]

[0047] In the formula, L is the objective function, L value To estimate the loss function, To estimate the dominance function at time t, π θ Labeled as CAV driving strategy based on parameter θ, S(π) θ (s t ))=E[π θ (s t logπ θ (s t [)] represents the entropy loss of the CAV policy, L clip The approximate loss value obtained by using the clip function:

[0048]

[0049] In the formula, ε is a hyperparameter that limits the range of allowed updates, ensuring that the new and old strategies are more similar. If the probability ratio is given, then it can be derived from... Determine CAV boot speed

[0050] The speed guidance module also includes a multi-objective additional reward function construction unit, which is used to set the driving safety reward function, driving stability reward function and driving traffic efficiency reward function, and adjust the corresponding weight vector of the reward function to obtain a CAV guidance speed comprehensive reward function that matches the dynamic environment.

[0051] Specific methods for constructing multi-objective additional reward functions: The determination of CAV guidance speed should consider multiple objectives, including driving stability, safety, economy, and traffic efficiency in dynamic environments. Therefore, a reasonable reward function should be designed to obtain the comprehensive optimal result. The reward function can be regarded as a training signal that motivates or inhibits specific actions of the CAV. The reward function consists of three sub-functions:

[0052]

[0053] In the formula, r CAV (t) is the CAV action reward function. The reward function for guiding speed.CAV (t) is further divided into the driving safety reward function r safe (t), driving stability reward function r sta (t), traffic efficiency reward function r effi (t), w i i = 1, 2, 3 are the weights of driving safety, driving stability and traffic efficiency factors respectively, and w = [w1, w2, w3] is the weight vector. By adjusting the weight values, a comprehensive reward function for the guidance speed that matches the dynamic environment can be obtained. For example, when it is desired to generate guidance speed that focuses on improving the safety of CAV, w = [2, 1, 1] can be set.

[0054] During operation, a CAV will inevitably encounter static or dynamic obstacles, including surrounding vehicles and pedestrians. Therefore, the safety risk coefficient should be minimized while considering front and rear safety distances, and the driving safety reward function should be set as follows:

[0055]

[0056] In the formula, x0 and y0 are the vehicle's position coordinates.

[0057] Driving stability is another important factor to consider. Frequent acceleration and deceleration are detrimental to machine reliability and driver comfort. Therefore, the reward function for driving stability can be designed as follows:

[0058]

[0059] In the formula, For CAV longitudinal acceleration, This refers to the lateral acceleration of CAV.

[0060] Traffic efficiency is positively correlated with CAV speed to some extent. Therefore, the speed variation should be determined based on the guide speed and the maximum speed. Considering the above factors, the driving traffic efficiency reward function is designed as follows:

[0061]

[0062] In the formula, v max This is the maximum speed of the CAV, determined according to the road speed limit.

[0063] The vehicle system receives the optimal guidance speed from the mobile edge computing center through the onboard unit (OBU) and sends it to the CAV control center in the form of instructions through the event transceiver center. The CAV executes the commands and updates the reward strategy based on environmental feedback.

[0064] This invention provides a CAV speed guidance system based on deep reinforcement learning. The system is designed based on a mobile edge and vehicle-road cooperative cloud architecture. It includes a traffic accident risk calculation module based on mobile edge computing. The system can process large amounts of data collected from various devices through distributed computing and highly scalable computing resources, thereby reducing the actual communication latency between the server infrastructure and the CAV.

[0065] A CAV speed guidance model based on the DPPO algorithm is constructed. The model utilizes a Markov decision process with a multi-objective additional reward function to calculate the optimal guidance speed for the CAV. This achieves precise matching between the CAV speed and the system's desired guidance speed, comprehensively integrating multi-objective environmental reward mechanisms to enhance the impact of CAV guidance speed on traffic safety, stability, and traffic efficiency.

[0066] In the first embodiment described above, a CAV speed guidance system based on deep reinforcement learning is provided. Correspondingly, this application also provides a CAV speed guidance method based on deep reinforcement learning. Please refer to... Figure 3 This is a structural block diagram of a CAV speed guidance system based on deep reinforcement learning provided in the second embodiment of the present invention. Since the method embodiment is basically similar to the device embodiment, it is described simply; relevant details can be found in the description of the device embodiment. The method embodiment described below is merely illustrative.

[0067] Please refer to Figure 3 This invention provides a CAV speed guidance method based on deep reinforcement learning, applicable to mobile edge computing centers in the system described in the first embodiment above. The method includes:

[0068] Receive vehicle driving data and vehicle status information data sent by the vehicle-road cooperative cloud computing center;

[0069] Obtain traffic condition information and CAV vehicle information, screen vehicles with accident risk in road segments based on vehicle speed dispersion, and conduct a comprehensive risk assessment of road accident risk.

[0070] A Markov decision process with a multi-objective additional reward function and a speed guidance model based on a deep proximal policy gradient algorithm are constructed. The multi-objective additional reward function is used to calculate the optimal guidance speed for CAV and the optimal guidance speed is sent to the vehicle system.

[0071] Detecting vehicle speed dispersion Represented as:

[0072]

[0073] In the formula, To detect the vehicle's current speed, To detect the average vehicle speed, n represents the number of times the vehicle speed is detected.

[0074] When the speed dispersion of vehicles on a detected road segment exceeds the boundary value, the vehicle is determined to pose an accident risk, and high-risk vehicles are screened out.

[0075]

[0076] Vehicles with varying degrees of risk have different impacts on traffic. Certain abnormal behaviors, such as sudden acceleration and deceleration (Level 1), are more likely to cause traffic accidents than sudden lane changes. When multiple abnormal behaviors are present in a single vehicle, that vehicle is considered high-risk. Therefore, different parameter values ​​are assigned to different abnormal behaviors, and these values ​​are used for subsequent road accident risk assessments. The comprehensive risk parameter is calculated as follows:

[0077]

[0078] In the formula, λ1 is the speed abnormality vehicle risk parameter, which refers to the abnormal value assigned to a vehicle with a large speed deviation, such as during emergency acceleration / deceleration; λ2 is the emergency lane change abnormality vehicle risk parameter; and λ3 is the mixed abnormality vehicle risk parameter, which refers to the abnormal value assigned to a vehicle that combines the first two types of abnormalities occurring simultaneously. 风险 The number of risky vehicles selected, ρ is the traffic flow density obtained from the intelligent transportation system cloud management platform, and l LINK To detect the length of the road segment.

[0079] This paper constructs a Markov decision process with a multi-objective additional reward function. The continuous state and action transitions of the CAV can be viewed as a Markov decision process. To consider the optimal guidance velocity generation condition, this study constructs a Markov decision process with a multi-objective additional reward function. A Markov process with an additional reward function is represented by the following tuple: (S t ,a t ,P t ,r t ,γ), where S t a represents all states in which the vehicle interacts with its driving environment. t For the actions performed by the vehicle, P t Let r be the transition probability function from the current state to the next state. t Let t be the environmental feedback reward at time t. Here, the reward function is a multi-objective additional reward function, and γ is a discount factor.

[0080] Construct a velocity guidance model based on the DPPO algorithm. DPPO is a model-free method that includes an Actor-Critic module and a policy gradient, updating the policy by maximizing the reward value. DPPO simplifies the computation of proximal gradients by using first-order optimization and mainly consists of two sub-modules: the Actor module generates the policy (or policy distribution) based on the current state, and the Critic module calculates the expected value of the current state. The model is based on the state-value function V. s and operation value function Q(a t |s t The model is then evaluated and updated. The advantage function A is then... t =Q(a t |s t )-V s Update PPO using the following objective function:

[0081]

[0082] In the formula, L is the objective function, L value To estimate the loss function, To estimate the dominance function at time t, π θ Labeled as CAV driving strategy based on parameter θ, S(π) θ (s t ))=E[π θ (s t logπ θ (s t [)] represents the entropy loss of the CAV policy, L clip The approximate loss value obtained by using the clip function:

[0083]

[0084] In the formula, ε is a hyperparameter that limits the range of allowed updates, ensuring that the new and old strategies are more similar. If the probability ratio is given, then it can be derived from... Determine CAV boot speed

[0085] Construction of multi-objective additional reward functions.

[0086] The determination of CAV guidance speed should consider multiple objectives, including driving stability, safety, and traffic efficiency in dynamic environments. Therefore, a reasonable reward function should be designed to obtain the overall optimal result. The reward function can be viewed as a training signal that motivates or inhibits specific CAV actions. The reward function consists of three sub-functions:

[0087]

[0088] In the formula, rCAV (t) is the CAV action reward function. The reward function for guiding speed. CAV (t) is further divided into the driving safety reward function r safe (t), driving stability reward function r sta (t), traffic efficiency reward function r effi (t), w i i = 1, 2, 3 are the weights of the above driving safety, driving stability and traffic efficiency factors respectively, and w = [w1, w2, w3] is the weight vector. By adjusting the weight values, a comprehensive reward function for the guidance speed that matches the dynamic environment can be obtained. For example, when it is desired to generate guidance speed that focuses on improving the safety of CAV, w = [2, 1, 1] can be set.

[0089] During operation, a CAV will inevitably encounter static or dynamic obstacles, including surrounding vehicles and pedestrians. Therefore, the safety risk coefficient should be minimized while considering front and rear safety distances, and the driving safety reward function should be set as follows:

[0090]

[0091] In the formula, x0 and y0 are the vehicle's position coordinates.

[0092] Driving stability is another important factor to consider. Frequent acceleration and deceleration are detrimental to machine reliability and driver comfort. Therefore, the reward function for driving stability can be designed as follows:

[0093]

[0094] In the formula, These are the longitudinal and lateral accelerations of CAV, respectively.

[0095] Traffic efficiency is positively correlated with CAV speed to some extent. Therefore, the speed variation should be determined based on the guide speed and the maximum speed. Considering the above factors, the driving traffic efficiency reward function is designed as follows:

[0096]

[0097] In the formula, v max The maximum driving speed of the CAV is determined based on the road speed limit. The calculated optimal guidance speed is then sent to the onboard system.

[0098] This invention provides a CAV speed guidance method based on deep reinforcement learning. By constructing a CAV speed guidance model using the DPPO algorithm, the model constructs a Markov decision process with a multi-objective additional reward function, designing an additional reward function that considers multiple objectives to calculate the optimal guidance speed for the CAV. This achieves precise matching between the CAV speed and the system's desired guidance speed, comprehensively integrating multi-objective environmental reward mechanisms, and comprehensively improving the impact of CAV guidance speed on traffic safety, stability, and traffic efficiency.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A CAV speed guidance system based on deep reinforcement learning, characterized in that, include: In-vehicle systems, roadside monitoring systems, vehicle-road cooperative cloud, and mobile edge computing centers, among which, The in-vehicle system is used to detect vehicle driving data and vehicle status information data, and transmit the detected data to the vehicle-road cooperative cloud computing center. The roadside monitoring system detects the driving environment, vehicle information data and pedestrian information data on different roads, matches the pedestrian and vehicle information data with the lane-level road links defined in the high-definition map, obtains pedestrian parameter and road link data packets and vehicle parameter and road link data packets, and sends the data packets to the vehicle-road cooperative cloud computing center. The vehicle-road cooperative cloud computing center is used to receive detection data transmitted by the vehicle system, data packets sent by the roadside monitoring system, and real-time traffic data obtained from the intelligent transportation system cloud management platform. The vehicle-road cooperative cloud computing center processes and stores the received data and sends the processed data to the mobile edge computing center. The mobile edge computing center is used to receive data sent by the vehicle-road cooperative cloud computing center, obtain traffic condition information and CAV vehicle information, screen vehicles with accident risks in road segments based on vehicle speed dispersion, conduct a comprehensive risk assessment of road accident risks, construct a Markov decision process with a multi-objective additional reward function and a speed guidance model based on a deep proximal policy gradient algorithm, construct a multi-objective additional reward function to calculate the optimal guidance speed for CAV, and send the optimal guidance speed of CAV to the vehicle system; The mobile edge computing center includes an accident risk calculation module and a speed guidance module. The speed guidance module includes a multi-objective additional reward function construction unit. The multi-objective additional reward function construction unit is used to set a driving safety reward function, a driving stability reward function, and a driving traffic efficiency reward function, and adjust the corresponding weight vectors of the reward functions to obtain a CAV guidance speed comprehensive reward function that matches the dynamic environment. The reward function constructed using the multi-objective additional reward function consists of three sub-functions: In the formula, For CAV action reward function, The reward function for guiding speed; Further divided into driving safety reward functions Driving stability reward function Traffic efficiency reward function , The weights for driving safety, driving stability, and traffic efficiency are respectively. Given a weight vector, a comprehensive reward function for guiding speed that matches the dynamic environment is obtained by adjusting the weight values. During driving, CAVs inevitably encounter static or dynamic obstacles. Considering the front and rear safe distances, the driving safety reward function is set as follows: In the formula, , The coordinates of the vehicle's position; Driving stability is another important factor. Frequent acceleration and deceleration are contrary to machine reliability and driver comfort. The reward function for driving stability is designed as follows: In the formula, For CAV longitudinal acceleration, This refers to the lateral acceleration of the CAV. The driving efficiency reward function is designed as follows: In the formula, This is the maximum speed of the CAV, determined according to the road speed limit.

2. The system as described in claim 1, characterized in that, The accident risk calculation module is used to calculate the vehicle speed dispersion based on the speed of the detected vehicles, and to screen out vehicles with accident risks in the road segment based on the vehicle speed dispersion. It also performs a comprehensive risk assessment on the behavior of vehicles with accident risks. The speed guidance module is used to construct a speed guidance model based on a deep proximal policy gradient algorithm. It uses different scenarios for simulation training to obtain the guidance speed for different road segments as the upper limit of the longitudinal speed of the corresponding road segment. It constructs a Markov decision process with a multi-objective additional reward function to calculate the optimal guidance speed for CAV.

3. The system as described in claim 2, characterized in that, The accident risk calculation module includes a vehicle speed dispersion analysis unit, which is used to calculate the vehicle speed dispersion based on the speed and average speed of the detected vehicle.

4. The system as described in claim 3, characterized in that, The accident risk calculation module also includes a vehicle speed risk discrimination unit. The vehicle speed risk discrimination unit is used to compare the vehicle speed dispersion with a preset threshold. If it is greater than or equal to the threshold, the vehicle is determined to have an accident risk. If it is less than the threshold, the vehicle is determined not to have an accident risk.

5. The system as described in claim 4, characterized in that, The accident risk calculation module also includes an abnormal speed screening unit, which is used to screen out vehicles with accident risks and count the number of risky vehicles.

6. The system as described in claim 5, characterized in that, The accident risk calculation module also includes a risk assessment unit, which is used to assign different risk parameter values ​​to different abnormal behaviors of vehicles and assess the comprehensive risk based on the risk parameter values, the number of risky vehicles, the length of the detection section, and the real-time traffic flow density.

7. The system as described in claim 2, characterized in that, The speed guidance module includes a Markov decision-making unit, which is used to construct a Markov decision process with a multi-objective additional reward function to determine the optimal guidance speed generation conditions.

8. The system as described in claim 7, characterized in that, The velocity guidance module also includes a CAV velocity guidance model construction unit, which is used to construct a velocity guidance model based on the deep proximal policy gradient algorithm, evaluate and update the model through state value function and operation value function, and determine the CAV guidance velocity.

9. A CAV speed guidance method based on deep reinforcement learning, characterized in that, The system applied to the deep reinforcement learning-based CAV speed guidance system of claim 1, and applicable to mobile edge computing centers, includes: Receive vehicle driving data and vehicle status information data sent by the vehicle-road cooperative cloud computing center; Obtain traffic condition information and CAV vehicle information, screen vehicles with accident risk in road segments based on vehicle speed dispersion, and conduct a comprehensive risk assessment of road accident risk. A Markov decision process with a multi-objective additional reward function and a speed guidance model based on a deep proximal policy gradient algorithm are constructed. The multi-objective additional reward function is used to calculate the optimal guidance speed for CAV and the optimal guidance speed is sent to the vehicle system.

Images