Network connection vehicle signal lamp control intersection passing method and system based on safety reinforcement learning

Abstract

The invention discloses a network connection vehicle signal lamp control intersection passing method and system based on safety reinforcement learning, and relates to the technical field of intelligent traffic intelligent network connection vehicle control. The method comprises the following steps: receiving traffic environment information, wherein the traffic environment information comprises signal lamp state information and state information of a front vehicle and a controlled vehicle; and a reinforcement learning intelligent controller is constructed, the traffic environment information is input into the reinforcement learning intelligent controller, and the driving acceleration control quantity or the braking deceleration control quantity of the controlled vehicle at the next moment is decided and planned in real time by adopting a self-adaptive soft action evaluation algorithm. The problem that a traditional safety control strategy and a traditional learning type control strategy are poor in adaptability under the high-dimensional complex condition is solved, and the safety and the real-time performance of decision control over vehicle passing at the intersection under the condition that forward traffic flow interference is included are achieved through organic combination of the safety control strategy and the learning type control strategy.

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicle control technology in smart transportation, specifically to a method and system for traffic signal control at intersections based on safety reinforcement learning for connected vehicles. Background Technology

[0002] Traffic flow at traffic lights in urban areas is one of the most challenging driving scenarios for autonomous driving. The complex traffic environment makes it difficult for traditional control algorithms to make effective decisions, and the constant stopping and starting of vehicles results in a large amount of energy consumption.

[0003] Reinforcement learning algorithms do not learn from manually labeled data. Instead, the controller generates data during interaction with the environment, forms values ​​based on environmental feedback signals, and updates the strategy for the next interaction by comparing these values. After multiple training iterations, an optimal strategy is gradually found that maximizes the value of a series of actions. An agent can autonomously explore the environment according to a pre-defined reward function, but this involves a large amount of ineffective exploration, such as violating traffic rules, making it difficult to guarantee the safety of training with real-world environments in practical applications. Therefore, general reinforcement learning carries exploration risks during policy planning. Ensuring the agent's safe exploration and improving the efficiency of experience-based learning sampling is an indispensable topic in both the theory and application of reinforcement learning.

[0004] On the other hand, safety domain control methods based on Control Barrier Function and Control Lyapunov Function have shown excellent control performance in certain autonomous driving scenarios. These methods constrain the controlled object to not exceed safety boundaries by defining a Barrier Function and utilize Lyapunov functions to stabilize the control signal at a target value, allowing the controlled object to asymptotically stabilize to a specified value. Finally, the optimal control is obtained by transforming the original control problem into solving a set of quadratic problems. However, because this method requires specific conditions to be met, it has poor adaptability to decision-making problems in high-dimensional, dynamic, and complex environments. Therefore, this invention proposes a safety reinforcement learning-based method and system for connected vehicle traffic signal control at intersections. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for connected vehicles to pass through signal-controlled intersections based on safety reinforcement learning. This method overcomes the problem of poor adaptability of traditional safety control strategies and learning-based control strategies to high-dimensional and complex conditions. By organically combining the two, it enables connected vehicles to optimize their economic speed in a safe training environment.

[0006] According to a first aspect of the present invention, in order to achieve the above-mentioned objective, the present invention provides the following technical solution: a method for connected vehicles to pass through signal-controlled intersections based on security reinforcement learning, comprising the following steps: Receive traffic environment information, which includes traffic light status information, and status information of the vehicle in front and the controlled vehicle; A reinforcement learning intelligent controller is constructed. Traffic environment information is input into the reinforcement learning intelligent controller, and an adaptive soft motion evaluation algorithm is used to make real-time decisions and plan the driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle in the next moment. A safety monitoring module is constructed, which includes an environmental state prediction network and a quadratic programming solver. The environmental state prediction network predicts the next state information of the controlled vehicle and the traffic environment based on the planned driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle. When the predicted environmental state does not meet the traffic light green wave passage safety constraint or does not meet the safe distance constraint, the quadratic programming solver is used to solve for the optimal safety control quantity that meets the safety constraint. The optimal safety control quantity obtained by solving is input into the controlled vehicle to realize one iteration, which is used to train the reinforcement learning intelligent controller until the reward function of the reinforcement learning intelligent controller converges and the training is completed. The trained reinforcement learning intelligent controller is used to realize the safety control of traffic at the intersection.

[0007] Furthermore, the traffic environment information includes: Controlled vehicle status information: Vehicle speed, driving acceleration, and braking deceleration information obtained based on the controlled vehicle's own CAN communication protocol; Forward vehicle status information: Speed ​​and position information of the vehicle ahead obtained through the onboard sensors of the controlled vehicle; Traffic light status information: The current status of the traffic lights and the remaining passage time are obtained based on wireless shortwave communication or LTE / 5G technology, and the distance information of the controlled vehicles to the intersection is obtained by V2I technology-assisted communication positioning.

[0008] Furthermore, a reinforcement learning intelligent controller is constructed. Traffic environment information is input into the reinforcement learning intelligent controller, and an adaptive soft motion evaluation algorithm is used to make real-time decisions and plan the driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle in the next moment, as follows: (31) Receive real-time signal timing information of the intersection where the controlled vehicle is located, obtained through wireless shortwave communication or LTE / 5G communication technology. and the speed of the vehicle ahead detected by the millimeter-wave radar mounted on the controlled vehicle. Location information And the driving acceleration obtained through the controlled vehicle's own speed sensor. Braking deceleration ,speed Displacement information ; (32) The reinforcement learning intelligent controller includes a parameterized network, which consists of a network with a parameterized network and a parameterized network. Parameterized Actor network, 3 Critic networks, respectively Parameterized target value network, with Parameterized Q1 network and Parameterized Q2 network; The reinforcement learning intelligent controller inputs the state information received in step (31) into the Actor network, and the neural network outputs the mean and variance of a Gaussian distribution. Vehicle acceleration is obtained by sampling according to probability distribution. The system obtains the environment state at the next moment and combines the environment state at the next moment with the obtained reward value to form a dataset. The dataset is then stored in the experience cache pool of the reinforcement learning intelligent controller; (33) Sample data from the experience cache pool Perform Actor and Critic network updates. The update process is as follows: (33.1) Q network: The environment state is obtained based on the actor network actions. The target value network outputs the real-time value of this state. Through the objective function: Calculate the gradient: To update the parameters of the two Q networks; In the formula, Indicated by The Q-value of the parameterized network under its state and action at time t. Indicated by The V value of the parameterized network at time t. This is the discount factor. To parameterize strategy, This indicates that the state and action conform to the expectation of D. Let be the coefficient of entropy, whose value at different times can be adaptively updated using the following formula: (33.2) Actor network: The objective equation is obtained by directly minimizing the KL divergence to obtain the relationship between policy entropy and Q value: Based on the gradient: Update policy network parameters, where The value is chosen to be the smallest estimated value between the two networks Q1 and Q2 to avoid overestimation; (33.3) Target value network: Input the current state Output value and through To update the Value network parameters ,in Furthermore, a safety monitoring module is constructed, comprising an environmental state prediction network and a quadratic programming solver. The environmental state prediction network predicts the next state information of the controlled vehicle and the traffic environment based on the planned driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle. When the predicted environmental state does not meet the traffic light green wave passage safety constraint or the safe distance constraint, the quadratic programming solver is used to obtain the optimal safety control quantity that meets the safety constraints, as detailed below: (41) In step (31), the vehicle-mounted sensors collect multiple sets of driving and environmental data as training data. The environmental state prediction network fits the environmental state through a set of pre-trained neural networks containing LSTM layers. The input data of the environmental state prediction network is the set of vehicle and environmental states at a certain moment. The output data is This is used to train a neural network to predict the environmental state in the next moment; (42) The trained environment state prediction network is applied to reinforcement learning training. Whenever the reinforcement learning intelligent controller is in a certain environment state at a certain moment... Output a control decision Then, all the vehicle and environmental state data at this moment are input into the environmental state prediction network to predict the environmental state at the next moment. ; (43) When the predicted environmental state does not meet the traffic light constraint or the safety distance constraint to be maintained with the vehicle in front, a quadratic programming solver is used to minimize the constraint and the reinforcement learning intelligent controller. The objective is the square of the difference: In the formula To obtain the permissible control set, we need to find the minimum control that satisfies both the traffic light passage constraint and the safe distance constraint. .

[0009] Furthermore, the optimal safety control quantity obtained from the solution is input into the controlled vehicle to perform one iteration, which is used to train the reinforcement learning intelligent controller until the reward function of the reinforcement learning intelligent controller converges and the training is completed. The trained reinforcement learning intelligent controller is then used to implement traffic safety control at the intersection, as follows: (51) The constraints on controlled vehicles during their passage through traffic light-controlled intersections mainly take the following forms: in Indicates the vehicle's speed. and These represent the minimum and maximum speeds a vehicle can travel at during a given traffic light cycle, respectively. This represents the relative distance between the vehicle being controlled and the vehicle in front. This indicates the minimum safe distance that must be maintained when a vehicle is stationary; (52) The reward function includes the set of safety soft constraints defined by the Control Barrier Function and Control Lyapunov Function methods, namely: In the formula This is the conversion factor for the rotational mass of a vehicle. The target speed is between the maximum and minimum passing speeds. This refers to the sum of the vehicle's resistance to movement; It is a constant; To optimize the objective, The smaller the value, the more stable the tracking of the target vehicle speed; The reward function is as follows: in The energy consumption of a vehicle while driving is calculated using the vehicle's longitudinal dynamics and kinematics models. This is the difference between the actual measured acceleration of the controlled vehicle and the acceleration output by the reinforcement learning intelligent controller.

[0010] According to a second aspect of the present invention, the present invention provides a traffic signal control intersection system for connected vehicles based on safety reinforcement learning, used to implement the traffic signal control intersection method for connected vehicles based on safety reinforcement learning described in the first aspect, comprising: The traffic environment information receiving module is used to receive traffic environment information, which includes traffic light status information, and the status information of the vehicle in front and the controlled vehicle. The first construction module is used to build a reinforcement learning intelligent controller. Traffic environment information is input into the reinforcement learning intelligent controller, and an adaptive soft motion evaluation algorithm is used to make real-time decisions and plan the driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle in the next moment. The second construction module is used to construct the safety supervision module. The safety supervision module includes an environmental state prediction network and a quadratic programming solver. The environmental state prediction network predicts the next state information of the controlled vehicle and the traffic environment based on the planned driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle. When the predicted environmental state does not meet the traffic light green wave passage safety constraint or does not meet the safe distance constraint, the quadratic programming solver is used to solve for the optimal safety control quantity that meets the safety constraint. The iterative training module is used to input the solved optimal safety control quantity into the controlled vehicle to achieve one iteration, and to train the reinforcement learning intelligent controller until the reward function of the reinforcement learning intelligent controller converges and the training is completed. The trained reinforcement learning intelligent controller is used to realize the intersection traffic safety control.

[0011] The present invention has at least the following beneficial effects: 1) This invention overcomes the problem of poor adaptability of traditional security control strategies and learning-based control strategies to high-dimensional complex conditions. By organically combining the two, it has good security performance.

[0012] 2) This invention employs a safe reinforcement learning algorithm for intersection passage decisions of intelligent connected vehicles, achieving both safety and real-time performance in decision-making and control of vehicle passage at intersections under conditions including forward traffic flow interference.

[0013] 3) While considering safety, this invention enables controlled vehicles to pass through intersections without waiting, and takes into account the impact of energy consumption factors, thus achieving the goal of efficient, safe and energy-saving travel.

[0014] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0015] Figure 1 This is a system flowchart of the passage method described in this invention.

[0016] Figure 2This is a schematic diagram of the reinforcement learning safety and stability algorithm based on Lyapunov functions in this invention.

[0017] Figure 3 This is a simulation diagram comparing the training process performance of the reinforcement learning safe and stable algorithm and the algorithm without stable constraints described in this invention. Figure 4 This is a simulation diagram of the passage method described in this invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] Example 1: refer to Figure 1 The traffic environment information acquisition module mainly includes acquiring the controlled vehicle's own state information via CAN, namely the vehicle's driving acceleration and braking deceleration. Other environmental information mainly includes traffic light timing information acquired via wireless shortwave communication, namely the current traffic light status and remaining passage time, as well as the distance information of the controlled vehicle to the intersection acquired through V2I device interaction, and the preceding vehicle traffic flow information acquired through radar sensors, namely the preceding vehicle's position and speed. The reinforcement learning algorithm module mainly includes using the Soft Actor Critic algorithm to make real-time decisions on control quantities and training the optimal strategy with the goal of optimal energy efficiency. The safety supervision module mainly includes predicting the next moment's information based on the current information of the controlled vehicle and the environment to calculate whether the passage constraints and safe distance constraints will be violated. If a violation is found, a quadratic programming solver is used to find the safe optimal control to replace the decision quantity of the agent in the reinforcement learning module, thereby training a safe and efficient passage strategy.

[0020] Please see Figures 1-4 This invention provides a technical solution: a method for connected vehicles to pass through signal-controlled intersections based on security reinforcement learning, comprising the following steps: S1. Receive traffic environment information, including traffic light status information, and status information of the vehicle in front and the controlled vehicle; Obtain traffic environment information, including: Controlled vehicle status information: Vehicle speed, driving acceleration, and braking deceleration information obtained based on the controlled vehicle's own CAN communication protocol; Forward vehicle status information: Speed ​​and position information of the vehicle ahead obtained through the onboard sensors of the controlled vehicle; Traffic light status information: Current status and remaining passage time of traffic lights obtained based on wireless shortwave communication or LTE / 5G technology; distance information of controlled vehicles to the intersection obtained through V2I technology-assisted communication and positioning. S2. Construct a reinforcement learning intelligent controller. Input traffic environment information into the reinforcement learning intelligent controller, and use an adaptive soft motion evaluation algorithm to make real-time decisions and plan the driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle in the next moment, as follows: (S21) Receive the traffic light timing information at the intersection where the controlled vehicle is located in real time, obtained through wireless shortwave communication or LTE / 5G communication technology. and the speed of the vehicle ahead detected by the millimeter-wave radar mounted on the controlled vehicle. Location information And the driving acceleration obtained through the controlled vehicle's own speed sensor. Braking deceleration ,speed Displacement information ; (S22) The reinforcement learning intelligent controller includes a parameterized network, which consists of a [network name missing] and a [network name missing]. Parameterized Actor network, 3 Critic networks, respectively Parameterized target value network, with Parameterized Q1 network and Parameterized Q2 network; The reinforcement learning intelligent controller inputs the state information received in step (31) into the Actor network, and the neural network outputs the mean and variance of a Gaussian distribution. Vehicle acceleration is obtained by sampling according to probability distribution. The system obtains the environment state at the next moment and combines the environment state at the next moment with the obtained reward value to form a dataset. The dataset is then stored in the experience cache pool of the reinforcement learning intelligent controller; (S23) Sample data from the experience cache pool Perform Actor and Critic network updates. The update process is as follows: (S23.1) Q network: Obtain the environment state based on the actor network actions. The target value network outputs the real-time value of this state. Through the objective function: Calculate the gradient: To update the parameters of the two Q networks; In the formula, Indicates The Q-value of the parameterized network under its state and action at time t. Indicates The V value of the parameterized network at time t. This is the discount factor. To parameterize strategy, This indicates that the state and action conform to the expectation of D. Let be the coefficient of entropy, whose value at different times can be adaptively updated using the following formula: (S23.2) Actor network: The objective equation is obtained by directly minimizing the KL divergence to obtain the relationship between policy entropy and Q-value. Based on the gradient: Update policy network parameters, where The value is chosen to be the smallest estimated value between the two networks Q1 and Q2 to avoid overestimation; (S23.3) Target value network: Input current state Output value and through To update the Value network parameters ,in S3. Construct a safety monitoring module, which includes an environmental state prediction network and a quadratic programming solver. The environmental state prediction network predicts the next state information of the controlled vehicle and the traffic environment based on the planned driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle. When the predicted environmental state does not meet the traffic light green wave passage safety constraint or the safe distance constraint, the quadratic programming solver is used to obtain the optimal safety control quantity that meets the safety constraints. Figure 2 As shown, the details are as follows: (S31) In step (S21), the vehicle-mounted sensors collect multiple sets of driving and environmental data as training data. The environmental state prediction network fits the environmental state through a set of pre-trained neural networks containing LSTM layers. The input data of the environmental state prediction network is the set of vehicle and environmental states at a certain moment. The output data is This is used to train a neural network to predict the environmental state in the next moment; (S32) The trained environment state prediction network is applied to reinforcement learning training. Whenever the reinforcement learning intelligent controller is in a certain environment state at a certain moment... Output a control decision Then, all the vehicle and environmental state data at this moment are input into the environmental state prediction network to predict the environmental state at the next moment. ; (S33) When the predicted environmental state does not meet the traffic light constraint or the safe distance constraint to be maintained with the vehicle in front, a quadratic programming solver is used to minimize the constraint with the reinforcement learning intelligent controller. The objective is the square of the difference: In the formula To obtain the permissible control set, we need to find the minimum control that satisfies both the traffic light passage constraint and the safe distance constraint. ; S4. Input the solved optimal safety control quantity into the controlled vehicle to achieve one iteration, which is used to train the reinforcement learning intelligent controller until the reward function of the reinforcement learning intelligent controller converges and the training is completed. The trained reinforcement learning intelligent controller is used to realize the intersection traffic safety control. In reinforcement learning training, the control quantity solved by the quadratic programming solver is input to the controlled vehicle to maintain safe training of the vehicle. At this time, the reinforcement learning agent performs trial and error learning within the safety constraints of the vehicle, avoiding ineffective exploration and accelerating training convergence. The constraints on controlled vehicles passing through traffic light-controlled intersections mainly take the following forms: in and These represent the minimum and maximum speeds a vehicle can travel at during a given traffic light cycle, respectively. This represents the relative distance between the vehicle being controlled and the vehicle in front. This indicates the minimum safe distance that must be maintained when a vehicle is stationary; The reinforcement learning reward function includes the set of safe soft constraints defined by the Control Barrier Function and Control Lyapunov Function methods, namely: in This is the conversion factor for the rotational mass of a vehicle. The target speed is between the maximum and minimum passing speeds. This refers to the sum of the vehicle's resistance to movement; Let it be a certain constant; To optimize the target, the smaller this value, the more stable the tracking of the target vehicle speed; The reward function is as follows: in The energy consumption of a vehicle while driving is calculated using the vehicle's longitudinal dynamics and kinematics models. This is the difference between the actual measured acceleration of the controlled vehicle and the acceleration output by the intelligent agent. This iterative training continues until the reward value output by the reinforcement learning converges, at which point the training is complete.

[0021] like Figure 3 As shown, the algorithm architecture comprises five main technical parts. First, the environmental constraints include road speed limits, safety constraints, and green wave traffic constraints. The dynamics and energy consumption model part includes the vehicle longitudinal dynamics model, motor efficiency model, and power demand model. These two parts are transmitted to the policy constraint guidance part, which includes fitting the constraint state equation, real-time supervision of state constraints, and QP solution policy replacement. Then, combined with the reward function part, the established control function, control Lyapunov function, battery energy consumption, and other reward models can transform the original optimal control problem into a reinforcement learning problem.

[0022] To achieve practical application results, this invention can employ a virtual simulation model for training. By modeling the vehicle and environment using simulation software, the trained strategy can be deployed to a real vehicle using the implementation method, thereby achieving safe passage in real traffic environments. Figure 4 The diagram shown is a simulation result of the traffic decision algorithm in this embodiment. According to the proposed method, vehicles can efficiently pass through all traffic intersections while maintaining a safe traffic distance, thus achieving green wave traffic.

[0023] In summary, this invention first acquires road traffic timing information based on vehicle-to-everything (V2X) communication protocols and uses onboard sensors to acquire vehicle operating status and the movement status of traffic flow ahead. Secondly, it trains a neural network to fit the environmental transition states based on real-world traffic and driving data. Then, based on the acceleration control quantity output by the reinforcement learning decision training module during each training process, a safety supervision module is used to detect whether the next state of the controlled object is safe. If it is unsafe, a quadratic programming solver is used to solve for the optimal safe control quantity with the objective of minimizing the difference between the control quantity and the decision control quantity of the reinforcement learning controller, thereby satisfying the safety constraints of driving in road traffic. The controlled vehicle is then controlled by replacing the control quantity, thus safely passing through the intersection. Therefore, this invention applies a safety reinforcement learning algorithm to actual vehicle intersection traffic control, addressing the low sampling efficiency problem caused by the trial-and-error nature of the reinforcement learning training process, and exhibits good safety and efficiency.

[0024] Example 2: This embodiment provides a traffic signal control intersection passage system for connected vehicles based on safe reinforcement learning, used to implement the traffic signal control intersection passage method for connected vehicles based on safe reinforcement learning described in Embodiment 1, including: The traffic environment information receiving module is used to receive traffic environment information, which includes traffic light status information, and the status information of the vehicle in front and the controlled vehicle. The first construction module is used to build a reinforcement learning intelligent controller. Traffic environment information is input into the reinforcement learning intelligent controller, and an adaptive soft motion evaluation algorithm is used to make real-time decisions and plan the driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle in the next moment. The second construction module is used to construct the safety supervision module. The safety supervision module includes an environmental state prediction network and a quadratic programming solver. The environmental state prediction network predicts the next state information of the controlled vehicle and the traffic environment based on the planned driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle. When the predicted environmental state does not meet the traffic light green wave passage safety constraint or does not meet the safe distance constraint, the quadratic programming solver is used to solve for the optimal safety control quantity that meets the safety constraint. The iterative training module is used to input the solved optimal safety control quantity into the controlled vehicle to achieve one iteration, and to train the reinforcement learning intelligent controller until the reward function of the reinforcement learning intelligent controller converges and the training is completed. The trained reinforcement learning intelligent controller is used to realize the intersection traffic safety control.

[0025] Example 3: The present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it adopts the above-mentioned method for traffic signal control of connected vehicles based on security reinforcement learning.

[0026] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server, and the terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0027] Furthermore, the processor can be a central processing unit (CPU). Of course, depending on the actual use, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be used. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it in this regard.

[0028] Example 4: The present invention provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the above-described method for traffic signal control at intersections for connected vehicles based on security reinforcement learning.

[0029] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0030] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0031] For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on," "mounted on," "fixed to," or "set on" another element, it may be directly on the other element or there may be an intermediate element present. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element present. The terms "vertical," "horizontal," "upper," "lower," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible embodiments.

[0032] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0033] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

Claims

1. A method for connected vehicles to pass through signal-controlled intersections based on safety reinforcement learning, characterized in that, Includes the following steps: Receive traffic environment information, which includes traffic light status information, and status information of the vehicle in front and the controlled vehicle; A reinforcement learning intelligent controller is constructed. Traffic environment information is input into the reinforcement learning intelligent controller, and an adaptive soft motion evaluation algorithm is used to make real-time decisions and plan the driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle in the next moment. A safety monitoring module is constructed, which includes an environmental state prediction network and a quadratic programming solver. The environmental state prediction network predicts the next state information of the controlled vehicle and the traffic environment based on the planned driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle. When the predicted environmental state does not meet the traffic light green wave passage safety constraint or does not meet the safe distance constraint, the quadratic programming solver is used to solve for the optimal safety control quantity that meets the safety constraint. The optimal safety control quantity obtained by solving is input into the controlled vehicle to realize one iteration, which is used to train the reinforcement learning intelligent controller until the reward function of the reinforcement learning intelligent controller converges and the training is completed. The trained reinforcement learning intelligent controller is used to realize the safety control of traffic at the intersection.

2. The method for traffic flow at signal-controlled intersections for connected vehicles based on security reinforcement learning according to claim 1, characterized in that: The traffic environment information includes: Controlled vehicle status information: Vehicle speed, driving acceleration, and braking deceleration information obtained based on the controlled vehicle's own CAN communication protocol; Forward vehicle status information: Speed ​​and position information of the vehicle ahead obtained through the onboard sensors of the controlled vehicle; Traffic light status information: The current status of the traffic lights and the remaining passage time are obtained based on wireless shortwave communication or LTE / 5G technology, and the distance information of the controlled vehicles to the intersection is obtained by V2I technology-assisted communication positioning.

3. The method for traffic flow at signal-controlled intersections for connected vehicles based on security reinforcement learning according to claim 2, characterized in that: A reinforcement learning intelligent controller is constructed. Traffic environment information is input into the controller, and an adaptive soft motion evaluation algorithm is used to make real-time decisions and plan the driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle in the next moment, as follows: (31) Receive real-time signal timing information of the intersection where the controlled vehicle is located, obtained through wireless shortwave communication or LTE / 5G communication technology. and the speed of the vehicle ahead detected by the millimeter-wave radar mounted on the controlled vehicle. Location information And the driving acceleration obtained through the controlled vehicle's own speed sensor. Braking deceleration ,speed Displacement information ; (32) The reinforcement learning intelligent controller includes a parameterized network, which consists of a network with a parameterized network and a parameterized network. Parameterized Actor network, 3 Critic networks, respectively Parameterized target value network, with Parameterized Q1 network and Parameterized Q2 network; The reinforcement learning intelligent controller inputs the state information received in step (31) into the Actor network, and the neural network outputs the mean and variance of a Gaussian distribution. Vehicle acceleration is obtained by sampling according to probability distribution. The system obtains the environment state at the next moment and combines the environment state at the next moment with the obtained reward value to form a dataset. The dataset is then stored in the experience cache pool of the reinforcement learning intelligent controller. (33) Sample data from the experience cache pool Perform Actor and Critic network updates. The update process is as follows: (33.1) Q network: The environment state is obtained based on the actor network actions. The target value network outputs the real-time value of this state. Through the objective function: Calculate the gradient: To update the parameters of the two Q networks; In the formula, Indicated by The Q-value of the parameterized network under its state and action at time t. Indicated by The V value of the parameterized network at time t. This is the discount factor. To parameterize strategy, This indicates that the state and action conform to the expectation of D. Let be the coefficient of entropy, whose value at different times can be adaptively updated using the following formula: in Represents the policy entropy. (33.2) Actor network: The objective equation is obtained by directly minimizing the KL divergence to obtain the relationship between policy entropy and Q value: Based on the gradient: Update policy network parameters, where The value is chosen to be the smallest estimated value between the two networks Q1 and Q2 to avoid overestimation; (33.3) Target value network: Input the current state Output value and through To update the Value network parameters ,in 。 4. The method for traffic flow at signal-controlled intersections for connected vehicles based on security reinforcement learning according to claim 1, characterized in that, A safety monitoring module is constructed, comprising an environmental state prediction network and a quadratic programming solver. The environmental state prediction network predicts the next state information of the controlled vehicle and the traffic environment based on the planned driving acceleration control or braking deceleration control quantity of the controlled vehicle. When the predicted environmental state does not meet the traffic light green wave passage safety constraint or the safe distance constraint, the quadratic programming solver is used to obtain the optimal safety control quantity that meets the safety constraints, as detailed below: (41) In step (31), the vehicle-mounted sensors collect multiple sets of driving and environmental data as training data. The environmental state prediction network fits the environmental state through a set of pre-trained neural networks containing LSTM layers. The input data of the environmental state prediction network is the set S of vehicle and environmental states at a certain moment. t The output data is S t+1 This is used to train a neural network to predict the environmental state in the next moment; (42) The trained environment state prediction network is applied to reinforcement learning training. Whenever the reinforcement learning intelligent controller is in the environment state s at a certain moment... t Output a control decision u t Then, all the vehicle and environmental state data at this moment are input into the environmental state prediction network to predict the environmental state s at the next moment. t+1 ; (43) When the predicted environmental state does not meet the traffic light constraint or the safe distance constraint to be maintained with the vehicle in front, a quadratic programming solver is used to minimize the error in the reinforcement learning intelligent controller u. t The objective is the square of the difference: In the formula, U is the permissible control set, which is used to find the minimum control u that satisfies the traffic light passage constraint and the safe distance constraint. t .

5. A method for traffic flow at signal-controlled intersections for connected vehicles based on security reinforcement learning, as described in claim 4, is characterized in that... The optimal safety control quantity obtained from the solution is input into the controlled vehicle to perform one iteration, which is used to train the reinforcement learning intelligent controller until the reward function of the reinforcement learning intelligent controller converges and the training is completed. The trained reinforcement learning intelligent controller is then used to implement traffic safety control at the intersection, as detailed below: (51) The constraints on controlled vehicles passing through traffic light-controlled intersections mainly take the following forms: in min ≤in ego ≤in max from rel ≥1.4v ego +l safe Where v ego The speed of the vehicle is represented by v. min and v max These represent the minimum and maximum speeds a vehicle can travel at during a given traffic light cycle, respectively. rel The l represents the relative distance between the controlled vehicle and the vehicle in front. safe This indicates the minimum safe distance that must be maintained when a vehicle is stationary; (52) The reward function includes the set of safe soft constraints defined by the Control Barrier Function and Control Lyapunov Function methods, namely: In the formula, δ is the vehicle rotational mass conversion factor; v tar The target speed is between the maximum and minimum passing speeds; F r It refers to the sum of vehicle driving resistance; c0 is a constant; ψ is the optimization objective, and the smaller the value of ψ, the more stable the tracking of the target vehicle speed; The reward function is as follows: Where E represents the energy consumption of the vehicle during operation, calculated from the vehicle's longitudinal dynamics model and kinematic model, e a This is the difference between the actual measured acceleration of the controlled vehicle and the acceleration output by the reinforcement learning intelligent controller.

6. A traffic signal control system for connected vehicles based on safety reinforcement learning, used to implement the traffic signal control method for connected vehicles based on safety reinforcement learning as described in any one of 1 to 5, characterized in that, include: The traffic environment information receiving module is used to receive traffic environment information, which includes traffic light status information, and the status information of the vehicle in front and the controlled vehicle. The first construction module is used to build a reinforcement learning intelligent controller. Traffic environment information is input into the reinforcement learning intelligent controller, and an adaptive soft motion evaluation algorithm is used to make real-time decisions and plan the driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle in the next moment. The second construction module is used to construct the safety supervision module. The safety supervision module includes an environmental state prediction network and a quadratic programming solver. The environmental state prediction network predicts the next state information of the controlled vehicle and the traffic environment based on the planned driving acceleration control quantity or braking deceleration control quantity of the controlled vehicle. When the predicted environmental state does not meet the traffic light green wave passage safety constraint or does not meet the safe distance constraint, the quadratic programming solver is used to solve for the optimal safety control quantity that meets the safety constraint. The iterative training module is used to input the solved optimal safety control quantity into the controlled vehicle to achieve one iteration, and to train the reinforcement learning intelligent controller until the reward function of the reinforcement learning intelligent controller converges and the training is completed. The trained reinforcement learning intelligent controller is used to realize the intersection traffic safety control.

Images