Traffic signal and vehicle trajectory cooperative control method in intelligent connected hybrid traffic flow environment

Abstract

This invention discloses a collaborative control method for traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment. This method constructs traffic signal control agents and vehicle trajectory control agents based on deep reinforcement learning, forming heterogeneous decision-making units. Multi-agent collaborative control of traffic signal control and vehicle trajectory control is achieved through a bidirectional state coupling design. A spatiotemporal multi-head attention mechanism and a reward-sharing mechanism are introduced to coordinate signal control decisions at multiple intersections. For intelligent connected mixed traffic flow scenarios, a vehicle-road-cloud integrated architecture is deployed. The agents are optimized and trained in the cloud, while the traffic signal control agent is deployed on the roadside for real-time traffic signal control, and the vehicle trajectory control agent is deployed on the vehicle terminal for dynamic optimization of intelligent connected vehicle trajectories. This invention eliminates the need for modeling the intelligent connected mixed traffic flow environment, achieving end-to-end collaborative optimization and effectively improving traffic efficiency while reducing energy consumption and emissions.

Description

Technical Field

[0001] This invention relates to the fields of autonomous driving and vehicle networking, and in particular to a method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment. Background Technology

[0002] Currently, with the continuous increase in global car ownership, compared with resource-intensive improvement measures such as widening roads, Traffic Signal Control (TSC) is an effective strategy for preventing and alleviating traffic congestion by regulating traffic flow in the spatiotemporal dimensions. With the rapid development of wireless communication, detection technology, artificial intelligence, and autonomous driving technology, Connected and Automated Vehicles (CAVs) have emerged. However, due to insufficient technological maturity and long infrastructure transformation cycles, urban roads will continue to maintain a mixed traffic flow state where human-driven vehicles (HDVs) and connected and automated vehicles (connected automated vehicles) coexist for a considerable period. In this new mixed traffic flow environment, TSC and Vehicle Trajectory Control (VTC) influence and interact with each other, and existing methods cannot establish an end-to-end collaborative control approach.

[0003] Deep Reinforcement Learning (DRL), as an end-to-end control method integrating perception, control, and execution, has been widely applied to solve complex system decision-making problems. However, existing research often employs DRL for single-dimensional optimization, focusing on vehicle trajectory planning or signal timing, neglecting the system-level benefits of coordinated control. Existing technologies mainly suffer from the following shortcomings:

[0004] Lack of Joint Optimization Solutions: Existing DRL-based solutions are generally limited to single-dimensional optimization, focusing either solely on traffic signal control or vehicle trajectory control, failing to achieve synergy between the two. The lack of end-to-end solutions that integrate both makes it difficult to maximize system-level benefits, resulting in limited effectiveness in improving urban road network traffic congestion. In fact, in an intelligent connected environment, traffic signal control and vehicle trajectory control are deeply coupled and inextricably linked: the timing of traffic signals directly affects vehicle driving decisions and trajectory planning, while the real-time trajectory and traffic demand of vehicles, in turn, determine the direction of traffic signal optimization.

[0005] Lack of coordination mechanisms: Existing TSC methods are mostly limited to single intersections and do not fully consider the network effects and coordination requirements between adjacent intersections. The lack of effective coordination mechanisms among agents leads to low regional traffic control efficiency.

[0006] Efficiency and real-time bottlenecks: At present, most technical solutions only propose methods to solve traffic signal control or vehicle trajectory control, but do not consider the problems of low training efficiency of DRL models and control response lag.

[0007] In summary, research on the coordinated control of TSC and VTC within the DRL and vehicle-road-cloud integrated technology framework under intelligent connected mixed traffic flow environments remains a technological gap. This area not only has urgent practical research needs but also broad engineering application prospects. Summary of the Invention

[0008] In view of this, the present invention provides a cooperative control approach for traffic signals and vehicle trajectories using a multi-agent actor-critic (CCTV-MAC) in an intelligent connected mixed traffic flow environment that at least solves some of the above-mentioned technical problems. The present invention does not require modeling of the intelligent connected mixed traffic flow environment and is an end-to-end cooperative control method with "perception-control-execution" capabilities. It helps to comprehensively improve the overall operating efficiency and safety of the traffic system, while effectively reducing vehicle travel time, delay time, fuel consumption and carbon emissions.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] In a first aspect, the present invention provides a method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment, the method mainly comprising:

[0011] Construct traffic signal control intelligent agents and vehicle trajectory control intelligent agents based on deep reinforcement learning;

[0012] A multi-agent collaborative control of traffic signal control and vehicle trajectory control is achieved through a bidirectional state coupling design. The traffic signal control agent uses the intelligent connected vehicle trajectory prediction data output by the vehicle trajectory control agent as the decision state input, and the vehicle trajectory control agent incorporates the signal timing scheme generated by the traffic signal control agent into the decision parameters.

[0013] For intelligent connected mixed traffic flow scenarios, the traffic signal control intelligent agent and the vehicle trajectory control intelligent agent are deployed in a vehicle-road-cloud integrated architecture. The traffic signal control intelligent agent and the vehicle trajectory control intelligent agent are optimized and trained in the cloud. The traffic signal control intelligent agent is deployed on the roadside for real-time control of traffic signals, and the vehicle trajectory control intelligent agent is deployed on the vehicle terminal for dynamic optimization of intelligent connected vehicle trajectories.

[0014] Furthermore, a traffic signal control agent is constructed based on deep reinforcement learning. The specific process includes:

[0015] By using V2I technology, real-time trajectory data of intelligent connected vehicles and manually driven vehicles can be acquired, the queue length of vehicles at the entrance lanes of signalized intersections can be calculated, and the state of the traffic signal control agent can be constructed.

[0016] A continuous action space design method with signal period as the update frequency is adopted to define the actions of the traffic signal control agent;

[0017] Using the intersection pressure difference as an indicator to measure the control effect of the traffic signal control agent, a reward and punishment function for the traffic signal control agent is defined.

[0018] Furthermore, a vehicle trajectory control agent is constructed based on deep reinforcement learning. The specific process includes:

[0019] By acquiring the preceding vehicle's state and signal phase data through V2V and V2I technologies, the state of the vehicle trajectory control agent is constructed.

[0020] The acceleration of intelligent connected vehicles defines the trajectory of the vehicle and controls the actions of the intelligent agent.

[0021] Based on the speed, acceleration, and distance from the parking line of the intelligent connected vehicle, a reward and punishment function for the vehicle trajectory control agent is defined.

[0022] Furthermore, a spatiotemporal multi-head attention mechanism is introduced into the feature extraction network of the traffic signal control agent; the spatiotemporal multi-head attention mechanism includes:

[0023] Using a Transformer-based spatial multi-head self-attention module, the spatial dynamic dependency relationship of intelligent connected hybrid traffic flow between adjacent intersections is modeled.

[0024] By utilizing a Transformer-based temporal multi-head self-attention module, we model the temporal dynamic dependencies of intelligent connected hybrid traffic flows in continuous time series.

[0025] Furthermore, a shared reward mechanism is adopted among multiple traffic signal control agents; the shared reward mechanism is as follows: the reward of each traffic signal control agent is the weighted sum of its own reward and the rewards of neighboring traffic signal control agents.

[0026] Furthermore, in the aforementioned vehicle-road-cloud integrated architecture, the cloud stores data samples and model parameters collected by the traffic signal control intelligent agent and the vehicle trajectory control intelligent agent. On the one hand, it accelerates model training through parallel computing; on the other hand, it distributes the updated model parameters of the traffic signal control intelligent agent to each roadside unit, and simultaneously pushes the updated model parameters of the vehicle trajectory control intelligent agent to each vehicle terminal. The roadside deploys the traffic signal control intelligent agent, receives the model parameters sent from the cloud and updates its local model. Combined with the collected intelligent connected mixed traffic flow data, it generates traffic signal timing decisions in real time. The vehicle terminal deploys the vehicle trajectory control intelligent agent, receives the model parameters from the cloud to complete the local model update, and then, based on the collected intelligent connected vehicle operation data and traffic signal timing information, realizes the dynamic optimization of the intelligent connected vehicle's driving trajectory.

[0027] Secondly, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to realize the above-described method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment.

[0028] As can be seen from the above technical solution, the present invention discloses a method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment, which has at least the following beneficial effects compared with the prior art:

[0029] (1) Traditional solutions often adopt a “segmented” technical approach, that is, first solve the data acquisition and environmental perception problems of the perception layer independently, and then process the control strategy generation and command issuance of the execution layer in stages. There is obvious separation between each link. However, the method of this invention deeply integrates the “perception-control-execution” functions to build an end-to-end control system, which can realize seamless connection and collaborative optimization from environmental information perception to control command execution. The method of this invention fills the technical gap in the existing methods that lack research on collaborative control of vehicle trajectory and signal timing in intelligent connected mixed traffic flow environment. This method does not require modeling of complex intelligent connected mixed traffic flow environment and is an end-to-end solution with “perception-control-execution” capabilities.

[0030] (2) This invention proposes a Transformer-based spatiotemporal multi-head attention mechanism and a shared reward mechanism. The spatiotemporal multi-head attention network overcomes the limitations of existing methods that only model traffic features in a single dimension. It can accurately capture the temporal volatility and periodicity of intelligent connected mixed traffic flows composed of CAVs and HDVs, and identify complex traffic association structures between intersections. The shared reward mechanism incentivizes the collaborative control of multiple TSC agents within the neighborhood, coordinating signal control decisions across multiple intersections. This method effectively aggregates the spatiotemporal dynamics and feedback rewards of adjacent intersections, improving overall traffic control performance.

[0031] (3) This invention adopts a bidirectional state coupling design to realize the collaborative control of heterogeneous decision-making agents composed of TSC and VTC. The TSC agent takes the CAV trajectory prediction data output by the VTC agent as the decision state input and adaptively generates a signal timing scheme. The VTC agent uses the preceding vehicle state and signal phase data obtained by V2V and V2I technologies to incorporate them into the decision parameters and dynamically adjust the driving behavior of the CAV. This is different from the serial structure in the prior art where the decision optimization of the TSC agent and the VTC agent are separated. Since the TSC agent adopts a continuous action space design with the signal period as the update frequency, the VTC agent can perform long-term optimization of the CAV driving trajectory, solve the problem of frequent changes in driving conditions in the existing methods, and realize multi-agent collaborative control of TSC and VTC.

[0032] (4) This invention proposes a vehicle-road-cloud integrated optimization architecture for multi-agent collaborative control of TSC and VTC. It shortens the training cycle of the agent's DRL model through parallel computing in the cloud, and simultaneously distributes the updated DRL model parameters of the agent to each terminal agent. Each terminal agent receives the cloud-based DRL model parameters to update its local DRL model, and then generates control decisions in real time based on the collected data. This structure solves the problems of low model training efficiency and delayed control response in multi-agent collaborative control of TSC and VTC in intelligent connected mixed traffic flow scenarios.

[0033] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0034] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0037] Figure 1 A schematic diagram of the overall structure of the coordinated control of traffic signals and intelligent connected vehicle trajectories provided by the present invention;

[0038] Figure 2 A schematic diagram of the spatiotemporal multi-head attention network structure for TSC multi-agent cooperative control provided by the present invention;

[0039] Figure 3 A schematic diagram of the integrated vehicle-road-cloud architecture for collaborative control of intelligent connected hybrid traffic flow TSC and VTC provided by the present invention.

[0040] Figure 4 This is a schematic diagram of the traffic network of the East Third Ring Road in Beijing provided by the present invention;

[0041] Figure 5 A schematic diagram showing the speedup curves of the parallel training algorithm for the traffic signal control intelligent agent provided by the present invention using different numbers of Worker nodes in the cloud.

[0042] Figure 6 A schematic diagram of the speedup curves of the parallel training algorithm for the vehicle trajectory control agent provided by the present invention using different numbers of Worker nodes in the cloud.

[0043] Figure 7 A schematic diagram illustrating the average vehicle travel time for different control methods provided by the present invention;

[0044] Figure 8 A schematic diagram illustrating the average vehicle delay time for different control methods provided by the present invention;

[0045] Figure 9 Schematic diagram of vehicle fuel consumption for different control methods provided by the present invention;

[0046] Figure 10 Schematic diagram of vehicle carbon dioxide emissions for different control methods provided by the present invention;

[0047] Figure 11This diagram illustrates the average vehicle travel time of the CCTV-MAC method under different CAV penetration rates provided by the present invention.

[0048] Figure 12 This is a schematic diagram of the electronic device structure provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0050] In the description of this invention, it should be noted that some processes described in this application specification and drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may be performed in any order or in parallel. Furthermore, various numbers are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0051] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0052] See Figures 1 to 3 As shown, this invention discloses a cooperative control approach for traffic signals and vehicle trajectory in an intelligent connected mixed traffic flow environment (CCTV-MAC). Figure 1As shown, the overall structure of the traffic signal and intelligent connected vehicle trajectory collaborative control in this invention is mainly divided into two parts: one is an intelligent connected hybrid traffic flow environment composed of CAVs and HDVs, and the other is a heterogeneous decision-making unit composed of a Traffic Signal Control (TSC) agent and a Vehicle Trajectory Control (VTC) agent. The TSC agent uses V2I technology to acquire trajectory data of CAVs and HDVs in real time and uses Deep Reinforcement Learning (DRL) to generate signal timing strategies adapted to the current traffic conditions. The VTC agent uses V2V and V2I technologies to acquire the status of preceding vehicles and signal phase data in real time and uses DRL to generate the optimal trajectory planning scheme. To achieve multi-agent collaboration in TSC, a spatiotemporal multi-head attention mechanism and a reward sharing mechanism are introduced to coordinate signal control decisions at multiple intersections. Furthermore, a bidirectional state coupling design is used to achieve deep collaboration between the TSC and VTC agents: on the one hand, the TSC agent uses the CAV trajectory prediction data output by the VTC as the input to the decision state; on the other hand, the VTC agent incorporates the signal timing scheme generated by the TSC into the decision parameters. The agent system adopts a three-level deployment architecture of vehicle-road-cloud: the cloud is responsible for the distributed DRL model training of the TSC and VTC agents; the TSC agent is deployed on the roadside to realize real-time control of traffic signals; and the VTC agent is deployed on the vehicle terminal to complete the dynamic optimization of CAV trajectories.

[0053] The specific embodiments of the present invention will be described in detail below:

[0054] (1) Establishing a TSC agent based on DRL:

[0055] Step 1.1: Construct the state of the TSC agent:

[0056] The TSC agent uses V2I technology to acquire real-time trajectory data of CAV and HDV, and then calculates the vehicle queue length at the signalized intersection approach lanes. Therefore, the traffic state of the i-th TSC agent at time step t is... The definition is as follows:

[0057]

[0058] Where I represents the number of TSC agents, and Let represent the number of vehicles queuing in the first to fourth phases of the i-th TSC agent at time step t, where 1 represents the east-west straight phase, 2 represents the east-west left-turn phase, 3 represents the north-south straight phase, and 4 represents the north-south left-turn phase.

[0059] Step 1.2: Define the actions of the TSC agent:

[0060] Most DRL-based traffic signal control methods use maintaining the current phase or switching to the next phase as the action of the TSC agent. This discrete action design limits the control accuracy and flexibility of the agent in intelligent connected mixed traffic flow scenarios. To solve these problems, this invention adopts a continuous action space design method with the signal period as the update frequency. The action of the i-th TSC agent at time step t... Its constraints are defined as follows:

[0061]

[0062] in, This represents the signal period of the i-th TSC agent at time step t. and Let these represent the minimum and maximum signal periods of the i-th TSC agent, respectively. This represents the green light ratio of the j-th green light phase of the i-th TSC agent at time step t. and Let represent the minimum green signal ratio and the maximum green signal ratio of the i-th TSC agent, respectively.

[0063] Step 1.3: Define the reward and punishment functions for the TSC agent:

[0064] The pressure difference at the intersection is a crucial indicator of the control effectiveness of the TSC agent. A smaller pressure difference indicates that the TSC agent's control strategy better meets actual needs and exhibits superior control efficiency. Therefore, the reward / penalty function of the i-th TSC agent at time step t... The definition is as follows:

[0065]

[0066] in, This represents the number of vehicles queuing in the l-th entrance lane of the i-th TSC agent at time step t. This represents the maximum number of vehicles queuing in the l-th entry lane for the i-th TSC agent. This represents the number of vehicles queuing at time step t for the l-th exit lane of the i-th TSC agent. This represents the maximum number of vehicles queuing at time step t for the l-th exit lane of the i-th TSC agent. This represents the total number of lanes for the i-th TSC agent.

[0067] (2) Establishing a VTC agent based on DRL:

[0068] Step 2.1: Construct the state of the VTC agent:

[0069] The VTC agent uses V2V and V2I technologies to acquire the state and signal phase data of the vehicle ahead, establishing the state vector needed for trajectory decision-making. Therefore, the state of the j-th VTC agent at time step t... The definition is as follows:

[0070]

[0071] in, and Let represent the velocity and acceleration of the current CAV collected by the j-th VTC agent at time step t, respectively. and Let represent the velocity and acceleration of the CAV ahead, collected by the j-th VTC agent at time step t, respectively. This represents the distance between the current CAV and the vehicle in front, collected by the j-th VTC agent at time step t. This represents the distance between the current CAV and the parking line collected by the j-th VTC agent at time step t. This represents the traffic signal status of the signalized intersection ahead, obtained by the j-th VTC agent via V2I at time step t. If the signalized intersection ahead has a green light, then... , This indicates the remaining time of the current green light phase; if the signalized intersection ahead is not green, then... , This indicates the time when the next green light phase will arrive.

[0072] Step 2.2, Define the actions of the VTC agent:

[0073] The action of the j-th VTC agent at time step t Its constraints are defined as follows:

[0074]

[0075] in, This represents the acceleration of CAV output by the j-th VTC agent at time step t. and Let represent the minimum and maximum acceleration of the CAV that the j-th VTC agent can output, respectively.

[0076] Step 2.3: Define the reward and punishment functions for the VTC agent:

[0077] The reward / penalty function of the j-th VTC agent at time step t The definition is as follows:

[0078]

[0079] in, This represents the CAV speed collected by the j-th VTC agent at time step t. This component is set to force the CAV to pass through the intersection at a higher speed. This represents the absolute value of the CAV acceleration collected by the j-th VTC agent at time step t. The setting of this component aims to guide the CAV to maintain a smooth driving state and reduce frequent rapid acceleration and deceleration. This represents the distance between the CAV and the parking line collected by the j-th VTC agent at time step t. This component is set to encourage the CAV to continuously approach the intersection. Let represent the penalty term for the j-th VTC agent at time step t. When the CAV velocity is 0, if the acceleration output by the VTC agent at this time is negative, then a penalty is imposed. Penalty: If the traffic light at the intersection ahead is green, and the CAV cannot leave the intersection at its current speed, and the VTC agent outputs a negative acceleration value at this time, then a penalty is imposed. Punishment. Furthermore... and These represent the weight coefficients of each item in the reward and punishment function of the VTC agent.

[0080] (3) Multi-agent cooperative control method of TSC:

[0081] like Figure 2 As shown, to achieve collaborative control of multiple agents in TSC, a spatiotemporal multi-head attention mechanism is introduced into the feature extraction network of DRL, and a reward sharing mechanism is introduced into the feedback evaluation of DRL, so as to coordinate the signal control decisions of multiple intersections. The specific process is as follows:

[0082] Step 3.1, Spatial Multi-Head Attention Mechanism:

[0083] like Figure 2 As shown, a Transformer-based Spatial Multi-Head Self-Attention Module (SMSA) is constructed to model the spatial dynamic dependencies of intelligent connected hybrid traffic flows composed of CAVs and HDVs between adjacent intersections. First, the joint spatial state of the i-th TSC agent and its neighboring agents at time step t is determined. The structure is as follows:

[0084]

[0085] in, This represents the state of the i-th TSC agent at time step t. Let I represent the state of the j-th neighboring agent of the i-th TSC agent at time step t, and let I represent the number of TSC agents. This represents the number of adjacent intersections for the i-th TSC agent. Following this, the joint spatial state... After passing through a fully connected layer, it is transformed into a high-dimensional feature vector. Then it is fed into the spatial multi-head attention layer to extract the dynamic spatial features of neighboring TSC agents. The calculation process is as follows:

[0086]

[0087] in, The spatial attention score generated by the h-th attention head of the l-th layer network of the i-th TSC agent at time step t is used to measure the degree of correlation between neighboring TSC agents in the spatial dimension. , and These represent the parameter matrices for the query, key, and value of the h-th attention head in the l-th layer of the neural network. This represents the matrix transpose operation. The dimension representing the key. A parameter matrix representing the aggregated multi-head attention values. For example... Figure 2 As shown, the output of the SMSA module and input After aggregation by the residual connection layer, a new feature vector is formed. This vector contains the state features of itself and its neighboring TSC agents.

[0088] Step 3.2, Temporal Multi-Head Attention Mechanism:

[0089] like Figure 2 As shown, following the SMSA module, a Transformer-based Temporal Multi-Head Self-Attention Module (TMSA) is constructed to model the temporal dynamic dependencies of intelligent connected hybrid traffic flows composed of CAVs and HDVs in continuous time series. First, after the spatial features are extracted by the SMSA module, the i-th TSC agent starts from time... Timing state up to time t The structure is as follows:

[0090]

[0091] in, This represents the number of time steps prior to time step t. This indicates that the i-th TSC agent is at time step [i]. The feature vectors. Then, the temporal states. The data is fed into a temporal multi-head attention layer to extract the dynamic temporal features of the TSC agent in continuous time series. The calculation process is as follows:

[0092]

[0093] in, Represents the i-th TSC agent's... The temporal attention score generated by the h-th attention head of the layered neural network at time step t is used to evaluate the degree of correlation between the TSC agent and the temporal dimension. For example... Figure 2 As shown, the output of the TMSA module and input After aggregation by the residual connection layer, a new feature vector is formed. This vector contains state features of itself at several time steps.

[0094] Step 3.3, Shared Reward Mechanism:

[0095] To incentivize collaborative control among multiple TSC agents within the neighborhood and coordinate signal control decisions at multiple intersections, a reward-sharing mechanism is adopted. The reward function for the i-th TSC agent at time step t is... The definition is as follows:

[0096]

[0097] in, and This represents the weighting coefficient that balances the rewards of the TSC agent and the rewards of the neighboring TSC agents; This represents the reward of the j-th neighboring agent of the i-th TSC agent at time step t; This represents the number of neighboring agents of the i-th TSC agent.

[0098] (4) Multi-agent cooperative control method of TSC and VTC:

[0099] In a heterogeneous decision-making unit composed of TSC and VTC agents, multi-agent collaborative control of TSC and VTC is achieved through a bidirectional state coupling design. The TSC agent uses the CAV trajectory prediction data output by VTC as the decision state input, extracts the dynamic spatiotemporal characteristics of the intelligent connected mixed traffic flow composed of CAV and HDV through a spatiotemporal multi-head attention network, and adaptively generates a signal timing scheme to achieve more efficient local response and regional collaborative control. The VTC agent incorporates the preceding vehicle state and signal phase data obtained by V2V and V2I technologies into the decision parameters, dynamically adjusts the driving behavior of CAV to coordinate with the dynamically changing traffic signal timing scheme, and autonomously decides acceleration and deceleration behavior to ensure safe, efficient and comfortable passage of CAV at intersections. Because the TSC agent adopts a continuous action space design with the signal cycle as the update frequency, the VTC agent can obtain the remaining time of the current green light phase or the waiting time of the next green light phase in real time through V2I technology, and perform long-term optimization of the CAV driving trajectory. At the same time, the intelligent connected hybrid traffic flow environment composed of CAV and HDV will provide corresponding reward feedback for traffic signal timing and vehicle trajectory decision-making of heterogeneous multi-agents, comprehensively evaluate the control effect of agent decision-making behavior, and help agents train end-to-end autonomous decision-making intelligence with "perception-decision-evaluation" capabilities.

[0100] (5) The vehicle-road-cloud integrated optimization architecture for multi-agent collaborative control of TSC and VTC:

[0101] To address the issues of low model training efficiency and delayed control response in multi-agent collaborative control of TSC and VTC in intelligent connected mixed traffic flow scenarios, this invention proposes an optimized Vehicle-Road-Cloud Integration Structure (VRCIS) architecture. For example... Figure 3 As shown, the cloud stores data samples and DRL model parameters collected by the TSC and VTC agents. This shortens the training cycle of the DRL model through parallel computing and distributes the updated TSC model parameters to each roadside unit (RSU), while simultaneously pushing the updated VTC model parameters to each vehicle terminal. The roadside TSC agent receives the model parameters from the cloud and updates its local model. Combined with the collected intelligent connected mixed traffic flow data, it generates real-time traffic signal timing decisions. The vehicle terminal deploys the VTC agent, which also receives the cloud model parameters to update its local model. Based on the collected CAV operating data and traffic signal timing information, it dynamically optimizes the CAV's trajectory. The specific training process includes:

[0102] Step 5.1: Set the total number of training rounds. Control the time step in each round The size of the experience pool of the TSC agent The size of the experience pool of the VTC agent .

[0103] Step 5.2: Initialize the Actor network and Critic network of the TSC agent located in the RSU, and the Actor network and Critic network of the VTC agent located in the CAV vehicle terminal.

[0104] Step 5.3: At each TSC control time step t, the TSC agent i located at the RSU uses V2I technology to acquire the trajectory data of the CAV and HDV at the adjacent intersection in real time, and constructs the joint state vector required by the agent according to formulas (1) and (11). Based on formulas (12)-(16), the dynamic spatiotemporal characteristics of intelligent connected mixed traffic flow at adjacent intersections are extracted, and then a coordinated TSC strategy is generated based on formulas (2)-(4). After the strategy is applied to the intelligent connected mixed traffic flow environment, the feedback reward is calculated according to formula (5). And obtain the state of the next control time step from the environment. These samples Experience replay pool stored in TSC agents middle.

[0105] Step 5.4: At each VTC control time step t, the VTC agent j located at the CAV on-board terminal acquires the preceding vehicle state and traffic signal phase data using V2V and V2I technologies, and establishes the state vector required for CAV trajectory decision-making according to formula (6). The CAV trajectory control strategy is generated according to formulas (7)-(8). After the strategy is applied to the intelligent connected mixed traffic flow environment, the feedback reward is calculated according to formulas (9)-(10). And obtain the state of the next control time step from the environment. These samples Experience replay pool stored in VTC agents middle.

[0106] Step 5.5: Repeat steps 5.3-5.4 until the amount of sample data reaches the size of the respective experience playback pool. Upload the sample data collected from the TSC agent and VTC agent from the roadside and vehicle terminals to the corresponding Worker nodes in the cloud.

[0107] Step 5.6: All Worker nodes in the cloud compute the gradients of the TSC and VTC intelligent agents Actor and Critic networks in parallel based on local sample data, and then aggregate these gradient values ​​to the cloud Master node.

[0108] Step 5.7: The cloud Master node calculates the average gradient of all Worker nodes and uses the Adam adaptive gradient descent algorithm to update the DRL model parameters of the TSC and VTC agents respectively.

[0109] Step 5.8: The cloud sends the updated TSC agent model parameters to each RSU to update the DRL model of the TSC agent; at the same time, it pushes the updated VTC agent model parameters to each vehicle terminal to update the DRL model of the VTC agent.

[0110] Step 5.9: Repeat steps 5.3-5.8 until the total number of training rounds E is reached. The RSU saves the DRL model parameters of the trained TSC agent, and the CAV vehicle terminal saves the DRL model parameters of the trained VTC agent.

[0111] Furthermore, to evaluate the CCTV-MAC method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected environment proposed in this invention, Fixed-Time Control (FTC), Max-Pressure, Pressure Difference-Based DRL (PressLight), Vehicle-Agent considering only CAV trajectory optimization, and Signal-Agent considering only traffic signal timing optimization are used as comparison objects. Figure 4 As shown, the traffic network of the East Third Ring Road in Beijing serves as the simulation network for the experiment, and this network has a total of 41 signalized intersections.

[0112] like Figure 5 and Figure 6 As shown, the parallel training algorithm proposed in this invention gradually reduces training time with the increase of computing nodes, and the speedup ratio curve shows a gradually increasing trend, indicating that the vehicle-road-cloud integrated training algorithm proposed in this invention can effectively reduce the training time loss of TSC and VTC agents. However, due to the influence of factors such as communication load and resource scheduling between computing nodes, the trend of speedup ratio change is non-linear. Secondly, the control effect is evaluated by statistically analyzing the vehicle's travel time, delay time, fuel consumption, and carbon emissions under different control methods.

[0113] like Figure 7 , Figure 8 , Figure 9 and Figure 10As shown, the CCTV-MAC method proposed in this invention outperforms other comparative methods in terms of reducing vehicle travel time, delay time, fuel consumption, and carbon emissions.

[0114] like Figure 11 As shown, the increase in CAV penetration rate is significantly correlated with the decrease in average vehicle driving time. This data not only intuitively reflects the positive effect of CAV application, but also strongly verifies the adaptability and robustness of the CCTV-MAC method in intelligent connected environments for different CAV penetration rate scenarios.

[0115] As described in the above embodiments, those skilled in the art will understand that the present invention provides a method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment. The main technical means employed in this method include:

[0116] 1. This invention proposes a multi-agent Actor-Critic deep reinforcement learning (CCTV-MAC) method for collaborative control of traffic signals and vehicle trajectories in intelligent connected mixed traffic flow environments. Unlike the segmented approach of existing technologies that first perform detailed environmental modeling and then process data through perception and decision-making, this method does not require modeling of complex intelligent connected mixed traffic flow environments. Instead, it is an end-to-end solution integrating the entire process of perception, control, and execution.

[0117] 2. This invention proposes a Transformer-based spatiotemporal multi-head attention mechanism and a shared reward mechanism. The spatiotemporal multi-head attention network overcomes the limitations of existing methods that extract traffic features only in a single time or spatial dimension. It can accurately capture the temporal volatility and periodicity of intelligent connected mixed traffic flows composed of CAVs and HDVs, and identify complex traffic association structures between intersections. The shared reward mechanism incentivizes collaborative control among multiple TSC agents within the neighborhood, coordinating signal control decisions across multiple intersections. This method effectively aggregates the spatiotemporal dynamics and feedback rewards of adjacent intersections, improving overall traffic control performance.

[0118] 3. This invention proposes a bidirectional state coupling design method for achieving deep collaboration between heterogeneous intelligent agents TSC and VTC. The TSC agent uses the CAV trajectory prediction data output by the VTC agent as the decision state input to adaptively generate traffic signal timing schemes. The VTC agent incorporates the preceding vehicle state obtained through V2V and V2I technologies and the traffic signal phase data output by the TSC agent into the decision parameters to dynamically adjust the CAV's driving behavior. The TSC agent dynamically optimizes the signal period and green light ratio using an update frequency based on the signal period to enable the CAV to perform longer-term trajectory planning, thereby achieving a more efficient, comfortable, and energy-saving driving experience.

[0119] 4. This invention addresses the coordinated control of traffic signals and vehicle trajectories in intelligent connected mixed traffic flow environments, employing an integrated vehicle-road-cloud optimization approach. Its technical solution mainly comprises three parts: First, cloud-based optimized training, using parallel optimization algorithms to accelerate the DRL model parameter training of both TSC and VTC agents, effectively shortening training time; second, distributed terminal deployment, deploying TSC agents on roadside terminals and VTC agents on vehicle-mounted terminals, utilizing a "nearby data collection + nearby decision execution" mechanism to minimize response latency and improve real-time decision-making efficiency; third, bidirectional information interaction, establishing information channels between the cloud, roadside, and vehicle-mounted terminals: the cloud distributes trained model parameters to the roadside to update the DRL model of the TSC agent; it also distributes them to the vehicle-mounted terminals to update the DRL model of the VTC agent; the roadside and vehicle-mounted terminals upload intelligent connected mixed traffic flow data, CAV trajectory data, and environmental feedback to the cloud, ensuring real-time data flow and continuous model optimization.

[0120] Compared with the prior art, the method of the present invention achieves significant technical effects in the following aspects:

[0121] (1) The method of the present invention fills the technical gap in the existing methods that lack research on the coordinated control of vehicle trajectory and signal timing in intelligent connected mixed traffic flow environment. The method does not require modeling of complex intelligent connected mixed traffic flow environment and is an end-to-end solution with "perception-control-execution" capability.

[0122] (2) The present invention proposes a Transformer-based spatiotemporal multi-head attention mechanism and a shared reward mechanism. The spatiotemporal multi-head attention network overcomes the limitations of existing methods that only model traffic features in a single dimension. It can accurately capture the temporal volatility and periodicity of intelligent connected mixed traffic flows composed of CAVs and HDVs, and identify complex traffic association structures between intersections. The shared reward mechanism incentivizes the collaborative control of multiple TSC agents within the neighborhood, coordinating signal control decisions across multiple intersections. This method effectively aggregates the spatiotemporal dynamics and feedback rewards of adjacent intersections, improving overall traffic control performance.

[0123] (3) This invention adopts a bidirectional state coupling design to realize the collaborative control of heterogeneous decision-making agents composed of TSC and VTC. The TSC agent uses the CAV trajectory prediction data output by VTC as the decision state input and adaptively generates a signal timing scheme. The VTC agent uses the preceding vehicle state and signal phase data obtained by V2V and V2I technologies to incorporate them into the decision parameters and dynamically adjust the driving behavior of CAV. This is different from the serial structure in the prior art where the decision optimization of TSC agent and VTC agent is separated. Since the TSC agent adopts a continuous action space design with the signal period as the update frequency, the VTC agent can perform long-term optimization of CAV driving trajectory, solve the problem of frequent changes in driving conditions in the existing method, and realize multi-agent collaborative control of TSC and VTC.

[0124] (4) The proposed multi-agent collaborative control vehicle-road-cloud integrated optimization architecture for TSC and VTC shortens the training cycle of the agent's DRL model through parallel computing in the cloud. At the same time, it distributes the updated DRL model parameters of the agent to each terminal agent. Each terminal agent receives the DRL model parameters from the cloud to complete the local DRL model update, and then generates control decisions in real time based on the collected data. This structure solves the problems of low model training efficiency and delayed control response in the multi-agent collaborative control of TSC and VTC in intelligent connected mixed traffic flow scenarios.

[0125] Furthermore, referring to Figure 12 As shown, this embodiment of the invention also provides an electronic device that can perform the above-described method. The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and capable of running on the processor 10.

[0126] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0127] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, electronic devices, or computer program products, etc. Therefore, the present invention can take the form of a completely software embodiment, a completely hardware embodiment, or an embodiment combining software and hardware aspects.

[0128] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer.

[0129] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0130] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment, characterized in that, The method includes: Construct traffic signal control intelligent agents and vehicle trajectory control intelligent agents based on deep reinforcement learning; A spatiotemporal multi-head attention mechanism is introduced into the feature extraction network of the traffic signal control agent; the spatiotemporal multi-head attention mechanism includes: Using a Transformer-based spatial multi-head self-attention module, the spatial dynamic dependencies of intelligent connected hybrid traffic flows between adjacent intersections are modeled; firstly, the... i A traffic signal control agent and its neighboring agents at time step t joint space state The construction is as follows: in, Indicates the first i A traffic signal control agent at time step t state, Indicates the first i The first traffic signal control intelligent agent j Neighboring agents at time step t state, I This indicates the number of traffic signal control agents. Indicates the first i The number of adjacent intersections for each traffic signal control agent; followed by, the joint spatial state. After passing through a fully connected layer, it is transformed into a high-dimensional feature vector. Then it is fed into the spatial multi-head attention layer to extract the dynamic spatial features of adjacent traffic signal control agents. The calculation process is as follows: in, Indicates the first i The first traffic signal control intelligent agent l The first layer of the network h Attention is focused on the time step t The generated spatial attention score is used to measure the degree of correlation between adjacent traffic signal control agents in the spatial dimension; , and They represent the first l The first layer of a neural network h The parameter matrix of the query, key, and value for each attention head. This represents the matrix transpose operation. The dimension representing the key. The parameter matrix representing the aggregated multi-head attention values; the output of the spatial multi-head self-attention module. and input After aggregation by the residual connection layer, a new feature vector is formed. This vector contains the state features of itself and its neighboring traffic signal control agents; Using a Transformer-based temporal multi-head self-attention module, the temporal dynamic dependencies of intelligent connected hybrid traffic flow in continuous time series are modeled. First, spatial features are extracted using a spatial multi-head self-attention module, and then... i A traffic signal control agent from time At the time t Timing state The construction is as follows: in, Indicates time step t Previous time steps, Indicates the first i A traffic signal control agent at time step The feature vectors; then, the temporal states. The data is fed into a multi-head attention layer to extract the dynamic temporal features of the traffic signal control agent in a continuous time series. The calculation process is as follows: in, Indicates the first i The first traffic signal control intelligent agent The first layer of a neural network h Attention is focused on the time step t The generated temporal attention score is used to evaluate the degree of correlation between traffic signal control agents in the temporal dimension; the output of the temporal multi-head self-attention module. and input After aggregation by the residual connection layer, a new feature vector is formed. This vector contains state features of itself at several time steps; A multi-agent collaborative control of traffic signal control and vehicle trajectory control is achieved through a bidirectional state coupling design. The traffic signal control agent uses the intelligent connected vehicle trajectory prediction data output by the vehicle trajectory control agent as the decision state input, and the vehicle trajectory control agent incorporates the signal timing scheme generated by the traffic signal control agent into the decision parameters. For intelligent connected mixed traffic flow scenarios, the traffic signal control intelligent agent and the vehicle trajectory control intelligent agent are deployed in a vehicle-road-cloud integrated architecture. The traffic signal control intelligent agent and the vehicle trajectory control intelligent agent are optimized and trained in the cloud. The traffic signal control intelligent agent is deployed on the roadside for real-time control of traffic signals, and the vehicle trajectory control intelligent agent is deployed on the vehicle terminal for dynamic optimization of intelligent connected vehicle trajectories.

2. The method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment according to claim 1, characterized in that, The process of building a traffic signal control agent based on deep reinforcement learning includes: By using V2I technology, real-time trajectory data of intelligent connected vehicles and manually driven vehicles can be acquired, the queue length of vehicles at the entrance lanes of signalized intersections can be calculated, and the state of the traffic signal control agent can be constructed. A continuous action space design method with signal period as the update frequency is adopted to define the actions of the traffic signal control agent; Using the intersection pressure difference as an indicator to measure the control effect of the traffic signal control agent, a reward and punishment function for the traffic signal control agent is defined.

3. The method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment according to claim 1, characterized in that, The process of building a vehicle trajectory control agent based on deep reinforcement learning includes: By acquiring the preceding vehicle's state and signal phase data through V2V and V2I technologies, the state of the vehicle trajectory control agent is constructed. The acceleration of intelligent connected vehicles defines the trajectory of the vehicle and controls the actions of the intelligent agent. Based on the speed, acceleration, and distance from the parking line of the intelligent connected vehicle, a reward and punishment function for the vehicle trajectory control agent is defined.

4. The method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment according to claim 1, characterized in that, A shared reward mechanism is adopted among multiple traffic signal control agents; the shared reward mechanism is as follows: the reward of each traffic signal control agent is the weighted sum of its own reward and the rewards of its neighboring traffic signal control agents.

5. The method for coordinated control of traffic signals and vehicle trajectories in an intelligent connected mixed traffic flow environment according to claim 1, characterized in that, In the aforementioned vehicle-road-cloud integrated architecture, the cloud stores data samples and model parameters collected by the traffic signal control intelligent agent and the vehicle trajectory control intelligent agent. On the one hand, it accelerates model training through parallel computing; on the other hand, it distributes the updated model parameters of the traffic signal control intelligent agent to each roadside unit, and simultaneously pushes the updated model parameters of the vehicle trajectory control intelligent agent to each vehicle terminal. The roadside deploys the traffic signal control intelligent agent, receives the model parameters sent from the cloud and updates its local model. Combined with the collected intelligent connected mixed traffic flow data, it generates traffic signal timing decisions in real time. The vehicle terminal deploys the vehicle trajectory control intelligent agent, receives the model parameters from the cloud to complete the local model update, and then, based on the collected intelligent connected vehicle operation data and traffic signal timing information, realizes the dynamic optimization of the intelligent connected vehicle's driving trajectory.

6. An electronic device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement a traffic signal and vehicle trajectory cooperative control method in an intelligent connected mixed traffic flow environment as described in any one of claims 1-5.

Images