Cloud side-end integrated collaborative digital and intelligent traffic collaborative management and control method, system, equipment and medium

Abstract

The invention relates to a cloud side-end integrated collaborative digital-intelligent traffic collaborative management and control method, system and device and a medium, and the method comprises the steps: dynamically adjusting the main line speed limit and ramp control rate through driving main line variable speed limit and traffic induction collaborative management and control based on a hierarchical control architecture, so as to enable the performance of a regional road network to be optimal; a collaborative optimization framework is constructed and utilized under a hierarchical control framework, an optimal control scheme of a single ramp is expanded to each partition of a road network in a random environment, and an optimal joint behavior strategy and a behavior selection strategy of each agent are determined, so that adaptive collaborative control of each agent is realized; and a traffic flow optimal equalization strategy of the whole road network is obtained. Therefore, by integrating a main line variable speed limit and traffic induction collaborative management and control technology, a main line and multi-ramp collaborative optimization framework and intelligent agent configuration, optimization of regional road network performance, optimal balance of overall road network traffic flow and improvement of traffic management and control accuracy and response speed are achieved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a cloud-edge-end integrated collaborative digital transportation collaborative management method, system, equipment and medium. Background Art

[0002] In the current traffic management system, with the continuous increase in traffic volume on expressways, rapid response and precise control of traffic incidents have become the key to improving road traffic efficiency and ensuring driving safety. Traditional expressway traffic management mainly relies on manual monitoring and post-event processing, which is not only slow to respond, but also difficult to fully cover complex traffic conditions, leading to frequent problems such as traffic congestion and untimely accident handling.

[0003] In order to improve the intelligence level of traffic management, existing technologies have begun to try to introduce vehicle-to-infrastructure collaboration (V2X) and cloud computing technologies. These technologies can improve the efficiency and accuracy of traffic management to a certain extent through information exchange between vehicles and infrastructure, as well as data processing and analysis in the cloud. However, since these technologies often operate independently and lack effective integration and coordination, there are still many shortcomings in practical applications.

[0004] Specifically, although the existing vehicle-road cooperative system can realize information exchange between vehicles and roadside equipment, it is limited by the communication range and data processing capabilities, making it difficult to achieve real-time perception and accurate prediction of the global traffic status of high-speed roads. At the same time, although the cloud computing platform has powerful data processing capabilities, due to network delays and large data transmission volumes, there is a significant time lag between cloud-based decision-making and roadside execution, which affects the immediacy and effectiveness of traffic control.

[0005] In addition, the existing traffic management system also has major hidden dangers in terms of data security and network security. Due to the lack of adequate protection measures during data transmission and processing, it is vulnerable to malicious attacks and data leakage risks, thus threatening the overall operational safety of the traffic system.

[0006] Therefore, there is an urgent need for a new technology system that can integrate the resources of vehicles, roads, and clouds to achieve precise control of high-speed road traffic incidents. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides a cloud-edge-end integrated collaborative digital traffic collaborative management method, system, equipment and medium, which solves the technical problems of incomplete perception, untimely decision-making and unguaranteed data security existing in the prior art.

[0009] (II) Technical solution

[0010] In order to achieve the above object, the main technical solutions adopted by the present invention include:

[0011] In a first aspect, an embodiment of the present invention provides a cloud-edge-end integrated collaborative digital intelligent traffic collaborative control method, including:

[0012] Based on the cloud-edge-end integrated multi-agent hierarchical control architecture, the main line speed limit and ramp control rate are dynamically adjusted in different scenarios by driving the main line variable speed limit and traffic induction collaborative control to optimize the performance of the regional road network.

[0013] Under the hierarchical control architecture, a mainline and multi-ramp collaborative optimization framework is constructed and utilized to extend the optimization control scheme of a single ramp to various partitions of the road network under a random environment, determine the optimal joint behavior strategy and behavior selection strategy of each intelligent agent, so as to achieve adaptive collaborative control of each intelligent agent and obtain the optimal traffic flow balance strategy of the entire road network;

[0014] Among them, the intelligent agent is an autonomous decision-making entity deployed on the road network, which is configured to perceive the traffic status in real time, select, execute, track and optimize the received control decisions.

[0015] Optionally, based on the constructed cloud-edge-end integrated multi-agent hierarchical control architecture, by driving the main line variable speed limit and traffic induction collaborative control, the main line speed limit and ramp control rate are dynamically adjusted in different scenarios to optimize the performance of the regional road network, including:

[0016] Build a cloud-edge-end integrated expressway traffic collaborative control system with a central cloud strategy generation module, an edge node collaborative optimization module, and a front-end device solution execution module;

[0017] Based on the cloud-edge-device integrated expressway traffic collaborative control system, a multi-agent based hierarchical control architecture with a cloud strategy layer and an edge solution layer is constructed;

[0018] Aiming at optimizing the performance of the regional road network, the cloud-based strategy layer outputs the mainline and ramp coordinated control strategy;

[0019] According to the main line and ramp coordinated control strategy, the intelligent agent searches for the best through the edge solution layer, generates an edge real-time optimization solution for the main line variable speed limit and ramp intelligent control coordination, sends it to the front-end device for execution, and receives the status feedback of the intelligent agent;

[0020] The variable speed limit and traffic induction coordinated control of the main line are driven according to the edge real-time optimization solution. The main line speed limit and ramp control rate are dynamically adjusted for different scenarios to optimize the performance of the regional road network.

[0021] Optionally, according to the main line and ramp coordinated control strategy, the intelligent agent searches for the best through the edge solution layer, generates an edge real-time optimization solution for main line variable speed limit and ramp intelligent control coordination, and sends it to the front-end device for execution, including:

[0022] Divide the main line and ramps into multiple control units with status information;

[0023] Based on the mainline and ramp coordinated control strategy from the cloud strategy layer and the control unit with status information, a mainline variable speed limit and ramp coordinated control model is constructed with road traffic flow and vehicle delay as the joint optimization control objectives;

[0024] Based on the mainline variable speed limit and ramp coordinated control model, the intelligent agent is used in the edge solution layer to search for the optimal combination of the mainline variable speed limit value and the entrance ramp control rate to maximize the value of the joint optimization control objective;

[0025] Based on the results of the intelligent agent search and optimization, an edge real-time optimization solution for the main line variable speed limit and ramp intelligent control collaboration is generated;

[0026] Send the generated optimization plan to the front-end device for execution, and monitor the execution effect of the optimization plan in real time;

[0027] Among them, the main line variable speed limit and ramp coordinated control model is:

[0028]

[0029]

[0030] In the formula, TTF is the road traffic flow model, TTD is the vehicle delay model, is the main road unit traffic flow density at time k, is the unit traffic flow speed of the main road at time k, λ m is the number of lanes on the main road at time k, is the k ramp control rate, is the traffic flow density of the ramp unit at time k, is the ramp unit traffic flow speed at time k, is the number of ramp lanes at time k, Δl m is the length of each control unit section, v f，m is the free flow velocity of the control unit m, r m is the number of unit ramps, m is the number of unit main lines;

[0031] The objective function of the joint optimization control objective is: ω = αTTF-(1-α)TTD. When ω is the largest, it reaches the optimal value. α is the adjustment coefficient, and its value range is (0,1).

[0032] Optionally, a mainline and multi-ramp collaborative optimization framework is constructed and utilized under a hierarchical control architecture to extend the optimization control scheme of a single ramp to each partition of the road network under a random environment, determine the optimal joint behavior strategy and behavior selection strategy of each agent, so as to realize adaptive collaborative control of each agent, and obtain the optimal traffic flow balance strategy of the overall road network, including:

[0033] Abstract each ramp signal in the road network as an intelligent agent;

[0034] Based on the hierarchical control architecture, the multi-agent behavior and evaluation strategy of the traffic network is introduced to build a multi-agent reinforcement learning framework for the coordinated optimization of the main line and multiple ramps;

[0035] Train all agents in the multi-agent reinforcement learning framework through centralized training;

[0036] After the multi-agent reinforcement learning framework is trained, the learned strategies are deployed to each agent in a distributed execution mode, so that each agent can make decisions and perform corresponding control actions independently according to the real-time traffic flow status of the partition area.

[0037] Extend the optimization control scheme of a single ramp to various sections of the road network in a random environment;

[0038] Through the linkage and coordination between the various intelligent agents in different partitions, the joint optimal behavior strategy and behavior selection strategy of each intelligent agent are determined, and through repeated iterations, the unique equilibrium in the random environment is found to obtain the optimal equilibrium strategy for the traffic flow of the entire road network.

[0039] Optionally, through the linkage and coordination between the agents in different partitions, the joint optimal behavior strategy and behavior selection strategy of each agent are determined, and the unique equilibrium in the random environment is found through repeated iterations to obtain the optimal equilibrium strategy for the traffic flow of the overall road network, including:

[0040] The D3QN algorithm is used to build a reinforcement learning model of multi-ramp agents, in which the ramp agents in the same partition share traffic status and experience and have a common control goal;

[0041] Replace the states and actions of a single agent with joint states and joint actions in a dynamic random environment respectively;

[0042] In each game strategy stage, according to the current joint state and joint action, through linkage and coordination of various intelligent agents, the value function of the equilibrium strategy is estimated to achieve joint decision-making of the multi-agent reinforcement learning system;

[0043] By repeatedly iterating and approximating the optimal strategy, the joint optimal behavior strategy and behavior selection strategy of each agent are determined;

[0044] According to the current joint state and joint action, the control action is selected and handed over to the agent for execution, and the control effect is continuously tracked and evaluated;

[0045] According to the goal of maximizing the cumulative return, the optimization is continued until the optimal behavior strategy and behavior selection strategy converge to the optimal probability mapping of state and action, and finally the optimal equilibrium strategy of traffic flow of the entire road network is obtained;

[0046] Among them, the expression of the optimal balance strategy of traffic flow in the overall road network is:

[0047]

[0048] Where, s={s1,…,s N} represents the joint state vector of N agents, including its own state and the states of other agents in the group, s k represents the joint state vector of N agents at time k, a={a1,…,a N} represents the joint action vector of N agents, including their own state-behavior pairs and each state-behavior pair of other agents, a k represents the joint action vector of N agents at time k, α k is the learning rate at time k, ranging from [0,1], r i k is the reward of agent i at time k, γV i k-1 is the state value of agent i at time k-1, V i (s) is the state value function of agent i in the joint state s, NE i is the Nash equilibrium strategy of agent i, π * is the optimal strategy, π is the strategy, π(a) is the probability that N agents choose a joint action, and A is the joint action space.

[0049] Optionally, a D3QN algorithm is used to construct a reinforcement learning model of multiple ramp agents, in which the ramp agents in the same partition share traffic status and experience and have common control objectives including:

[0050] The D3QN algorithm is used to build a reinforcement learning model of multi-ramp agents, where the reinforcement learning model includes a main DQN network and a target DQN network;

[0051] Configure the ramp agents in the same partition to share traffic status and experience data, and assign common control targets to the ramp groups in the same partition;

[0052] Train multiple ramp agents and select the control action with the maximum Q value in the main DQN network;

[0053] Introduce the target DQN network to obtain the Q value of the control action in the target DQN network, with the goal of minimizing the target Q value fluctuation and avoiding the selection of non-optimal Q values;

[0054] The Q value of the state-behavior pair is split into two output parts: Value and Advantage. Value represents the value of the traffic environment state itself, and Advantage represents the additional value brought by dynamically selecting a certain control action.

[0055] According to the training results, the control strategy of each ramp agent is updated. At each time step, each ramp agent uses the updated strategy to make decisions based on the current state and shared information and select the optimal control action;

[0056] Continuously perform iterative training of the reinforcement learning model, monitor the expected value of the cumulative reward of the agent in the training process under the current control strategy, until the preset conditions are met, and obtain the trained reinforcement learning model;

[0057] The Q value is:

[0058] Q(s t ,a t )=V(s t )+A(a t );

[0059] In the formula, Q(s t ,a t ) is the state-behavior function, which is used to evaluate the expected return of adopting the strategy in the current state, s t Represents the current state, a t represents the action taken in the current state, t is the time step, V(s t ) is the value of the traffic environment state itself, A(a t ) is the additional value brought by dynamically selecting a certain control action, Q(s t ,a t ) is larger, the larger the expected value of the cumulative reward of the agent in state s by following the strategy.

[0060] In a second aspect, an embodiment of the present invention provides a cloud-edge-end integrated collaborative digital intelligent traffic collaborative management and control system, including:

[0061] The regional road network optimization module is used to dynamically adjust the main line speed limit and ramp control rate in different scenarios based on the cloud-edge-end integrated multi-agent hierarchical control architecture to optimize the performance of the regional road network by driving the main line variable speed limit and traffic induction collaborative control;

[0062] The overall road network optimization module is used to build and utilize the mainline and multi-ramp collaborative optimization framework under the hierarchical control architecture, expand the optimization control scheme of a single ramp to various partitions of the road network under a random environment, determine the optimal joint behavior strategy and behavior selection strategy of each intelligent agent, so as to achieve adaptive collaborative control of each intelligent agent and obtain the optimal traffic flow balance strategy of the overall road network;

[0063] Among them, the intelligent agent is an autonomous decision-making entity deployed on the road network, which is configured to perceive the traffic status in real time, select, execute, track and optimize the received control decisions.

[0064] In a third aspect, an embodiment of the present invention provides a cloud-edge-end integrated collaborative digital intelligent traffic collaborative control device, including:

[0065] A central cloud, used to execute the method as described above;

[0066] Front-end equipment, used to collect multi-source fusion data in real time and transmit it to edge nodes for processing;

[0067] The intelligent agent is deployed at the edge node and receives policy guidance from the central cloud. It generates real-time optimization solutions for variable speed limits on the main line and intelligent ramp control through search and optimization. During the execution process, it feeds back its own status to the central cloud to achieve dynamic selection, execution, tracking and optimization of strategies.

[0068] Edge nodes are equipped with computing and storage resources to receive and process data transmitted from front-end devices. At the same time, they coordinate with intelligent agents to implement strategies and issue plans based on the policy scheduling of the central cloud, thereby achieving collaborative optimization of edge support for cloud applications and cloud support for edge localization needs.

[0069] Optionally, the central cloud, front-end devices, edge nodes and intelligent agents form a cloud-edge-end integrated expressway traffic collaborative control system with a central cloud strategy generation module, an edge node collaborative optimization module and a front-end device solution execution module;

[0070] Based on the cloud-edge-end integrated expressway traffic collaborative control system, a hierarchical control architecture with a cloud strategy layer and an edge solution layer based on multi-agents is formed;

[0071] Among them, the cloud strategy layer outputs data-driven mainline and ramp collaborative control strategies, and the edge solution layer generates edge real-time optimization solutions for mainline variable speed limits and ramp intelligent control collaboration through intelligent agent search and optimization, and feeds back the intelligent agent status to the central cloud to build a closed-loop computing architecture for cloud-edge collaboration.

[0072] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium on which computer executable instructions are stored. When the executable instructions are executed by a processor, the digital and intelligent traffic collaborative management method with integrated cloud-edge-end collaboration as described above is implemented.

[0073] (III) Beneficial effects

[0074] The beneficial effects of the present invention are:

[0075] First, by driving the variable speed limit of the main line and the traffic induction collaborative control technology, the present invention can dynamically adjust the main line speed limit and ramp control rate in different scenarios. This flexibility not only helps to alleviate the frequent congestion on expressways, but also effectively responds to the deterioration of traffic conditions caused by emergencies. Through the "induction-control" double-layer optimization control strategy, the overall performance of the regional road network is optimized, thereby significantly improving the traffic efficiency of high (fast) expressways.

[0076] Secondly, under the hierarchical control architecture, the present invention constructs a main line and multi-ramp collaborative optimization framework, and successfully extends the optimization control scheme of a single ramp to each partition of the road network under a random environment. Thus, the optimal joint behavior strategy and behavior selection strategy of each intelligent agent are determined. This collaborative optimization method not only realizes the adaptive collaborative control between each intelligent agent, but also obtains the optimal traffic flow balance strategy for the entire road network. This helps to solve the problem of uneven capacity of ramps along the expressway and further improves the traffic efficiency of the road network.

[0077] It should be emphasized that the intelligent agent in the present invention, as an autonomous decision-making entity deployed on the road network, is configured to perceive the traffic status in real time and select, execute, track and optimize the received control decisions. This intelligent configuration not only improves the accuracy and response speed of traffic control, but also further improves the safety and traffic efficiency of high (fast) highways.

[0078] Therefore, the present invention achieves the optimization of regional road network performance, the optimal balance of overall road network traffic flow, and the improvement of traffic control accuracy and response speed by integrating the main line variable speed limit and traffic induction collaborative control technology, the main line and multi-ramp collaborative optimization framework, and intelligent intelligent body configuration. It further realizes the intelligentization, precision and efficiency of traffic control, and provides strong technical support for improving the overall traffic flow of high (fast) roads and reducing delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 A flowchart of a cloud-edge-end integrated collaborative digital traffic control method provided in an embodiment of the present invention;

[0080] Figure 2A specific flow chart of step S1 of a cloud-edge-end integrated collaborative digital traffic collaborative control method provided in an embodiment of the present invention;

[0081] Figure 3 A specific flow chart of step S14 of a cloud-edge-end integrated collaborative digital traffic collaborative control method provided in an embodiment of the present invention;

[0082] Figure 4 A specific flow chart of step S2 of a cloud-edge-end integrated collaborative digital traffic control method provided in an embodiment of the present invention;

[0083] Figure 5 A specific flow chart of step S26 of a cloud-edge-end integrated collaborative digital traffic collaborative control method provided in an embodiment of the present invention;

[0084] Figure 6 A specific flow chart of step S261 of a cloud-edge-end integrated collaborative digital traffic collaborative control method provided in an embodiment of the present invention;

[0085] Figure 7 A strategy-layer ramp multi-agent collaborative control process of a digital traffic collaborative management method with cloud-edge-end integrated collaboration is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0087] like Figure 1 As shown, an embodiment of the present invention proposes a cloud-edge-end integrated collaborative digital traffic collaborative control method, including: based on the constructed cloud-edge-end integrated multi-agent hierarchical control architecture, by driving the main line variable speed limit and traffic induction collaborative control, dynamically adjusting the main line speed limit and ramp control rate in different scenarios to optimize the performance of the regional road network; constructing and utilizing the main line and multi-ramp collaborative optimization framework under the hierarchical control architecture, extending the optimization control scheme of a single ramp to each partition of the road network under a random environment, determining the optimal joint behavior strategy and behavior selection strategy of each agent, so as to realize adaptive collaborative control of each agent, and obtain the optimal traffic flow balancing strategy for the overall road network; wherein, the agent is an autonomous decision-making entity deployed on the road network, configured to perceive the traffic status in real time, select, execute, track and optimize the received control decisions.

[0088] First, by driving the variable speed limit of the main line and the traffic induction collaborative control technology, the present invention can dynamically adjust the main line speed limit and ramp control rate in different scenarios. This flexibility not only helps to alleviate the frequent congestion on expressways, but also effectively responds to the deterioration of traffic conditions caused by emergencies. Through the "induction-control" double-layer optimization control strategy, the overall performance of the regional road network is optimized, thereby significantly improving the traffic efficiency of high (fast) expressways.

[0089] Secondly, under the hierarchical control architecture, the present invention constructs a main line and multi-ramp collaborative optimization framework, and successfully extends the optimization control scheme of a single ramp to each partition of the road network under a random environment. Thus, the optimal joint behavior strategy and behavior selection strategy of each intelligent agent are determined. This collaborative optimization method not only realizes the adaptive collaborative control between each intelligent agent, but also obtains the optimal traffic flow balance strategy for the entire road network. This helps to solve the problem of uneven capacity of ramps along the expressway and further improves the traffic efficiency of the road network.

[0090] It should be emphasized that the intelligent agent in the present invention, as an autonomous decision-making entity deployed on the road network, is configured to perceive the traffic status in real time and select, execute, track and optimize the received control decisions. This intelligent configuration not only improves the accuracy and response speed of traffic control, but also further improves the safety and traffic efficiency of high (fast) highways.

[0091] Therefore, the present invention achieves the optimization of regional road network performance, the optimal balance of overall road network traffic flow, and the improvement of traffic control accuracy and response speed by integrating the main line variable speed limit and traffic induction collaborative control technology, the main line and multi-ramp collaborative optimization framework, and intelligent intelligent body configuration. It further realizes the intelligentization, precision and efficiency of traffic control, and provides strong technical support for improving the overall traffic flow of high (fast) roads and reducing delays.

[0092] In order to better understand the above technical solution, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer and more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0093] Specifically, an embodiment of the present invention provides a cloud-edge-end integrated collaborative digital intelligent traffic collaborative control method, which includes:

[0094] S1. Based on the constructed cloud-edge-integrated multi-agent hierarchical control architecture, by driving the main line variable speed limit and traffic induction collaborative control, the main line speed limit and ramp control rate are dynamically adjusted in different scenarios to optimize the performance of the regional road network.

[0095] Furthermore, if Figure 2 As shown, step S1 includes:

[0096] S11. Build a cloud-edge integrated expressway traffic collaborative control system with a central cloud strategy generation module, an edge node collaborative optimization module, and a front-end equipment solution execution module.

[0097] S12. Based on the cloud-edge integrated expressway traffic collaborative control system, a multi-agent based hierarchical control architecture with a cloud strategy layer and an edge solution layer is constructed.

[0098] This step is to further build a more refined hierarchical control architecture based on the already established cloud-edge-end integrated system. This architecture includes two main layers:

[0099] Cloud strategy layer: This layer focuses on global strategy formulation and optimization, and achieves the optimal overall traffic efficiency of the expressway network through the collaborative work of multiple intelligent agents. It is responsible for generating macro control strategies and providing guidance for the next layer, the edge solution layer.

[0100] Edge solution layer: This layer focuses more on local and real-time optimization. Based on the guidance of the cloud strategy layer, it generates specific and real-time optimization solutions through intelligent agent search and optimization, and sends them to the front-end devices for execution. At the same time, it can also receive status feedback from the front-end devices to achieve closed-loop control.

[0101] S13. With the goal of optimizing the performance of the regional road network, the mainline and ramp collaborative control strategy is output through the cloud strategy layer.

[0102] S14. According to the main line and ramp coordinated control strategy of the cloud strategy layer, the edge solution layer is handed over to the intelligent agent to search and optimize, and an edge real-time optimization solution for the main line variable speed limit and ramp intelligent control coordination is generated and sent to the front-end device for execution, and the status feedback of the intelligent agent is received.

[0103] This step describes the workflow of the edge solution layer and its interaction with the cloud strategy layer and front-end devices. Based on the guidance of the cloud strategy layer, the edge solution layer uses the intelligent agent to search and optimize, and generates specific, real-time optimization solutions. These solutions include collaborative measures such as variable speed limits on the main line and intelligent ramp control, which are designed to make real-time adjustments and optimizations based on current traffic conditions. These optimization solutions are then sent to the front-end devices for execution, and the edge solution layer also receives status feedback from the intelligent agent to further adjust and optimize the solutions and achieve closed-loop control.

[0104] Furthermore, if Figure 3 As shown, step S14 includes:

[0105] S141. Divide the main line and the ramp into m control units with status information.

[0106] S142. Based on the mainline and ramp coordinated control strategy from the cloud strategy layer and the control unit with status information, a mainline variable speed limit and ramp coordinated control model is constructed with road traffic flow and vehicle delay as joint optimization control targets.

[0107] S143. Based on the main line variable speed limit and ramp coordinated control model, an intelligent agent is used in the edge solution layer to search for the optimal combination of the main line variable speed limit value and the entrance ramp control rate to maximize the value of the joint optimization control target.

[0108] S144. Based on the results of the intelligent agent search and optimization, generate an edge real-time optimization solution for the main line variable speed limit and ramp intelligent control collaboration.

[0109] S145. Send the generated optimization plan to the front-end device for execution, and monitor the execution effect of the optimization plan in real time.

[0110] In a specific embodiment, in order to optimize the road traffic flow (TTF) and reduce vehicle delay (TTD), a main line variable speed limit and ramp coordinated control model is constructed. In this model, the main line and ramp are first divided into m control units for more refined management.

[0111] The construction of the model revolves around the two joint optimization control objectives of TTF and TTD. TTF represents the road capacity, while TTD reflects the driving efficiency of vehicles on the road. In order to accurately describe these two objectives, multiple variables are introduced, including the traffic flow density, traffic flow speed, number of lanes of the ramp unit at time k, and the traffic flow density of the main road unit at time k. In addition, the ramp control rate and the section length of each control unit are also important parameters in the model.

[0112] When constructing the TTF objective function, special consideration is given to road traffic status indicators, and weights are assigned to the traffic flows on the main road and ramps based on these indicators. This is done to ensure that the model can more accurately reflect the dynamic changes in actual road traffic flows.

[0113] At the same time, the TTD model is also used to describe the traffic flow of the upstream main road and the ramp traffic flow. Through this model, the maximum benefit of road traffic flow can be calculated, thus providing strong support for optimization control.

[0114] In the main line variable speed limit and ramp coordinated control model, the control variables are set as the main line variable speed limit value and the entrance ramp control rate. By adjusting these two variables, effective regulation of road traffic flow can be achieved, thereby achieving the goal of optimizing TTF and reducing TTD:

[0115] Specifically, the main line variable speed limit and ramp coordinated control model is:

[0116]

[0117] In the formula, TTF is the road traffic flow model, TTD is the vehicle delay model, is the main road unit traffic flow density at time k, is the unit traffic flow speed of the main road at time k, λ m is the number of lanes on the main road at time k, is the k ramp control rate, is the traffic flow density of the ramp unit at time k, is the ramp unit traffic flow speed at time k, is the number of ramp lanes at time k, Δl m is the length of each control unit section, v f，m is the free flow velocity of the control unit m, r m is the number of unit ramps, and m is the number of unit main lines.

[0118] The objective function of the joint optimization control objective is: ω = αTTF-(1-α)TTD. When ω is the largest, it reaches the optimal value. α is the adjustment coefficient, and its value range is (0,1).

[0119] S35. Drive the main line variable speed limit and traffic induction collaborative control according to the edge real-time optimization solution. Dynamically adjust the main line speed limit and ramp control rate for different scenarios to optimize the performance of the regional road network.

[0120] In normal scenarios, the mainline speed limit and ramp control rate are continuously monitored and fine-tuned to maintain the smooth and efficient operation of the road network. When encountering special events, such as traffic accidents, large-scale events or bad weather, a more proactive control strategy is adopted, such as temporarily reducing the mainline speed limit, increasing ramp regulation, issuing emergency detour suggestions, etc., to restore the normal traffic capacity of the road network as quickly as possible and reduce the impact of the event on the overall traffic.

[0121] S2. Construct and utilize the mainline and multi-ramp collaborative optimization framework under the hierarchical control architecture, expand the optimization control scheme of a single ramp to various partitions of the road network under a random environment, determine the optimal joint behavior strategy and behavior selection strategy of each intelligent agent, so as to achieve adaptive collaborative control of each intelligent agent and obtain the optimal traffic flow balance strategy for the entire road network.

[0122] Furthermore, if Figure 4 As shown, step S2 includes:

[0123] S21. Each ramp signal in the road network is abstracted as an intelligent agent. These intelligent agents will serve as the basic units of decision-making and execution, responsible for monitoring and regulating the traffic flow of the ramp where they are located.

[0124] S22. Based on the hierarchical control architecture, the multi-agent behavior and evaluation strategy of the traffic network is introduced to construct a multi-agent reinforcement learning framework for the coordinated optimization of the main line and multiple ramps.

[0125] Based on the established "strategy-solution" hierarchical control architecture, the cooperative multi-agent actor-critic (CMAC) strategy for traffic network is further introduced. Through this method, a multi-agent reinforcement learning framework for the coordinated optimization of the main line and multiple ramps is constructed to achieve effective collaboration and information sharing among the agents. By defining the state space, action space and reward function of the agent, an environment can be provided for the agent to learn how to make the best decision based on the real-time traffic conditions.

[0126] S23. All agents in the multi-agent reinforcement learning framework are trained in a centralized way. This means that all agents are simulated and trained under a central controller, which provides global traffic status information and allows agents to exchange information and collaborate with each other. In this way, agents can learn how to coordinate actions in different traffic scenarios to achieve optimal performance of the overall road network.

[0127] S24. After the multi-agent reinforcement learning framework is trained, the learned strategies are deployed to each agent in a distributed execution manner, so that each agent can independently make decisions and perform corresponding control actions according to the real-time traffic flow status of the partition area.

[0128] After training is completed, the learned strategies are deployed to each agent in a distributed manner. This means that each agent can independently make decisions and perform corresponding control actions based on the real-time traffic flow status of its partition area. This distributed execution method improves the flexibility and responsiveness of the system, allowing the agent to adapt to changes in traffic conditions more quickly.

[0129] S25. Expand the optimization control scheme of a single ramp to various partitions of the road network under a random environment.

[0130] The reinforcement learning control of a single ramp is extended to the regional high-speed (express) road network partition in a random environment, in order to achieve the optimal balance strategy of the expressway network traffic flow through the linkage coordination between multiple ramp agents. Therefore, in the reinforcement learning framework for establishing the multi-agent integrated control of the ramp, its state, action and feedback not only consider the coordination of a single ramp, but also need to consider the influence of the corresponding elements of other ramps in the coordinated ramp group.

[0131] S26. Through the linkage and coordination between the intelligent agents in different partitions, the joint optimal behavior strategy and behavior selection strategy of each intelligent agent are determined, and the unique equilibrium in the random environment is found through repeated iterations to obtain the optimal equilibrium strategy for the traffic flow of the entire road network.

[0132] Through the linkage and coordination between the agents in different partitions, the joint optimal behavior strategy and behavior selection strategy of each agent are determined. This step involves information exchange and collaboration between agents to ensure that they can work together to achieve the optimal performance of the overall road network. Through repeated iterations and trial and error, the only equilibrium point in a random environment can be found, at which point the traffic flow of the overall road network reaches the optimal equilibrium state. This state corresponds to the highest traffic efficiency and the lowest congestion, thus achieving the optimization goal.

[0133] Furthermore, if Figure 5 As shown, step S26 includes:

[0134] S261. The D3QN algorithm is used to construct a reinforcement learning model of multiple ramp agents, in which the ramp agents in the same partition share traffic status and experience and have a common control goal.

[0135] Furthermore, if Figure 6 As shown, step S261 includes:

[0136] S2611. Use the D3QN algorithm to build a reinforcement learning model for multi-ramp agents, where the reinforcement learning model includes a main DQN network and a target DQN network.

[0137] S2612. Configure the ramp agents in the same partition to share traffic status and experience data, and assign common control targets to the ramp groups in the same partition.

[0138] S2613. Train the multi-ramp agents and select the control action with the maximum Q value in the main DQN network.

[0139] S2614, introduce the target DQN network, obtain the Q value of the control action in the target DQN network, minimize the target Q value fluctuation, and avoid the selection of non-optimal Q value.

[0140] S2615. Split the Q value of the state-behavior pair into two output parts: Value and Advantage. Value represents the value of the traffic environment state itself, and Advantage represents the additional value brought by dynamically selecting a certain control action.

[0141] S2616. Based on the training results, the control strategy of each ramp agent is updated. At each time step, each ramp agent makes decisions based on the current state and shared information using the updated strategy to select the optimal control action.

[0142] S2617. Continuously perform iterative training of the reinforcement learning model, monitor the expected value of the cumulative reward of the intelligent agent in the state according to the current control strategy during the training process, until the preset conditions are met, and obtain a trained reinforcement learning model.

[0143] S262. Replace the state and action of a single agent with the joint state and joint action in a dynamic random environment respectively.

[0144] S263. In each game strategy stage, according to the current joint state and joint action, each intelligent agent is coordinated through linkage, the value function of the equilibrium strategy is estimated, and the joint decision-making of the multi-agent reinforcement learning system is achieved.

[0145] S264. Determine the joint optimal behavior strategy and behavior selection strategy of each intelligent agent by repeatedly iterating and approximating the optimal strategy.

[0146] S265. According to the current joint state and joint action, select the control action and hand it over to the intelligent agent for execution, and continuously track and evaluate the control effect.

[0147] S266. Based on the goal of maximizing the cumulative return, continue to optimize until the optimal behavior strategy and behavior selection strategy converge to the optimal probability mapping of state and action, and finally obtain the optimal traffic flow equilibrium strategy for the entire road network.

[0148] In another specific embodiment, the present invention adopts the D3QN (Double Deep Q-Learning Network with Dueling Architecture) algorithm to implement the above control strategy, wherein two DQN networks are used to train the control strategy. On the basis of the control action with the maximum Q value selected by the main DQN network, a target network with the target Q value of the control strategy is added, and then the Q value of this control action in the target DQN network is obtained, with the goal of minimizing the fluctuation of the target Q value of the control strategy, thereby avoiding the non-optimal Q value of the control action selected by the main DQN network. At the same time, the D3QN algorithm splits the Q value of the state-behavior pair into two output parts, one part represents the value of the traffic environment state itself, called Value; the other part represents the additional value brought by the dynamic selection of a certain control action, called Advantage. If the Advantage value of the action is a positive number, it means that the value brought by the action is higher than the average value of other actions. The definition of Q value is as follows:

[0149] Q(s t ,a t )=V(s t )+A(a t );

[0150] In the formula, Q(s t ,a t ) is the state-behavior function, which is used to evaluate the expected return of adopting the strategy in the current state, s t Represents the current state, a t represents the action taken in the current state, t is the time step, V(s t ) is the value of the traffic environment state itself, A(a t ) is the additional value brought by dynamically selecting a certain control action, Q(s t ,a t ) is larger, the larger the expected value of the cumulative reward of the agent in state s by following the strategy.

[0151] The ramp group in the same partition is taken as the overall control object, and the traffic status and experience are shared between the ramps, and the same ramp group has a common control target. Under the above premise, a reinforcement learning model of multi-ramp agents is constructed, and the state and action of the single agent are replaced by the joint state and joint action in a dynamic random environment. In each game strategy stage, the value function of the equilibrium strategy is estimated to achieve the joint decision-making of the multi-agent reinforcement learning system. By repeatedly iterating and approximating the optimal strategy, the unique equilibrium of the system in a random environment is found. The expression of the optimal equilibrium strategy of the traffic flow of the overall road network is obtained as follows:

[0152]

[0153] Where, s={s1,…,s N} represents the joint state vector of N agents, including its own state and the states of other agents in the group, s k represents the joint state vector of N agents at time k, a={a1,…,a N} represents the joint action vector of N agents, including their own state-behavior pairs and each state-behavior pair of other agents, a k represents the joint action vector of N agents at time k, α k is the learning rate at time k, ranging from [0,1], r i k is the reward of agent i at time k, γV i k-1 is the state value of agent i at time k-1, V i (s) is the state value function of agent i in the joint state s, NE i is the Nash equilibrium strategy of agent i, π * is the optimal strategy, π is the strategy, π(a) is the probability that N agents choose a joint action, and A is the joint action space.

[0154] Next, refer to Figure 7 The strategy-level ramp multi-agent collaborative control process is given. The above model realizes the safety control of urban expressways through a multi-agent control system that integrates reinforcement learning and coordination mechanisms. Specifically, on the one hand, the model considers more the control needs of expressway safety control in the improved design of the reward function, and can automatically switch the corresponding coordination goals dynamically following the changes in the on-site safety system evaluation results. On the other hand, in the application, it makes full use of the effective data accumulated in the experience pool, regularly retrains and optimizes the Q-value network estimation of the expressway change characteristics, improves the algorithm coordination success rate and application efficiency, and makes full use of the online learning function to enable the Agent to respond quickly to changes in the traffic environment and changes in other Agents.

[0155] In addition, an embodiment of the present invention provides a cloud-edge-end integrated collaborative digital intelligent traffic collaborative control system, including:

[0156] The regional road network optimization module is used to dynamically adjust the main line speed limit and ramp control rate in different scenarios by driving the main line variable speed limit and traffic induction collaborative management based on the constructed cloud-edge-end integrated multi-agent hierarchical control architecture to optimize the performance of the regional road network.

[0157] The overall road network optimization module is used to construct and utilize the main line and multi-ramp collaborative optimization framework under the hierarchical control architecture, expand the optimization control scheme of a single ramp to various partitions of the road network under a random environment, determine the optimal joint behavior strategy and behavior selection strategy of each intelligent agent, so as to achieve adaptive collaborative control of each intelligent agent and obtain the optimal traffic flow balance strategy for the overall road network.

[0158] Among them, the intelligent agent is an autonomous decision-making entity deployed on the road network, which is configured to perceive the traffic status in real time, select, execute, track and optimize the received control decisions.

[0159] In addition, an embodiment of the present invention provides a cloud-edge-end integrated digital traffic collaborative control device, including: a central cloud, used to execute the method described above; a front-end device, used to collect multi-source fusion data in real time, and transmit it to the edge node for processing; an intelligent body, deployed at the edge node, receives policy guidance from the central cloud, generates real-time optimization solutions for main line variable speed limit and ramp intelligent control through search and optimization, and feeds back its own status to the central cloud during execution, to achieve dynamic selection, execution, tracking and optimization of strategies; an edge node, equipped with computing and storage resources, for receiving and processing data transmitted from the front-end device, and at the same time, according to the policy scheduling of the central cloud, collaborates with the intelligent body to complete the implementation of the strategy and the issuance of the solution, to achieve collaborative optimization of edge support for cloud applications and cloud assistance for edge localization needs.

[0160] It is important to understand that the central cloud, front-end devices, edge nodes and intelligent agents form a cloud-edge integrated expressway traffic collaborative control system with a central cloud strategy generation module, an edge node collaborative optimization module and a front-end device solution execution module.

[0161] Based on the cloud-edge integrated expressway traffic collaborative control system, a hierarchical control architecture based on multi-agent is formed with a cloud strategy layer and an edge solution layer.

[0162] Among them, the cloud strategy layer outputs data-driven mainline and ramp collaborative control strategies, and the edge solution layer generates edge real-time optimization solutions for mainline variable speed limits and ramp intelligent control collaboration through intelligent agent search and optimization, and feeds back the intelligent agent status to the central cloud to build a closed-loop computing architecture for cloud-edge collaboration.

[0163] Furthermore, an embodiment of the present invention provides a computer-readable medium on which computer executable instructions are stored. When the executable instructions are executed by the processor, the digital intelligent traffic collaborative control method for cloud-edge-end integrated collaboration as described above is implemented. What is stored on the medium is not just ordinary data, but an instruction set that can be recognized and executed by a computer processor. These instruction sets follow specific programming languages ​​and logic when they are written to ensure that they can complete the expected functions when they are executed.

[0164] The instruction set stored in it is designed to realize a cloud-edge-end integrated digital intelligent traffic collaborative control method. This means that when these instructions are executed by the processor, they can drive the entire traffic system to achieve efficient collaboration between the cloud, edge and device ends, and improve the intelligence and response speed of traffic management.

[0165] In summary, the embodiments of the present invention provide a cloud-edge-end integrated collaborative digital intelligent traffic control method, system, device and medium, which constitute a complete and efficient technical solution. In this technical system, the core content of high-speed road cloud-edge-end integrated collaborative control technology is particularly prominent, which mainly includes the following two key technologies:

[0166] First, the main line variable speed limit and traffic induction coordinated control technology aims to optimize the overall performance of the regional road network by implementing a "induction-control" double-layer optimization control strategy to address the local traffic deterioration caused by frequent congestion and emergencies on expressways. This technology can not only effectively alleviate traffic pressure, but also improve road traffic efficiency.

[0167] Secondly, the multi-ramp collaborative optimization control technology based on multi-agent reinforcement learning focuses on the problem of uneven ramp capacity along the expressway. It adopts a coordinated strategy of centralized training and distributed execution, and through the autonomous learning and collaborative decision-making of the agents, it significantly improves the traffic efficiency of the overall network and achieves a balanced distribution of traffic flow.

[0168] These two key technologies work closely together during implementation to build a seamless and organically integrated technology system. This system not only strengthens the path guidance and safety warning capabilities of the "car", but also improves the efficiency of congestion diagnosis and treatment and accident handling of the "road", while enhancing the road network control and collaborative optimization capabilities of the "cloud", providing solid technical support for high-speed (express) road traffic control business.

[0169] From the technical details, a "strategy-scheme" hierarchical control architecture for the expressway network based on multi-agents was constructed, through which the mainline and ramp collaborative control strategy can be obtained. On this basis, event-driven high (fast) highway variable speed limit and induction collaborative control technology was studied to ensure effective induction and collaborative control of the mainline in both conventional and special event scenarios. At the same time, a breakthrough was made in the multi-ramp collaborative optimization control technology based on multi-agent reinforcement learning, so that the overall traffic efficiency is optimized.

[0170] Obviously, this technical system is guided by the demand for precise control of highway (high-speed) road traffic events, and is built around the three major elements of vehicle, road, and cloud. Among them, the high-speed (high-speed) road stereoscopic perception and vehicle-road-cloud integrated collaborative control platform is the core basic platform, which closely connects various types of cloud-edge stereoscopic perception hardware equipment and smart high-speed (high-speed) road software services, providing a foothold and implementation support for key technologies for smart traffic control on high-speed (high-speed) roads. This platform not only provides rich digital, panoramic traffic management information services to end users such as drivers and on-duty traffic police, but also improves the immediacy of data processing and the speed of algorithm response through edge hardware devices such as edge smart stations, reduces the amount of data transmitted over the network, shares the cloud computing load, and ensures network and data security.

[0171] Since the system / device described in the above embodiments of the present invention is a system / device used to implement the method of the above embodiments of the present invention, a person skilled in the art can understand the specific structure and deformation of the system / device based on the method described in the above embodiments of the present invention, and thus will not be described in detail here. All systems / devices used in the method of the above embodiments of the present invention belong to the scope of protection of the present invention.

[0172] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions.

[0174] It should be noted that in the claims, any reference numerals placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In the claims enumerating several means, several of these means may be embodied by the same hardware. The use of the words first, second, third, etc., is for convenience of expression only and does not indicate any order. These words may be understood as part of the component name.

[0175] In addition, it should be noted that, in the description of this specification, the description of the terms "one embodiment", "some embodiments", "embodiment", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0176] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments after knowing the basic creative concept. Therefore, the claims should be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0177] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention should also include these modifications and variations.

Claims

1. A cloud-edge-end integrated digital intelligent traffic collaborative control method, characterized in that: include: Based on the cloud-edge-end integrated multi-agent hierarchical control architecture, the main line speed limit and ramp control rate are dynamically adjusted in different scenarios by driving the main line variable speed limit and traffic induction collaborative control to optimize the performance of the regional road network. Under the hierarchical control architecture, a mainline and multi-ramp collaborative optimization framework is constructed and utilized to extend the optimization control scheme of a single ramp to various partitions of the road network under a random environment, determine the optimal joint behavior strategy and behavior selection strategy of each intelligent agent, so as to achieve adaptive collaborative control of each intelligent agent and obtain the optimal traffic flow balance strategy of the entire road network; Among them, the intelligent agent is an autonomous decision-making entity deployed on the road network, which is configured to perceive the traffic status in real time, select, execute, track and optimize the received control decisions.

2. The cloud-edge-end integrated collaborative digital intelligent traffic collaborative control method according to claim 1 is characterized in that: Based on the cloud-edge-end integrated multi-agent hierarchical control architecture, the main line speed limit and ramp control rate are dynamically adjusted in different scenarios by driving the main line variable speed limit and traffic induction collaborative control to optimize the performance of the regional road network, including: Build a cloud-edge-end integrated expressway traffic collaborative control system with a central cloud strategy generation module, an edge node collaborative optimization module, and a front-end device solution execution module; Based on the cloud-edge-device integrated expressway traffic collaborative control system, a multi-agent based hierarchical control architecture with a cloud strategy layer and an edge solution layer is constructed; Aiming at optimizing the performance of the regional road network, the cloud-based strategy layer outputs the mainline and ramp coordinated control strategy; According to the main line and ramp coordinated control strategy, the intelligent agent searches for the best through the edge solution layer, generates an edge real-time optimization solution for the main line variable speed limit and ramp intelligent control coordination, sends it to the front-end device for execution, and receives the status feedback of the intelligent agent; The variable speed limit and traffic induction coordinated control of the main line are driven according to the edge real-time optimization solution. The main line speed limit and ramp control rate are dynamically adjusted for different scenarios to optimize the performance of the regional road network.

3. The cloud-edge-end integrated collaborative digital intelligent traffic control method according to claim 2 is characterized in that: According to the main line and ramp coordinated control strategy, the intelligent agent searches for the best through the edge solution layer, generates an edge real-time optimization solution for the main line variable speed limit and ramp intelligent control coordination, and sends it to the front-end device for execution, including: Divide the main line and ramps into multiple control units with status information; According to the mainline and ramp coordinated control strategy from the cloud strategy layer and the control unit with status information, the mainline variable speed limit and ramp coordinated control model is constructed with road traffic flow and vehicle delay as the joint optimization control target; Based on the mainline variable speed limit and ramp coordinated control model, the intelligent agent is used in the edge solution layer to search for the optimal combination of the mainline variable speed limit value and the entrance ramp control rate to maximize the value of the joint optimization control objective. Based on the results of the intelligent agent search and optimization, an edge real-time optimization solution for the main line variable speed limit and ramp intelligent control collaboration is generated; Send the generated optimization plan to the front-end device for execution, and monitor the execution effect of the optimization plan in real time; Among them, the main line variable speed limit and ramp coordinated control model is: In the formula, TTF is the road traffic flow model, TTD is the vehicle delay model, is the main road unit traffic flow density at time k, is the unit traffic flow speed of the main road at time k, λ m is the number of lanes on the main road at time k, is the k ramp control rate, is the traffic flow density of the ramp unit at time k, is the ramp unit traffic flow speed at time k, is the number of ramp lanes at time k, Δl m is the length of each control unit section, v f，m is the free flow velocity of the control unit m, r m is the number of unit ramps, m is the number of unit main lines; The objective function of the joint optimization control objective is: ω = αTTF-(1-α)TTD. When ω is the largest, it reaches the optimal value. α is the adjustment coefficient, and its value range is (0,1).

4. The cloud-edge-end integrated collaborative digital intelligent traffic collaborative control method according to claim 1 is characterized in that: Under the hierarchical control architecture, a mainline and multi-ramp collaborative optimization framework is constructed and utilized to extend the optimization control scheme of a single ramp to various partitions of the road network under a random environment, determine the optimal joint behavior strategy and behavior selection strategy of each agent, and realize adaptive collaborative control of each agent. The optimal traffic flow balance strategy of the overall road network is obtained, including: Abstract each ramp signal in the road network as an intelligent agent; Based on the hierarchical control architecture, the multi-agent behavior and evaluation strategy of the traffic network is introduced to build a multi-agent reinforcement learning framework for the coordinated optimization of the main line and multiple ramps; Train all agents in the multi-agent reinforcement learning framework through centralized training; After the multi-agent reinforcement learning framework is trained, the learned strategies are deployed to each agent in a distributed execution mode, so that each agent can make decisions and perform corresponding control actions independently according to the real-time traffic flow status of the partition area. Extend the optimization control scheme of a single ramp to various sections of the road network in a random environment; Through the linkage and coordination between the various intelligent agents in different partitions, the joint optimal behavior strategy and behavior selection strategy of each intelligent agent are determined, and through repeated iterations, the unique equilibrium in the random environment is found to obtain the optimal equilibrium strategy for the traffic flow of the entire road network.

5. The cloud-edge-end integrated collaborative digital intelligent traffic collaborative control method according to claim 4 is characterized in that: Through the linkage and coordination between the agents in different partitions, the joint optimal behavior strategy and behavior selection strategy of each agent are determined, and the unique equilibrium in the random environment is found through repeated iterations. The optimal equilibrium strategy for the traffic flow of the overall road network includes: The D3QN algorithm is used to build a reinforcement learning model of multi-ramp agents, in which the ramp agents in the same partition share traffic status and experience and have a common control goal; Replace the states and actions of a single agent with joint states and joint actions in a dynamic random environment respectively; In each game strategy stage, according to the current joint state and joint action, through linkage and coordination of various intelligent agents, the value function of the equilibrium strategy is estimated to achieve joint decision-making of the multi-agent reinforcement learning system; By repeatedly iterating and approximating the optimal strategy, the joint optimal behavior strategy and behavior selection strategy of each agent are determined; According to the current joint state and joint action, the control action is selected and handed over to the agent for execution, and the control effect is continuously tracked and evaluated; According to the goal of maximizing the cumulative return, the optimization is continued until the optimal behavior strategy and behavior selection strategy converge to the optimal probability mapping of state and action, and finally the optimal equilibrium strategy of traffic flow of the entire road network is obtained; Among them, the expression of the optimal balance strategy of traffic flow in the overall road network is: Where, s={s1,…,s N } represents the joint state vector of N agents, including its own state and the states of other agents in the group, s k represents the joint state vector of N agents at time k, a={a1,…,a N } represents the joint action vector of N agents, including their own state-behavior pairs and each state-behavior pair of other agents, a k represents the joint action vector of N agents at time k, α k is the learning rate at time k, ranging from [0,1], is the reward of agent i at time k, is the state value of agent i at time k-1, V i (s) is the state value function of agent i in the joint state s, NE i is the Nash equilibrium strategy of agent i, π * is the optimal strategy, π is the strategy, π(a) is the probability that N agents choose a joint action, and A is the joint action space.

6. The cloud-edge-end integrated digital intelligent traffic collaborative control method according to claim 5 is characterized in that: The D3QN algorithm is used to build a reinforcement learning model for multiple ramp agents, in which the ramp agents in the same partition share traffic status and experience and have common control objectives including: The D3QN algorithm is used to build a reinforcement learning model of multi-ramp agents, where the reinforcement learning model includes a main DQN network and a target DQN network; Configure the ramp agents in the same partition to share traffic status and experience data, and assign common control targets to the ramp groups in the same partition; Train multiple ramp agents and select the control action with the maximum Q value in the main DQN network; Introduce the target DQN network to obtain the Q value of the control action in the target DQN network, with the goal of minimizing the target Q value fluctuation and avoiding the selection of non-optimal Q values; The Q value of the state-behavior pair is split into two output parts: Value and Advantage. Value represents the value of the traffic environment state itself, and Advantage represents the additional value brought by dynamically selecting a certain control action. According to the training results, the control strategy of each ramp agent is updated. At each time step, each ramp agent uses the updated strategy to make decisions based on the current state and shared information and select the optimal control action; Continuously perform iterative training of the reinforcement learning model, monitor the expected value of the cumulative reward of the agent in the training process under the current control strategy, until the preset conditions are met, and obtain a trained reinforcement learning model; The Q value is: Q(s t ,a t )=V(s t )+A(a t ); In the formula, Q(s t ,a t ) is the state-behavior function, which is used to evaluate the expected return of adopting the strategy in the current state, s t Represents the current state, a t represents the action taken in the current state, t is the time step, V(s t ) is the value of the traffic environment state itself, A(a t ) is the additional value brought by dynamically selecting a certain control action, Q(s t ,a t ) is larger, the larger the expected value of the cumulative reward of the agent in state s by following the strategy.

7. A cloud-edge-end integrated digital intelligent traffic collaborative control system, characterized in that: include: The regional road network optimization module is used to dynamically adjust the main line speed limit and ramp control rate in different scenarios based on the cloud-edge-end integrated multi-agent hierarchical control architecture to optimize the performance of the regional road network by driving the main line variable speed limit and traffic induction collaborative control; The overall road network optimization module is used to build and utilize the mainline and multi-ramp collaborative optimization framework under the hierarchical control architecture, expand the optimization control scheme of a single ramp to various partitions of the road network under a random environment, determine the optimal joint behavior strategy and behavior selection strategy of each intelligent agent, so as to achieve adaptive collaborative control of each intelligent agent and obtain the optimal traffic flow balance strategy of the overall road network; Among them, the intelligent agent is an autonomous decision-making entity deployed on the road network, which is configured to perceive the traffic status in real time, select, execute, track and optimize the received control decisions.

8. A cloud-edge-end integrated digital intelligent traffic collaborative control device, characterized in that: include: A central cloud, used to execute the method according to any one of claims 1 to 6; Front-end equipment, used to collect multi-source fusion data in real time and transmit it to edge nodes for processing; The intelligent agent is deployed at the edge node and receives policy guidance from the central cloud. It generates real-time optimization solutions for variable speed limits on the main line and intelligent ramp control through search and optimization. During the execution process, it feeds back its own status to the central cloud to achieve dynamic selection, execution, tracking and optimization of strategies. Edge nodes are equipped with computing and storage resources to receive and process data transmitted from front-end devices. At the same time, they coordinate with intelligent agents to implement strategies and issue plans based on the policy scheduling of the central cloud, thereby achieving collaborative optimization of edge support for cloud applications and cloud support for edge localization needs.

9. The cloud-edge-end integrated digital intelligent traffic collaborative control device according to claim 8 is characterized in that: The central cloud, front-end devices, edge nodes and intelligent agents form a cloud-edge integrated expressway traffic collaborative control system with a central cloud strategy generation module, an edge node collaborative optimization module and a front-end device solution execution module; Based on the cloud-edge-end integrated expressway traffic collaborative control system, a hierarchical control architecture with a cloud strategy layer and an edge solution layer based on multi-agents is formed; Among them, the cloud strategy layer outputs data-driven mainline and ramp collaborative control strategies, and the edge solution layer generates edge real-time optimization solutions for mainline variable speed limits and ramp intelligent control collaboration through intelligent agent search and optimization, and feeds back the intelligent agent status to the central cloud to build a closed-loop computing architecture for cloud-edge collaboration.

10. A computer-readable medium having computer-executable instructions stored thereon, characterized in that: When the executable instructions are executed by the processor, the cloud-edge-end integrated collaborative digital traffic control method as described in any one of claims 1-6 is implemented.

Images