A vehicle-road cooperation strategy deployment method and device

Abstract

The application discloses a kind of vehicle-road cooperation strategy deployment method and device, the method includes: based on vehicle-road cooperation perception data, constructs digital twin simulation model;Based on the intelligent agent associated with each road side unit and each intelligent vehicle, construct multi-agent system;In digital twin simulation model, the multi-agent system is reinforced learning training, and the strategy network of each intelligent agent is obtained;Each strategy network is deployed to each road side unit and each intelligent vehicle to carry out real-time resource scheduling.The method provides a zero-risk training environment for multi-agent reinforcement learning through digital twin simulation model, changes vehicle-road cooperation resource scheduling from a static mode that depends on pre-set rules to a decision-making mode of autonomous learning and dynamic collaboration of each intelligent agent, significantly improving the decision-making quality and operational efficiency in complex traffic scenarios.

Description

Technical Field

[0001] This invention relates to the field of vehicle-road cooperative technology, and in particular to a method and apparatus for deploying vehicle-road cooperative strategies. Background Technology

[0002] Currently, with the acceleration of urbanization and the continuous growth of vehicle ownership, problems such as traffic congestion and frequent accidents are becoming increasingly prominent. As an important component of intelligent transportation, vehicle-road cooperative systems have become a key means to improve traffic efficiency through data interaction and collaborative decision-making between roadside units (RSUs), vehicles and the cloud.

[0003] In vehicle-road cooperative systems, data fusion strategies and the scheduling of communication / computing resources directly impact the system's real-time performance and decision-making quality. Existing solutions often rely on pre-defined data fusion methods and resource allocation rules based on expert experience, representing static or simple rule-based scheduling mechanisms. This approach struggles to dynamically adapt to real-time traffic conditions in large-scale, highly random urban traffic scenarios and lacks the ability to learn and optimize autonomously from historical data.

[0004] Furthermore, while some studies have attempted to introduce artificial intelligence algorithms into transportation systems, these have mostly been applied to isolated, specific tasks, lacking a unified framework capable of simulating, evaluating, and collaboratively optimizing the behavior of the entire vehicle-infrastructure system in a safe environment. Directly testing new scheduling strategies in real-world traffic environments carries high trial-and-error costs and safety risks. Summary of the Invention

[0005] To enable vehicle-road cooperative systems to have autonomous learning and dynamic decision-making capabilities, this invention provides a method and apparatus for deploying vehicle-road cooperative strategies.

[0006] In a first aspect, embodiments of the present invention provide a method for deploying vehicle-road cooperative strategies, which may include: A digital twin simulation model is constructed based on vehicle-road cooperative perception data; A multi-agent system is constructed based on the agents associated with each roadside unit and each intelligent vehicle. The multi-agent system is trained using reinforcement learning in the digital twin simulation model to obtain the policy network of each agent. The aforementioned strategy networks are deployed to each roadside unit and each intelligent vehicle for real-time resource scheduling.

[0007] In one or more optional embodiments of this application, the step of performing reinforcement learning training on the multi-agent system in the digital twin simulation model to obtain the policy network of each agent includes: Determine the initial policy network for each of the agents; Based on the digital twin simulation model, the initial policy network of each agent is subjected to multiple rounds of reinforcement learning training until a preset stopping condition is reached, thereby obtaining the policy network of each agent. The reinforcement learning training operations include: Based on the digital twin simulation model and the initial policy network of each agent, the execution actions of each agent are determined; Based on the actions performed by each of the intelligent agents, the digital twin simulation model is updated to obtain the updated digital twin simulation model. The reward value is obtained based on the updated digital twin simulation model. The parameters of the initial policy network of each agent are optimized based on the reward value, and the initial policy network of each agent after parameter adjustment is used as the initial policy network of the corresponding agent in the next round of reinforcement learning training.

[0008] In one or more optional embodiments of this application, obtaining the reward value based on the updated digital twin simulation model includes: Based on the updated digital twin simulation model, the average vehicle speed increase, congestion reduction, communication latency, and excessive consumption of computing resources are obtained. The reward value is obtained by weighted summing of the average vehicle speed increase, the congestion reduction, the communication latency, and the excessive consumption of computing resources.

[0009] In one or more optional embodiments of this application, determining the execution actions of each agent based on the digital twin simulation model and the initial policy network of each agent includes: Based on the digital twin simulation model, the state space of each of the intelligent agents is determined; For each agent, the agent's execution action is obtained based on the agent's state space and the agent's initial policy network.

[0010] In one or more optional embodiments of this application, determining the state space of each intelligent agent based on the digital twin simulation model includes: For each of the aforementioned intelligent agents, if the intelligent agent is associated with any roadside unit, then based on the digital twin simulation model, the local traffic density, average vehicle speed, queue length, load, and communication link quality of the area under the jurisdiction of the roadside unit are determined; The state space of the agent is defined by at least one of the local traffic density, average vehicle speed, queue length, load, and communication link quality within the area governed by the roadside unit.

[0011] In one or more optional embodiments of this application, obtaining the agent's execution action based on the agent's state space and the agent's initial policy network includes: If the agent is associated with any roadside unit, the state space of the agent is input into the initial policy network of the agent to obtain each candidate action and its corresponding probability; wherein, the candidate action includes adjusting the data upload priority, selecting the data fusion algorithm, and allocating local computing resources; The agent's action is determined based on each candidate action and its corresponding probability.

[0012] In one or more optional embodiments of this application, determining the state space of each intelligent agent based on the digital twin simulation model includes: For each of the aforementioned intelligent agents, if the intelligent agent is associated with any intelligent vehicle, then based on the digital twin simulation model, the location, driving speed, surrounding traffic environment, and on-board computing load of the intelligent vehicle are determined. The intelligent vehicle's location, driving speed, surrounding traffic environment, and onboard computing load are used as the state space of the intelligent agent.

[0013] In one or more optional embodiments of this application, obtaining the agent's execution action based on the agent's state space and the agent's initial policy network includes: If the agent is associated with any intelligent vehicle, the state space of the agent is input into the initial policy network of the agent to obtain each candidate action and its corresponding probability; wherein, the candidate actions include adjusting driving speed and selecting vehicle perception data; The agent's action is determined based on each candidate action and its corresponding probability.

[0014] In a second aspect, embodiments of the present invention provide a vehicle-road cooperative strategy deployment device, which may include: The model building module is used to build digital twin simulation models based on vehicle-road cooperative perception data; The system construction module is used to build a multi-agent system based on the agents associated with each roadside unit and each intelligent vehicle; The training module is used to perform reinforcement learning training on the multi-agent system in the digital twin simulation model to obtain the policy network of each agent. The deployment module is used to deploy each of the aforementioned policy networks to each roadside unit and each intelligent vehicle for real-time resource scheduling.

[0015] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the vehicle-road cooperative strategy deployment method as described above.

[0016] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the vehicle-road cooperative strategy deployment method as described above.

[0017] Fifthly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the vehicle-road cooperative strategy deployment method as described above.

[0018] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: This invention provides a method for deploying vehicle-road cooperative strategies. This method constructs a digital twin simulation model, providing a zero-risk simulation training environment synchronized with the real physical world for deploying vehicle-road cooperative strategies. This avoids the safety risks and trial-and-error costs associated with directly testing new strategies in real traffic systems. A multi-agent system is constructed based on agents associated with roadside units and intelligent vehicles. The policy network of each agent is obtained through reinforcement learning training, transforming the static resource scheduling that originally relied on preset rules into a collaborative scheduling mechanism where each agent learns autonomously and makes dynamic decisions. The trained policy network is deployed to the corresponding roadside units and intelligent vehicles in the physical world, enabling dynamic adjustments to data fusion strategies and communication / computing resource allocation based on traffic conditions during real-time operation. This significantly improves the decision-making quality and operational efficiency of the vehicle-road cooperative system in complex and ever-changing traffic scenarios.

[0019] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 A flowchart illustrating the vehicle-road cooperative strategy deployment method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the vehicle-road cooperative strategy deployment device provided in an embodiment of the present invention. Detailed Implementation

[0022] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0023] The inventors discovered that in existing technologies, data fusion methods and resource allocation rules are mostly pre-set by expert experience, belonging to static or simple rule-based scheduling mechanisms. This approach struggles to dynamically adapt to real-time traffic conditions when facing large-scale, highly random urban traffic scenarios, and lacks the ability to autonomously learn and optimize from historical data. Furthermore, while some research has attempted to introduce artificial intelligence algorithms into traffic systems, these are mostly used for isolated, specific tasks, lacking a unified framework capable of simulating, evaluating, and collaboratively optimizing the behavior of the entire vehicle-road system in a safe environment. Directly testing new scheduling strategies in real traffic environments incurs high trial-and-error costs and safety risks. Based on this, the inventors, through further research and development, created this invention, providing a method and apparatus for deploying vehicle-road cooperative strategies.

[0024] Example 1 Embodiment 1 of the present invention provides a method for deploying vehicle-road cooperative strategies, referring to... Figure 1 As shown, the method may include the following steps S101-S104: S101: Construct a digital twin simulation model based on vehicle-road cooperative perception data.

[0025] S102: Construct a multi-agent system based on the agents associated with each roadside unit and each intelligent vehicle.

[0026] S103: Reinforcement learning training is performed on the multi-agent system in the digital twin simulation model to obtain the policy network of each agent.

[0027] S104: Deploy various policy networks to various roadside units and intelligent vehicles for real-time resource scheduling.

[0028] This invention provides a method for deploying vehicle-road cooperative strategies. This method constructs a digital twin simulation model, providing a zero-risk simulation training environment synchronized with the real physical world for deploying vehicle-road cooperative strategies. This avoids the safety risks and trial-and-error costs associated with directly testing new strategies in real traffic systems. A multi-agent system is constructed based on agents associated with roadside units and intelligent vehicles. The policy network of each agent is obtained through reinforcement learning training, transforming the static resource scheduling that originally relied on preset rules into a collaborative scheduling mechanism where each agent learns autonomously and makes dynamic decisions. The trained policy network is deployed to the corresponding roadside units and intelligent vehicles in the physical world, enabling dynamic adjustments to data fusion strategies and communication / computing resource allocation based on traffic conditions during real-time operation. This significantly improves the decision-making quality and operational efficiency of the vehicle-road cooperative system in complex and ever-changing traffic scenarios.

[0029] In step S101 above, a digital twin simulation model is constructed based on vehicle-road cooperative perception data.

[0030] Specifically, it can be to access vehicle-road cooperative perception data in real time from the physical world. Vehicle-road cooperative perception data can include streaming data collected by roadside cameras or lidar, vehicle global positioning system (GPS) data or controller area network (CAN) bus data, traffic signal status data, and Internet traffic data, etc.

[0031] Based on vehicle-road cooperative perception data, a digital twin simulation model corresponding to the physical traffic environment is constructed in virtual space. This model includes digital mirrors of roads, vehicles, roadside facilities, and traffic participants. Simultaneously, the digital twin simulation model can be updated synchronously with the physical world according to a preset simulation step size (e.g., 100 milliseconds), enabling the virtual environment to realistically reflect the microscopic state of real-world traffic and providing a high-fidelity simulation environment for subsequent reinforcement learning training.

[0032] In step S102 above, a multi-agent system is constructed based on the agents associated with each roadside unit and each intelligent vehicle.

[0033] Specifically, this can be achieved by associating a corresponding agent with each roadside unit and each intelligent vehicle in the physical world. Here, an agent is an independent decision-making unit within a reinforcement learning framework, capable of autonomously selecting and executing actions based on observed environmental states. Each agent possesses an independent state space, action space, and policy network. In this method, the agent associated with each roadside unit is responsible for making decisions regarding data fusion and resource scheduling within the jurisdiction of that roadside unit, while the agent associated with each intelligent vehicle is responsible for making decisions regarding the preprocessing of its own perception data and its driving behavior.

[0034] Combining the agents associated with each roadside unit and the agents associated with each intelligent vehicle constitutes a multi-agent system. A multi-agent system refers to a group of agents that share the same operating environment (i.e., a digital twin simulation model), and their actions influence each other. Unlike individual agents making independent decisions, agents in a multi-agent system need to learn cooperative strategies while considering the behaviors of other agents to achieve global optimum.

[0035] It should be noted that step S102 above specifically refers to intelligent vehicles, not all vehicles. Intelligent vehicles are those with onboard communication and computing capabilities, capable of receiving and executing actions output by their associated intelligent agents, and thus able to participate in collaborative decision-making as members of a multi-agent system. Ordinary vehicles on the road that lack these capabilities are observed and predicted as environmental elements in the digital twin simulation model and the physical world, but do not participate in decision-making as intelligent agents in a multi-agent system.

[0036] In step S103 above, reinforcement learning training is performed on the multi-agent system in the digital twin simulation model to obtain the policy network of each agent. Specifically, this includes the following steps S1031-S1032: S1031: Determine the initial policy network for each agent.

[0037] Specifically, this could involve establishing an initial policy network for each agent in a multi-agent system. The policy network is the core component used by the agent to make decisions based on the environmental state; its internal parameters determine which actions the agent is inclined to perform in a given state.

[0038] The specific structure of the initial policy network can be selected according to actual needs, such as using network structures suitable for sequential decision-making tasks, such as Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), or Transformer.

[0039] The parameters of the initial policy network can be determined either through random initialization or by loading parameters from a pre-trained model. The initial policy network structure for each agent matches its corresponding state space dimension and action space dimension.

[0040] S1032: Based on the digital twin simulation model, perform multiple rounds of reinforcement learning training on the initial policy network of each agent until the preset stopping condition is reached, and obtain the policy network of each agent.

[0041] Specifically, it can be that each initial policy network determined in step S1031 is connected to the digital twin simulation model, so that each initial policy network can make decisions based on the environmental state characteristics provided by the digital twin simulation model, and apply the execution actions generated by the decisions to the digital twin simulation model.

[0042] Through multiple rounds of iterative reinforcement learning training, each agent gradually learns and optimizes its corresponding initial policy network through repeated trial and error.

[0043] In each round of reinforcement learning training, each agent adjusts the parameters of its initial policy network based on feedback from the digital twin simulation model, enabling it to make better decisions in the same or similar environments. As the number of training rounds increases, the actions output by each initial policy network gradually approach their optimal values, and training terminates when a preset stopping condition is met. The preset stopping condition can be that the change in reward value over multiple consecutive training rounds is less than a preset threshold, or that the maximum number of training rounds has been reached.

[0044] When training terminates, the initial policy network parameters of each agent are used as the final result to obtain the policy network of each agent after training is completed.

[0045] The reinforcement learning training operation includes the following steps S10321-S10324: S10321: Based on the digital twin simulation model and the initial policy network of each agent, determine the actions to be performed by each agent. Specifically, this includes the following steps S103211-S103212: S103211: Based on the digital twin simulation model, determine the state space of each agent. This specifically includes the following steps: S1032111-S1032114: S1032111: For each agent, if the agent is associated with any roadside unit, then based on the digital twin simulation model, determine the local traffic density, average vehicle speed, queue length, load and communication link quality of the area under the jurisdiction of the roadside unit.

[0046] Specifically, for an agent associated with a roadside unit, real-time traffic status data and equipment operation status data within the jurisdiction of the roadside unit can be obtained from the digital twin simulation model, including local traffic density, average vehicle speed, queue length, load, and communication link quality.

[0047] Among these, local traffic density refers to the number of vehicles per unit road segment within the coverage area of ​​the roadside unit; average vehicle speed refers to the average speed of all vehicles within the coverage area; queue length refers to the total length of vehicles queuing in each lane; load refers to the utilization rate of the roadside unit's own computing resources; and communication link quality refers to one or more of the following: signal strength, delay, or bandwidth of wireless communication between the roadside unit and vehicles within its coverage area. All these data are synchronously generated by the digital twin simulation model based on real-time vehicle-road cooperative sensing data accessed from the physical world.

[0048] S1032112: Use at least one of the following factors in the jurisdiction of the roadside unit: local traffic density, average vehicle speed, queue length, load, and communication link quality as the state space of the agent.

[0049] Specifically, one or more of the local traffic density, average vehicle speed, queue length, load, and communication link quality obtained in step S1032111 can be combined into the state space of the agent in the current simulation time slot.

[0050] The state space defines the range of environmental information that the agent bases on when making decisions. For roadside units, the state space is the aforementioned local traffic density, average vehicle speed, queue length, load, and communication link quality, providing sufficient environmental information for collaborative decision-making in data fusion and resource scheduling.

[0051] S1032113: For each intelligent agent, if the intelligent agent is associated with any intelligent vehicle, then based on the digital twin simulation model, determine the location, speed, surrounding traffic environment, and onboard computing load of the intelligent vehicle.

[0052] Specifically, for intelligent agents associated with intelligent vehicles, the driving status data, surrounding environment data, and on-board equipment status data of the intelligent vehicle itself can be obtained from the digital twin simulation model.

[0053] Here, location refers to the real-time coordinates of the intelligent vehicle in the digital twin simulation model; driving speed refers to the current speed of the intelligent vehicle; surrounding traffic environment refers to the number of other vehicles, their relative distances, and road conditions within a preset range around the intelligent vehicle; and onboard computing load refers to the current utilization rate of the intelligent vehicle's onboard computing unit. These data are synchronously generated by the digital twin simulation model based on the vehicle-road cooperative perception data uploaded in real-time by the physical vehicle corresponding to the intelligent vehicle.

[0054] S1032114: Use at least one of the following as the state space of the intelligent vehicle: its location, speed, surrounding traffic environment, and onboard computing load:

[0055] Specifically, one or more of the location, driving speed, surrounding traffic environment, and on-board computing load obtained in step S1032113 can be combined into the state space of the intelligent agent in the current simulation time slot.

[0056] The state space defines the range of environmental information that an intelligent agent bases on when making decisions. For intelligent vehicles, the state space includes location, driving speed, surrounding traffic environment, and onboard computing load, providing sufficient environmental information for making decisions on the selection of perception data preprocessing methods and the adjustment of driving speed.

[0057] S103212: For each agent, based on the agent's state space and initial policy network, the agent's execution action is obtained. This specifically includes the following steps S1032121-S1032122: S1032121: If the agent is associated with any roadside unit, input the agent's state space into the agent's initial policy network to obtain each candidate action and its corresponding probability. Based on each candidate action and its corresponding probability, determine the agent's action. Candidate actions include adjusting data upload priority, selecting a data fusion algorithm, and allocating local computing resources.

[0058] Specifically, for an agent associated with a roadside unit, the state space determined in step S1032112 is used as input and fed into the agent's initial policy network. The initial policy network performs inference based on the input state space and outputs the probability value corresponding to each candidate action in the agent's action space.

[0059] The candidate actions in the action space include adjusting data upload priority, selecting a data fusion algorithm, and allocating local computing resources. Adjusting data upload priority refers to the roadside unit dynamically setting the data upload order or frequency of each vehicle within its jurisdiction. Selecting a data fusion algorithm refers to the roadside unit choosing the target fusion algorithm for the current time slot from a set of preset algorithms. Allocating local computing resources refers to the roadside unit allocating its computing power to different data processing tasks.

[0060] Based on the probability corresponding to each candidate action, the action to be executed by the agent in the current simulation time slot is determined from the candidate actions according to a preset rule. For example, the candidate action with the highest probability value is selected as the action to be executed, or random sampling is performed according to the probability distribution to determine the action to be executed.

[0061] S1032122: If the agent is associated with any intelligent vehicle, input the agent's state space into the agent's initial policy network to obtain each candidate action and its corresponding probability. Based on each candidate action and its corresponding probability, determine the agent's action. Candidate actions include adjusting driving speed and selecting onboard perception data.

[0062] Specifically, for an agent associated with an intelligent vehicle, the state space determined in step S1032114 is used as input and fed into the agent's initial policy network. The initial policy network performs reasoning based on the input state space and outputs the probability value corresponding to each candidate action in the agent's action space.

[0063] The candidate actions in the action space include adjusting driving speed and selecting a preprocessing method for onboard perception data. Adjusting driving speed refers to the intelligent vehicle dynamically adjusting its own speed based on the current traffic conditions and suggestions from roadside units. Selecting a preprocessing method for onboard perception data refers to the intelligent vehicle choosing the target preprocessing method for the current time slot from a number of preset data preprocessing methods, such as choosing to directly upload the raw data or choosing to preprocess it locally before uploading.

[0064] Based on the probability corresponding to each candidate action, the action to be executed by the agent in the current simulation time slot is determined from the candidate actions according to a preset rule. For example, the candidate action with the highest probability value is selected as the action to be executed, or random sampling is performed according to the probability distribution to determine the action to be executed.

[0065] It should be noted that in steps S1032121 and S1032122 above, the execution action output by the initial policy network includes not only the selected execution action but also the corresponding execution parameters. For example, for an agent associated with a roadside unit, if the execution action is to allocate local computing resources, the policy network simultaneously outputs the specific resource allocation ratio; for an agent associated with an intelligent vehicle, if the execution action is to adjust the driving speed, the policy network simultaneously outputs the recommended specific speed value. This output method of action type + execution parameters allows the policy network's decision results to be directly used to guide actual execution without requiring additional parameter determination steps.

[0066] Furthermore, the ability of the initial policy network to simultaneously output action types and corresponding execution parameters is related to its network structure design. Specifically, the output layer of the initial policy network can contain two parts: the first part outputs discrete action types, such as outputting the probability distribution of each candidate action through a Softmax function; the second part outputs continuous execution parameters, such as directly outputting specific values ​​through a linear layer. Both parts share the feature extraction results of the same input layer and intermediate layers, but are obtained through different output branches. That is, during training, the initial policy network, through backpropagation of reward values, needs to simultaneously learn the action selection strategy and the execution parameter determination strategy. For example, when the environment state of the agent associated with the intelligent vehicle requires deceleration, the policy network learns to select the action type "adjust driving speed" and also learns to output a specific speed value matching the state, without needing to discretize the speed value into multiple independent candidate actions. This structure enables the policy network to make decisions efficiently in a mixed action space, avoiding both the action space expansion problem caused by discretizing continuous parameters and the policy bias problem caused by making discrete actions continuous.

[0067] S10322: Based on the actions executed by each intelligent agent, update the digital twin simulation model to obtain the updated digital twin simulation model.

[0068] Specifically, the actions of each intelligent agent determined in step S10321 can be synchronously applied to the digital twin simulation model. The digital twin simulation model, based on the actions performed by the intelligent agents associated with each roadside unit (such as adjusting data upload priority, selecting data fusion algorithms, and allocating local computing resources) and the actions performed by the intelligent agents associated with each intelligent vehicle (such as adjusting driving speed and selecting vehicle-mounted perception data preprocessing methods), uses its internal physics engine and traffic flow model to perform simulations and calculate the impact of these actions on traffic and resource conditions.

[0069] For example, a digital twin simulation model can simulate traffic flow changes caused by vehicles operating at adjusted speeds, communication latency changes caused by data transmission according to adjusted upload priorities, and task completion time changes caused by data processing according to allocated computing resources. After the simulation is completed, the environmental state of the digital twin simulation model is updated, resulting in an updated digital twin simulation model. This updated model reflects the new traffic environment and resource state generated after each agent performs its actions in the current simulation time slot.

[0070] S10323: Based on the updated digital twin simulation model, obtain the reward value. This specifically includes the following steps S103231-S103232: S103231: Based on the updated digital twin simulation model, the average vehicle speed increase, congestion reduction, communication delay, and excessive consumption of computing resources are obtained.

[0071] Specifically, after obtaining the updated digital twin simulation model in step S10322, various indicators reflecting the execution effect of the current training action can be extracted from the model, including the increase in average vehicle speed, the reduction in congestion, the amount of communication delay, and the amount of excessive consumption of computing resources.

[0072] The following parameters are defined as follows: Average speed improvement refers to the difference between the average speed of each road segment in the updated digital twin simulation model and the average speed before the current training exercise; a larger value indicates a more significant improvement in traffic efficiency. Congestion reduction refers to the reduction in queue length or congestion duration of each road segment in the updated digital twin simulation model compared to before the current training exercise; a larger value indicates a better congestion mitigation effect. Communication latency refers to the end-to-end data transmission latency between each intelligent vehicle and the roadside unit in the updated digital twin simulation model; a smaller latency indicates higher communication efficiency. Excessive computing resource consumption refers to the portion of computing resource utilization of each roadside unit in the updated digital twin simulation model that exceeds a preset reasonable range; a larger value indicates more severe resource waste.

[0073] The above four indicators collectively reflect the combined impact of the actions performed by each intelligent agent in this round on traffic operation efficiency and resource utilization efficiency.

[0074] S103232: The reward value is obtained by weighted summing of the increase in average vehicle speed, the reduction in congestion, the amount of communication delay, and the amount of excessive consumption of computing resources.

[0075] Specifically, the reward value for this round of reinforcement learning training can be calculated by weighting and summing the average vehicle speed increase, congestion reduction, communication latency, and excessive computing resource consumption obtained in step S103231 according to preset weight coefficients. The calculation formula is shown in Formula 1 below:

[0076] In the formula, As a reward value, This represents the increase in average vehicle speed. To reduce congestion, This refers to the communication delay. To calculate excessive resource consumption, , , and These are preset positive weight coefficients, and the magnitude of each weight coefficient reflects the importance of the corresponding indicator in the reward function.

[0077] In Formula 1, the positive signs preceding the increases in average vehicle speed and congestion reduction indicate that improvements in these two indicators will increase the reward value, guiding the agent to learn in the direction of improving traffic efficiency. The negative signs preceding the increases in communication latency and excessive consumption of computing resources indicate that increases in these two indicators will decrease the reward value, guiding the agent to optimize traffic efficiency while also considering the rational use of communication and computing resources. Through this reward value, the agent can quantitatively evaluate the merits of its actions, providing clear feedback signals for the subsequent optimization of policy network parameters.

[0078] S10324: Optimize the parameters of the initial policy network of each agent based on the reward value, and use the initial policy network of each agent after parameter adjustment as the initial policy network of the corresponding agent in the next round of reinforcement learning training operation.

[0079] Specifically, the reward value can be fed back to each agent in the multi-agent system. Since the reward value is a unified evaluation value calculated based on the global traffic state and resource state, each agent receives the same reward signal.

[0080] Based on the reward value, each agent calculates the gradient of each parameter in its initial policy network using the backpropagation algorithm, and adjusts the network parameters along the gradient direction so that it can output better execution actions in the same or similar environmental conditions.

[0081] After parameter adjustment, the adjusted initial policy network is used as the agent's initial policy network for the next round of reinforcement learning training. In the next round, the agent will use the adjusted initial policy network to re-observe the environmental state, output actions, and obtain new reward values, further optimizing the parameters. Thus, as the training rounds increase, the parameters of each agent's initial policy network are continuously iterated and updated, and its output actions gradually approach optimality, ultimately enabling the multi-agent system to learn strategies for collaborative data fusion and resource scheduling in complex traffic scenarios.

[0082] It should be noted that in the reinforcement learning training process described in step S1032 above, the digital twin simulation model provides two major capabilities for training: repetition and unlimited acceleration. Repetition refers to the fact that within the same simulation time slot, each agent can repeatedly try different actions based on the same state space and compare the corresponding reward values ​​to find the optimal action for the current simulation time slot. Unlimited acceleration means that the running speed of the digital twin simulation model is not limited by the real-time clock of the physical world, and can perform simulations at a speed far exceeding the passage of real time, allowing a large number of training rounds to be completed in a short time.

[0083] The reinforcement learning training operation described in S10321 to S10324 above corresponds to a simulation time slot in the digital twin simulation model. A simulation time slot is the smallest time unit for the digital twin simulation model to perform a state update. During training, when the agent completes the reinforcement learning training operation in the current simulation time slot and obtains the initial policy network with adjusted parameters, it can either roll back to the initial state of the current simulation time slot and execute the next round of reinforcement learning training operation using the initial policy network with adjusted parameters, or it can advance to the next simulation time slot after executing the optimal action and continue the next round of reinforcement learning training operation based on the updated digital twin simulation model. The entire training process is carried out entirely in the virtual environment of the digital twin simulation model, without the need to actually perform any trial and error actions in the real traffic system. Therefore, it will not cause any accident risk or congestion impact on real-world traffic operations, achieving zero-risk policy optimization.

[0084] In step S104 above, each policy network is deployed to each roadside unit and each intelligent vehicle for real-time resource scheduling.

[0085] Specifically, this can be achieved by deploying the policy networks trained in step S103 to the corresponding roadside units and intelligent vehicles in the physical world. Deployment refers to fixing the parameters and structure of the trained policy networks and loading them into the computing modules of the corresponding roadside units or intelligent vehicles, enabling them to invoke the policy networks for real-time decision-making during actual operation.

[0086] Each roadside unit loads the policy network corresponding to its associated intelligent agent, and each intelligent vehicle loads the policy network corresponding to its associated intelligent agent. Each policy network operates independently and does not interfere with each other. After deployment, each roadside unit and intelligent vehicle will have the ability to autonomously make resource scheduling decisions based on real-time traffic conditions.

[0087] In actual operation, each roadside unit, through its deployed policy network, determines and adjusts data upload priority, selects data fusion algorithms, and allocates local computing resources in real time based on the local traffic density, average vehicle speed, queue length, load, and communication link quality of the currently perceived jurisdiction area. Each intelligent vehicle, through its deployed policy network, selects and adjusts its driving speed and selects onboard perception data in real time based on its own location, driving speed, surrounding traffic environment, and onboard computing load.

[0088] Since the policy network has been fully trained and validated in the digital twin simulation model, its output decision commands can balance traffic operation efficiency and resource utilization efficiency, without requiring further trial and error exploration in the physical world. Meanwhile, for ordinary vehicles that lack the ability to load the policy network, they are still perceived and predicted by roadside units as part of the traffic environment, but do not participate in policy network-based collaborative decision-making. Through this deployment, policy transfer from the simulation environment to the physical world is achieved, enabling the vehicle-road cooperative system to acquire dynamic resource scheduling capabilities optimized by reinforcement learning in a zero-risk manner.

[0089] In this embodiment of the application, after completing the above step S104 and deploying the policy network to each roadside unit and each intelligent vehicle in the physical world, each roadside unit and intelligent vehicle will continuously generate new operating data during real-time operation. This data can be fed back to the digital twin simulation model for further fine-tuning and optimization of the deployed policy network, so that the system can adapt to long-term changes in traffic patterns.

[0090] Specifically, this can involve continuously or periodically feeding back actual traffic state data, resource usage data, and decision-making effect data output by the policy network, collected in real-time by various roadside units and intelligent vehicles in the physical world, to the digital twin simulation model. Based on this feedback data, the deployed policy network can be fine-tuned or retrained within the digital twin simulation model.

[0091] Fine-tuning refers to using the deployed policy network parameters as initial values ​​and continuing reinforcement learning training in the digital twin simulation model using feedback data. This allows the policy network to adapt to gradually changing factors such as seasonal variations in traffic flow or new traffic patterns within a small range. Retraining refers to reconstructing the corresponding simulation scenario in the digital twin simulation model when the traffic environment in the physical world undergoes significant changes, such as the opening of a new road, the reconstruction of intersection structures, or the addition of roadside facilities. Reinforcement learning training is then performed again starting from randomly initialized or existing parameters to obtain a policy network adapted to the new environment, which is then redeployed to the physical world.

[0092] Therefore, the policy network can continuously evolve with the evolution of traffic patterns in the physical world, so that the vehicle-road cooperative resource scheduling strategy can always maintain optimal performance that is adapted to the current traffic environment and avoid policy failure due to changes in the traffic environment.

[0093] Example 2 Based on the same inventive concept, embodiments of the present invention also provide a vehicle-road cooperative strategy deployment device, referring to... Figure 2 As shown, the device includes: Model building module 101 is used to build a digital twin simulation model based on vehicle-road cooperative perception data; System construction module 102 is used to construct a multi-agent system based on the agents associated with each roadside unit and each intelligent vehicle; Training module 103 is used to perform reinforcement learning training on the multi-agent system in the digital twin simulation model to obtain the policy network of each agent; The deployment module 104 is used to deploy each of the policy networks to each roadside unit and each intelligent vehicle for real-time resource scheduling.

[0094] Example 3 Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the vehicle-road cooperative strategy deployment method as described in Embodiment 1 above.

[0095] Example 4 Based on the same inventive concept, embodiments of the present invention also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the vehicle-road cooperative strategy deployment method as described in Embodiment 1 above.

[0096] Example 5 Based on the same inventive concept, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the vehicle-road cooperative strategy deployment method as described in Embodiment 1 above.

[0097] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0098] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0099] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0100] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0101] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for deploying a vehicle-road cooperative strategy, characterized in that, include: A digital twin simulation model is constructed based on vehicle-road cooperative perception data; A multi-agent system is constructed based on the agents associated with each roadside unit and each intelligent vehicle. The multi-agent system is trained using reinforcement learning in the digital twin simulation model to obtain the policy network of each agent. The aforementioned strategy networks are deployed to each roadside unit and each intelligent vehicle for real-time resource scheduling.

2. The method according to claim 1, characterized in that, The step of training the multi-agent system through reinforcement learning in the digital twin simulation model to obtain the policy networks of each agent includes: Determine the initial policy network for each of the agents; Based on the digital twin simulation model, the initial policy network of each agent is subjected to multiple rounds of reinforcement learning training until a preset stopping condition is reached, thereby obtaining the policy network of each agent. The reinforcement learning training operations include: Based on the digital twin simulation model and the initial policy network of each agent, the execution actions of each agent are determined; Based on the actions performed by each of the intelligent agents, the digital twin simulation model is updated to obtain the updated digital twin simulation model. The reward value is obtained based on the updated digital twin simulation model. The parameters of the initial policy network of each agent are optimized based on the reward value, and the initial policy network of each agent after parameter adjustment is used as the initial policy network of the corresponding agent in the next round of reinforcement learning training.

3. The method according to claim 2, characterized in that, The reward value obtained based on the updated digital twin simulation model includes: Based on the updated digital twin simulation model, the average vehicle speed increase, congestion reduction, communication latency, and excessive consumption of computing resources are obtained. The reward value is obtained by weighted summing of the average vehicle speed increase, the congestion reduction, the communication latency, and the excessive consumption of computing resources.

4. The method according to claim 2, characterized in that, The determination of the actions to be performed by each agent, based on the digital twin simulation model and the initial policy network of each agent, includes: Based on the digital twin simulation model, the state space of each of the intelligent agents is determined; For each agent, the agent's execution action is obtained based on the agent's state space and the agent's initial policy network.

5. The method according to claim 4, characterized in that, The determination of the state space of each intelligent agent based on the digital twin simulation model includes: For each of the aforementioned intelligent agents, if the intelligent agent is associated with any roadside unit, then based on the digital twin simulation model, the local traffic density, average vehicle speed, queue length, load, and communication link quality of the area under the jurisdiction of the roadside unit are determined; The state space of the agent is defined by at least one of the local traffic density, average vehicle speed, queue length, load, and communication link quality within the area governed by the roadside unit.

6. The method according to claim 4, characterized in that, The process of obtaining the agent's execution actions based on the agent's state space and initial policy network includes: If the agent is associated with any roadside unit, the state space of the agent is input into the initial policy network of the agent to obtain each candidate action and its corresponding probability; wherein, the candidate action includes adjusting the data upload priority, selecting the data fusion algorithm, and allocating local computing resources; The agent's action is determined based on each candidate action and its corresponding probability.

7. The method according to claim 4, characterized in that, The determination of the state space of each intelligent agent based on the digital twin simulation model includes: For each of the aforementioned intelligent agents, if the intelligent agent is associated with any intelligent vehicle, then based on the digital twin simulation model, the location, driving speed, surrounding traffic environment, and on-board computing load of the intelligent vehicle are determined. The intelligent vehicle's location, driving speed, surrounding traffic environment, and onboard computing load are used as the state space of the intelligent agent.

8. The method according to claim 4, characterized in that, The process of obtaining the agent's execution actions based on the agent's state space and initial policy network includes: If the agent is associated with any intelligent vehicle, the state space of the agent is input into the initial policy network of the agent to obtain each candidate action and its corresponding probability; wherein, the candidate actions include adjusting driving speed and selecting vehicle perception data; The agent's action is determined based on each candidate action and its corresponding probability.

9. A vehicle-road cooperative strategy deployment device, characterized in that, include: The model building module is used to build digital twin simulation models based on vehicle-road cooperative perception data; The system construction module is used to build a multi-agent system based on the agents associated with each roadside unit and each intelligent vehicle; The training module is used to perform reinforcement learning training on the multi-agent system in the digital twin simulation model to obtain the policy network of each agent. The deployment module is used to deploy each of the aforementioned policy networks to each roadside unit and each intelligent vehicle for real-time resource scheduling.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the vehicle-road cooperative strategy deployment method according to any one of claims 1-8.

Images