Regional parking system supply and demand flow adaptation and optimization regulation and control method driven by reinforcement learning

Abstract

The invention relates to a reinforcement learning-driven regional parking system supply and demand flow adaptation and optimization regulation and control method. The regional parking system operation optimization method comprises the steps of user-side-oriented parking distribution opportunity problem optimization; and optimizing a resource optimal utilization problem oriented to a management end. According to two system problems of parking distribution opportunity and resource optimal utilization, different regional parking system operation optimization processes without model data driving are given through a multi-agent deep reinforcement learning method, the driving distance of users in the regional parking system can be effectively reduced, and the system operation efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation systems and parking management, and in particular to a reinforcement learning-driven regional parking system supply and demand flow adaptation and optimization control method. Background Art

[0002] At present, regional parking systems have complex and random demand characteristics, and there are significant differences in parking demand at different times and spaces, which poses a challenge to the application of traditional model-based methods in large-scale complex systems. These models often lack flexibility and accuracy when dealing with the dynamics and uncertainties of the system. In addition, parking is a time-consuming resource scheduling problem. When matching supply and demand, it is necessary not only to consider the immediate system return, but also to evaluate the long-term benefits of the system to ensure long-term resource utilization efficiency.

[0003] For the user side, when the vehicle approaches the destination, it usually requests the allocation of available parking spaces through the central system. The center matches the vehicle and parking resources based on the real-time collected parking space status information. If the complex mechanism design is ignored, the problem can be simplified to a bipartite graph matching problem, that is, optimizing the matching by minimizing the driving distance between the vehicle and the parking space. However, in the actual parking process, users often face a decision-making dilemma: when there are no available parking spaces near the destination, the user can learn through the system that there are vacant parking spaces farther away. At this time, the user will hesitate whether to choose to park in the distance immediately or wait for a while, hoping that the parking space near the destination may be vacant at the next moment. For users with different risk preferences, there may be different choices: conservative users will directly choose to park in the distance, while aggressive users may choose to wait, but this is accompanied by different risk-benefit trade-offs. In the context of connected autonomous vehicles, similar decision-making problems also exist in the allocation of vehicles by the management center. Since connected autonomous vehicles can drive autonomously after passengers get off the car, the time cost is greatly reduced. Therefore, there is a possibility of delayed matching in the system, that is, the management center can choose to postpone the matching of vehicles at certain times in order to obtain a better parking location.

[0004] For the management side, parking demand is usually composed of long-term parking and short-term parking. Especially in mixed-use areas such as urban complexes or transportation hubs, the coexistence of long-term and short-term parking demands is particularly obvious. Parking resources have different levels of quality due to their different distances from the destination. The closer the distance, the better the quality of the parking space. When these high-quality resources are occupied by long-term parking, short-term parking vehicles are often forced to park far away, increasing the driving distance of the system and reducing the overall operating efficiency of the system. In order to solve this problem, many large transportation hubs and urban complexes have implemented zoning management strategies, allocating poorer parking resources to long-term parking needs, and reserving high-quality resources for short-term parking vehicles to improve resource utilization.

[0005] Traditional methods based on modeling or supervised learning have limitations in building long-term dynamic relationships between decisions and the environment. It is difficult to accurately capture the continuous impact of current decisions on the environment in subsequent periods, especially in the rapid processing of large-scale complex systems. In recent years, deep reinforcement learning, as a cutting-edge technology that combines the advantages of deep learning and reinforcement learning, can directly learn the optimal behavior strategy of intelligent agents through high-dimensional perceptual input, providing a more effective solution.

[0006] Therefore, there is an urgent need for a reinforcement learning-driven regional parking system supply and demand flow adaptation and optimization control method, which can realize a model-free data-driven optimization process through multi-agent deep reinforcement learning to cope with the complex dynamic supply and demand matching problems in the regional parking system, thereby improving the overall operation efficiency of the system. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned traditional methods and provide a reinforcement learning driven regional parking system supply and demand flow adaptation and optimization control method.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A reinforcement learning-driven regional parking system supply and demand flow adaptation and optimization control method, the method comprising the following parts:

[0010] Optimization of parking allocation timing problem for users;

[0011] Optimization of optimal resource utilization issues for management.

[0012] Furthermore, the user-side parking allocation timing problem optimization includes the following steps:

[0013] S1. Define the intelligent delayed parking matching problem of connected autonomous vehicles as a multi-agent Markov decision process;

[0014] S2. Design an online matching simulator for regional parking;

[0015] S3. Intelligent delay matching is performed based on multi-agent DQN strategy or multi-agent deep deterministic strategy gradient to achieve regional parking system operation optimization.

[0016] Furthermore, the research area of ​​the user-oriented parking allocation timing problem is divided into a grid matrix to ensure that each square has at most one parking space and one parking demand attraction center, i.e., a passenger activity center; the networked autonomous driving vehicle accepts parking allocation from the management center, the vehicle sends a parking request and a destination, the dispatch center decides whether the vehicle will be matched immediately or delayed to the next time, and after a successful match, the user is first sent to the destination and then drives to the assigned parking space to wait for the user's call.

[0017] Furthermore, the S1 is specifically:

[0018] S101, each vehicle requesting parking is regarded as an intelligent agent, and the number of intelligent agents in the system changes dynamically at any time;

[0019] S102, the agent state is formed by global information and local information;

[0020] S103, defining the action space, and determining whether the agent enters the allocation pool for matching within the period;

[0021] S104, setting the corresponding state transition of the intelligent agent;

[0022] S105. Design corresponding reward function for state transition.

[0023] Furthermore, the global information of S102 includes existing unmatched parked vehicles in each grid in the time period, parking occupancy status in each grid in the time period, occupied parking spaces that may be released in the next time period, and expected visit parking demand of each activity destination in the next time period.

[0024] Furthermore, the local information of S102 includes the location information of the current intelligent agent, the accumulated waiting time after the parking request is issued, and the location information of the activity destination corresponding to the intelligent agent.

[0025] Furthermore, the reward function of S105 is divided into two stages. The first stage is the total driving time of the agent and the idle driving time from the destination to the parking space and back. The second stage is updated based on the reward of the first stage.

[0026] Furthermore, the S2 is specifically:

[0027] S201, inputting historical parking transaction data and parking space layout of the parking area;

[0028] S202, initializing the joint state;

[0029] S203, extracting parking data from the historical transaction data set as requests generated by the system during this period, and placing these parking requests in a demand-side waiting list;

[0030] S204, collecting global information of the environment and local information of each agent to form a state;

[0031] S205, generating the action of the system agent in the time period, determining whether the agent enters the matching pool to receive allocation in this time period, or delays the decision-making to the next time period;

[0032] S206, matching the parking requests on the demand side and the idle parking spaces on the supply side in the matching pool;

[0033] S207, removing the matched parking request from the waiting list. If an idle parking space is successfully matched, its status changes from "idle" to "occupied".

[0034] S208, entering the next allocation period, the parking time of each vehicle in the simulation follows a certain distribution characteristic, when the vehicle reaches the set parking time, the vehicle leaves the system, and the corresponding "occupied" parking space status changes to "free";

[0035] S209. From the initial moment to the end moment of the simulation, each intelligent allocation decision period repeats S203 to S208.

[0036] Furthermore, the optimization of the optimal resource utilization problem for the management end includes the following steps:

[0037] S4. The system online allocation problem considering parking duration is defined as a multi-agent Markov decision process;

[0038] S5. Design a regional parking system operation simulator;

[0039] S6. Based on the multi-agent DQN strategy, intelligent matching is performed considering the parking time of the system to achieve the optimization of the operation of the regional parking system.

[0040] Furthermore, the research area of ​​the resource optimal utilization problem for the management end is divided into a grid matrix, each grid represents a spatial location, and each grid can accommodate at most one parking space or one arriving parking demand; time is discretized, and the simulation is evenly divided from the start time to the end time; at most one new parking demand arrives in each time period; the total number of vacant parking spaces changes dynamically; and there are parking spaces available for the newly arriving parking demands in each time period.

[0041] Furthermore, the S4 is specifically:

[0042] S401, treating the vacant parking spaces as intelligent agents, and the number of intelligent agents in the system changes dynamically over time;

[0043] S402, the agent state is formed by global information and local information;

[0044] S403, defining the action space, and determining whether the agent enters the allocation pool for matching within the period;

[0045] S404, setting the corresponding state transition of the intelligent agent;

[0046] S405. Design corresponding reward function for state transition.

[0047] Furthermore, the global information of S402 includes the simulation period to which the current moment belongs, the parking supply spatiotemporal data matrix, and the parking demand spatiotemporal data matrix.

[0048] Furthermore, the local information of S402 includes the travel time between the location information of the current intelligent agent and the location of the newly arrived parking demand.

[0049] Furthermore, the reward function of S405 is defined in different situations. When there is no parking demand, the reward of the agent is r t i = 0, when there is a parking demand, the potential reward of each agent is defined as

[0050] Furthermore, the S5 is specifically:

[0051] S501, input parking demand center and regional parking space layout;

[0052] S502, initializing system status;

[0053] S503, generating parking demand according to historical data or distribution characteristics, including arrival time and parking duration;

[0054] S504, collecting global information of the environment and local information of each agent to form a state;

[0055] S505, generating the action of the time period system agent, determining whether the parking space agent can accept the current parking demand in this time period, if it can accept it, it will enter the matching pool to receive allocation, if it cannot accept it, it will continue to wait for the parking demand of the next time period before making a decision;

[0056] S506, matching parking requests and acceptable free parking spaces in the matching pool based on a minimum driving distance criterion;

[0057] S507, removing the matched parking request from the waiting list, and the parking request that has not been successfully matched leaves the parking system, and the corresponding request is also deleted from the waiting list. If an idle parking space is successfully matched, its status changes from idle to occupied;

[0058] S508, entering the next allocation period, the parking time of each vehicle in the simulation follows a certain distribution characteristic, when the vehicle reaches the set parking time, the vehicle leaves the system, and the corresponding "occupied" parking space status changes to "free";

[0059] S509. From the initial moment to the end moment of the simulation, each intelligent allocation decision period repeats S503 to S508.

[0060] Furthermore, the S6 adopts a centralized training and distributed execution strategy, utilizes a multi-agent DQN method, uses the same neural network to describe the behavior strategies of all agents, and the trained neural network parameters are shared by all agents.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention realizes model-free data-driven regional parking system optimization through a multi-agent deep reinforcement learning method. This method combines intelligent delay matching on the user side and resource allocation on the management side, and can dynamically adapt to complex parking needs, improve parking allocation efficiency, and reduce vehicle driving distance. At the same time, the resource utilization of long-term and short-term parking is optimized through zoning management, which significantly improves the overall operation efficiency and user experience of the parking system. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The parking grid management model and parking flow chart of the present invention;

[0064] Figure 2 A schematic diagram of the parking user intelligent delay matching system of the present invention;

[0065] Figure 3 A schematic diagram of an intelligent regional parking allocation system considering the impact of parking duration according to the present invention;

[0066] Figure 4 A schematic diagram showing comparison of results of different methods in different scenarios of the present invention;

[0067] Figure 5 A schematic diagram of the parking demand center and parking space layout distribution of the present invention;

[0068] Figure 6 Flowchart of the multi-agent DQN algorithm framework of the present invention. DETAILED DESCRIPTION

[0069] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0070] like Figure 1 As shown, a reinforcement learning driven regional parking system supply and demand flow adaptation and optimization control method, the method includes the following parts:

[0071] Optimization of parking allocation timing problem for users;

[0072] Optimization of optimal resource utilization issues for management.

[0073] Furthermore, the user-side parking allocation timing problem optimization includes the following steps:

[0074] S1. Define the intelligent delayed parking matching problem of connected autonomous driving vehicles as a multi-agent Markov decision process. Figure 2 This is a schematic diagram of the intelligent delayed matching system for this problem. The study area is a rectangular parking area with a side length of 1km×1km. Grid processing is performed. The side length of each small square is 10m, which contains a parking space. There are a total of 10,000 parking spaces.

[0075] The region includes five activity destinations as attraction points for parking demand, which are evenly distributed in the study area.

[0076] We set the input and output of the environment to be stable, the arrival rate of parked vehicles in the environment follows the Poisson distribution, the parking time of all vehicles follows the negative exponential distribution, and the departure rate also follows the Poisson distribution. The location of the vehicle when it requests parking is randomly generated, and the vehicle leaves the parking space based on the parking time of the vehicle. When the parking time set in the simulation is reached, the vehicle automatically leaves. The simulation time is 5 hours and the simulation warm-up time is 1 hour.

[0077] At the same time, assuming that the maximum tolerable waiting time of all vehicles is evenly distributed in the interval [10,20] seconds, if a vehicle has not successfully matched within its maximum tolerable time, it will give up parking and automatically leave the system. The driving distance of the vehicle is calculated according to the Manhattan distance, the average speed of the vehicle is 4 meters per second, and the system allocation time interval is 1 second.

[0078] S101. Consider each vehicle requesting parking as an intelligent agent. At any time t, the number of intelligent agents in the system is N. t Make dynamic changes;

[0079] S102, the state of agent i is formed by global information and local information

[0080] Global Information It includes the existing unmatched parking vehicles in each grid in the t period, the parking occupancy status in each grid in the t period, the occupied parking spaces that may be released in the next period, and the expected parking demand for each activity destination in the next period; the global information is shared by all agents in the system in the t period, and is a 3D data matrix of [m, n, 4], that is,

[0081] Local information The location information of the current agent is encoded using one-hot coding, the accumulated waiting time after the parking request is issued, and the location information of the activity destination corresponding to the agent are encoded using one-hot coding; the local information contains the demand information of the agent i itself, which is a three-dimensional data matrix of [m, n, 3], that is,

[0082] S103, define the action space as A(t)=[0,1], and decide whether the agent enters the allocation pool for matching within the period.

[0083] in Indicates that in period t, agent i enters the allocation pool to accept allocation; and It means that in time period t, agent i delays entering the allocation pool and waits for the next moment before deciding whether to enter the allocation pool.

[0084] The actions of all agents in period t can be expressed as

[0085] S104, setting the corresponding state transition of the agent.

[0086] The environment is accepting action a t After the joint state from s t Convert to s t ', for which the state of agent i changes from Convert to There are 4 situations:

[0087] 1) Agent i decides to enter the allocation pool for matching, and if the matching is successful, the corresponding parking demand will be deleted from the waiting sequence, completing the matching

[0088] 2) Agent i decides to enter the allocation pool for matching, but due to the imbalance between supply and demand, no matching is successful. The corresponding parking demand is kept in the waiting list and the matching is not completed.

[0089] 3) Agent i delays entering the allocation pool to accept the match, and the user does not withdraw the request, The state at time t+1 is That is, the corresponding parking demand remains in the list and the matching is not completed.

[0090] 4) Agent i fails to match successfully. Since the user's waiting time exceeds the maximum waiting time that the user can tolerate, the user cancels the parking request by himself. The corresponding parking demand is deleted from the list and the matching is not completed. In particular, we believe that the action of agent i at this time is In particular, due to the successful matching of parking requests in the waiting list and the emergence of new requests, the system t 'With s t+1 The number of agents in may not be equal;

[0091] S105. Design corresponding reward function for state transition.

[0092] The reward function of S105 is divided into two stages. The first stage is the total driving time of the agent. The time between the destination and the parking space and back. The second stage is updated based on the rewards of the first stage. When the match is successful, the reward is the total driving time. When the match is unsuccessful, the reward is V. When the delay is normal to the next moment, the reward is 0. When the user cancels, the reward is V.

[0093] Among them, V is a negative value less than 0, indicating the penalty for unsuccessful matching.

[0094] Try to avoid long-distance supply and demand matching, that is, the loss of user requests, then define for

[0095]

[0096] TT represents the benchmark travel time.

[0097] The total average return of the system during period t is In order to balance the personal benefits of agent i and the total average revenue of the system To achieve the cooperation between agents to optimize the total system revenue, define the revenue of agent i in period t as

[0098] S2. Design an online matching simulator for regional parking.

[0099] S201, inputting historical parking transaction data and parking space layout of the parking area;

[0100] S202, initialize joint state s t ;

[0101] S203, extracting parking data from the historical transaction data set as requests generated by the system during this period, and placing these parking requests in a demand-side waiting list;

[0102] S204, collect the global information of the environment and the local information s of each agent i t Composition status;

[0103] S205: Generate actions of the system agent during the period Decide whether the agent enters the matching pool to receive allocation during this period, or delays the decision until the next period t+1;

[0104] S206, using a bipartite graph matching method to match the parking requests on the demand side and the idle parking spaces on the supply side in the matching pool;

[0105] S207, removing the matched parking request from the waiting list. If an idle parking space is successfully matched, its status changes from "idle" to "occupied".

[0106] S208, entering the next allocation period t=t+1, the parking time of each vehicle in the simulation follows a certain distribution characteristic, when the vehicle reaches the set parking time, the vehicle leaves the system, and the corresponding "occupied" parking space status changes to "free";

[0107] S209, from the initial time 0 to the end time T of the simulation, each time the intelligent allocation decision period t, repeat S203 to S208.

[0108] S3. Intelligent delay matching is performed based on multi-agent DQN strategy or multi-agent deep deterministic strategy gradient to achieve regional parking system operation optimization.

[0109] To study the effect of the intelligent delayed matching method on different parking supply and demand patterns, we set up three different scenarios as shown in Table 1.

[0110] Scenario 1 describes the parking process during a period with a low parking demand arrival rate, a long parking duration, a low parking departure rate, and a low turnover rate, and a relatively balanced supply and demand, such as business visitors in some areas during the day;

[0111] Scenario 2 describes the parking process during the commuting arrival period, with a high arrival rate and a long parking time, that is, a low departure rate and a greater demand than supply;

[0112] Scenario 3 describes the parking process in a large shopping mall or integrated transportation hub, with a high arrival rate, short parking time, high departure rate and high turnover rate, and balanced supply and demand.

[0113] Table 1 Simulation parameter settings

[0114]

[0115] We compared the following three different strategies in our results:

[0116] 1) When a vehicle arrives at the simulation area and makes a parking request, it immediately enters the matching pool and is matched using the bipartite graph matching algorithm. We use this result as a comparison benchmark for deep reinforcement learning and refer to this strategy as Greedy.

[0117] 2) Consider the strategy of intelligent delay matching, among which the intelligent delay matching based on multi-agent DQN strategy is referred to as DQN.

[0118] 3) The intelligent delayed matching strategy based on multi-agent deep deterministic policy gradient is referred to as DDPG. To avoid the randomness of the results, we run the trained network for 50 rounds and analyze the results.

[0119] Figure 4 Box plot of the test results for the average successful matching rate and average driving distance of parking demands in different scenarios.

[0120] In scenario one, where the parking turnover rate is slow and parking supply and demand are relatively balanced, the use of deep reinforcement learning methods driven by spatiotemporal parking data for intelligent delayed matching can reduce the average driving time of vehicles to a certain extent. However, the average successful matching rate of demand will decrease to a certain extent.

[0121] In scenario 2, where parking arrivals are dense and departures are sparse, and parking supply is in short supply, although the average successful matching rate does not change significantly, the average vehicle driving time will increase, making it unsuitable for intelligent delayed matching.

[0122] In scenario 3, where the parking turnover rate is fast and the parking supply and demand are relatively balanced, using deep reinforcement learning for intelligent delayed matching can reduce the average vehicle driving time. At the same time, the DDPG algorithm will reduce the average successful matching rate, while DQN will not reduce the successful matching rate.

[0123] In summary, compared with the immediate matching strategy, the intelligent delayed matching strategy based on deep reinforcement learning can reduce the average parking driving time of vehicles to a certain extent, but it will affect the average successful matching rate of demand. Therefore, during the peak commuting period, vehicles arrive densely and the parking turnover rate is low, which is not suitable for intelligent delayed matching; in other cases, with the increasing popularity of connected autonomous driving, a waiting area with a certain capacity can be set near each activity destination, which can increase the maximum tolerable time of the vehicle. Based on the delayed matching strategy, the average driving distance can be further reduced without reducing the successful matching rate of the system, achieving the purpose of optimizing system operation efficiency and energy conservation and emission reduction.

[0124] Furthermore, the optimization of the optimal resource utilization problem for the management end includes the following steps:

[0125] S4. The system online allocation problem considering parking duration is defined as a multi-agent Markov decision process. Figure 3 This is a schematic diagram of the intelligent parking allocation system for this problem.

[0126] The study area is divided into M×N grids, each grid represents a spatial location, and it is assumed that each grid can accommodate at most one parking space or one arriving parking demand; time is discretized, and it is assumed that the simulation is divided into K segments from the starting time 0 to the ending time T, t∈[0,T], k∈[1,2,...,K]; it is assumed that at most one new parking demand arrives in each time period; the total number of vacant parking spaces in the area at time t is N t , is dynamically changing. Assuming that the parking supply in the system is sufficient, that is, to ensure N t > 0, in each time period t, all newly arrived parking demands have parking spaces available.

[0127] S401, consider the vacant parking spaces as intelligent agents. The number of intelligent agents in the system changes dynamically over time. In time period t, each vacant parking space within the M×N area is considered as one intelligent agent, with a total of N t agents, i = {1,2,...,N t}, the number of agents changes dynamically at different times;

[0128] S402, based on global information and local information Constructing the state of an agent

[0129] The global information includes the simulation period k to which the current moment belongs; the M×N×K parking supply spatiotemporal data matrix, and each three-dimensional data block ps m,n,k =0 means that there is no available parking resource in this space and time period (no parking space or the parking space is occupied), ps m,n,k=1 indicates that there are available parking resources; and an M×N×K parking demand spatiotemporal data matrix, each three-dimensional data block pd m,n,k =0 means that there is no new parking demand arriving in this space and time period, ps m,n,t =1 indicates that new parking resources have arrived.

[0130] Global information is shared by all agents in the system at time t, that is, It is a 1+M×N×K×2 data matrix.

[0131] Local information The local information includes the location information of the current agent i using one-hot encoding; the travel time between the location of the newly arrived parking demand and the location of the current parking demand. The local information contains the relevant information of agent i, that is,

[0132] S403, define the action space A(t) = [0,1], and decide whether the agent enters the allocation pool for matching within the period. The actions of all agents in period t can be expressed as

[0133] S404, set the corresponding state transition of the agent, and accept action a t After the joint state from s t Convert to s t '.

[0134] If a new parking demand arrives in time period t, 1) for the agent that can accept the current parking demand and enter the allocation pool for matching, and the matching is successful, then the agent will be deleted from the list in time period t+1; if the matching is not successful, then the agent will remain in the list in time period t+1; 2) If the actions of all agents are 0, then the parking demand cannot be matched, and the parking demand can only leave the system, and all agents will transfer to the next time state. 3) If vehicles in some parking spaces leave the system in time period t, the corresponding parking spaces will change from occupied state to idle state, and the corresponding new agents will be added to the list. 4) If no new parking demand arrives and no new vehicles leave the system, then s t '=s t . From period t to period t+1, the total number of agents in the system may be different;

[0135] S405. Design corresponding reward function for state transition.

[0136] Use pa t Indicates whether there is a new parking demand arriving during period t. If pa t =1 indicates that there is a parking demand, pa t = 0 means no parking demand has arrived. We use represents the potential travel time between agent i and the parking demand.

[0137] The reward function is defined in different situations. When there is no parking demand, the reward of the agent is r t i = 0, when there is a parking demand, the potential reward of each agent is defined as

[0138] Matching supply and demand based on the minimum driving distance can calculate

[0139] The system's revenue is

[0140] In the formula, When , it means that all the agent actions of the system are 0, that is, none of them can accept the parking demand, then the parking demand agent is forced to leave the system, and the system will be punished accordingly, V is a large positive real number; when the agent action is 1, the system will select the smallest Matching, and then the reward of each agent i can be set to

[0141] S5. Design a regional parking system operation simulator;

[0142] S501, input parking demand center and regional parking space layout;

[0143] S502, initializing system status t ;

[0144] S503, generating parking demand according to historical data or distribution characteristics, including arrival time and parking duration;

[0145] S504: Collect the global information of the environment and the local information of each agent to form the state s t ;

[0146] S505: Generate actions of the system agent during the period Determine whether the parking space agent can accept the current parking demand in this period. If it can accept it, it will enter the matching pool to receive the allocation. If it cannot accept it, it will continue to wait for the parking demand in the next period t+1 before making a decision.

[0147] S506, matching parking requests and acceptable free parking spaces in the matching pool based on a minimum driving distance criterion;

[0148] S507, removing the matched parking request from the waiting list, and the parking request that has not been successfully matched leaves the parking system, and the corresponding request is also deleted from the waiting list. If an idle parking space is successfully matched, its status changes from idle to occupied;

[0149] S508, entering the next allocation period t=t+1, the parking time of each vehicle in the simulation follows a certain distribution characteristic, when the vehicle reaches the set parking time, the vehicle leaves the system, and the corresponding "occupied" parking space status changes to "free";

[0150] S509. From the initial moment to the end moment of the simulation, each intelligent allocation decision period repeats S503 to S508.

[0151] S6. Based on the multi-agent DQN strategy, intelligent matching is performed considering the parking time of the system to achieve the optimization of the operation of the regional parking system.

[0152] It adopts a centralized training and distributed execution strategy, uses a multi-agent DQN method, uses the same neural network to describe the behavior strategies of all agents, and the trained neural network parameters are shared by all agents. Figure 6 This is the framework process of the multi-agent DQN algorithm.

[0153] Construct an 11×11 square grid area with a parking demand center in the center grid of the entire area, such as Figure 5 As shown, that is, in each period, the newly arrived parking vehicles appear from the grid where the demand center is located. There are three circles of parking spaces around the parking demand center, with a total of 72 parking spaces. The driving time between two adjacent grids is 1 minute, and the distance between any two grids is calculated according to the Manhattan distance. The number of parking spaces corresponding to the demand center at different driving times is shown in Table 2. The entire simulation time is set to 8 hours, each time step is 1 minute, and there are 480 time steps in total. At the initial moment, each parking space is idle. In order to study the impact of the arrival law and duration characteristics of parking demand on the results, three different parking scenarios will be analyzed: 1) There are differences in parking duration, and the arrival of vehicles has obvious periodicity. The parameters are set as shown in Table 3; 2) There are differences in parking duration, and there is no obvious periodicity in the arrival of vehicles. The parameters are set as shown in Table 4; 3) There is no difference in parking duration, and there is an obvious periodicity in the arrival of vehicles. The parameters are set as shown in Table 5;

[0154] Table 2 Statistics of parking spaces corresponding to different driving times

[0155] Driving time (min) 1 2 3 4 5 Number of parking spaces 4 4 4 8 12 Driving time (min) 6 7 8 9 10 Number of parking spaces 12 8 8 8 4

[0156] Table 3 Parking demand arrival and parking duration parameter settings

[0157] Simulation period 0～1 1～2 2～3 3～4 Arrival rate (vehicles / min) 0.4 0.4 0.4 0.6 Duration Type Long China Times Short Short Driving time (min) 4～5 5～6 6～7 7～8 Arrival rate (vehicles / min) 0.4 0.4 0.5 0.5 Duration Type Long time:Medium time=1:1 Medium time: short time = 1:1 Short Short

[0158] Table 4 Parking demand arrival and parking duration parameter settings

[0159]

[0160]

[0161] Table 5 Parking demand arrival and parking duration parameter settings

[0162] Simulation period 0～1 1～2 2～3 3～4 Arrival rate (vehicles / min) 0.4 0.4 0.4 0.4 Duration Type China Times China Times China Times China Times Driving time (min) 4～5 5～6 6～7 7～8 Arrival rate (vehicles / min) 0.4 0.4 0.4 0.4 Duration Type China Times China Times China Times China Times

[0163] Table 6 shows the test results of the average driving time of the system in all test rounds, where Optimization corresponds to the theoretical optimal value with all demand information known, FIFS is the test result corresponding to the first-come-first-served allocation strategy, and DQN is the system intelligent allocation result corresponding to the multi-agent deep reinforcement learning training. The numbers in brackets represent the percentage increase compared to the theoretical optimal value.

[0164] Table 6 Comparative analysis of test results of different strategies

[0165] System average travel time (min) Optimization FIFS DQN Scenario 1 (Duration varies, arrival has periodicity) 855.5 1029.6(+20.3％) 918.2(+7.3％) Scenario 2 (Time varies, arrival has no periodic pattern) 555.4 701.0(+26.2％) 687.5(+23.8％) Scenario 3 (no difference in duration, arrival has periodic regularity) 734.6 825.7(+12.4％) 806.7(+9.8％)

[0166] The results show that in scenarios where parking time varies and vehicle arrivals have obvious periodic patterns, the intelligent allocation strategy using deep reinforcement learning has the most significant improvement on system operation, and the result is better than the FIFS strategy and closer to the theoretical optimal value. In scenarios where parking time varies but vehicle arrivals have no obvious periodic patterns, deep reinforcement learning can learn the FIFS strategy, but it is still some distance away from the theoretical optimal value. In scenarios where there is no difference in parking time but vehicle arrivals have obvious periodic patterns, the strategy learned by deep reinforcement learning is better than the FIFS strategy and closer to the theoretical optimal value, but since there is no difference in parking time, the system's optimization space is limited, so the result is not much improved.

[0167] In summary, the deep reinforcement learning method can learn the system operation allocation strategy that takes long-term benefits into consideration based on data. It is suitable for business districts, comprehensive transportation hubs, urban complexes and other areas where parking characteristics have certain periodic regularities, and can further improve the operation efficiency of the system.

Claims

1. A reinforcement learning driven regional parking system supply and demand flow adaptation and optimization control method, characterized in that: The following steps are involved: Optimization of parking allocation timing problem for users; Optimization of optimal resource utilization issues for management.

2. According to claim 1, a reinforcement learning driven regional parking system supply and demand flow adaptation and optimization control method is characterized in that: The user-side parking allocation timing problem optimization includes the following steps: S1. Define the intelligent delayed parking matching problem of connected autonomous vehicles as a multi-agent Markov decision process; S2. Design an online matching simulator for regional parking; S3. Intelligent delay matching is performed based on multi-agent DQN strategy or multi-agent deep deterministic strategy gradient to achieve regional parking system operation optimization.

3. According to claim 2, a reinforcement learning driven regional parking system supply and demand flow adaptation and optimization control method is characterized in that: The research area of ​​the user-oriented parking allocation timing problem is divided into a grid matrix, ensuring that each square has at most one parking space and one parking demand attraction center, i.e., a passenger activity center; The networked autonomous driving vehicle accepts parking allocation from the management center. The vehicle sends a parking request and destination. The dispatch center decides whether the vehicle will be matched immediately or delayed to the next time. After a successful match, the user is first sent to the destination and then drives to the allocated parking space to wait for the user's call.

4. According to claim 2, a reinforcement learning driven regional parking system supply and demand flow adaptation and optimization control method is characterized in that: The S1 is specifically: S101, each vehicle requesting parking is regarded as an intelligent agent, and the number of intelligent agents in the system changes dynamically over time; S102, the agent state is formed by global information and local information; S103, defining the action space, and determining whether the agent enters the allocation pool for matching within the period; S104, setting the corresponding state transition of the intelligent agent; S105. Design corresponding reward function for state transition.

5. According to claim 4, a reinforcement learning driven regional parking system supply and demand flow adaptation and optimization control method is characterized in that: The global information of S102 includes existing unmatched parked vehicles in each grid in the time period, parking occupancy status in each grid in the time period, occupied parking spaces that may be released in the next time period, and expected visit parking demand of each activity destination in the next time period.

6. The method for adapting and optimizing supply and demand flows of a regional parking system driven by reinforcement learning according to claim 4 is characterized in that: The local information of S102 includes the location information of the current intelligent agent, the accumulated waiting time after the parking request is issued, and the location information of the activity destination corresponding to the intelligent agent.

7. The method for adapting and optimizing supply and demand flows of a regional parking system driven by reinforcement learning according to claim 4 is characterized in that: The reward function of S105 is divided into two stages. The first stage is the total driving time of the agent and the round-trip idle travel time from the destination to the parking space, and the second stage is updated based on the reward of the first stage.

8. The method for adapting and optimizing supply and demand flows of a regional parking system driven by reinforcement learning according to claim 2 is characterized in that: The S2 is specifically: S201, inputting historical parking transaction data and parking space layout of the parking area; S202, initializing the joint state; S203, extracting parking data from the historical transaction data set as requests generated by the system during this period, and placing these parking requests into a demand-side waiting list; S204, collecting global information of the environment and local information of each agent to form a state; S205, generating the action of the system agent in the time period, determining whether the agent enters the matching pool to receive allocation in this time period, or delays the decision-making to the next time period; S206, matching the parking requests on the demand side and the idle parking spaces on the supply side in the matching pool; S207, removing the matched parking request from the waiting list, if the free parking space is successfully matched, its status changes from free to occupied; S208, entering the next allocation period, the parking time of each vehicle in the simulation follows a certain distribution characteristic, when the vehicle reaches the set parking time, the vehicle leaves the system, and the corresponding occupied parking space becomes free; S209, from the initial moment to the end moment of the simulation, each time the intelligent allocation decision period is repeated, S203 to S208 are repeated.

9. The method for adapting and optimizing supply and demand flow of regional parking system driven by reinforcement learning according to claim 1 is characterized in that: The optimization of the optimal resource utilization problem for the management end includes the following steps: S4. The system online allocation problem considering parking duration is defined as a multi-agent Markov decision process; S5. Design a regional parking system operation simulator; S6. Based on the multi-agent DQN strategy, intelligent matching is performed considering the parking time of the system to achieve the optimization of the operation of the regional parking system.

10. The method for adapting and optimizing supply and demand flows of a regional parking system driven by reinforcement learning according to claim 9 is characterized in that: The research area of ​​the resource optimal utilization problem facing the management end is divided into a grid matrix, each grid represents a spatial location, and each grid can accommodate at most one parking space or one arriving parking demand; Time is discretized, and the simulation is evenly divided from the start time to the end time; At most one new parking demand can arrive in each period; The total number of available parking spaces changes dynamically; There are parking spaces available to accommodate new parking demands in each time period.

11. The method for adapting and optimizing supply and demand flows of a regional parking system driven by reinforcement learning according to claim 9 is characterized in that: The S4 is specifically: S401, treating the vacant parking spaces as intelligent agents, and the number of intelligent agents in the system changes dynamically over time; S402, the agent state is formed by global information and local information; S403, defining the action space, and determining whether the agent enters the allocation pool for matching within the period; S404, setting the corresponding state transition of the intelligent agent; S405. Design corresponding reward function for state transition.

12. A reinforcement learning driven regional parking system supply and demand flow adaptation and optimization control method according to claim 11, characterized in that: The global information of S402 includes the simulation period to which the current moment belongs, the parking supply spatiotemporal data matrix, and the parking demand spatiotemporal data matrix.

13. The method for adapting and optimizing supply and demand flow of regional parking system driven by reinforcement learning according to claim 11 is characterized in that: The local information of S402 includes the travel time between the location information of the current intelligent body and the location of the newly arrived parking demand.

14. The method for adapting and optimizing supply and demand flow of regional parking system driven by reinforcement learning according to claim 11 is characterized in that: The reward function of S405 is defined in different situations. When there is no parking demand, the reward of the agent is r t i = 0, when there is a parking demand, the potential reward of each agent is defined as 15. The method for adapting and optimizing supply and demand flow of regional parking system driven by reinforcement learning according to claim 9 is characterized in that: The S5 is specifically: S501, input parking demand center and regional parking space layout; S502, initializing system status; S503, generating parking demand according to historical data or distribution characteristics, including arrival time and parking duration; S504, collecting global information of the environment and local information of each agent to form a state; S505, generating the action of the system agent in the time period, determining whether the parking space agent can accept the current parking demand in this time period, if it can accept it, it will enter the matching pool to receive allocation, if it cannot accept it, it will continue to wait for the parking demand in the next time period before making a decision; S506, matching parking requests and acceptable free parking spaces in the matching pool based on a minimum driving distance criterion; S507, removing the matched parking request from the waiting list, and the parking request that has not been successfully matched leaves the parking system, and the corresponding request is also deleted from the waiting list. If an idle parking space is successfully matched, its status changes from idle to occupied; S508, entering the next allocation period, the parking time of each vehicle in the simulation follows a certain distribution characteristic, when the vehicle reaches the set parking time, the vehicle leaves the system, and the corresponding occupied parking space becomes free; S509, from the initial moment to the end moment of the simulation, each time the intelligent allocation decision period is repeated, S503 to S508 are repeated.

16. The method for adapting and optimizing supply and demand flow of regional parking system driven by reinforcement learning according to claim 9 is characterized in that: The S6 adopts a centralized training and distributed execution strategy, utilizes a multi-agent DQN method, uses the same neural network to describe the behavior strategies of all agents, and the trained neural network parameters are shared by all agents.

Images