A deep reinforcement learning approach for real-time recommendation of conflict-free parking spaces

Abstract

The present invention discloses a deep reinforcement learning method for real-time recommendation of conflict-free parking spaces, the method comprising the following steps: using real-time collected vacant parking position data as parking perception data, inputting it into a reinforcement learning PPO model built on a VCS server, extracting features through a feature network, and predicting the social environment of the vehicle; then obtaining the corresponding strategy probability distribution of the vehicle through a strategy network, and calculating the V value using a value network, and then updating the two networks until convergence; in this process, the VCS server calculates the reward of the vehicle's parking perception state and uses the predicted next state of the vehicle to provide a suitable parking strategy for the vehicle; in addition, the present invention takes the actual situation into consideration and uses game rules to provide alternative solutions for the vehicle, thereby solving the problems of difficulty in finding parking spaces and parking space conflicts caused by an increase in vehicles and traffic congestion in real scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of Internet of Things and relates to a deep reinforcement learning method for real-time recommendation of conflict-free parking spaces. Background Art

[0002] As the number of cars in cities increases, it is very time-consuming to find a parking space close to the driving destination. These problems are particularly extreme in on-demand parking in metropolitan centers around the world. In order to meet the parking needs of users, a large number of off-street parking lots are equipped with the necessary infrastructure or personnel to provide the location and number of public vacant parking spaces. However, this approach not only lacks timely information about public parking spaces such as streets and curbs, but also lacks the reliability of crowd worker data, which exacerbates congestion and provides unnecessary time and money expenditure for drivers looking for vacant parking spaces. Drivers can find parking locations through location-based services (LBS). However, due to the complexity of the traffic environment and the nonlinearity of vehicle movement within the city, there is a lack of effective prediction of vehicle routes. There is not enough information about potential parking spaces, so it is difficult to find a suitable parking space, especially during peak hours. The most direct way to provide such information is to publish it on the Internet, which can be done by the owner of the parking space. However, this method is not suitable for scenarios such as free street parking and roadside parking because there is no motivation to publish this data. Data can be derived by building a spatiotemporal model and applying it to historical data with geographical relevance, but the published data is usually collected early and inaccurate.

[0003] Many urban parking systems rely on specialized infrastructure, embedded in roads or vehicles to capture real-time parking space occupancy. But when expanding to large cities, the cost of deploying these sensors or short-range communication technologies is large. With the rapid development of smart mobile devices, vehicular crowd sensing (VCS) has become a popular paradigm. Typically, VCS recruits a group of vehicles to complete perception tasks such as road condition monitoring, traffic environment assessment, and optimization of impact diffusion. However, current research mainly adopts sensor-based solutions, which can only detect the existence of parking spaces, but not the parking space, parking time, hourly price, and whether it is private or public. In addition, these methods cannot estimate future availability information, hindering drivers from planning their trips in advance based on future availability. Therefore, how to achieve real-time recommendation of parking spaces in actual engineering scenarios with numerous interference factors and complex backgrounds has become a difficult problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to propose a deep reinforcement learning method for real-time recommendation of conflict-free parking spaces, so as to solve the problems of difficulty in finding parking spaces due to traffic congestion and parking space conflicts between vehicles in real scenarios.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] A deep reinforcement learning method for real-time recommendation of conflict-free parking spaces includes the following steps:

[0007] Step 1. The VCS server obtains user parking requirements in real time, generates and publishes parking perception tasks;

[0008] Step 2. The VCS server uses a utility function related to the parking perception level and the quality of the parking perception data to attract vehicles to participate in the parking perception task; the vehicles participating in the parking perception task upload the parking perception data to the VCS server;

[0009] Step 3. The process of the VCS server obtaining a suitable parking decision based on all parking perception data collected in the tth time slot is regarded as a non-cooperative game process, and the non-cooperative game process is modeled as a Markov decision process;

[0010] Step 4. Build the reinforcement learning PPO model on the VCS server;

[0011] Among them, the constructed reinforcement learning PPO model consists of feature network, strategy network and value network;

[0012] The processing process of parking perception data in the reinforcement learning PPO model is as follows:

[0013] First, the parking perception data enters the feature network, which extracts the features of the one-dimensional parking perception data and puts them into the strategy network and value network. In the strategy network and value network, the agent is a parking decision maker.

[0014] The agent observes parking perception states from the environment, executes actions derived from its policy network, and then receives the corresponding rewards and next parking states from the environment, finally obtaining the optimal parking perception result for each user;

[0015] Step 5. Calculate the utility of all participating vehicles and form a utility matrix;

[0016] If there is only one Nash equilibrium point in the utility matrix, each user obtains the optimal parking perception result;

[0017] If there are two or more Nash equilibrium points in the utility matrix, multiple users will obtain the same parking perception result. The Pareto principle is added to the non-cooperative game process to simplify multiple Nash equilibrium points into one, providing users with suboptimal parking perception results.

[0018] Step 6. The VCS server returns parking report information to the user, including the parking space location, parking time, and parking fee.

[0019] The present invention has the following advantages:

[0020] As described above, the present invention relates to a deep reinforcement learning method for real-time recommendation of conflict-free parking spaces. The method can better formulate parking strategies, collect vacant parking location information by predicting the social environment of the vehicle, recommend the best parking space to the user in real time, and provide alternative solutions for conflicting vehicles. The method of the present invention solves the problem of difficulty in finding parking spaces due to traffic congestion and parking space conflicts between vehicles in real scenarios, thereby providing real-time parking locations for vehicles, significantly reducing the time and resource waste of drivers in finding parking spaces, and has a wide range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of a deep reinforcement learning method for real-time recommendation of non-conflict parking spaces in an embodiment of the present invention;

[0022] Figure 2 Schematic diagram of the feature extraction network in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0024] This embodiment describes a deep reinforcement learning method for real-time recommendation of conflict-free parking spaces, so as to solve the problem of difficulty in finding parking spaces due to traffic congestion and parking space conflicts between vehicles in real scenarios.

[0025] like Figure 1 As shown, the deep reinforcement learning method for real-time recommendation of conflict-free parking spaces includes the following steps:

[0026] Step 1. The VCS server obtains user parking requirements in real time, generates and publishes parking perception tasks.

[0027] Define a set of vehicles V = {1,…,M} that are randomly deployed in a two-dimensional area around a user's destination.

[0028] Wherein, M is a natural number, M≥2.

[0029] When the distance between the user and the destination is less than the preset distance threshold, the user sends a parking request to the VCS server. The VCS receives the user's request and generates corresponding parking perception tasks, which are randomly published in the above two-dimensional area.

[0030] Step 2. The VCS server uses a utility function related to the parking perception level and the quality of the parking perception data to attract vehicles to participate in the parking perception task; the vehicles participating in the parking perception task upload the parking perception data to the VCS server.

[0031] Multiple vehicles will participate in the parking perception task and upload corresponding parking perception data, which includes parking location l, parking time h, parking price p and parking perception level Among them, parking perception level It is the perception data of how much it contributes to the specified task. In the present invention, it represents the length of time that vehicle i participates in the parking perception task.

[0032] Since more vehicle data is more beneficial to parking perception decisions, in order to expand the number of vehicles participating in parking perception tasks, the VCS server designs a utility function related to the parking perception level and the quality of parking perception data to further attract vehicle participation.

[0033] Specifically, the utility function is designed as shown in formula (1):

[0034]

[0035] Among them, the utility function It consists of two parts: reward function and cost function; the first two and Expressed as a reward function, the last two terms and ∑f i is the cost function. t is the revenue of the parking perception task, is the quality of the real-time parking perception data of vehicle i at time slot t, is the parking perception level of vehicle i at time slot t, represents the battery cost per sensing level, represents the sensing cost, ∑f i represents the total visit cost of vehicle i.

[0036] w ijIt is expressed as the influence coefficient of vehicle j on vehicle i, that is, the mutual dependence between the two vehicles, which can be simply understood as a weight. As vehicles travel in a complex and changing traffic environment, they will have the opportunity to communicate or connect with surrounding vehicles, and even a chain reaction may occur. That is, if vehicle i has a deep connection with vehicle j and vehicle i participates in the parking perception task, then vehicle j is affected by vehicle i and is likely to participate in this task.

[0037] In the embodiment of the present invention, this influence is quantified as w ij .

[0038] The quality of parking perception data is one of the most important parameters in the utility function. For vehicle i at time slot t, the quality of real-time parking perception data is The VCS server dynamically evaluates the authenticity of vehicle i's answers to a series of location-related point of interest (POI) control questions.

[0039] In the present invention, the POI question answered by the vehicle is whether it is near the destination parking lot.

[0040] X{x1,x2,……,x M} represents the answer to the POI question given by the vehicle, Y = y, y2, ..., y M} is the true label.

[0041] Among them, x i represents the answer of vehicle i to the POI question, x i ∈(-1,1), i∈M; that is, when x i =1 means vehicle i is close to the parking lot, x i =0 means vehicle i is far away from the parking lot. i It represents the real answer given by vehicle i to POI.

[0042] y i ∈(0,1, i∈M; that is, when y i =1 means vehicle i is close to the parking lot, y i =0 means vehicle i is far away from the parking lot.

[0043] Therefore, the calculation formula of the parking perception data quality q of vehicle i is shown in formula (2);

[0044]

[0045] Calculated by likelihood function.

[0046] Among them, β is the coefficient vector of the answer X to the POI question submitted by the vehicle in the likelihood estimation. In order to alleviate the overfitting problem that may occur during the calculation process, the L2 regularization term is added to the likelihood estimation for calculation.

[0047] Due to time constraints and user experience, the number of POI questions answered by participants was limited, which was too different from the number of training data, resulting in overfitting. Therefore, an L2 regularization term was added to the likelihood estimation to facilitate calculation.

[0048] Step 3. The process of the VCS server obtaining a suitable parking decision based on all parking perception data collected in the tth time slot is regarded as a non-cooperative game process, and the non-cooperative game process is modeled as a Markov decision process.

[0049] Since each vehicle has the right to make its own choice, for the parking perception task, they evaluate their own utility through formula (1) to decide whether to participate in this task. For example, if the vehicle's participation gets a negative utility, it will leave and not participate in the task. Otherwise, the vehicle will contribute a specific perception level to maximize its utility.

[0050] Step 3.1. The process in which the VCS server obtains a suitable parking decision based on all parking perception data collected in the tth time slot is regarded as a non-cooperative game process, namely, the VCS game process.

[0051] Non-cooperative game is the process in which vehicles compete with each other to maximize their respective utility.

[0052] The VCS game consists of three basic elements: vehicle i, parking perception result set S and utility function U.

[0053] Each vehicle i is regarded as a player participating in the game process. For each vehicle i in the tth time slot, it has a parking perception result set That is, the set of all parking strategies for vehicle i in time slot t.

[0054] VCS game uses formula (3) to express the utility function to maximize the utility of the vehicle.

[0055]

[0056] in, represents the utility of vehicle i in time slot t, represents the parking perception level of vehicle i at time t, represents the parking perception level of all other vehicles except vehicle i at time t.

[0057] Step 3.2. The VCS server models the VCS game as a Markov decision process, i.e., MDP.

[0058] Among them, the decision of each vehicle is made independently.

[0059] MDP is a four-tuple feature consisting of state, action, state transition probability and utility function, recorded as:

[0060] MDP=<S,A,P,U> ;

[0061] State space S: The state space is used to provide the complete parking perception data knowledge required for the agent’s parking decision.

[0062] Specifically, is defined as the parking perception state of vehicle i at time t.

[0063] It consists of the following four parts:

[0064] (1) parking time h, (2) parking fee p, (3) parking location l, (4) parking sensing level of other vehicles before

[0065] Action space A: parking perception level of all vehicles The set represents the action space, and when interacting with the environment, the agent needs to choose a parking awareness level to execute at each parking decision point.

[0066] State transition probability P: Refers to parking sensing status In the implementation of parking awareness level Transition to parking awareness state The probability is an N*N matrix.

[0067] The parking perception state probability at the next moment is calculated through the state transition probability P.

[0068] Using the parking perception state probability at the next moment, the agent is transferred to the next parking perception state

[0069] Utility function U: Formula (2) is used as the utility function to reward participating vehicles.

[0070] Step 4. Build the reinforcement learning PPO model on the VCS server.

[0071] The VCS server uses the reinforcement learning PPO model to analyze and process the data, that is, to calculate rewards for vehicles based on their contributions to obtain the maximum cumulative reward (optimal parking perception data).

[0072] Among them, the constructed reinforcement learning PPO model consists of feature network, strategy network and value network.

[0073] The processing process of parking perception data in the reinforcement learning PPO model is as follows:

[0074] First, the parking perception data enters the feature network, which extracts the features of the one-dimensional parking perception data and puts them into the strategy network and value network. In the strategy network and value network, the agent is a parking decision maker.

[0075] The agent observes parking perception states from the environment, executes actions derived from its policy network, and then receives the corresponding rewards and next parking states from the environment, finally obtaining the optimal parking perception result for each user.

[0076] like Figure 2 As shown in the figure, the feature network consists of a convolutional layer and a fully connected layer. The processing process of the feature network is as follows:

[0077] First, the parking perception state composed of parking perception data is received, and then local features are extracted in the convolutional layer. The fully connected layer integrates the parking space features in the previous layer of the network to obtain a one-dimensional parking space feature.

[0078] In the interaction with the environment, the agent maps the parking space features to a sample space containing many fully connected layers through the policy network, and applies softma to normalize the sample space to obtain the probability of each parking perception level.

[0079] It is worth noting that the policy network trains the next parking perception state from the vehicle’s current parking perception state; the environment will also return a reward value u and the updated parking perception state Constructing experience tuples

[0080] The experience tuple Deposit into the corresponding experience pool D.

[0081] When the length of the experience pool is greater than the preset value, batch group experience values ​​are sampled from the experience pool for updating the policy network and value network. As the game gradually converges, the parking perception status and parking perception level of all vehicles are obtained.

[0082] At the same time, the rewards of all vehicles are accumulated to the maximum to obtain an optimal parking perception result for users.

[0083] For the update of the policy network and the value network in the reinforcement learning PPO model, the time difference error method and the time difference error weighted method are used to optimize the value network and the policy network respectively.

[0084] First define the advantage estimator The expression is as follows:

[0085]

[0086] in, represents the utility of vehicle i at time t, T represents the sum of all time slots t, represents the parking perception level adopted by vehicle i at time slot t The value of , γ represents the discount factor.

[0087] If the discount factor γ = 0, then indicates that the advantage estimator only cares about the utility of the current parking perception state, while the discount factor γ = 1 means that the advantage estimator cares about the overall cumulative utility from the entire time period t.

[0088] For the value network, MSE is applied as the loss function, which is expressed as the difference between the calculated cumulative discounted reward A of the current parking perception state and the critic’s predicted value V. Its loss function L V The specific expressions are as follows:

[0089] For the value network, MSE is applied as the loss function, which is expressed as the difference between the calculated cumulative discounted reward A of the current parking perception state and the critic’s predicted value V. Its loss function L V The specific expressions are as follows:

[0090]

[0091] In formula (5), b represents the mini-batch, B represents the sampling interval, represents the value of vehicle i in mini-batch b, represents the overall cumulative utility of vehicle i in small batch b;

[0092] Because the policy network needs to output the largest possible action advantage, the loss function Lθ required for its training is expressed as follows:

[0093]

[0094] In formula (6),

[0095] Represented as the distribution from existing strategies Estimate the new strategy distribution from the sample The probability update range of ; and limit it to (1-∈, 1+∈), where ∈ represents the clipping hyperparameter;

[0096] represents the parking awareness level of vehicle i in mini-batch b;

[0097] represents the parking perception state of vehicle i in mini-batch b;

[0098] Finally, use formula (7) to update the policy network π, and use formula (8) to update the value network V;

[0099]

[0100]

[0101] Among them, θ represents the parameters of the value network, η θ represents the learning rate of the value network, represents the policy network gradient, represents the parameters of the policy network, represents the learning rate of the policy network, represents the value network gradient.

[0102] During the network update process, the learning strategy network increases V, where the supervision value V is provided by the value network, and then updates the value network to make the score close to the true value, and provides feedback based on the rewards given by the environment.

[0103] The updates of the value network and the policy network are both realized by the gradient descent algorithm. And the present invention derives the optimal parking perception result after the update network reaches convergence. Compared with the ordinary greedy algorithm and random algorithm, the deep reinforcement algorithm is used to extract effective features from the vehicle uploaded data, which can generate parking perception results with more application value.

[0104] The method of the present invention ultimately selects a deep reinforcement learning algorithm based on PPO, taking into account both stability and reliability.

[0105] Step 5. Calculate the utility of all participating vehicles and construct a utility matrix.

[0106] Considering the actual application scenario of vehicles, that is, a certain parking space meets the parking needs of two vehicles, the present invention solves the conflict problem by reasonably designing the utility function and game rules to reduce the waste of energy and time.

[0107] That is, the Pareto principle is added when designing non-cooperative game rules. If there is only one Nash equilibrium point in the utility matrix of the vehicle, it means that there is no conflict when formulating the vehicle parking perception results, and the vehicle can find the optimal parking space. If there are multiple Nash equilibrium points in the utility matrix, it means that the results of the game will conflict.

[0108] Therefore, in the presence of multiple Nash equilibria, the present invention specifies that each player selects a parking strategy based on the Pareto optimal principle, where each player calculates the total utility of participating in the game under the same circumstances, and then selects the optimal parking strategy and the suboptimal parking strategy based on the action tuple with the largest total utility.

[0109] Specifically, first calculate the utility of all participating vehicles and form the utility matrix S NE .

[0110]

[0111] Among them, u i t represents the utility of vehicle i at time t, i∈M, t∈T.

[0112] If there is only one Nash equilibrium point in the utility matrix, each user obtains the optimal parking perception result.

[0113] If there are two or more Nash equilibrium points in the utility matrix, multiple users will obtain the same parking perception result. When there is more than one Nash equilibrium point, it is necessary to ensure that each participant does not deviate from the Pareto optimal principle.

[0114] Assuming that the vehicle crowd-sensing game follows the Pareto optimal principle, we obtain an action tuple s = {s i ,s -i}. Among them, s i represents the parking perception result taken by vehicle i, s -i Represents the parking perception results obtained by all vehicles except vehicle i.

[0115] If the parking perception data calculation of a certain vehicle can obtain the maximum utility, the parking perception result is recommended to the user and the user is advised to apply it. Otherwise, the utility of the parking perception data of the remaining vehicles is calculated.

[0116] The calculated utilities are sorted by size to obtain the suboptimal utility, and the suboptimal parking perception result is recommended to the user.

[0117] Step 6. The VCS server returns the parking report information to the user, including the parking space location, parking time and parking fee. Of course, users can also exchange parking report content with each other.

[0118] Aiming at the parking demand of dynamic vehicles, the method of the present invention designs an incentive mechanism based on external task rewards and internal social network effects to attract vehicles to participate and upload parking perception data. Under limited budget conditions, the optimal parking perception results are provided to users in need, which can effectively reduce unnecessary time and space consumption of users in need. For parking conflicts that occur in real life, the present invention uses the Pareto principle to effectively solve them and provide alternative solutions for conflicting vehicles.

[0119] The method of the present invention realizes the basic function of optimal parking space recommendation by performing various data operations on parking space data in actual application scenarios with rich environmental information, thereby meeting the needs of actual parking users to the greatest extent.

[0120] The method of the present invention is applicable to actual scenarios with complex traffic environments and unstable demand, and solves the problem of parking conflicts in specific actual application scenarios, and performs good parking space recommendations, thus having a wide range of application scenarios.

[0121] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A deep reinforcement learning method for real-time recommendation of conflict-free parking spaces, characterized in that: The steps include: Step 1. The VCS server obtains user parking requirements in real time, generates and publishes parking perception tasks; Define a set of vehicles V = {1, ..., M} that are randomly deployed in a two-dimensional area around a user's destination; Wherein, M is a natural number, M≥2; When the distance between the user and the destination is less than a preset distance threshold, the user sends a parking request to the VCS server, and the VCS receives the user's request and generates a corresponding parking perception task, which is randomly published in the two-dimensional area; Step 2. The VCS server uses a utility function related to the parking perception level and the quality of the parking perception data to attract vehicles to participate in the parking perception task; the vehicles participating in the parking perception task upload the parking perception data to the VCS server; The step 2 is specifically as follows: Parking perception data includes parking location l, parking time h, parking price p and parking perception level The utility function related to the parking perception level and the parking perception data quality is shown in formula (1): Among them, the utility function It consists of two parts: reward function and cost function; the first two and Expressed as a reward function, the last two terms and ∑f i is the cost function; r t is the revenue of the parking perception task, is the quality of the real-time parking perception data of vehicle i at time slot t, is the parking perception level of vehicle i at time slot t, w ij It is expressed as the influence coefficient of vehicle j on vehicle i; represents the battery cost per sensing level, represents the sensing cost, ∑f i represents the total visit cost of vehicle i; The quality of real-time parking perception data for vehicle i at time slot t The VCS server dynamically evaluates the authenticity of vehicle i’s answers to a series of location-related points of interest, or POI control questions; The POI question answered by the vehicle is whether it is near the destination parking lot; X={x1,x2,……,x M } represents the answer to the POI question given by the vehicle, Y = {y, y2, ..., y M } is the true label; Among them, x i represents the answer of vehicle i to the POI question, x i ∈(-1,1), i∈M, that is, when x i =1 means vehicle i is close to the parking lot, x i =0 means vehicle i is far away from the parking lot; y i It represents the real answer given by vehicle i to POI; y i ∈(0,1), i∈M, that is, when y i =1 means vehicle i is close to the parking lot, y i =0 means vehicle i is far away from the parking lot; Therefore, the calculation formula of the parking perception data quality q of vehicle i is shown in formula (2); Calculated by likelihood function; Where β is the coefficient vector of the answer X to the POI question submitted by the vehicle in the likelihood estimation; L2 regularization term is added to the likelihood estimation for calculation; Step 3. The process of the VCS server obtaining a suitable parking decision based on all parking perception data collected in the tth time slot is regarded as a non-cooperative game process, and the non-cooperative game process is modeled as a Markov decision process; The step 3 is specifically as follows: Step 3.

1. The process of the VCS server obtaining a suitable parking decision based on all parking perception data collected in the tth time slot is regarded as a non-cooperative game process, namely, the VCS game process; The VCS game consists of three basic elements: vehicle i, parking perception result set S, and utility function U; Each vehicle i is regarded as a player participating in the game process. For each vehicle i in the tth time slot, it has a parking perception result set That is, the set of all parking strategies for vehicle i in time slot t; VCS game uses formula (3) to express the utility function to maximize the utility of the vehicle; in, represents the utility of vehicle i in time slot t, represents the parking perception level of vehicle i at time t, represents the parking perception level of all other vehicles except vehicle i at time t; Step 3.

2. The VCS server models the VCS game as a Markov decision process, i.e., MDP; Among them, the decision of each vehicle is made independently; MDP is a four-tuple feature consisting of state, action, state transition probability and utility function, recorded as: MDP=<S,A,P,U> ; State space S: The state space is used to provide the complete parking perception data knowledge required for the agent's parking decision; is defined as the parking perception state of vehicle i at time t; It consists of the following four parts: (1) parking time h, (2) parking fee p, (3) parking location l, (4) parking sensing level of other vehicles before Action space A: parking perception level of all vehicles The set represents the action space. When interacting with the environment, the agent needs to choose a parking awareness level to execute at each parking decision point. State transition probability Refers to parking perception status Parking awareness level Transition to parking awareness state The probability of , which is an N*N matrix; The parking perception state probability at the next moment is calculated through the state transition probability P; Using the parking perception state probability at the next moment, the agent is transferred to the next parking perception state Utility function U: Formula (2) is used as the utility function to reward participating vehicles; Step 4. Build the reinforcement learning PPO model on the VCS server; Among them, the constructed reinforcement learning PPO model consists of feature network, strategy network and value network; The processing process of parking perception data in the reinforcement learning PPO model is as follows: First, the parking perception data enters the feature network, which extracts the features of the one-dimensional parking perception data and puts them into the strategy network and value network. In the strategy network and value network, the agent is a parking decision maker. The agent observes parking perception states from the environment, executes actions derived from its policy network, and then receives the corresponding rewards and next parking states from the environment, finally obtaining the optimal parking perception result for each user; Step 5. Calculate the utility of all participating vehicles and form a utility matrix; If there is only one Nash equilibrium point in the utility matrix, each user obtains the optimal parking perception result; If there are two or more Nash equilibrium points in the utility matrix, multiple users will obtain the same parking perception result. The Pareto principle is added to the non-cooperative game process to simplify multiple Nash equilibrium points into one, providing users with suboptimal parking perception results. Step 6. The VCS server returns parking report information to the user, including the parking space location, parking time, and parking fee.

2. The deep reinforcement learning method for real-time recommendation of conflict-free parking spaces according to claim 1, characterized in that: The step 4 is specifically as follows: The feature network consists of a convolutional layer and a fully connected layer. The processing process of the feature network is as follows: First, the parking perception state composed of parking perception data is received, and then local features are extracted in the convolutional layer. The fully connected layer integrates the parking space features in the previous layer of the network to obtain a one-dimensional parking space feature. In the interaction with the environment, the agent maps the parking space features to a sample space containing many fully connected layers through the policy network, and applies softmax to normalize the sample space to obtain the probability of each parking perception level; The policy network trains the next parking perception state from the vehicle’s current parking perception state; the environment will also return a reward value u and the updated parking perception state Constructing experience tuples And store it in the corresponding experience pool D; When the length of the experience pool is greater than the preset value, the batch group experience value is sampled from the experience pool for updating the policy network and the value network. As the game gradually progresses and the network converges, the parking perception status and parking perception level of all vehicles are obtained. At the same time, the rewards of all vehicles are accumulated to the maximum to obtain an optimal parking perception result for users.

3. The deep reinforcement learning method for real-time recommendation of conflict-free parking spaces according to claim 2, characterized in that: For the update of the policy network and value network in the reinforcement learning PPO model, the time difference error method and the time difference error weighted method are used to optimize the value network and the policy network respectively; First define the advantage estimator The expression is as follows: in, represents the utility of vehicle i at time t, T represents the sum of all time slots t, represents the parking perception level adopted by vehicle i at time slot t The value of , γ represents the discount factor; If the discount factor γ = 0, then indicates that the advantage estimator only cares about the utility of the current parking perception state, while the discount factor γ = 1 means that the advantage estimator cares about the overall cumulative utility from the entire time period t; For the value network, MSE is applied as the loss function, which is expressed as the difference between the calculated cumulative discounted reward A of the current parking perception state and the critic’s predicted value V. Its loss function L v The specific expressions are as follows: In formula (5), b represents the mini-batch, B represents the sampling interval, represents the value of vehicle i in mini-batch b, represents the overall cumulative utility of vehicle i in small batch b; The loss function L(θ) is expressed as follows: Formula (6) Represented as the distribution from existing strategies Estimate the new strategy distribution from the sample The probability update range of ; and limit it to (1-∈, 1+∈), where ∈ represents the clipping hyperparameter; represents the parking awareness level of vehicle i in mini-batch b; represents the parking perception state of vehicle i in mini-batch b; Finally, use formula (7) to update the policy network π, and use formula (8) to update the value network V; Among them, θ represents the parameters of the value network, η θ represents the learning rate of the value network, represents the policy network gradient, represents the parameters of the policy network, represents the learning rate of the policy network, represents the value network gradient.

4. The deep reinforcement learning method for real-time recommendation of conflict-free parking spaces according to claim 1, characterized in that: In step 5, the utility of all participating vehicles is calculated and a utility matrix S is constructed. NE ; in, represents the utility of vehicle i at time t, i∈M, t∈T; If there is only one Nash equilibrium point in the utility matrix, each user obtains the optimal parking perception result; If there are two or more Nash equilibrium points in the utility matrix, multiple users will obtain the same parking perception result. When there is more than one Nash equilibrium point, it is necessary to ensure that each participant does not deviate from the Pareto optimal principle; Assuming that the vehicle crowd-sensing game follows the Pareto optimal principle, we obtain an action tuple s = {s i ,s -i }; where s i represents the parking perception result taken by vehicle i, s -i represents the parking perception results obtained by all vehicles except vehicle i; If the parking perception data calculation of a certain vehicle can obtain the maximum utility, then the parking perception result is recommended to the user and the user is advised to apply it; otherwise, the utility of the parking perception data of the remaining vehicles is calculated; The calculated utilities are sorted by size to obtain the suboptimal utility, and the suboptimal parking perception result is recommended to the user.

Images