Vehicle path planning method based on graph structure and reinforcement learning

Abstract

The invention provides a vehicle path planning method based on a graph structure and reinforcement learning, and the method comprises the steps: constructing a traffic map and training a reinforcement learning model, firstly reconstructing a real map, taking the waiting time of signal lamps, morning and evening peak hours and other factors into consideration, and selecting the signal lamps as nodes to construct an aging traffic map; then reinforcement learning model training is carried out, a Dirjkstra algorithm is utilized to construct a suboptimal path in the graph, and in the training process, the suboptimal path is utilized to guide learning how to select an optimal path, path planning is carried out, and the training speed is increased; and finally, real-time vehicle path planning is carried out through the trained reinforcement learning model. According to the method, reinforcement learning is introduced into path planning, the calculation cost is reduced, the calculation time is long, meanwhile, the trained reinforcement learning model can give an optimal path at different time according to different road conditions, and meanwhile, factors such as the distance and the road traffic condition are considered, so that the planned path is more accurate and optimized.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a vehicle path planning method based on graph structure and reinforcement learning. Background Art

[0002] With the rapid development of the national economy, the number of cars owned by residents has continued to rise sharply, and urban traffic jams have become more and more serious. It has become increasingly important to study how to reasonably plan vehicle routes to avoid congestion. Applying reinforcement learning methods to vehicle route planning is a major research direction in this field. Urban vehicle route planning faces many difficulties. Morning and evening rush hours, sudden traffic accidents, etc. will greatly affect the existing urban congestion, and then affect the real-time vehicle route. Secondly, a large number of roads and cities make the planning problem large-scale and difficult to solve.

[0003] As a type of machine learning, reinforcement learning can learn how to solve various complex tasks. With the increasing number of vehicles in cities, reasonable vehicle path planning is a key link in solving urban congestion. Currently, commonly used vehicle path planning algorithms include contraction hierarchy algorithms, path planning algorithms based on model prediction, etc. However, the above methods all have some disadvantages. First, the amount of calculation is relatively large and the calculation is complex; second, the real-time urban road traffic conditions are not taken into consideration. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a vehicle path planning method based on graph structure and reinforcement learning. By training the reinforcement learning model with traffic road graph and enhanced real-time data, an efficient model that can obtain the optimal path is obtained to perform real-time path planning for the vehicle.

[0005] A vehicle path planning method based on graph structure and reinforcement learning includes the following steps:

[0006] Step S1: Map data processing, including road data processing and road segmentation, standardizing road information, dividing road segments, marking traffic lights in the map, and then connecting these traffic lights in sequence according to the road, and the road between two traffic lights is regarded as a separate segment;

[0007] Step S2: Based on the road information, with traffic lights as nodes and time factors introduced, multiple time-effective traffic maps are constructed. First, a benchmark traffic map is constructed. The benchmark traffic map is constructed by using traffic lights as nodes to construct a node matrix storage map. The distance between traffic lights is used as the weight of the edge of the map. The building locations are then analyzed and clustered with traffic lights as the center. Then, a time-effective traffic map is constructed. The average passing time of different sections in different time periods is obtained through statistics based on real data. The actual passing length of sections in different time periods is calculated, and the edges of the map are updated to obtain a time-effective traffic map.

[0008] Step S3: training the reinforcement learning model, randomly selecting the starting point and the end point in the time efficiency diagram, simulating real vehicle driving, assigning the starting point and the end point to the nearest traffic light according to the distance, and then using the Dijkstra algorithm to obtain the shortest path between each time efficiency traffic diagram as the suboptimal path, training the reinforcement learning model, finding the optimal path between the starting point and the end point, and using the suboptimal path to speed up the training speed, and repeating the process continuously, and finally obtaining a reinforcement learning model that can plan the optimal path;

[0009] Step S4: Use the trained reinforcement learning model to perform path planning, convert the user input data into model input, obtain the optimal path through the reinforcement learning model, and update the optimal path according to the user's real-time location.

[0010] Further, step S1 includes:

[0011] Road data processing, unifying the storage formats of different road information, including road annotation, building the required benchmark road map for different formats of road maps, setting the unit length of the benchmark road map, identifying the roads through manual or intelligent identification methods, and recording them in the benchmark road map. If there is a situation of road overlap, different roads are distinguished according to the road numbers;

[0012] Road segmentation, based on the road segment length data required for the subsequent construction of the traffic map, the roads in the benchmark road map are segmented, specifically including traffic light position marking and segment construction and recording; traffic light position marking, using the latitude and longitude addresses of the traffic lights to mark the positions of the traffic lights in the benchmark road map in the form of points; segment construction and recording, connecting the traffic lights in the road in sequence, recording the road section between two traffic lights as a separate section, and recording the length of the section, the starting traffic light, the minimum speed limit, the maximum speed limit and the section type, where the minimum speed limit, the maximum speed limit and the section type are consistent with the road where the section is located, and the adjacent traffic lights but not on the same road are not connected.

[0013] Further, step S2 includes:

[0014] First, we construct the nodes of the benchmark traffic map. A node in the map represents a set, which is a set with a traffic light as the center and multiple buildings nearby. In order to obtain such a set, we use the Dis clustering algorithm to cluster based on the differences in longitude and latitude. The clustering results ensure that the traffic light closest to the building in different sets is the traffic light represented by the cluster center.

[0015] Secondly, a benchmark traffic map is constructed. Cartesian products are performed on the cluster sets obtained above. The result is used as a matrix. The initial value of all points in the matrix is ​​set to -1. According to the road section information, when there is a path directly corresponding to the corresponding traffic light, the value of the corresponding position in the matrix is ​​set to the road section length.

[0016] Finally, a time-sensitive traffic map is constructed. According to actual statistical data, the city is divided into peak hours, normal hours, and late night hours. Then, statistics are conducted through real data to obtain the average passing time of different sections at different times. Finally, the average passing time is multiplied by the maximum speed limit of the corresponding section, and the result is added as an additional distance to the edge weight of the corresponding section, so as to obtain traffic maps for different time periods, that is, time-sensitive traffic maps.

[0017] Furthermore, the Dis clustering algorithm is used for clustering, including:

[0018] Initialization: Select K initial centroids, which are all traffic lights, and each traffic light is a centroid;

[0019] Calculate the interval distance: For each point in the traffic light, calculate the distance between it and each directly connected traffic light. Assume that the longitude and latitude of the two traffic lights are represented as Traffic (lat1, lon1) and Traffic (lat2, lon2) respectively. The calculation formula of the interval distance Diff is as follows:

[0020]

[0021] Divide the building: For each traffic light, take it as the midpoint and calculate the division range. Suppose the latitude and longitude of the traffic light are represented as Traffic (lat1, lon1), then the division range is calculated as follows:

[0022] Latitude=lat1+sin(θ)*Diff

[0023] Longtitude=lon1+sin(θ)*Diff

[0024] Where θ represents the angle, ranging from 0° to 360°;

[0025] For all buildings, if their longitude and latitude are within the range of (Latitude, Longtitude), they are within the distance range with this traffic light as the center;

[0026] For the remaining buildings that are not within the division range, calculate their distances from all traffic lights, and then divide them into clusters corresponding to the nearest traffic lights.

[0027] Further, step S3 includes:

[0028] First, simulate real vehicle navigation, select the building as the starting point, and construct the path planning task and suboptimal path, including:

[0029] Building mapping: When selecting the starting point, the clusters to which the selected buildings belong must not be adjacent to avoid too short a distance. Then the starting point is processed and mapped to the node of the constructed benchmark traffic map. Specifically, the input longitude and latitude of the building are replaced with the longitude and latitude of the traffic light at the center of the corresponding cluster.

[0030] Suboptimal path construction, in the time-effective traffic map of each period, the Dijkstra algorithm is used to calculate the shortest path as the suboptimal path. The suboptimal path is recorded as follows:

[0031] P = {node start ,node1,...,node end , {period T}}

[0032] Among them, node is the traffic light in the path, specifically represented by the number of the traffic light, and time period T indicates which time-effective traffic map the path is obtained from, which is an enumeration type value;

[0033] Then the reinforcement learning model is trained. When training the reinforcement learning model, the constructed suboptimal path will be used to assist and accelerate the model training. First, the model input is constructed. The input of the model is a set, which includes the longitude and latitude of the starting and ending buildings of the path and the number of the current time period.

[0034] Furthermore, the model training is guided by the constructed suboptimal path, including:

[0035] Model input conversion, first process the model input, classify the input building longitude and latitude through the clustering model, find the corresponding distance, and then replace it with the traffic light number of the corresponding cluster center to meet the use conditions of the suboptimal path. After replacement, the model input is expressed as: {traffic light number 1, traffic light number 2, time period number}, where traffic light number 1 corresponds to the starting point and traffic light number 2 corresponds to the end point;

[0036] The model is trained through the suboptimal path. During the training process, the model uses the suboptimal path to give the next expected traffic light number with a probability of 1-α, and uses the output predicted by the model as the next expected traffic light number with a probability of α. α gradually increases with the increase of training rounds. The specific calculation formula is:

[0037] α=α+α*β

[0038] in, β is the growth parameter, which controls the rate of increase of α. Each round of training increases the size of α through this formula;

[0039] Output path construction: After the model obtains the input, it will give the next expected traffic light number and record the expected traffic light number as the relay point of the path. Then, the model input will be reconstructed with the given expected traffic light number. The specific operation is to replace the first item of the input, input the model again, calculate the next expected traffic light number, and reach the end point. All the expected traffic lights are connected on the map to obtain the path output by the model.

[0040] Further, step S4 includes:

[0041] Get input, use the trained clustering model to find the corresponding traffic light according to the starting point and end point selected by the user, and then select the corresponding time period according to the current time to construct the initial input of the model;

[0042] The optimal path is updated in real time. The system will obtain the longitude and latitude of the user's current location in real time, and then use the trained clustering model to find the corresponding traffic light. At the same time, it will re-obtain the time, construct a new input, and then input it into the model to obtain the latest optimal path given by the model.

[0043] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0044] The present invention processes the map information and constructs the required graph structure to meet the input results of subsequent steps. The starting point of the vehicle may be any building, and a large number of building locations will increase the difficulty of problem solving. Therefore, the present invention adopts a clustering method to represent multiple buildings with traffic lights, which effectively reduces the difficulty of training. Then, by combining the time periods and analyzing the specific traffic conditions in different time periods, different time-effect traffic maps are constructed, which can better symbolize the actual vehicle driving conditions. When training the model, by constructing a suboptimal path in the constructed time-effect traffic map and guiding the training in the early stage of training, it is effectively avoided that when the early model is insufficiently trained, there are many output errors, which reduces the training efficiency and the appearance of insufficient model capabilities. In actual use, the present invention obtains the latest location information at a regular interval and updates the current model output, so that the output construction is more flexible and in line with the actual driving conditions;

[0045] The present invention introduces reinforcement learning into path planning, thereby avoiding the problems of high computational cost and long computational time faced by traditional path planning. In actual use, the trained neural network model can be used to output the required path in a timely manner, and the computational amount and cost are lower. At the same time, the scheme of the present invention combines the time factor and can be more in line with the current situation of urban roads. The trained reinforcement learning model can give the optimal path according to different road conditions at different times, while taking into account factors such as distance and road traffic conditions, so that the planned path is more accurate and optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is an overall flow chart of a vehicle path planning method based on graph structure and reinforcement learning provided by an embodiment of the present invention;

[0048] Figure 2 is a flowchart of map data processing provided by an embodiment of the present invention;

[0049] Figure 3 It is a flow chart of constructing a time-effective traffic map provided by an embodiment of the present invention;

[0050] Figure 4 is a flow chart of reinforcement learning model training provided by an embodiment of the present invention;

[0051] Figure 5 It is a flow chart of using the model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The present invention will be further described below in conjunction with the accompanying drawings:

[0054] The data source of the present invention is the detailed map data of a certain place, including information about traffic lights, roads, buildings, etc. The specific analysis data includes traffic light related data: traffic light location, traffic light number, traffic light type, signal setting time; building related data: building location, building number, building type; road related data: road number, road type, road location, road length, maximum speed limit, minimum speed limit. The specific explanation of map data is as follows:

[0055]

[0056]

[0057] Figure 1 The present invention provides an overall flow chart of a vehicle path planning method based on graph structure and reinforcement learning, wherein the method includes map data processing, time-sensitive traffic map construction, model training and model use, wherein map data processing includes road data processing and road section division, time-sensitive traffic map construction is to take factors such as time into consideration of the model so that the planned path is more in line with the current situation, including baseline traffic map construction and time-sensitive traffic map construction; model training is to train a reinforcement learning model by randomly selecting a starting point on the time-sensitive traffic map to learn how to select the optimal path, wherein the training speed is accelerated by constructing a suboptimal path, and model use is to plan the optimal path in real time through input data.

[0058] Figure 2 This is a specific flow chart for map data processing. First, the source data is processed, including the signal light data and road data. Then these data are used to build the required environment map. The details are as follows:

[0059] Road data processing, unifying the storage formats of different road information, including: road marking, building the required benchmark road map for different formats of road maps, the unit length of the benchmark road map is 0.01 km, identifying the roads through manual or intelligent identification methods, and recording them in the benchmark road map. If there is a situation of overlapping roads, distinguish different roads according to the road numbers, and then record the different roads by building a new benchmark map;

[0060] Road segmentation, segmenting the roads in the baseline road map according to the road segment length data required for the subsequent construction of the environment map, specifically including traffic light position marking and segment construction and recording; traffic light position marking, marking the location of the traffic light in the baseline road map through the latitude and longitude address of the traffic light, specifically marking the traffic light in the form of points in the baseline road map; segment construction and recording, connecting the traffic lights in the road in sequence, recording the road section between two traffic lights as a separate section, and recording the length of the section, the starting traffic light, the minimum speed limit, the maximum speed limit and the section type, where the minimum speed limit, the maximum speed limit and the section type are consistent with the road where the section is located, and the traffic lights that are adjacent but not on the same road are not connected.

[0061] Through the above processing, the necessary data traffic lights and road section information for constructing the benchmark traffic map are obtained, and then the required fields are converted into numerical data.

[0062] Figure 3 This is the construction process of the time-effective traffic map of the present invention. The map is composed of nodes and edges. First, the map nodes of the benchmark traffic map are constructed through traffic lights, and then the relationship between the nodes and the edges of the map are constructed through road section information. The specific operations are as follows:

[0063] First, we construct the nodes of the benchmark traffic map. A node in the map represents a set, which is specifically represented as a set centered on a traffic light and multiple buildings nearby. In order to obtain such a set, the Dis clustering algorithm is used to perform clustering based on the differences in longitude and latitude. The clustering result ensures that the traffic light closest to the building in different sets is the traffic light represented by the cluster center.

[0064] Among them, the implementation steps of the Dis clustering algorithm are:

[0065] Initialization: Select K initial centroids, which are all traffic lights, and each traffic light is a centroid;

[0066] Calculate the interval distance: For each point in the traffic light, calculate the distance between it and each directly connected traffic light. Assuming that the longitude and latitude of the two traffic lights are represented as Traffic (lat1, lon1) and Traffic (lat2, lon2), the interval distance Diff calculation formula is as follows:

[0067]

[0068] This distance applies to both lights.

[0069] Divide the building: For each traffic light, calculate the division range with it as the midpoint. Assuming the latitude and longitude of the traffic light are expressed as Traffic (lat1, lon1), the calculation formula for the division range is as follows:

[0070] Latitude=lat1+sin(θ)*Diff

[0071] Longtitude=lon1+sin(θ)*Diff

[0072] Where θ represents the angle, ranging from 0° to 360°;

[0073] For all buildings, if its longitude and latitude are within the range of (Latitude, Longtitude), it belongs to the distance range with this traffic light as the center;

[0074] For the remaining buildings that are not within the division range, calculate their distances from all traffic lights, and then divide them into clusters corresponding to the nearest traffic lights.

[0075] Then, a benchmark traffic map is constructed. Cartesian products are performed on the cluster sets obtained above. The result is used as a matrix. The initial value of all points in the matrix is ​​set to -1. According to the road section information, when there is a path directly corresponding to the corresponding traffic light, the value of the corresponding position in the matrix is ​​set to the road section length.

[0076] Finally, we construct a time-sensitive traffic map. First, we divide the city into morning and evening rush hours, normal hours, and late night hours based on actual statistical data. Then, we use real data to conduct statistics to obtain the average passing time of different sections in different periods. Finally, we multiply the average passing time by the maximum speed limit of the corresponding section, and add the multiplication result as an additional distance to the edge weight of the corresponding section, so as to obtain traffic maps for different periods, that is, time-sensitive traffic maps.

[0077] Figure 4 It is the training process of the reinforcement learning model. First, randomly select a building as the starting point, simulate real vehicle planning, and accelerate model training by building a suboptimal path, while enhancing the model effect. The specific operations are as follows:

[0078] First, simulate real vehicle navigation, select the building as the starting point, and construct the path planning task and suboptimal path, including:

[0079] Building mapping: When selecting the starting point, the clusters to which the selected buildings belong must not be adjacent to avoid too short a distance. Then the starting point is processed and mapped to the node of the constructed benchmark traffic map. Specifically, the input longitude and latitude of the building are replaced with the longitude and latitude of the traffic light at the center of the corresponding cluster.

[0080] Suboptimal path construction, in the time-effective traffic map of each period, the Dijkstra algorithm is used to calculate the shortest path as the suboptimal path. The suboptimal path is recorded as follows:

[0081] P = {node start , node1,...,node end , {period T}}

[0082] Among them, node is the traffic light in the path, specifically represented by the number of the traffic light. Time period T indicates which time-effective traffic map the path is obtained from, and is an enumeration type value, specifically represented as follows:

[0083] Period Number Corresponding meaning 1 Morning rush hour 2 Evening peak period 3 Normal time 4 Late Night

[0084] Then, the reinforcement learning model training is carried out. When training the reinforcement learning model, the constructed suboptimal path will be used to assist and accelerate the model training. First, the model input is constructed. The input of the model is a set, which is specifically expressed as follows:

[0085] Input number Meaning Example 1 The latitude and longitude of the building at the start of the path [104.92193082,30.75117200] 2 The latitude and longitude of the building at the end of the path [103.88046799,30.82336421] 3 The number of the current period 1

[0086] In the early stage of training the model, the training is guided by the constructed suboptimal path. The specific operations are as follows:

[0087] Model input conversion,First, process the model input, classify the input building longitude and latitude through the clustering model, find the corresponding distance, and then replace it with the traffic light number of the corresponding cluster center to meet the use conditions of the suboptimal path. After replacement, the model input is expressed as: {traffic light number 1, traffic light number 2, time period number}, where traffic light number 1 corresponds to the starting point and traffic light number 2 corresponds to the end point.

[0088] The model is trained through the suboptimal path. During the training process, the model uses the suboptimal path to give the next expected traffic light number with a probability of 1-α, and uses the output predicted by the model as the next expected traffic light number with a probability of α. α gradually increases with the increase of training rounds. The specific calculation formula is:

[0089] α=α+α*β

[0090] in, β is a growth parameter that controls the rate of increase of α. With each round of training, the size of α is increased by this formula.

[0091] Output path construction: After the model obtains the input, it will give the next expected traffic light number. At this time, the expected traffic light number is recorded as the relay point of the path, and then the model input is reconstructed with the given expected traffic light number. The specific operation is to replace the first item of the input, input the model again, and calculate the next expected traffic light number until the end point is reached. At this time, all the expected traffic lights are connected on the map to obtain the path given by the model.

[0092] Figure 5 This is the actual use process of the model of the present invention. In actual use, the trained reinforcement learning model is called regularly to update the path information in real time to better meet actual needs. The specific operations are:

[0093] Get input. In actual use, let the user select the starting point and the end point, then use the trained clustering model to find the corresponding traffic light, and then select the corresponding time period according to the current time to construct the initial input of the model;

[0094] The optimal path is updated in real time. In actual use, the system obtains the user's current location latitude and longitude every minute, and then uses the trained clustering model to find the corresponding traffic light. At the same time, it re-obtains the time, constructs a new input, and then inputs it into the model to obtain the latest optimal path given by the model. For a trained model, this process is very fast.

[0095] The present invention is based on reinforcement learning and integrates graph-related ideas (paths, clustering, etc.) into traditional path planning methods, replacing the original algorithms such as contraction levels that cannot be updated in real time and have huge computational complexity, thereby making the results more timely. Experiments have shown that this method has a good evaluation effect, a wide range of applications, and a short running time.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle path planning method based on graph structure and reinforcement learning, characterized in that: The method comprises: Step S1: Map data processing, including road data processing and road segmentation, standardizing road information, dividing road segments, marking traffic lights in the map, and then connecting these traffic lights in sequence according to the road, and the road between two traffic lights is regarded as a separate segment; Step S2: Based on the road information, with traffic lights as nodes and time factors introduced, multiple time-effective traffic maps are constructed. First, a benchmark traffic map is constructed. The benchmark traffic map is constructed by using traffic lights as nodes to construct a node matrix storage map. The distance between traffic lights is used as the weight of the edge of the map. The building locations are then analyzed and clustered with traffic lights as the center. Then, a time-effective traffic map is constructed. The average passing time of different sections in different time periods is obtained through statistics based on real data. The actual passing length of sections in different time periods is calculated, and the edges of the map are updated to obtain a time-effective traffic map. Step S3: training the reinforcement learning model, randomly selecting the starting point and the end point in the time efficiency diagram, simulating real vehicle driving, assigning the starting point and the end point to the nearest traffic light according to the distance, and then using the Dijkstra algorithm to obtain the shortest path between each time efficiency traffic diagram as the suboptimal path, training the reinforcement learning model, finding the optimal path between the starting point and the end point, and using the suboptimal path to speed up the training speed, and repeating the process continuously, and finally obtaining a reinforcement learning model that can plan the optimal path; Step S4: Use the trained reinforcement learning model to perform path planning, convert the user input data into model input, obtain the optimal path through the reinforcement learning model, and update the optimal path according to the user's real-time location.

2. The method according to claim 1, characterized in that The step S1 further comprises: Road data processing, unifying the storage formats of different road information, including road annotation, building the required benchmark road map for different formats of road maps, setting the unit length of the benchmark road map, identifying the roads through manual or intelligent identification methods, and recording them in the benchmark road map. If there is a situation of road overlap, different roads are distinguished according to the road numbers; Road segmentation, based on the road segment length data required for the subsequent construction of the traffic map, the roads in the benchmark road map are segmented, specifically including traffic light position marking and segment construction and recording; traffic light position marking, using the latitude and longitude addresses of the traffic lights to mark the positions of the traffic lights in the benchmark road map in the form of points; segment construction and recording, connecting the traffic lights in the road in sequence, recording the road section between two traffic lights as a separate section, and recording the length of the section, the starting traffic light, the minimum speed limit, the maximum speed limit and the section type, where the minimum speed limit, the maximum speed limit and the section type are consistent with the road where the section is located, and the adjacent traffic lights but not on the same road are not connected.

3. The method according to claim 1, characterized in that: The step S2 further comprises: First, we construct the nodes of the benchmark traffic map. A node in the map represents a set, which is a set with a traffic light as the center and multiple buildings nearby. In order to obtain such a set, we use the Dis clustering algorithm to cluster based on the differences in longitude and latitude. The clustering results ensure that the traffic light closest to the building in different sets is the traffic light represented by the cluster center. Secondly, a benchmark traffic map is constructed. Cartesian products are performed on the cluster sets obtained above. The result is used as a matrix. The initial value of all points in the matrix is ​​set to -1. According to the road section information, when there is a path directly corresponding to the corresponding traffic light, the value of the corresponding position in the matrix is ​​set to the road section length. Finally, a time-sensitive traffic map is constructed. According to actual statistical data, the city is divided into peak hours, normal hours, and late night hours. Then, statistics are conducted through real data to obtain the average passing time of different sections at different times. Finally, the average passing time is multiplied by the maximum speed limit of the corresponding section, and the result is added as an additional distance to the edge weight of the corresponding section, so as to obtain traffic maps for different time periods, that is, time-sensitive traffic maps.

4. The method according to claim 3, characterized in that The Dis clustering algorithm is used for clustering, including: Initialization: Select K initial centroids, which are all traffic lights, and each traffic light is a centroid; Calculate the interval distance: For each point in the traffic light, calculate the distance between it and each directly connected traffic light. Suppose the longitude and latitude of the two traffic lights are represented as Traffic (lat1, lon1) and Traffic (lat2, lon2) respectively. The calculation formula of the interval distance Diff is as follows: Divide the building: For each traffic light, take it as the midpoint and calculate the division range. Suppose the latitude and longitude of the traffic light are represented as Traffic (lat1, lon1), then the division range is calculated as follows: Latitude=lat1+sin(θ)*Diff Longtitude=lon1+sin(θ)*Diff Where θ represents the angle, ranging from 0° to 360°; For all buildings, if their longitude and latitude are within the range of (Latitude, Longtitude), they are within the distance range with this traffic light as the center; For the remaining buildings that are not within the division range, calculate their distances from all traffic lights, and then divide them into clusters corresponding to the nearest traffic lights.

5. The method according to claim 1, characterized in that The step S3 further comprises: First, simulate real vehicle navigation, select the building as the starting point, and construct the path planning task and suboptimal path, including: Building mapping: When selecting the starting point, the clusters to which the selected buildings belong must not be adjacent to avoid too short a distance. Then the starting point is processed and mapped to the node of the constructed benchmark traffic map. Specifically, the input longitude and latitude of the building are replaced with the longitude and latitude of the traffic light at the center of the corresponding cluster. Suboptimal path construction, in the time-effective traffic map of each period, the Dijkstra algorithm is used to calculate the shortest path as the suboptimal path. The suboptimal path is recorded as follows: P = {node start , node1,...,node end , {period T}} Among them, node is the traffic light in the path, specifically represented by the number of the traffic light, and time period T indicates which time-effective traffic map the path is obtained from, which is an enumeration type value; Then the reinforcement learning model is trained. When training the reinforcement learning model, the constructed suboptimal path will be used to assist and accelerate the model training. First, the model input is constructed. The input of the model is a set, which includes the longitude and latitude of the starting and ending buildings of the path and the number of the current time period.

6. The method according to claim 5, characterized in that Guide model training through constructed suboptimal paths, including: Model input conversion, first process the model input, classify the input building longitude and latitude through the clustering model, find the corresponding distance, and then replace it with the traffic light number of the corresponding cluster center to meet the use conditions of the suboptimal path. After replacement, the model input is expressed as: {traffic light number 1, traffic light number 2, time period number}, where traffic light number 1 corresponds to the starting point and traffic light number 2 corresponds to the end point; The model is trained through the suboptimal path. During the training process, the model uses the suboptimal path to give the next expected traffic light number with a probability of 1-α, and uses the output predicted by the model as the next expected traffic light number with a probability of α. α gradually increases with the increase of training rounds. The specific calculation formula is: α＝α+α*β in, β is the growth parameter, which controls the rate of increase of α. Each round of training increases the size of α through this formula; Output path construction: After the model obtains the input, it will give the next expected traffic light number and record the expected traffic light number as the relay point of the path. Then, the model input will be reconstructed with the given expected traffic light number. The specific operation is to replace the first item of the input, input the model again, calculate the next expected traffic light number, and reach the end point. All the expected traffic lights are connected on the map to obtain the path output by the model.

7. The method according to claim 1, characterized in that The step S4 further comprises: Get input, use the trained clustering model to find the corresponding traffic light according to the starting point and end point selected by the user, and then select the corresponding time period according to the current time to construct the initial input of the model; The optimal path is updated in real time. The system will obtain the longitude and latitude of the user's current location in real time, and then use the trained clustering model to find the corresponding traffic light. At the same time, it will re-obtain the time, construct a new input, and then input it into the model to obtain the latest optimal path given by the model.

Images