A personalized travel route recommendation method based on reinforcement learning

Abstract

The present invention discloses a personalized tourist route recommendation method based on reinforcement learning. First, the mobile transportation time between different scenic spots and the average visit time of each scenic spot are obtained, and a scenic spot-scenic spot transition probability model and a user preference model are established according to the user's historical visit tourist routes. Then, based on the user preference model, the scenic spot-scenic spot transition probability model and the scenic spot heat model, a scenic spot utility function model is constructed. Finally, based on the obtained scenic spot utility function model and the personalized tourist route constraints, a reinforcement learning algorithm is designed to obtain the tourist route with the highest matching degree as the final recommended tourist route to the user. The present invention solves the problem that the traditional tourist route recommendation method cannot meet the personalized needs of tourists, while taking into account user preferences and the popularity of scenic spots, and avoiding the cold start problem of scenic spot categories.

Description

Technical Field

[0001] The present invention relates to the technical field of travel route recommendation, and mainly to a personalized travel route recommendation method based on reinforcement learning. Background Art

[0002] With the development of high-precision positioning technology and the maturity and widespread application of various mobile devices (smartphones, sports bracelets, and car-mounted positioning devices, etc.), all aspects of people's lives have been greatly changed. As a new Internet platform connecting cyberspace and the physical world, more and more social media websites have sprung up like mushrooms after a rain. Users actively or passively leave their geographic location information in location-based social networks, which generates large-scale spatiotemporal trajectory data (location check-in data, travel trajectory data, etc.). Massive spatiotemporal trajectory data records users' mobile activities in the real world and contains rich personalized interest preference information. How to mine users' interest preferences for attractions through geographic location data and provide users with personalized travel route recommendation services has become an important problem that needs to be solved in the recommendation field.

[0003] Although traditional travel recommendation systems can provide tourists with standard travel packages, these packages usually only include popular attractions and cannot meet the personalized needs of tourists. Existing mainstream methods attempt to transform the travel route recommendation problem into a variant of the directional problem, and consider user personalized preferences and spatiotemporal constraints at the same time, and then use optimization algorithms to recommend group routes. Specifically, user personalized preferences are calculated based on category information, that is, the longer the user stays in a certain attraction category or the more times they visit, the more interested the user is in this attraction category, which will cause the cold start problem of the attraction category.

[0004] Most current personalized itinerary recommendation methods have the following problems: (1) User travel experience refers to the user's satisfaction with the entire travel process. However, previous personalized itinerary recommendation work mostly believes that travel experience only involves the visit of scenic spots, and only considers the popularity of POIs and the user's interest in scenic spots when designing a suitable itinerary. (2) The visit pattern is asymmetric. For example, the conversion probability of "restaurant" → "cafe" is much greater than the conversion probability of "cafe" → "restaurant". (3) The currently visited scenic spot is not only affected by the last visited scenic spot, but also by other scenic spots in the travel route. Summary of the invention

[0005] Purpose of the invention: In view of the problems existing in the above-mentioned background technology, the present invention provides a personalized travel route recommendation method based on reinforcement learning. First, the travel time between different scenic spots and the average visit time of each scenic spot are obtained; then, the user preference model and the scenic spot-scenic spot transfer probability model are established according to the user's historical visit to the tourist route; then, the scenic spot utility function model is constructed according to the user preference model, the scenic spot-scenic spot transfer probability model and the scenic spot heat model; finally, according to the scenic spot utility function model and the personalized tourist route constraints, a reinforcement learning algorithm is used to obtain the most matching tourist route as the final recommended tourist route to the user.

[0006] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] A personalized travel route recommendation method based on reinforcement learning includes the following steps:

[0008] Step S1, obtaining the mobile transportation time between different scenic spots and the average visit time of each scenic spot;

[0009] Step S2: Establishing a scenic spot-scenic spot transfer probability model and a user preference model based on the user's historical travel routes;

[0010] Step S3, constructing a scenic spot heat model, and constructing a scenic spot utility function model based on the user preference model, the scenic spot-scenic spot transfer probability model and the scenic spot heat model;

[0011] Step S4: Based on the obtained scenic spot utility function model and personalized travel route constraints, a reinforcement learning algorithm is designed to obtain the travel route with the highest matching degree as the final travel route recommended to the user.

[0012] Furthermore, the specific steps of obtaining the traffic travel time and the average visit time of different scenic spots in step S1 include:

[0013] Step S1.1, obtaining the travel time between different scenic spots;

[0014] According to the distance between two scenic spots, the travel modes between scenic spots are divided into walking, cycling and driving. When the distance between scenic spots is less than 2km, the walking time is calculated by default; when the distance between scenic spots is between 2km-5km, the cycling time is calculated by default; when the distance between scenic spots is greater than 5km, the driving time is calculated by default; the travel time between two scenic spots when different cycling modes are used is obtained based on the third-party map API; the scenic spots are p i and attractions j The moving time between i ,p j );

[0015] Step S1.2, obtaining the average visit time of each scenic spot;

[0016] According to the historical travel routes of users and groups, the average visit time of each scenic spot is calculated; the visit time of a scenic spot is the time difference between arriving at a scenic spot and leaving the scenic spot. i The average access time is recorded as D(p i ).

[0017] Furthermore, the specific steps of establishing the scenic spot-scenic spot transfer probability model and the user preference model in step S2 include:

[0018] Step S2.1: Number the scenic spots and user data to obtain the scenic spot p i and user u i The one-hot encoding is passed through the embedding layer to obtain the initial representation vector P of the scenic spot i and the user's initial representation vector U i ;

[0019] Step S2.2, learning the output representation vector of the scenic spot through the graph attention network;

[0020] User visits attractions i Next visit attractions j , learning attractions i The output representation vector of the lth layer of the neural network The calculation is as follows:

[0021]

[0022] in Indicates attractions i The output representation vector at layer l is, Initialize the output representation vector to be the attraction point p i The initial representation vector P i , α ij The weights of other scenic spots when representing the output vector are calculated as follows:

[0023]

[0024] Where W e Represents the weight parameter of the attention network, and the weights of other scenic spots are calculated by using the normalized exponential function; LeakyReLU() is the activation function, which gives a non-zero slope to all negative values. The calculation process is as follows:

[0025]

[0026] Among them, a is a fixed parameter in the interval (1,∞).

[0027] Step S2.3, learning the input representation vector of the scenic spot through the graph attention network;

[0028] According to the attractions i Previous visit attractions j , learning attractions i The input representation vector at layer l The calculation is as follows:

[0029]

[0030] Indicates attractions i The input representation vector at layer l is Initialize the input representation vector to be the attraction p i The initial representation vector P i , β ij The weights of other scenic spots when the input represents a vector are calculated as follows:

[0031]

[0032] W b Represents the weight parameters of the attention network, and uses the normalized exponential function to calculate the weights of other attractions;

[0033] Step S2.4: According to the scenic spot p i The output representation vector and the scenic spot p in the Lth layer j The input representation vector at layer L is used to calculate p i -p j The transition probability model is as follows:

[0034]

[0035] in, For attractions i The output representation vector at layer L is For attractions j The input representation vector at layer L is are learnable parameters;

[0036] Step S2.5: According to the scenic spot p i Based on the output representation vector and input representation vector of layer l, calculate the scenic spot p i The representation vector at layer l is as follows:

[0037]

[0038] in, It is a tourist attractioni In the representation vector at the lth layer, || is the concatenation operation;

[0039] Step S2.6: Based on the representation vector of the scenic spot at the lth layer, the representation vector of the user at the lth layer is calculated as follows:

[0040]

[0041] in, and For user u i The representation vectors at the lth and l+1th layers are, It is a tourist attraction i The representation vector at the lth layer; V l represents the weight matrix of the lth layer of the graph neural network, b l represents the bias vector of the lth layer of the graph neural network, Aggreate(·) is the average aggregation function;

[0042] Step S2.7: Based on the representation vectors of the scenic spots and the users, the user preference model for the scenic spots is calculated as follows:

[0043]

[0044] Furthermore, the specific steps of constructing the scenic spot utility function model in step S3 include:

[0045] Step S3.1: Construct a scenic spot popularity model Pop(p j );

[0046] Step S3.2: Based on the user preference model, the attraction-attraction transfer probability model and the attraction popularity model, a scenic spot utility function model is constructed as follows:

[0047] U(u,p j )=Pop(p j )+R uj +S ij

[0048] Among them, Pop(p j ) represents the scenic spot heat model, R uj represents the user preference model, S ij Indicates that from the scenic spot p i Transfer to attractions j The transition probability model.

[0049] Furthermore, the reinforcement learning algorithm in step S4 specifically includes:

[0050] Step S4.1: Based on the reinforcement learning perspective, the parameters in the personalized travel route problem are defined as follows:

[0051] State Space The current state of the system is the last intermediate scenic spot of the line, that is, the previous scenic spot before the end of the line;

[0052] Action Space For state s, take candidates from the set other than s Select another attraction to add between s and the end of the line;

[0053] Reward value r: in state s t Next, perform action a t The rewards obtained are calculated as follows:

[0054]

[0055] Among them U(u,a t ) is the scenic spot utility function model, T(a t ,s t ) is to add attractions a t To state attractions t The travel time of D(a t ) is a scenic spot t Average visit time;

[0056] State Transfer The next state after executing the operation is to select the next attraction to be added;

[0057] Q-value: By learning the feasible solution information generated during the iterative search process, the Q-value is adaptively adjusted as follows:

[0058]

[0059] Among them, γ is the discount factor and α is the learning rate;

[0060] Step S4.2: Based on the above definition, a Q-Learning algorithm is designed to obtain the tourist route with the highest final matching degree as the tourist route recommended to the user; specifically,

[0061] Enter the departure point p1; destination p N ; Total time constraint T total ; User interest preference R; Attraction-attraction conversion probability S; Average attraction stay time function D(.); Transportation time function between attractions T(.); Number of iterations maxLoop; ∈-greedy parameter ∈; Discount coefficient γ; Learning rate α;

[0062] Initialize Q-value Q(p i-1 ,p i )=0, initialize the total time of the route totalCost←0, initialize the travel route I0←(p1,p N ); when TotalCost <T total When observing at the destination p N Previous attraction i-1 , select the attraction p by ∈-greedy method i , and place the attractions i Insert to attractions i-1 After that, the total time of updating the line is as follows:

[0063] TotalCost←TotalCost+T(p i-1 ,p i )+D(p i )

[0064] The current reward value is obtained according to the reward value r calculation method in step S4.1, and Q(p i-1 ,p i ); continue to iterate until the maximum number of iterations maxLoop is reached; finally output the personalized itinerary route I = (p1, p2, ..., p N ).

[0065] Beneficial effects:

[0066] The personalized travel route recommendation method based on reinforcement learning provided by the present invention first obtains the mobile transportation time between different scenic spots and the average visit time of each scenic spot, and establishes a scenic spot-scenic spot transfer probability model and a user preference model based on the user's historical visit to the tourist route. Then, based on the user preference model, the scenic spot-scenic spot transfer probability model and the scenic spot heat model, a scenic spot utility function model is constructed. Finally, based on the obtained scenic spot utility function model and the personalized tourist route constraints, a reinforcement learning algorithm is designed to obtain the tourist route with the highest matching degree as the tourist route finally recommended to the user. The present invention solves the problem that the traditional tourist route recommendation method cannot meet the personalized needs of tourists, while taking into account user preferences and the popularity of scenic spots, and avoiding the cold start problem of scenic spot categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of a personalized travel route recommendation method based on reinforcement learning provided by the present invention;

[0068] Figure 2 4 is a diagram of a recommendation result example provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The personalized travel route recommendation method based on reinforcement learning provided by the present invention is as follows Figure 1 As shown, the following steps are included:

[0071] Step S1: Obtain the travel time between different scenic spots and the average visit time of each scenic spot.

[0072] Step S1.1, obtaining the travel time between different scenic spots.

[0073] According to the distance between two scenic spots, the travel modes between scenic spots are divided into walking, cycling and driving. When the distance between scenic spots is less than 2km, the walking time is calculated by default. When the distance between scenic spots is between 2km-5km, the cycling time is calculated by default. When the distance between scenic spots is greater than 5km, the driving time is calculated by default. Based on the third-party map API, the travel time between two scenic spots is obtained when different cycling methods are used. i and attractions j The moving time between i ,p j ).

[0074] Step S1.2: Obtain the average visit time of each scenic spot.

[0075] According to the historical travel routes of users and groups, the average visit time of each scenic spot is calculated. The visit time of a scenic spot is the time difference between arriving at the scenic spot and leaving the scenic spot. i The average access time is recorded as D(p i ).

[0076] Step S2: Establish a scenic spot-scenic spot transfer probability model and a user preference model based on the user's historical travel routes. Specifically,

[0077] Step S2.1: Number the scenic spots and user data to obtain the scenic spot p i and user u i The one-hot encoding is passed through the embedding layer to obtain the initial representation vector P of the scenic spot i and the user's initial representation vector U i .

[0078] Step S2.2: Learn the output representation vector of the scenic spot through the graph attention network.

[0079] User visits attractions i Next visit attractions j , learning attractions i The output representation vector of the lth layer of the neural network The calculation is as follows:

[0080]

[0081] in Indicates attractions i The output representation vector at layer l is, Initialize the output representation vector to be the attraction point p i The initial representation vector P i , α ij The weights of other scenic spots when representing the output vector are calculated as follows:

[0082]

[0083] Where W e Represents the weight parameter of the attention network, and calculates the weights of other scenic spots by using the normalized exponential function. LeakyReLU() is the activation function, which gives a non-zero slope to all negative values. The calculation process is as follows:

[0084]

[0085] Among them, a is a fixed parameter in the interval (1,∞).

[0086] Step S2.3: Learn the input representation vector of the scenic spot through the graph attention network.

[0087] According to the attractions i Previous visit attractions j , learning attractions i The input representation vector at layer l The calculation is as follows:

[0088]

[0089] Indicates attractions i The input representation vector at layer l is Initialize the input representation vector to be the attraction p i The initial representation vector P i , β ij The weights of other scenic spots when the input represents a vector are calculated as follows:

[0090]

[0091]

[0092] W b Represents the weight parameter of the attention network, and uses the normalized exponential function to calculate the weights of other attractions.

[0093] Step S2.4: According to the scenic spot p i The output representation vector and the scenic spot p in the Lth layer j The input representation vector at layer L is used to calculate p i -p j The transition probability model is as follows:

[0094]

[0095] in, For attractions i The output representation vector at layer L is For attractions j The input representation vector at layer L is is a learnable parameter. In this embodiment, the value of L is 2.

[0096] Step S2.5: According to the scenic spot p i Based on the output representation vector and input representation vector of layer l, calculate the scenic spot p i The representation vector at layer l is as follows:

[0097]

[0098] in, It is a tourist attraction i The representation vector at the lth layer, || is the concatenation operation.

[0099] Step S2.6: Based on the representation vector of the scenic spot at the lth layer, the representation vector of the user at the lth layer is calculated as follows:

[0100]

[0101] in, and For user u i The representation vectors at the lth and l+1th layers are, It is a tourist attraction i The representation vector at the lth layer. V l represents the weight matrix of the lth layer of the graph neural network, b l represents the bias vector of the lth layer of the graph neural network, and Aggreate(·) is the average aggregation function.

[0102] Step S2.7: Based on the representation vectors of the scenic spots and the users, the user preference model for the scenic spots is calculated as follows:

[0103]

[0104] Step S3: construct a scenic spot popularity model, and construct a scenic spot utility function model based on the user preference model, the scenic spot-scenic spot transfer probability model and the scenic spot popularity model. Specifically,

[0105] Step S3.1: Construct a scenic spot popularity model Pop(p j ).

[0106] Step S3.2: Based on the user preference model, the attraction-attraction transfer probability model and the attraction popularity model, a scenic spot utility function model is constructed as follows:

[0107] U(u,p j )=Pop(p j )+R uj +S ij

[0108] Among them, Pop(p j ) represents the scenic spot heat model, R uj represents the user preference model, S ij Indicates that from the scenic spot p i Transfer to attractions j The transition probability model.

[0109] Step S4: Based on the obtained scenic spot utility function model and personalized travel route constraints, a reinforcement learning algorithm is designed to obtain the travel route with the highest matching degree as the final travel route recommended to the user. Specifically,

[0110] Step S4.1: Based on the reinforcement learning perspective, the parameters in the personalized travel route problem are defined as follows:

[0111] State Space The current state of the system is the last intermediate scenic spot of the line, that is, the previous scenic spot before the end of the line.

[0112] Action Space For state s, take candidates from the set other than s , select another point of interest to add between s and the end of the line.

[0113] Reward value r: in state s t Next, perform action a t The rewards obtained are calculated as follows:

[0114]

[0115] Among them U(u,a t ) is the scenic spot utility function model, T(a t ,s t ) is to add attractions a t To state attractions t The travel time. t ) is a scenic spot t The average visit time.

[0116] State Transfer The next state after performing an action is to select the next attraction to add.

[0117] Q-value: By learning the feasible solution information generated during the iterative search process, the Q-value is adaptively adjusted as follows:

[0118]

[0119] Among them, γ is the discount factor and α is the learning rate.

[0120] Step S4.2: Based on the above definition, a Q-Learning algorithm is designed to obtain the tourist route with the highest final matching degree as the tourist route recommended to the user. The specific algorithm is shown in Table 1 below.

[0121] Table 1 Algorithms based on Q-Learning

[0122] Input: Departure point p1. Destination p N . Total time constraint T total . User interest preference R. Attraction-attraction transition probability S. Average attraction stay time function D(.). Transportation time function between attractions T(.). Number of iterations maxLoop. ∈-Greedy parameter ∈. Discount factor γ. Learning rate α.

[0123] Output: Personalized itinerary I = (p1, p2, ..., p N )

[0124] Initialize Q table Q(p i-1 ,p i )=0

[0125] FOR number of iterations from 0 to maxLoop DO

[0126] Initialize totalCost←0, I0←(p1,p N )

[0127] WHILE TotalCost <Ttotal DO

[0128] Observe at destination N The last state before (attraction)p i-1

[0129] Select actions (attractions) p by ∈-greedy method i

[0130] Place attractions i Insert to attractions i-1 after

[0131] Update the total time of the route TotalCost←TotalCost+T(p i-1 ,p i )+D(p i )

[0132] According to formula (1), the current reward value is obtained

[0133] According to formula (2), the value Q(p i-1 ,p i )

[0134] p i-1 ←p i

[0135] END WHILE

[0136] END FOR

[0137] In order to verify the effect of the algorithm, the following experiments were conducted. The experimental data came from three public data sets. The data sets were extracted from Flickr and contained the tourist route trajectories in Italian cities. In order to verify the performance of the project recommendation, we randomly selected 80% and 20% of the group travel routes as the test set and validation set, and the remaining data as the training set. The overall experimental results are shown in Table 2.

[0138] Table 2 Experimental comparison results

[0139]

[0140] It can be seen that the GRM-PersIRE method provided by the present invention achieves the best performance in terms of recall and precision, and improves by 11.9%-14.7%, 10.8%-13.7%, and 7.2%-9.3% on the FLOR, ROME, and PISA datasets compared with the second-best method C-ALNS. In terms of LDS, GRM-PersIRE outperforms the baseline method in 14 out of 18 comparisons.

[0141] like Figure 2The figure shows an example based on the ROME dataset, which visualizes the recommendation results. It can be seen that the GRM-PersIRE algorithm provided by the present invention is consistent with the Ground Truth (real result).

[0142] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A personalized travel route recommendation method based on reinforcement learning, characterized in that: The following steps are involved: Step S1, obtaining the mobile transportation time between different scenic spots and the average visit time of each scenic spot; Step S2: Establishing a scenic spot-scenic spot transfer probability model and a user preference model based on the user's historical travel routes; Step S3, constructing a scenic spot heat model, and constructing a scenic spot utility function model based on the user preference model, the scenic spot-scenic spot transfer probability model and the scenic spot heat model; Step S4: Based on the obtained scenic spot utility function model and personalized travel route constraints, a reinforcement learning algorithm is designed to obtain the travel route with the highest matching degree as the final travel route recommended to the user; The specific steps of establishing the scenic spot-scenic spot transfer probability model and the user preference model in step S2 include: Step S2.1: Number the scenic spots and user data to obtain the scenic spot p i and user u i The one-hot encoding is passed through the embedding layer to obtain the initial representation vector P of the scenic spot i and the user's initial representation vector U i ; Step S2.2, learning the output representation vector of the scenic spot through the graph attention network; User visits attractions i Next visit attractions j , learning attractions i The output representation vector of the lth layer of the neural network The calculation is as follows: in Indicates attractions i The output representation vector at layer l is, Initialize the output representation vector to be the attraction point p i The initial representation vector P i , α ij The weights of other scenic spots when representing the output vector are calculated as follows: Where W e Represents the weight parameter of the attention network, and the weights of other scenic spots are calculated by using the normalized exponential function; LeakyReLU() is the activation function, which gives a non-zero slope to all negative values. The calculation process is as follows: Among them, a is a fixed parameter in the interval (1,∞); Step S2.3, learning the input representation vector of the scenic spot through the graph attention network; According to the attractions i Previous visit attractions j , learning attractions i The input representation vector at layer l The calculation is as follows: Indicates attractions i The input representation vector at layer l is Initialize the input representation vector to be the attraction p i The initial representation vector P i , β ij The weights of other scenic spots when the input represents a vector are calculated as follows: W b Represents the weight parameters of the attention network, and uses the normalized exponential function to calculate the weights of other attractions; Step S2.4: According to the scenic spot p i The output representation vector and the scenic spot p in the Lth layer j The input representation vector at layer L is used to calculate p i -p j The transition probability model is as follows: in, For attractions i The output representation vector at layer L is For attractions j The input representation vector at layer L is are learnable parameters; Step S2.5: According to the scenic spot p i Based on the output representation vector and input representation vector of the lth layer, calculate the scenic spot p i The representation vector at layer l is as follows: in, It is a tourist attraction i In the representation vector at the lth layer, || is the concatenation operation; Step S2.6: Based on the representation vector of the scenic spot at the lth layer, the representation vector of the user at the lth layer is calculated as follows: in, and For user u i The representation vectors at the lth and l+1th layers are, It is a tourist attraction i The representation vector at the lth layer; V l represents the weight matrix of the lth layer of the graph neural network, b l represents the bias vector of the lth layer of the graph neural network, Aggreate(·) is the average aggregation function; Step S2.7: Based on the representation vectors of the scenic spots and the users, the user preference model for the scenic spots is calculated as follows: The specific steps of constructing the scenic spot utility function model in step S3 include: Step S3.1: Construct a scenic spot popularity model Pop(p j ); Step S3.2: Based on the user preference model, the attraction-attraction transfer probability model and the attraction popularity model, a scenic spot utility function model is constructed as follows: U(u,p j )=Pop(p j )+R uj +S ij Among them, Pop(p j ) represents the scenic spot heat model, R uj represents the user preference model, S ij Indicates that from the scenic spot p i Transfer to attractions j The transition probability model of The reinforcement learning algorithm in step S4 specifically includes: Step S4.1: Based on the reinforcement learning perspective, the parameters in the personalized travel route problem are defined as follows: State Space The current state of the system is the last intermediate scenic spot of the line, that is, the previous scenic spot before the end of the line; Action Space For state s, take candidates from the set other than s Select another attraction to add between s and the end of the line; Reward value r: in state s t Next, perform action a t The rewards obtained are calculated as follows: Among them U(u,a t ) is the scenic spot utility function model, T(a t ,s t ) is to add attractions a t To state attractions t The travel time of D(a t ) is a scenic spot t Average visit time; State Transfer The next state after executing the operation is to select the next attraction to be added; Q-value: By learning the feasible solution information generated during the iterative search process, the Q-value is adaptively adjusted as follows: Among them, γ is the discount factor and α is the learning rate; Step S4.2: Based on the above definition, a Q-Learning algorithm is designed to obtain the tourist route with the highest final matching degree as the tourist route recommended to the user; specifically, Enter the departure point p1; destination p N ; Total time constraint T total ; User interest preference R; Attraction-attraction conversion probability S; Average attraction stay time function D(.); Transportation time function between attractions T(.); Number of iterations maxLoop; ∈-greedy parameter ∈; Discount coefficient γ; Learning rate α; Initialize Q-value Q(p i-1 ,p i )=0, initialize the total time of the route totalCost←0, initialize the travel route I0←(p1,p N ); when TotalCost <T total When observing at the destination p N Previous attraction i-1 , select the attraction p by ∈-greedy method i , and place the attractions i Insert to attractions i-1 After that, the total time of updating the line is as follows: TtotalCost←TtotalCost+T(p i-1 ,p i )+D(p i ) The current reward value is obtained according to the reward value r calculation method in step S4.1, and Q(p i-1 ,p i ); continue to iterate until the maximum number of iterations maxLoop is reached; finally output the personalized itinerary route I = (p1, p2, ..., p N ).

2. According to claim 1, a personalized travel route recommendation method based on reinforcement learning is characterized in that: The specific steps of obtaining the traffic travel time and the average visit time of different scenic spots in step S1 include: Step S1.1, obtaining the travel time between different scenic spots; According to the distance between two scenic spots, the travel modes between scenic spots are divided into walking, cycling and driving. When the distance between scenic spots is less than 2km, the walking time is calculated by default; when the distance between scenic spots is between 2km-5km, the cycling time is calculated by default; when the distance between scenic spots is greater than 5km, the driving time is calculated by default; the travel time between two scenic spots when different cycling modes are used is obtained based on the third-party map API; the scenic spots are p i and attractions j The moving time between i ,p j ); Step S1.2, obtaining the average visit time of each scenic spot; According to the historical travel routes of users and groups, the average visit time of each scenic spot is calculated; the visit time of a scenic spot is the time difference between arriving at a scenic spot and leaving the scenic spot. i The average access time is recorded as D(p i ).

Images