A hierarchical multi-agent reinforcement learning trip recommendation method and system

Abstract

This invention discloses a hierarchical multi-agent reinforcement learning-based trip recommendation method and system, belonging to the field of trip recommendation technology. The hierarchical multi-agent reinforcement learning-based trip recommendation method includes: acquiring basic POI data, user historical behavior data, and contextual information; constructing a dynamic heterogeneous POI graph and generating POI embedding representations; constructing multi-type user preference representations and calculating the Bayesian user preference posterior distribution, and calculating candidate POI preference scores; constructing a hierarchical multi-agent reinforcement learning model, including, from high to low, a region selection agent, an agent action set generation unit, a POI selection agent, a time scheduling agent, and a path optimization agent; and outputting the final personalized trip route based on the hierarchical multi-agent reinforcement learning model. This invention can efficiently generate executable trips that balance personalization, path quality, and hard constraint satisfaction in large-scale POI scenarios, significantly improving recommendation accuracy and user experience.

Description

Technical Field

[0001] This invention belongs to the field of trip recommendation technology, specifically relating to a hierarchical multi-agent reinforcement learning method and system for trip recommendation. Background Technology

[0002] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art.

[0003] With the development of mobile internet, location services, and smart tourism platforms, users can access a wealth of information on attractions, restaurants, accommodations, and transportation before traveling. While this increased information availability improves the accessibility of tourism resources, it also makes it more difficult for users to complete attraction selection, route prioritization, budget estimation, and time management within a limited timeframe. Personalized itinerary recommendation technology aims to automatically generate an executable sequence of Points of Interest (POIs) for users, given their departure point, destination, available time, budget range, and preference type.

[0004] Existing itinerary planning methods mainly include traditional path search methods, multi-objective optimization methods, deep recommendation methods, and reinforcement learning methods. However, the inventors have found that traditional path search methods typically rely on manually set objective weights, making them less adaptable to changes in user preferences, dynamic contexts, and complex constraints. When deep recommendation methods are directly used for itinerary route generation, they are prone to issues such as recommending attractions that match interests but have excessively long routes, obvious detours, or failure to arrive on time. Reinforcement learning methods have excessively large action spaces in large-scale POI environments, resulting in low efficiency in reinforcement learning model training and online inference, making it difficult to meet the rapid response requirements of real-world tourism platforms. Multi-objective optimization methods often involve parallel objective decomposition, lacking a hierarchical decision-making process from global regional planning to local path adjustments, making it easy for generated routes to have detours, backtracking, time conflicts, or budget overruns. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a hierarchical multi-agent reinforcement learning method and system for route recommendation.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: In a first aspect, the technical solution of the present invention provides a hierarchical multi-agent reinforcement learning method for route recommendation, comprising: Acquire basic POI data, user historical behavior data, and contextual information, and perform preprocessing; Based on the acquired data, a dynamic heterogeneous POI graph containing multiple types of nodes and multiple types of dynamic edges is constructed, and POI embedding representations of the corresponding nodes are generated. Based on user historical behavior data and contextual information, construct multi-type user preference representations and calculate the Bayesian user preference posterior distribution, and calculate candidate POI preference scores; A hierarchical multi-agent reinforcement learning model was constructed and trained. The model, from high to low layers, includes a region selection agent, an agent action set generation unit, a POI selection agent, a time scheduling agent, and a path optimization agent. The region selection agent determines the next area to explore. The agent action set generation unit performs candidate action pruning and constraint masking, filtering candidate POIs based on their preference scores to obtain the agent action set. The POI selection agent selects specific POIs from the agent action set, and the path optimization agent optimizes the local order of the generated itinerary. The final personalized travel route is generated based on the trained hierarchical multi-agent reinforcement learning model.

[0007] In at least one embodiment, the POI basic data includes POI number, name, latitude and longitude, category, rating, number of reviews, consumption level, business hours, whether it is suitable for rainy days, and whether it is suitable for nighttime visits; User historical behavior data includes user ID, access POI ID, access time, dwell time, rating, clicks, favorites or comments; Contextual information includes current time, date, weather, traffic conditions, congestion level, holiday information, and user travel mode.

[0008] In at least one embodiment, the acquired POI basic data is preprocessed, specifically including: cleaning the data and deleting records with missing latitude and longitude, category, timestamp, or severely abnormal business hours; deleting users with too few check-ins and POIs with too few visits; and uniformly mapping POI categories to form a category system.

[0009] In at least one embodiment, in a dynamic heterogeneous POI graph with multiple types of nodes and multiple types of dynamic edges, the multiple types of nodes include user nodes, POI nodes, region nodes, time slice nodes, and context nodes; the multiple types of dynamic edges include POI-POI spatial distance edges, POI-POI semantic similarity edges, user-POI interest edges, POI-region affiliation edges, POI-temporal accessibility edges, and POI-context adaptation edges.

[0010] In at least one embodiment, generating the POI embedding representation of the corresponding node specifically includes: generating the POI embedding representation based on the constructed dynamic heterogeneous POI graph, using the initial attribute vector of the node as input, and employing a relation-aware graph neural network or a graph attention network.

[0011] In at least one embodiment, constructing multi-type user preference representations based on user historical behavior data and contextual information, and calculating the Bayesian user preference posterior distribution, specifically includes: Based on users' historical behavior data and current context information, we perform probabilistic modeling of users' multidimensional preferences and construct multi-type user preference representations, including category preferences, budget preferences, distance preferences, popularity preferences, and diversity preferences. For any candidate POI, the posterior distribution of user preferences is calculated using Bayes' theorem, and its personalized expected utility and preference uncertainty are calculated based on the posterior distribution of user preferences. Based on personalized expected utility and preference uncertainty, and combined with the uncertainty exploration coefficient, the candidate POI preference score is calculated by weighting.

[0012] In at least one embodiment, the agent action set generation unit is used to perform candidate action pruning and constraint masking, and to obtain an agent action set based on the candidate POI preference score. Specifically, it includes: generating a dynamic candidate POI set based on the POI status of the next-stage priority visit area selected by the region selection agent, through strategies including hard constraint filtering and geographic neighborhood recall, and obtaining an agent action set based on the candidate POI preference score; then performing constraint masking on the candidate actions in the agent action set to restrict the agent's action selection.

[0013] In at least one embodiment, when multiple agents in a hierarchical multi-agent reinforcement learning model make collaborative decisions, a comprehensive reward function is used for guidance, which is specifically expressed as follows:

[0014] In the formula, Match rewards to preferences; For POI quality awards; Rewards for diversity; Rewards for path efficiency; Exploring rewards based on preference for uncertainty; To constrain violations and impose penalties; , , , , and These are non-negative weighting coefficients.

[0015] In at least one embodiment, the final personalized travel route is output based on a trained hierarchical multi-agent reinforcement learning model, specifically including: First, the region selection agent chooses the priority area to visit in the next stage; Then, the agent action set generation unit obtains the agent action set based on the current state of the selected tour area through candidate action pruning and constraint masking. Selecting the next specific POI from the set of agent actions based on POI; The time-scheduling agent determines whether the POI selected by the POI selection agent is suitable for access and decides whether to allow a short stay, a normal stay, a long stay, or skip the visit. Add feasible points of interest (POIs) determined by the time scheduling agent to the current travel sequence, and update the current position, remaining time, remaining budget, and context state. The route optimization agent adjusts the order of local route windows composed of the most recent POIs, calculates the traffic cost before and after the adjustment, and accepts the adjustment if the adjusted route still meets the time, budget and business hours constraints; otherwise, it keeps the original order. The state space is updated based on the optimized itinerary sequence of the path optimization agent, and the comprehensive reward function is calculated. It is then determined whether the termination condition is met. If not, the agent returns to the area selection agent to reselect the tour area. If the termination condition is met, the optimized itinerary sequence is output as the final personalized itinerary route.

[0016] Secondly, the technical solution of the present invention also provides a hierarchical multi-agent reinforcement learning-based trip recommendation system, comprising: The data acquisition module is configured to: acquire basic POI data, user historical behavior data, and contextual information, and perform preprocessing. The dynamic heterogeneous POI graph module is configured to: construct a dynamic heterogeneous POI graph containing multiple types of nodes and multiple types of dynamic edges based on the acquired data, and generate POI embedding representations of the corresponding nodes; The Bayesian preference posterior module is configured to: construct multi-type user preference representations based on user historical behavior data and context information, calculate the Bayesian user preference posterior distribution, and calculate candidate POI preference scores; The model building module is configured to: construct and train a hierarchical multi-agent reinforcement learning model; the hierarchical multi-agent reinforcement learning model includes, from high to low layers, a region selection agent, an agent action set generation unit, a POI selection agent, a time scheduling agent, and a path optimization agent; wherein, the region selection agent is used to determine the tour area for the next stage; the agent action set generation unit is used to perform candidate action pruning and constraint masking, and to obtain the agent action set based on the candidate POI preference score; the POI selection agent is used to select specific POIs from the agent action set; and the path optimization agent is used to optimize the local order of the generated itinerary; The output module is configured to output the final personalized travel route based on the trained hierarchical multi-agent reinforcement learning model.

[0017] The beneficial effects of the above-described technical solution of the present invention are as follows: 1) The hierarchical multi-agent reinforcement learning-based itinerary recommendation method of the present invention effectively improves the matching degree between recommended POIs and users' actual access interests through Bayesian preference posterior modeling. Through hierarchical multi-agent collaborative decision-making, it improves the global rationality and local executability of route planning. Through the candidate action pruning mechanism, it reduces the computational complexity and online recommendation time in large-scale POI scenarios. Through the constraint masking mechanism, it reduces the problem of constraint violations such as time, budget, business hours and repeated visits. It can generate more personalized, coherent, executable and efficient travel itineraries, and is suitable for application scenarios such as online travel platforms, smart scenic spot recommendation systems, urban travel planning systems and personalized route navigation systems.

[0018] 2) This invention adopts a hierarchical multi-agent collaborative decision-making architecture. By decomposing the complex journey planning task into four specialized agents—region selection, POI selection, time scheduling, and path optimization—and one action set generation unit, it effectively solves the multi-objective conflict problem between preference matching, path efficiency, and constraint satisfaction in traditional single-agent or parallel multi-agent architectures. This architecture achieves step-by-step optimization from macro-regional planning to micro-local adjustments, avoiding cross-regional jumps and large-scale detours, and further improving route smoothness through local fine-tuning. At the same time, the division of labor and cooperation among agents significantly reduces the problem complexity of a single decision step, making the reinforcement learning model easier to converge and more efficient in online inference, ultimately generating a high-quality journey that balances personalization, executability, and overall coherence.

[0019] 3) This invention unifies multiple optimization objectives, such as preference matching, POI quality, path efficiency, diversity, uncertainty exploration, and constraint violation penalties, into a single scalar signal by integrating the reward function. This provides a clear and consistent training guide for hierarchical multi-agent systems and effectively solves the problem of policy imbalance caused by conflicting multiple objectives in traditional methods. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a schematic diagram of a hierarchical multi-agent reinforcement learning route recommendation method disclosed in Embodiment 1 of the present invention; Figure 2This is a comparison chart of recommendation accuracy under different POI scales disclosed in Embodiment 1 of the present invention; Figure 3 This is a comparison chart of path detour rates under different travel lengths disclosed in Embodiment 1 of the present invention; Figure 4 This is a comparison chart of route generation time under different POI scales disclosed in Embodiment 1 of the present invention. Detailed Implementation

[0022] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] Terminology Explanation: POI: Point of Interest, refers to accessible locations in a tourism scenario, such as attractions, restaurants, accommodations, shopping, and transportation hubs.

[0024] Personalized itinerary recommendations: A technical process that automatically generates ordered access routes based on user preferences, origin and destination, time, budget, context, and POI resources.

[0025] Bayesian preference posterior: The preference distribution obtained by probabilistically updating the user's latent interest parameters under the conditions of user's historical behavior, explicit preferences, and context.

[0026] Dynamic heterogeneous POI graph: A graph structure composed of multiple types of nodes such as users, POIs, regions, time slices and contexts, as well as multiple types of relational edges, and whose edge weights are updated as time and environment change.

[0027] Hierarchical multi-agent reinforcement learning: It breaks down complex journey decisions into multiple layers of agents working together, including region selection, POI selection, time scheduling, and path optimization.

[0028] Candidate action pruning: A technique for selecting a small number of actionable actions from the full set of Points of Interest (POIs) before reinforcement learning makes a decision, based on factors such as geography, time, budget, and preferences.

[0029] Constraint mask: Sets candidate actions that violate hard constraints such as time, budget, business status, and repeated access to zero, preventing the agent from selecting that action.

[0030] Local path optimization: The order of local POIs in the generated route is swapped, inserted, or adjusted using 2-opt to reduce detours and backtracking.

[0031] Contextual information: External information that affects trip feasibility and user experience, such as current time, weather, traffic conditions, congestion, and holiday status.

[0032] MAPPO: Multi-Agent Proximal Policy Optimization, is an algorithm for optimizing policies in a proximal manner and can be used for centralized training and distributed execution.

[0033] Example 1 As introduced in the background section, existing itinerary planning methods generally include traditional path search methods, multi-objective optimization methods, deep recommendation methods, and reinforcement learning methods. Traditional path search methods typically transform itinerary planning into a traveling salesman problem, a shortest path problem, or a path optimization problem with a time window, using heuristic search, genetic algorithms, ant colony optimization, particle swarm optimization, etc., to obtain routes. While these methods have a clear structure, they often use fixed preference vectors or fixed reward weights, making them less adaptable to changes in user preferences, dynamic contexts, and complex constraints. They fail to fully express the uncertainty and dynamic changes in user preferences, resulting in recommendation results that are difficult to adapt to cold-start users and preference drift scenarios. For example, in real-world tourism scenarios, users may only provide vague descriptions such as "likes history and culture," "moderate budget," and "doesn't want to travel too far"; new users lack historical visit records, and the preferences of existing users can change with weather, time, travel companions, and travel stage. Without modeling preference uncertainty, recommendation systems easily become overly reliant on popular attractions or historical interests, leading to insufficient personalization of recommended routes. Deep recommendation methods learn user interests from historical check-ins, clicks, favorites, and ratings, and predict POIs the user is likely to visit. However, ordinary POI recommendations primarily focus on matching interests at individual locations and cannot adequately handle factors such as the order of visits between multiple POIs, travel distance, opening hours, budget constraints, and route coherence. Therefore, when these methods are directly used for itinerary generation, they are prone to problems such as recommending attractions that match interests but resulting in routes that are too far-fetched, involve significant detours, or prevent timely visits.

[0034] Reinforcement learning methods are well-suited for sequential decision-making problems, as they can consider long-term benefits during route generation. However, in large-scale urban tourism scenarios, the number of Points of Interest (POIs) can reach thousands to tens of thousands. If the agent selects the next visit point from all POIs at each step, the action space becomes excessively large, resulting in high training and online inference costs. Existing multi-objective or multi-agent route planning methods mostly employ single-agent or parallel multi-agent structures, which primarily involve parallel objective decomposition and lack a hierarchical decision-making process from global region planning to local path adjustment. This can easily lead to objective conflicts, and the generated routes may suffer from detours, backtracking, time conflicts, or budget overruns.

[0035] To overcome the shortcomings of the prior art, in a typical embodiment of the present invention, such as... Figures 1 to 4As shown in the figure, this embodiment discloses a hierarchical multi-agent reinforcement learning method for route recommendation, which specifically includes the following steps: S1. Obtain basic POI data, user historical behavior data, and contextual information, and perform preprocessing; S2. Based on the acquired data, construct a dynamic heterogeneous POI graph containing multiple types of nodes and multiple types of dynamic edges, and generate POI embedding representations of the corresponding nodes; S3. Construct multi-type user preference representations based on user historical behavior data and context information, calculate the Bayesian user preference posterior distribution, and calculate the candidate POI preference score; S4. Construct and train a hierarchical multi-agent reinforcement learning model; the hierarchical multi-agent reinforcement learning model includes a region selection agent, an agent action set generation unit, a POI selection agent, a time scheduling agent, and a path optimization agent from high to low levels; among them, the region selection agent is used to determine the tour area for the next stage; the agent action set generation unit is used to perform candidate action pruning and constraint masking, and to obtain the agent action set based on the candidate POI preference score; the POI selection agent is used to select specific POIs from the agent action set; and the path optimization agent is used to optimize the local order of the generated itinerary.

[0036] S5. Based on the trained hierarchical multi-agent reinforcement learning model, the final personalized travel route is generated.

[0037] The following detailed description of the above-mentioned hierarchical multi-agent reinforcement learning route recommendation method, with reference to specific implementation methods, provides a detailed explanation.

[0038] S1. Obtain basic POI data, user historical behavior data, and contextual information, and perform preprocessing.

[0039] In this step, the basic POI data includes attributes such as POI ID, name, latitude and longitude, category, rating, number of reviews, consumption level, opening hours, suitability for rainy days, and suitability for nighttime visits. User historical behavior data includes user ID, visited POI ID, visit time, dwell time, rating, clicks, favorites, or comments. For implicit behaviors, visits, clicks, favorites, and dwell time can be uniformly converted into interaction intensity. Contextual information includes current time, date, weather, traffic conditions, congestion level, holiday information, and user travel mode.

[0040] It should be noted that the collection, processing, and use of user historical behavior data and contextual information in this step are all conducted with the user's explicit knowledge and written / electronic authorization, and all information is kept strictly confidential. Before obtaining data, the system will prominently prompt the user to read the privacy policy and obtain the user's explicit consent to the data collection, storage, analysis, and use for trip recommendation purposes.

[0041] After obtaining the basic POI data, user historical behavior data, and contextual information, the data needs to be preprocessed, including but not limited to: cleaning the data, deleting records with missing latitude and longitude, categories, timestamps, or severely abnormal business hours; deleting users with too few check-ins and POIs with too few visits; and uniformly mapping POI categories to form a system of categories such as catering, tourism, accommodation, shopping, transportation, and entertainment.

[0042] The preprocessed data is sorted according to the user access time to construct the user access sequence, and the city is divided into several regions according to latitude and longitude using the K-means method.

[0043] S2. Based on the acquired data, construct a dynamic heterogeneous POI graph containing multiple types of nodes and multiple types of dynamic edges, and generate POI embedding representations of the corresponding nodes.

[0044] In this step, the dynamic heterogeneous POI graph is a dynamic graph containing multiple types of nodes and multiple types of dynamic edges. The multiple types of nodes include user nodes, POI nodes, region nodes, time slice nodes, and context nodes. The multiple types of dynamic edges include POI-POI spatial distance edges, POI-POI semantic similarity edges, user-POI interest edges, POI-region affiliation edges, POI-temporal accessibility edges, and POI-context adaptation edges.

[0045] Based on multiple node types and multiple dynamic edges, the dynamic heterogeneous POI graph of the target city is constructed as follows:

[0046] In the formula, Indicates time Dynamic heterogeneous POI diagram below; Represents a set of nodes. Represents a set of edges that change over time and with context; The feature set representing nodes and edges.

[0047] As a further implementation method, the principle for constructing POI-POI spatial distance edges is as follows: if the distance between two POIs is less than a preset threshold or they belong to the same area, then a spatial edge is established, and the edge weight can be determined by distance, travel time or estimated walking / driving time.

[0048] The principle for constructing POI-POI semantic similarity edges is as follows: calculate the similarity between POIs based on their categories, tags, comment keywords, or attribute vectors, and establish semantic edges between POIs with similar categories or semantics.

[0049] The principle for constructing the user-POI interest edge is to generate interest weights between users and POIs based on users' historical visits, clicks, favorites, ratings, or dwell time.

[0050] The principle for constructing POI-regional affiliation edges is as follows: based on the POI's latitude and longitude and the regional division results, each POI is connected to its affiliated regional node.

[0051] The principle for constructing POI-time-accessible edges is as follows: determine whether a POI is accessible based on its business hours and the current time slice. If it is accessible, assign a positive weight; if it is inaccessible, assign zero weight or a high penalty.

[0052] The principle of constructing POI-context-adaptive edges is to dynamically adjust the weight of POIs based on contextual information such as weather, congestion, and traffic conditions. For example, reduce the weight of outdoor attractions on rainy days, block attractions that are not open at night, and increase the cost of long-distance transportation edges during peak hours.

[0053] For POI nodes in the target city Its initial attribute vector is represented as:

[0054] In the formula, Indicates geographical location; Indicates category; Indicates a rating; Indicates the number of comments or the popularity of visits; Indicates the level of consumption; Indicates business hours; This indicates a context-adaptive attribute.

[0055] Based on the constructed dynamic heterogeneous POI graph, using the initial attribute vectors of nodes as input, a relation-aware graph neural network or graph attention network is used to generate POI embedding representations for any POI node. The generated POI embedding representation is specifically as follows:

[0056] In the formula, This represents the POI embedding representation, which integrates the POI's own attributes, spatial structure, user interest relationships, and dynamic context relationships into a unified representation, serving as the input for subsequent agent decisions.

[0057] S3. Construct multi-type user preference representations based on user historical behavior data and contextual information, calculate the Bayesian user preference posterior distribution, and calculate the candidate POI preference score.

[0058] In this step, define the user. The potential preferences include category preference, budget preference, distance preference, popularity preference, and diversity preference. The constructed multi-type user preference representation is specifically expressed as follows:

[0059] In the formula, Indicates category preference; Indicates budget preferences; Indicates distance preference; Indicates preference for popularity; This indicates a preference for diversity.

[0060] Based on user historical behavior data and current context information This involves probabilistic modeling of users' multidimensional preferences. Specifically, category preferences... Using Dirichlet distribution modeling, the system counts the number of times each category of POIs is accessed in the user's historical visits and updates the category preference parameters based on the user's explicit preferences. The distribution parameters corresponding to the preference categories explicitly selected by the user receive incremental increases. For example, if the user explicitly selects preference categories such as "museums, history and culture, natural attractions," the corresponding category parameters for "museums, history and culture, natural attractions" will increase.

[0061] Budget preferences Using Gaussian or empirical distribution modeling, the system estimates the mean and variance of budget preferences based on the user's historical spending levels at POIs and the user's current input budget range. A larger variance indicates greater uncertainty about the user's budget preferences.

[0062] Distance preference By using Gaussian distribution or piecewise statistical methods for modeling, the system estimates the user's acceptance of long-distance travel based on the movement distance of adjacent POIs in the user's historical visits.

[0063] Popularity preference Estimated based on user historical POI ratings, number of comments, and visit popularity.

[0064] Diversity preferences Estimated based on category changes in the user's historical routes.

[0065] The posterior distribution of user preferences is calculated using Bayes' theorem, and is specifically expressed as follows:

[0066] Based on the Bayesian user preference posterior distribution, the personalized expected utility and preference uncertainty are calculated for any candidate POI to form a Bayesian preference comprehensive score.

[0067] Among them, candidate POI nodes Compared to users Personalized expected utility refers to estimating the expected value of a POI to a user based on the posterior distribution of user preferences, which can be specifically expressed as:

[0068] In the formula, This represents the personalized expected utility value.

[0069] Candidate POI nodes Compared to users Preference uncertainty is used to measure the current user The degree of uncertainty regarding preference for a POI increases with the user's historical behavior, which can be specifically expressed as:

[0070] In the formula, This indicates the value of preference uncertainty.

[0071] Based on personalized expected utility and preference uncertainty, an uncertainty exploration coefficient is introduced, and a weighted Bayesian preference comprehensive score is calculated to obtain the candidate POI preference score, which is specifically expressed as follows:

[0072] In the formula, Indicates the candidate POI preference score; The uncertainty exploration coefficient is used to determine the degree of user preference uncertainty. When users have limited historical behavior, their preferences are more uncertain, and the system can appropriately explore potential points of interest (POIs). When users have sufficient historical behavior, the system is more inclined to make recommendations based on known preferences.

[0073] S4. Construct and train a hierarchical multi-agent reinforcement learning model; the hierarchical multi-agent reinforcement learning model includes a region selection agent, an agent action set generation unit, a POI selection agent, a time scheduling agent, and a path optimization agent from high to low levels; among them, the region selection agent is used to determine the tour area for the next stage; the agent action set generation unit is used to perform candidate action pruning and constraint masking, and to obtain the agent action set based on the candidate POI preference score; the POI selection agent is used to select specific POIs from the agent action set; and the path optimization agent is used to optimize the local order of the generated itinerary.

[0074] In this step, trip recommendation is modeled as a hierarchical multi-agent reinforcement learning problem, and a hierarchical multi-agent reinforcement learning model is constructed. Each time step of the hierarchical multi-agent reinforcement learning model is defined. state space It consists of the current POI embedding, user representation, user preference posterior, remaining time, remaining budget, visited sequence, and contextual information, specifically represented as follows:

[0075] In the formula, For the first The state representation at each time step; For the current POI node The embedded representation is used to characterize the spatial, categorical, and historical transition features of the current access location; For users The embedded representation is used to characterize the long-term historical behavioral features of users; Based on user history behavior and current context The Bayesian user preference posterior obtained through inference; This represents the remaining time for the current trip. The remaining budget for the current trip; As of the date Steps to the already generated access sequence; This is the current context feature vector, which includes information such as current time, current location, visited POIs, candidate regions, and environmental constraints.

[0076] This hierarchical multi-agent reinforcement learning model includes, from high to low layers, a region selection agent, an agent action set generation unit, a POI selection agent, a time scheduling agent, and a path optimization agent.

[0077] The region selection agent, located at a high level, is responsible for selecting the priority areas to visit in the next stage. Its role is to control the itinerary structure globally, reducing cross-region jumps and large-scale detours. The agent action set generation unit, located at the next higher level, is responsible for pruning candidate actions and performing constraint masking on the candidate POI set. Based on the candidate POI preference scores, it selects the agent action set, reducing the original action space from |V| to K, where K is much smaller than |V|. This reduces the single-step action evaluation complexity from O(|V|) to O(K), thereby significantly improving the online recommendation speed. This addresses the challenges of excessively large action spaces, training, and recommendation complexity in large-scale POI scenarios. To address the issue of low efficiency, the POI selection agent, located in the middle layer, is responsible for selecting the next specific POI from the set of agent actions. This agent is mainly used to optimize user preference matching, POI quality, and route diversity. The time scheduling agent, located in the second middle layer, is responsible for determining whether the POI selected by the POI selection agent is suitable for access and deciding on short stay, normal stay, long stay, or skip, thereby reducing issues such as business hour conflicts and exceeding total time limits. The route optimization agent, located in the lower layer, is responsible for adjusting the order of the local path window composed of the most recent POIs, including maintaining, exchanging, inserting, or 2-opt optimization, to reduce detours and backtracking.

[0078] When multiple agents in this hierarchical multi-agent reinforcement learning model make collaborative decisions, a comprehensive reward function is used for guidance. This comprehensive reward function is specifically expressed as follows:

[0079] In the formula, Match rewards to preferences; For POI quality awards; Rewards for diversity; Rewards for path efficiency; Exploring rewards based on preference for uncertainty; To constrain violations and impose penalties; , , , , and These are non-negative weighting coefficients, used to control the relative importance of preference matching reward, POI quality reward, diversity reward, path efficiency reward, preference uncertainty exploration reward, and constraint violation penalty in the comprehensive reward function. The larger the value, the stronger the model's penalty for constraint violations. Through this comprehensive reward function, the system can simultaneously optimize individual satisfaction, POI quality, path efficiency, category diversity, and constraint satisfaction.

[0080] As a further implementation, in the agent action set generation unit, a dynamic candidate POI set is first generated based on the current state using strategies including hard constraint filtering and geographical neighborhood recall, and the agent action set is obtained by filtering according to the candidate POI preference scores; then, the candidate actions in the agent action set are subjected to constraint masking to restrict the agent's action selection.

[0081] Specifically, firstly, based on the POI status of the next-stage priority visit area selected by the region selection agent, the original action space is subjected to hard constraint filtering and geographic neighborhood recall to generate a dynamic candidate POI set. Specifically, based on the current POI status, visited POIs, POIs currently inaccessible, POIs whose estimated arrival time exceeds the business hours window, and POIs that may cause total time or budget limits to be exceeded are deleted from the original action space. Then, centered on the current location, spatially reachable candidate POIs are recalled using methods such as KD-Tree, Ball-Tree, HNSW, GeoHash, or grid indexing to obtain the dynamic candidate POI set.

[0082] Then, the comprehensive score of the candidate POIs is calculated based on the candidate POI preference scores, specifically expressed as follows:

[0083] In the formula, Indicates candidate POI The overall score; Indicates user In the current context Next pair of candidate POIs Bayesian preference composite score; Indicates candidate POI With the current POI The geographical accessibility score between two entities is usually calculated based on their spatial distance; the closer they are, the higher the score. Indicates candidate POI At the current time The following business hours are available for access to scores; Indicates candidate POI In the remaining budget Budget feasibility score under constraints; Indicates candidate POI Compared to the generated route Novelty or duplication avoidance score; , , , and These are non-negative weighting coefficients, used to control the relative importance of user preference, spatial accessibility, business hour accessibility, budget feasibility, and route novelty in the overall score.

[0084] Based on the comprehensive score of candidate POIs The POIs in the dynamic candidate POI set are sorted, and a Top-K selection strategy is used to select the top K POIs with the highest scores as the agent's action set, specifically represented as follows:

[0085] In the formula, Indicates the first The set of candidate actions for the agent at each time step, i.e., the set of candidate POIs obtained after Top-K filtering; This represents the dynamic set of candidate POIs after filtering by spatial constraints, preference constraints, time constraints, and budget constraints. Indicates candidate POI The overall score; This indicates that the top scorers will be selected based on their overall scores, from highest to lowest. The process of filtering candidate POIs.

[0086] Finally, constraint masking is applied to the candidate actions in the agent's action set to restrict the agent's action selection. Specifically, a binary mask is applied to the candidate actions in the agent's action set. If a candidate POI violates a hard constraint, the mask value is 0; otherwise, the mask value is 1. The policy distribution after masking is then renormalized to ensure that the agent can only choose from available actions.

[0087] The training phase employs a centralized training and distributed execution approach. During training, each agent shares global state information and learns the joint decision-making effect through a centralized value function; during execution, each agent makes independent decisions based on local observations to ensure online recommendation efficiency. As a further implementation method, MAPPO, QMIX, MADDPG, IPPO, or multi-agent DQN can be used as training algorithms during the training phase. In this embodiment, MAPPO is preferred as the basic training method.

[0088] S5. Based on the trained hierarchical multi-agent reinforcement learning model, the final personalized travel route is generated.

[0089] In this step, multiple agents based on a pre-trained hierarchical multi-agent reinforcement learning model make collaborative decisions. First, the region selection agent chooses the priority region to visit in the next stage. Then, the agent action set generation unit obtains the agent action set based on the current state of the selected region through candidate action pruning and constraint masking. The POI selection agent selects the next specific POI from the agent action set. Next, the time scheduling agent determines whether the POI selected by the POI selection agent is suitable for visit and decides whether to stay briefly, stay normally, stay for a long time, or skip it. The feasible POIs determined by the time scheduling agent are added to the list. The current itinerary sequence is updated with the current location, remaining time, remaining budget, and context state. The path optimization agent adjusts the order of the local path window composed of the most recent POIs, calculates the traffic cost before and after the adjustment, and accepts the adjustment if the adjusted path still meets the time, budget, and business hours constraints; otherwise, the original order is maintained. Based on the itinerary sequence optimized by the path optimization agent, the state space is updated and the comprehensive reward function is calculated. It is then determined whether the termination condition is met. If not, the process returns to the area selection agent to reselect a tour area. If the termination condition is met, the optimized itinerary sequence is output as the final personalized itinerary route.

[0090] As a further implementation, termination conditions include insufficient remaining time, reaching the maximum number of POIs accessed, reaching the endpoint, or the agent's action set having no available actions.

[0091] As a further implementation method, the final personalized itinerary includes the POI visit order, estimated arrival time, stay time, travel time, budget estimate, and recommended explanation. The recommended explanation may include reasons such as "aligns with user's historical and cultural preferences," "accessible at the current time," "close to the current location," or "within budget."

[0092] To verify the effectiveness of the hierarchical multi-agent reinforcement learning-based trip recommendation method (hereinafter referred to as "this method") proposed in this embodiment in personalized trip planning, this method was experimentally compared with Triple-AgentDRL, MORL-Trip, and NSGA-II algorithms. A comprehensive evaluation was conducted on aspects such as recommendation accuracy, preference matching, path quality, constraint satisfaction, diversity and novelty, and operational efficiency. Recommendation accuracy was evaluated using Normalized Discount Cumulative Gain (NDCG@K); preference matching was evaluated using preference hit rate; path quality was evaluated using total path distance, total travel time, detour rate, and return rate; constraint satisfaction was evaluated using time violation rate, budget violation rate, business hours violation rate, and repeat visit rate; diversity and novelty were evaluated using category diversity and novelty; and operational efficiency was evaluated using single-step decision time, total route generation time, and speedup ratio. Experimental results are as follows: Figures 2 to 4 As shown.

[0093] In terms of recommendation accuracy, this method uses a Bayesian preference posterior model to jointly model user explicit preferences, historical access behavior, and contextual information, enabling the system to more accurately characterize users' personalized needs in terms of attraction category, consumption level, distance sensitivity, and attraction popularity. Compared with traditional recommendation methods based on fixed weights or a single preference vector, this method can dynamically adjust the preference distribution based on users' historical behavior, thereby improving the matching degree between recommended POIs and users' actual visited POIs. Figure 2 As shown, under experimental conditions with 1000, 3000, 5000, 8000, and 10000 POIs, the proposed method's Normalized Diminished Cumulative Gain (NDCG@10) is consistently higher than the comparative methods. When the number of POIs is 10000, the proposed method achieves an NDCG@10 of 0.735, representing an improvement of approximately 12.2% compared to Triple-AgentDRL, approximately 16.3% compared to MORL-Trip, and approximately 28.3% compared to NSGA-II. This demonstrates that the proposed method maintains good recommendation accuracy even in scenarios with a large number of POIs.

[0094] In preference matching, this method incorporates user preference category information, budget level, distance preference, and explicit input conditions into the Bayesian posterior modeling process, resulting in higher consistency between POIs in the recommended routes and user preferences. Especially when user historical data is limited or user preference representation is incomplete, this method can utilize prior preferences and group statistics to supplement user interest representation, reducing the impact of cold start on recommendation quality. Compared to reinforcement learning methods that rely solely on fixed preference weights, this method has a significant advantage in preference hit rate, reducing POIs irrelevant to user preferences in the recommendation results and improving the personalization of recommended routes.

[0095] Regarding path quality, this method introduces a region selection agent and a path optimization agent, enabling the system to consider not only the attractiveness of individual POIs but also the spatial coherence between POIs when selecting them. The region selection agent can control the overall tour range of the trip, avoiding frequent cross-regional jumps within the city; the path optimization agent can adjust the order of local visits, reducing detours and backtracking. Figure 3 As shown, under experimental conditions with recommended trip lengths of 4, 6, 8, 10, and 12, the detour rate of this method is lower than that of the comparative methods. When the trip length is 10, the detour rate of the method proposed in this embodiment is 0.145, which is about 28.6% lower than Triple-AgentDRL, about 37.2% lower than MORL-Trip, and about 48.9% lower than NSGA-II. This indicates that this method can generate more coherent, compact, and easy-to-execute travel routes while maintaining the personalized recommendation effect.

[0096] Regarding constraint satisfaction, this method employs a time-scheduled agent and a constraint mask mechanism to control the recommendation process. Before selecting a candidate POI at each step, the system filters candidate actions based on remaining time, remaining budget, operating hours, and whether the POI is being visited repeatedly, preventing the agent from selecting unexecutable POIs. Compared to reinforcement learning methods that do not explicitly introduce constraint masks, this method effectively reduces time violation rates, budget violation rates, operating hour violation rates, and repeated visit rates, improving the actual executability of recommended routes. This mechanism avoids issues such as users arriving at attractions that are already closed, exceeding the total trip time limit, exceeding the budget, or repeatedly visiting the same POI, making it more consistent with real-world tourism application scenarios.

[0097] Regarding diversity and novelty, this method introduces category diversity rewards and uncertainty exploration rewards into the reward function, enabling the model to appropriately recommend POIs with potential interest value but not excessive popularity while satisfying users' main preferences. This design avoids the problem of over-concentrating on a single category of attractions or over-relying on popular POIs in recommended routes, making the generated itineraries richer in terms of attraction categories, tour experience, and recommended content. Compared with traditional optimization methods that only pursue maximizing ratings or popularity, this method achieves a better balance between user preference matching and route diversity.

[0098] In terms of operational efficiency, this method proposes a preference-prior-guided candidate action pruning mechanism, compressing the entire POI action space into a dynamic Top-K candidate set. This mechanism first performs hard-constraint filtering based on business hours, budget, repeat visits, and time accessibility, then combines geographical neighborhood retrieval and Bayesian preference scores to rank the candidates, enabling the reinforcement learning agent to make decisions only from a smaller candidate set. Figure 4 As shown, as the number of POIs increases from 1000 to 10000, the overall route generation time of this method increases relatively gradually, while the generation time of the comparison method increases significantly. When the number of POIs is 10000, the route generation time of this method is 0.91 seconds, which is about 12.6% shorter than Triple-AgentDRL, about 33.2% shorter than MORL-Trip, and about 30.4% shorter than NSGA-II. This demonstrates that this method can maintain high online recommendation efficiency in large-scale POI scenarios.

[0099] Example 2 In a typical embodiment of the present invention, this embodiment discloses a hierarchical multi-agent reinforcement learning-based trip recommendation system, comprising: The data acquisition module is configured to: acquire basic POI data, user historical behavior data, and contextual information, and perform preprocessing. The dynamic heterogeneous POI graph module is configured to: construct a dynamic heterogeneous POI graph containing multiple types of nodes and multiple types of dynamic edges based on the acquired data, and generate POI embedding representations of the corresponding nodes; The Bayesian preference posterior module is configured to: construct multi-type user preference representations based on user historical behavior data and context information, calculate the Bayesian user preference posterior distribution, and calculate candidate POI preference scores; The model building module is configured to: construct and train a hierarchical multi-agent reinforcement learning model; the hierarchical multi-agent reinforcement learning model includes, from high to low layers, a region selection agent, an agent action set generation unit, a POI selection agent, a time scheduling agent, and a path optimization agent; wherein, the region selection agent is used to determine the tour area for the next stage; the agent action set generation unit is used to perform candidate action pruning and constraint masking, and to obtain the agent action set based on the candidate POI preference score; the POI selection agent is used to select specific POIs from the agent action set; and the path optimization agent is used to optimize the local order of the generated itinerary; The output module is configured to output the final personalized travel route based on the trained hierarchical multi-agent reinforcement learning model.

[0100] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hierarchical multi-agent reinforcement learning method for route recommendation, characterized in that, include: Acquire basic POI data, user historical behavior data, and contextual information, and perform preprocessing; Based on the acquired data, a dynamic heterogeneous POI graph containing multiple types of nodes and multiple types of dynamic edges is constructed, and POI embedding representations of the corresponding nodes are generated. Based on user historical behavior data and contextual information, construct multi-type user preference representations and calculate the Bayesian user preference posterior distribution, and calculate candidate POI preference scores; Construct and train a hierarchical multi-agent reinforcement learning model; The hierarchical multi-agent reinforcement learning model comprises, from high to low layers, a region selection agent, an agent action set generation unit, a POI selection agent, a time scheduling agent, and a path optimization agent. The region selection agent determines the tour area for the next stage; the agent action set generation unit performs candidate action pruning and constraint masking, filtering candidate POIs based on their preference scores to obtain the agent action set; the POI selection agent selects specific POIs from the agent action set; and the path optimization agent optimizes the local order of the generated itinerary. The final personalized travel route is generated based on the trained hierarchical multi-agent reinforcement learning model.

2. The hierarchical multi-agent reinforcement learning method for route recommendation as described in claim 1, characterized in that, POI basic data includes POI number, name, latitude and longitude, category, rating, number of reviews, consumption level, business hours, whether it is suitable for rainy days, and whether it is suitable for nighttime visits; User historical behavior data includes user ID, access POI ID, access time, dwell time, rating, clicks, favorites or comments; Contextual information includes current time, date, weather, traffic conditions, congestion level, holiday information, and user travel mode.

3. The hierarchical multi-agent reinforcement learning method for route recommendation as described in claim 2, characterized in that, The acquired POI basic data is preprocessed, specifically including: cleaning the data and deleting records with missing latitude and longitude, category, timestamp, or severely abnormal business hours; deleting users with too few check-ins and POIs with too few visits; and uniformly mapping POI categories to form a category system.

4. The hierarchical multi-agent reinforcement learning method for route recommendation as described in claim 1, characterized in that, In a dynamic heterogeneous POI graph with multiple types of nodes and multiple types of dynamic edges, the multiple types of nodes include user nodes, POI nodes, region nodes, time slice nodes, and context nodes; the multiple types of dynamic edges include POI-POI spatial distance edges, POI-POI semantic similarity edges, user-POI interest edges, POI-region affiliation edges, POI-temporal accessibility edges, and POI-context adaptation edges.

5. The hierarchical multi-agent reinforcement learning method for route recommendation as described in claim 1, characterized in that, The specific steps for generating POI embedding representations for corresponding nodes include: based on the constructed dynamic heterogeneous POI graph, using the initial attribute vectors of the nodes as input, and employing a relation-aware graph neural network or graph attention network to generate POI embedding representations.

6. The hierarchical multi-agent reinforcement learning method for route recommendation as described in claim 1, characterized in that, Based on user historical behavior data and contextual information, multiple types of user preference representations are constructed, and Bayesian user preference posterior distributions are calculated, specifically including: Based on users' historical behavior data and current context information, we perform probabilistic modeling of users' multidimensional preferences and construct multi-type user preference representations, including category preferences, budget preferences, distance preferences, popularity preferences, and diversity preferences. For any candidate POI, the posterior distribution of user preferences is calculated using Bayes' theorem, and its personalized expected utility and preference uncertainty are calculated based on the posterior distribution of user preferences. Based on personalized expected utility and preference uncertainty, and combined with the uncertainty exploration coefficient, the candidate POI preference score is calculated by weighting.

7. The hierarchical multi-agent reinforcement learning method for route recommendation as described in claim 1, characterized in that, The agent action set generation unit is used to perform candidate action pruning and constraint masking. Based on the candidate POI preference scores, the agent action set is obtained. Specifically, based on the POI status of the next-stage priority visit area selected by the region selection agent, a dynamic candidate POI set is generated through strategies including hard constraint filtering and geographic neighborhood recall. The agent action set is then obtained by filtering the candidate POI preference scores. Then, constraint masking is performed on the candidate actions in the agent action set to restrict the agent's action selection.

8. The hierarchical multi-agent reinforcement learning method for route recommendation as described in claim 1, characterized in that, When multiple agents in a hierarchical multi-agent reinforcement learning model make collaborative decisions, a comprehensive reward function is used for guidance. This comprehensive reward function is specifically expressed as follows: In the formula, Match rewards to preferences; For POI quality awards; Rewards for diversity; Rewards for path efficiency; Exploring rewards based on preference for uncertainty; To constrain violations and impose penalties; , , , , and These are non-negative weighting coefficients.

9. The hierarchical multi-agent reinforcement learning method for route recommendation as described in claim 1, characterized in that, The final personalized travel route is generated based on the trained hierarchical multi-agent reinforcement learning model, specifically including: First, the region selection agent chooses the priority area to visit in the next stage; Then, the agent action set generation unit obtains the agent action set based on the current state of the selected tour area through candidate action pruning and constraint masking. Selecting the next specific POI from the set of agent actions based on POI; The time-scheduling agent determines whether the POI selected by the POI selection agent is suitable for access and decides whether to allow a short stay, a normal stay, a long stay, or skip the visit. Add feasible points of interest (POIs) determined by the time scheduling agent to the current travel sequence, and update the current position, remaining time, remaining budget, and context state. The route optimization agent adjusts the order of local route windows composed of the most recent POIs, calculates the traffic cost before and after the adjustment, and accepts the adjustment if the adjusted route still meets the time, budget and business hours constraints; otherwise, it keeps the original order. The state space is updated based on the optimized itinerary sequence of the path optimization agent, and the comprehensive reward function is calculated. It is then determined whether the termination condition is met. If not, the agent returns to the area selection agent to reselect the tour area. If the termination condition is met, the optimized itinerary sequence is output as the final personalized itinerary route.

10. A hierarchical multi-agent reinforcement learning-based itinerary recommendation system, characterized in that, include: The data acquisition module is configured to: acquire basic POI data, user historical behavior data, and contextual information, and perform preprocessing. The dynamic heterogeneous POI graph module is configured to: construct a dynamic heterogeneous POI graph containing multiple types of nodes and multiple types of dynamic edges based on the acquired data, and generate POI embedding representations of the corresponding nodes; The Bayesian preference posterior module is configured to: construct multi-type user preference representations based on user historical behavior data and context information, calculate the Bayesian user preference posterior distribution, and calculate candidate POI preference scores; The model building module is configured to: build and train a hierarchical multi-agent reinforcement learning model; The hierarchical multi-agent reinforcement learning model comprises, from high to low layers, a region selection agent, an agent action set generation unit, a POI selection agent, a time scheduling agent, and a path optimization agent. The region selection agent determines the tour area for the next stage; the agent action set generation unit performs candidate action pruning and constraint masking, filtering candidate POIs based on their preference scores to obtain the agent action set; the POI selection agent selects specific POIs from the agent action set; and the path optimization agent optimizes the local order of the generated itinerary. The output module is configured to output the final personalized travel route based on the trained hierarchical multi-agent reinforcement learning model.

Images