A personalized scenic spot recommendation method based on multi-modal knowledge graph contrast learning

Abstract

This invention relates to the fields of artificial intelligence and intelligent recommendation technology, specifically disclosing a personalized tourist attraction recommendation method based on multimodal knowledge graph contrastive learning, comprising the following steps: constructing a multimodal knowledge graph for the tourism field; secondly, establishing a visual feature embedding module and a text feature embedding module respectively; then, modeling the graph structure containing user interests through a knowledge-aware graph convolutional network, fusing user preferences, entity semantics, and relational semantic information to obtain structured knowledge embedding; introducing a structure-aware contrastive learning mechanism to evaluate the importance of the graph structure and enhance perturbation; generating a unified multimodal tourist attraction representation through a gated fusion network; and outputting personalized recommendation results. This invention effectively fuses multimodal information under weak supervision and data sparsity conditions, achieving high-precision tourist attraction recommendations, improving the personalization and robustness of the recommendation system, and is applicable to application scenarios such as smart tourism, travel navigation, and cultural scenic spot recommendations.

Description

Technical Field

[0001] This invention relates to the field of Internet technology, and in particular to a personalized tourist attraction recommendation method based on multimodal knowledge graph comparative learning. Background Technology

[0002] As people's living standards improve, tourism has become an increasingly popular leisure and entertainment choice, leading to a continuous expansion of the tourism market. Against this backdrop, how to help users quickly and accurately obtain information about tourist attractions that meet their needs, and provide personalized recommendations, has become a research hotspot. Initially, users primarily obtained tourism information through traditional methods such as travel agency consultations and travel guidebooks. These methods not only offered limited information but also failed to meet users' personalized needs.

[0003] With the development of internet technology, online travel platforms have emerged, providing users with a wealth of information on tourist attractions. However, faced with such a massive amount of information, users often need to spend a significant amount of time and effort filtering it, which has spurred the application of personalized recommendation technology in the tourism sector. Early recommendation systems were mostly based on simple collaborative filtering algorithms, analyzing users' historical behavioral data, such as browsing and booking records, to recommend attractions favored by similar users. While this method can achieve personalized recommendations to some extent, relying solely on user behavior data fails to provide a comprehensive understanding of the characteristics of attractions and users' interests and preferences.

[0004] At the same time, data types are becoming increasingly diverse, with a surge in multimodal data such as images and audio, in addition to traditional text data. Multimodal data can describe objects from different perspectives, providing more comprehensive and richer information. For example, photos of tourist attractions can visually showcase their appearance and features, while user reviews can convey users' genuine feelings. In this context, multimodal data processing technologies are gradually emerging, aiming to integrate data from different modalities and extract more valuable information.

[0005] In the field of knowledge graphs, recommendation methods based on knowledge graphs are gradually gaining attention. Knowledge graphs characterize tourist attractions and related attributes, such as attraction category, geographical location, and ticket price, through triples of entities and relations, thereby providing semantic reasoning capabilities for recommendations.

[0006] In existing tourism recommendation systems, scholars and businesses have proposed using knowledge graphs (KG) for personalized recommendations of tourist attractions. The basic idea behind this approach is:

[0007] 1. Knowledge Graph Construction:

[0008] Entities involved in the tourism scenario are modeled as graph nodes, such as attractions, attraction categories, geographical locations, ticket price ranges, and related facilities; semantic relationships between entities are modeled as edges, such as "belongs to category," "located in location," "ticket price," and "adjacent attractions"; a graph containing triples (e) is constructed. i ,r,e j Heterogeneous graphs of ).

[0009] 2. Alignment between users and knowledge graphs:

[0010] By leveraging users' historical interaction behaviors (such as browsing, favorites, and ticket purchase records), "user-attraction" connection edges are established in the knowledge graph; in this way, user nodes are embedded in the knowledge graph, forming an extended graph together with attraction entities and related relationships.

[0011] 3. Learning through graph representation:

[0012] Knowledge graph embedding models (such as TransE, TransR) or graph neural networks (such as GCN, R-GCN) are used to learn embedding representations of users and attractions;

[0013] During the dissemination process, user interests can spread along paths such as "user—attraction—category—similar attractions," thereby obtaining richer semantic representations.

[0014] 4. Recommended generation:

[0015] The recommendation score is obtained by calculating the similarity between the user embedding and the candidate attraction embedding; the Top-N attraction recommendations are then output in order of score.

[0016] This method, by utilizing the structured information of knowledge graphs, can capture the semantic relationships between users and attractions to a certain extent, and improve the personalization and interpretability of recommendations.

[0017] Although knowledge graph-based recommendation methods (such as RippleNet, KGAT, and KGIN) have achieved some success in modeling entity structures, they still have the following shortcomings:

[0018] Lack of modal information: Existing methods mainly rely on structured triples for modeling, failing to utilize unstructured modalities such as scenic spot photos, text descriptions, and user reviews. This results in single-dimensional node representations, containing only relational structure information and lacking visual and semantic supplementation, leading to insufficient interpretability and comprehensiveness of recommendation results.

[0019] Insufficient expression of user interests: In the recommendation process, users are usually modeled as nodes interacting with items, but they are not included in the computation of graph convolutional propagation. Different users should have differentiated embeddings under the same relationship path, but existing methods cannot reflect personalization and easily generate "one-size-fits-all" recommendation results.

[0020] Insufficient robustness: In real-world tourism knowledge graphs, node connections are extremely unevenly distributed, with some attractions interacting frequently while long-tail attractions have sparse data. Some models exhibit unstable embedding quality under cold-start or sparse node conditions, leading to a significant decrease in recommendation performance.

[0021] Access intent is not considered: User interests evolve over time, but existing methods often use static user representations, ignoring the importance of recent interactions for recommendations. In travel recommendations, failing to consider interest migration will cause recommendation results to lag behind user needs.

[0022] While this existing technology solves the problem of the lack of semantic structure in traditional collaborative filtering, it has shortcomings such as insufficient modality fusion, insufficient user modeling, lack of robustness, and failure to consider user access intent. Summary of the Invention

[0023] To address the shortcomings of existing technologies in multimodal information fusion, personalized structural modeling, robustness in cold-start scenarios, and dynamic interest capture, this application provides a personalized tourist attraction recommendation method based on multimodal knowledge graph comparative learning. This method improves the accuracy and adaptability of tourism recommendation systems by fusing multimodal data, enhancing structural semantic modeling, improving robustness, and taking into account dynamic user interests.

[0024] This invention provides the following technical solution:

[0025] A personalized tourist attraction recommendation method based on multimodal knowledge graph comparative learning includes the following steps:

[0026] S1. Construct a multimodal knowledge graph and collect multi-source heterogeneous data of tourist attractions, including visual modal information, text modal information and structured attribute information;

[0027] S2. Construct a visual feature embedding model to extract visual information from the scenic spot photos of the visual modality information. Use a pre-trained CLIP model to obtain the high-dimensional visual feature vector corresponding to each photo to realize the embedded representation of the visual modality.

[0028] S3. Construct a text feature embedding model, process the text description information of the scenic spots in the text modality using the BERT model, extract semantic embedding vectors, and form the text semantic representation of the scenic spots;

[0029] S4. Construct a knowledge-aware graph convolutional neural network model, and model the triples containing user interests in the structured graph using the knowledge-aware graph convolutional neural network KA-GCN; the KA-GCN integrates user interests, entity semantics and relational semantics, and has dual perception capabilities of structure and personality;

[0030] S5. Comparative Learning Mechanism for Knowledge Graph Structure Awareness

[0031] The multimodal knowledge graph is subjected to structure-aware graph structure enhancement and contrastive learning to improve the robustness, generalization ability and discriminativeness of the knowledge-aware graph convolutional neural network model embedded in the weakly supervised environment.

[0032] S6. Multimodal fusion

[0033] The features embedded from the visual feature embedding model, the text feature embedding model, and the knowledge-aware graph convolutional neural network model are adaptively weighted and fused to generate a unified and highly discriminative multimodal attraction representation, providing input for subsequent user interest modeling and recommendation ranking.

[0034] S7. Construct a user interest model, combining the user-attraction historical interaction information with the candidate attraction features, to construct a personalized interest representation model, thereby capturing user access intentions and generating targeted user embedding representations during the recommendation process;

[0035] S8. Recommendation generation: Based on the matching relationship between the user interest model embedding and the candidate scenic spot multimodal embedding, calculate the recommendation score and output the final Top-N recommendation results.

[0036] According to some possible implementations, step S1 includes the following steps:

[0037] S1.1 Collect multi-source heterogeneous data in the tourism sector, including:

[0038] Visual modality refers to a set of images corresponding to the scenic spots.

[0039] Text modality, consisting of attraction titles, descriptions, and summaries of user reviews;

[0040] Structured attributes include category, geographic location, time tag, and price range;

[0041] User-attraction history interactions include browsing, rating, favorites, and check-in sequences;

[0042] S1.2 Constructing a multimodal knowledge graph: The collected multimodal information of scenic spots is used to construct a multimodal knowledge graph G for the tourism field in the form of triples. The nodes include scenic spot entities, category entities, time entities, price entities, and user entities, which are used to describe multidimensional semantic features; the edges are used to represent the semantic or behavioral relationships between different entities. The multimodal knowledge graph is used to uniformly represent and organize scenic spot information from multiple information sources.

[0043] According to some possible implementations, step S2 includes the following steps:

[0044] S2.1 Obtain the image set corresponding to each scenic spot entity from the multimodal knowledge graph G constructed in step S1, and input the image Image corresponding to each scenic spot;

[0045] S2.2 utilizes the image encoder ViT in the pre-trained CLIP model to extract visual features from each image, obtaining a high-dimensional visual vector. Multiple images of the same scenic spot are averaged or weighted to obtain the visual representation of that scenic spot. To unify the embedding dimension, principal component analysis (PCA) is used to reduce the dimensionality of the extracted feature vectors, and then the vectors are passed through a fully connected layer (FC). img Achieve nonlinear mapping to obtain visual embedding vectors ;

[0046] (1)

[0047] Where ReLU is the activation function;

[0048] S2.3 Visual Embedding The site nodes are mapped to the knowledge graph and used as one of the initial node representations to provide input features for the subsequent multimodal feature fusion step S6.

[0049] According to some possible implementations, step S3 includes the following steps:

[0050] S3.1 From the multimodal knowledge graph G constructed in step S1, obtain the text information corresponding to each scenic spot node, including title, introduction and user comment summary, and combine them to form a complete text input sequence Text;

[0051] S3.2 Employs a pre-trained BERT model to encode the input text; obtains contextual semantic representations through TokenEmbedding, PositionEmbedding, and SegmentEmbedding layers; and takes the output vector at the [CLS] position or performs average pooling on all token representations as the overall semantic vector of the text. The specific calculations are as follows:

[0052] (2)

[0053] S3.3 embeds the obtained text semantics The corresponding scenic spot nodes are mapped to the knowledge graph and used as the initial modal representation of the nodes, providing input features for the subsequent multimodal feature fusion step S6.

[0054] According to some possible implementations, step S4 includes the following steps:

[0055] S4.1 Input Structure: The input is the multimodal knowledge graph G=(E,R,U) constructed in step S1, where E is the set of entity nodes, R is the set of relations, and U is the set of user nodes; in the graph, each edge constitutes a quadruple (u,e). i ,e j ,r), representing user u and triple entity e i and entity e j And the interactive relevance of their relationship r; used for modeling incorporating user personality preferences;

[0056] S4.2 Message Passing Mechanism and Attention Modeling: The message propagation in the KA-GCN is based not only on inter-entity relationships but also jointly considers user interests and preferences; for any edge (e i ,r,e j In the context of user u, a multi-dimensional interactive attention mechanism is adopted, introducing joint representations of entity pairs, relations, and users, with attention weights. The calculation process is as follows:

[0057] (3)

[0058] in, It is an activation function. It is an attention vector. , , , and It is a learnable linear transformation matrix; This represents vector concatenation. Indicates with entity e i The set of connected neighboring nodes;

[0059] S4.3 Node Information Aggregation: Based on the calculated attention weights, perform weighted aggregation of neighbor node information and update node e. i The embedding representation h i for:

[0060] (4)

[0061] In the above formula, σ() represents the activation function; The weight matrix is ​​used; through multi-layer stacking and message passing, a high-order structured graph representation can be obtained, resulting in the final structure embedding vector of the node.

[0062] According to some possible implementations, step S5 includes the following steps:

[0063] S5.1 Edge Importance Assessment Mechanism: First, for each edge e=(e...) in the original multimodal knowledge graph G=(E,R,U)... i ,r,e j Calculate its structural importance score S. e To guide subsequent graph structure perturbations, considering both node connectivity and global propagation impact, the importance calculation formula is as follows:

[0064] (5)

[0065] in, and The degree of the node. Used to measure the strength of its connection in the graph; the edge-level PageRank value reflects the propagation influence of the edge throughout the entire graph;

[0066] S5.2 Structural Enhancement View Generation: After obtaining the importance scores of the edges, an importance ranking interval filtering mechanism is introduced in the edge perturbation stage to sort all edges according to their importance scores; only edges located in the quantile interval [0.15, 0.75] are selected as candidate perturbation edges to avoid deleting overly critical or redundant edges.

[0067] Random edge deletion is performed on the multimodal knowledge graph G to construct two different but structurally similar graph augmented views G1 and G2 to ensure semantic consistency and structural comparability. Subsequently, the two views are input into the graph encoder KA-GCN constructed in S4 for encoding to obtain the embedding representations of nodes in different views.

[0068] (6)

[0069] in, and Node e i Embedded representation in two views;

[0070] S5.3 Contrastive Loss Training: To enhance the consistency of node representations across different views, the InfoNCE loss function based on contrastive learning is used for embedding training; for each node e iThe representations of the nodes in the two views are considered positive sample pairs; the representations between other different nodes are considered negative sample pairs, and the loss function is compared accordingly. The definition is as follows:

[0071] (7)

[0072] Where sim() is the cosine similarity; Temperature parameters The entire set of nodes in the graph; obtain the node structure embedding optimized by structure contrast learning, which is used in the subsequent multimodal fusion step S6 to improve the robust representation and generalization performance of the scenic spot nodes.

[0073] According to some possible implementations, step S6 includes the following steps:

[0074] S6.1 Input modal representation, for each attraction entity e i The following three modal embeddings are obtained from the previous steps: Visual embedding: Image feature representation extracted by the CLIP model in step S2; text embedding: The text semantic representation generated by the BERT model in step S3; graph structure embedding: The structured knowledge representation output by the KA-GCN model in step S5;

[0075] S6.2 Gating Weight Generation: The gating fusion network dynamically generates the weight coefficients of each modality during the fusion process to achieve adaptive adjustment of information importance. The gating weights consider not only the representation of the modality itself, but also the consistency and complementarity between them. The calculation process is as follows:

[0076] (8)

[0077] in, Gated weight generation matrix; Bias term; This represents the weighting coefficients corresponding to the three modes, satisfying α v +α t +α s =1;

[0078] S6.3 Fusion Embedding Generation, the fused unified scenic spot embedding representation is as follows: :

[0079] (9)

[0080] This representation integrates the semantic expressive power, visual perception ability, and structural features of multimodal information; the resulting fusion embedding It integrates the perceptual features of the visual modality, the semantic information of the text modality, and the relational knowledge of the structural modality, providing input for the subsequent user interest modeling step S7 and recommendation generation step S8.

[0081] According to some possible implementations, step S7 includes the following steps:

[0082] S7.1 Construction of Historical Interaction Graph

[0083] Extract user nodes and their interaction records from the knowledge graph G, and construct a user interest subgraph G. u =(V u E u ), V u This includes user nodes and their interactive attraction nodes; E u An edge is represented between a user and the interactive attractions, indicating the interaction relationship; this subgraph is used for local modeling of the user's individual interests.

[0084] S7.2 User Access Intent Recognition

[0085] When scoring a candidate attraction i, the score is embedded from step S6. As a query, the structure-enhanced representation from each historical node k in step S6 is used. Perform semantic alignment calculations to assess query-aware attention, evaluate the influence weight of historical attractions on the current attraction, and characterize the user's primary access intent. :

[0086] (10)

[0087] in, and Learnable weights, q is used as the matching readout vector to output a scalar score. ; Softmax normalization is applied to the scores of all historical nodes to obtain the candidate relevant attention weights:

[0088] (11)

[0089] in, The weights representing user attention; the denominator is all historical sites. The sum of the scores of candidate site i and the total score of candidate site i is used to ensure that the total weight is 1, thus highlighting the influence of highly relevant historical nodes.

[0090] S7.3 User Interest Representation Aggregation

[0091] The attention weights obtained in S7.2 are used to perform a weighted summation of the embedding representations of all historical interaction sites to generate the final user interest embedding vector. :

[0092] (12)

[0093] The most relevant parts of the user's past behavior to the candidates are extracted and used as the user representation for rating and ranking. This representation dynamically aggregates the interest patterns most relevant to the current candidate attractions from the user's past behavior, providing input features for the subsequent recommendation generation step S8.

[0094] According to some possible implementations, step S8 includes the following steps:

[0095] S8.1 User Embedding and Scenic Spot Embedding Calculation

[0096] User embedding representation: Based on the user interest embedding vector output in step S7 ;

[0097] Scenic spot embedding representation: For candidate scenic spot nodes, a fused multimodal embedding vector is generated through an S6 gated fusion network. It includes visual, textual, and structural semantic features;

[0098] S8.2 Similarity Scoring Calculation

[0099] Using cosine similarity as a measure of interest matching between users and attractions, given user embeddings and candidate attraction embeddings, the recommendation score is defined as:

[0100] (13)

[0101] A higher Score() score indicates that the user is more likely to be interested in the attraction. The attractions are then sorted from highest to lowest score, and the Top-N highest-scoring attractions are selected to generate the final recommendation results.

[0102] Compared with the prior art, the present invention has the following beneficial effects:

[0103] This invention provides a personalized tourist attraction recommendation method based on multimodal knowledge graph contrastive learning. In the chain of "multimodal input—dynamic fusion—user-aware graph modeling—structure-aware contrastive learning—interest dynamic capture—personalized recommendation output," it utilizes user-embedded KA-GCN, a structure-aware contrastive learning strategy, and attention-based user interest modeling. Specifically:

[0104] 1. Comprehensive information utilization: Fully integrate multimodal information such as visual, text, and structured data to achieve comprehensive modeling of scenic spot features and enhance the comprehensiveness of scenic spot representation.

[0105] 2. Deep personalization: Achieve truly personalized recommendations through a knowledge-aware attention mechanism to meet the differentiated needs of different users. The knowledge-aware graph convolutional network introduces user embeddings and entity and relation semantics into graph propagation to achieve personalized structural modeling.

[0106] 3. Strong model robustness: The introduction of edge importance assessment and enhanced view generation, along with the contrastive learning-based training strategy, significantly improves the model's resistance to noise and perturbations.

[0107] 4. High timeliness of recommendation results: User interest modeling based on graph attention network (GAT) dynamically highlights recent behaviors and high-weight nodes, improving the timeliness of recommendations. Attached Figure Description

[0108] Figure 1 This is a schematic diagram illustrating the principle of a personalized tourist attraction recommendation method based on multimodal knowledge graph comparative learning, as provided in an embodiment of the present invention. Detailed Implementation

[0109] like Figure 1 The technical approach adopted by the present invention to solve the technical problem is as follows:

[0110] 1) Address the problem of insufficient utilization of multimodal information in existing recommendation methods.

[0111] Existing knowledge graph-based recommendation methods typically rely solely on structured triples (such as "attraction-category" and "attraction-location") for modeling, neglecting the role of unstructured modal information such as attraction photos, text descriptions, and user reviews. This results in a single dimension of attraction embedding, lacking visual and semantic supplementation, and failing to fully reflect the characteristics of attractions.

[0112] Solution: Construct a multimodal knowledge graph and employ a multimodal fusion mechanism. The CLIP (Contrastive Language–Image Pretraining) model extracts visual features from attraction photos, while the BERT (Bidirectional Encoder Representations from Transformers) model extracts semantic features from attraction text descriptions and reviews. Combining the structured attributes of the knowledge graph, a gated fusion network adaptively assigns importance weights to different modalities. This mechanism fully leverages the complementary information between different modalities to obtain a unified attraction representation encompassing visual, textual, and structural features, thereby improving the comprehensiveness and accuracy of recommendation results.

[0113] 2) Addressing the issue of insufficient personalization in knowledge graph representation learning.

[0114] Existing methods mostly employ uniform graph convolutional networks (GCN, R-GCN) for knowledge graph representation learning, neglecting the differences in interests and preferences among different users. For example, some users are more concerned with the geographical location of attractions, while others value the historical and cultural background of attractions. Existing uniform propagation mechanisms cannot reflect this difference, resulting in recommendations that lack personalization and present a "one-size-fits-all" approach.

[0115] Solution: Employ a Knowledge-Aware Graph Convolutional Neural Network (KA-GCN). User nodes are introduced into the knowledge graph propagation process, constructing a "user-entity-relationship-entity" quadruple, and jointly modeling the interaction features of users, entities, and relationships through an attention mechanism. This method dynamically adjusts the user's attention weights towards different entities and relationships, making the knowledge graph propagation process vary from user to user, thus achieving truly personalized site representation learning.

[0116] 3) Addressing the issues of insufficient robustness and generalization ability of the model.

[0117] Existing recommendation models primarily rely on supervised learning and lack effective self-supervised enhancement mechanisms. Their performance tends to degrade significantly when faced with sparse data, noise interference, malicious attacks, or distribution shifts. While some contrastive learning methods introduce augmented views, they often employ random edge dropping or node occlusion, which can disrupt the core topological structure of the knowledge graph, leading to unstable embedding representations or even the loss of crucial semantic information.

[0118] Solution: Introduce a structure-aware contrastive learning mechanism. First, the importance of edges is evaluated based on metrics such as node degree and PageRank. While preserving the core graph structure, edges are discarded according to their importance ranking to construct diverse augmented views. Then, the InfoNCE (Information Noise-Contrastive Estimation) contrastive loss function is used to constrain the embedding consistency of the same node across different views, thereby improving the discriminativeness and robustness of the representation. This mechanism not only avoids destroying the core semantic structure but also mines the inherent structure of graph data through self-supervised learning, enhancing the model's generalization ability in sparse scenarios.

[0119] 4) Solving the problem of static user interest modeling

[0120] Most existing user interest modeling methods are based on static aggregation, which simply averages or weighted averages the attractions that users have interacted with in their history, lacking the ability to model the dynamic evolution of interests. This method cannot reflect changes in users' interests at different times and in different contexts, resulting in lagging recommendation results and difficulty in accurately capturing users' real-time needs.

[0121] Solution: Introduce a Graph Attention Network (GAT) to model the user's historical interaction behavior graph. By dynamically assigning weights to different interaction points through an attention mechanism, a dynamic user representation that changes over time is constructed. This approach can promptly capture the evolution of user interests and primary access intentions, improving the timeliness and personalization of recommendation results.

[0122] The present invention will now be described in detail with reference to embodiments and accompanying drawings. However, it should be understood that the embodiments and drawings are for illustrative purposes only and do not constitute any limitation on the scope of protection of the present invention. All reasonable modifications and combinations included within the inventive spirit of the present invention fall within the scope of protection of the present invention.

[0123] The present invention will be further described below with reference to the accompanying drawings.

[0124] Example 1

[0125] like Figure 1 This embodiment provides a personalized tourist attraction recommendation method based on multimodal knowledge graph comparative learning, aiming to integrate visual, textual, and structured information to achieve more accurate user interest modeling and recommendation services. The method includes the following steps:

[0126] S1. Constructing a multimodal knowledge graph

[0127] A multimodal knowledge graph for the tourism sector is constructed to uniformly represent and organize attraction information from multiple information sources. The knowledge graph includes the following:

[0128] Visual modal information: including official photos of the attractions, user-uploaded images, etc.;

[0129] Text modal information: including basic introductions, detailed descriptions, and summaries of user reviews of the attractions;

[0130] Structured attribute information: including structured fields such as attraction opening hours, ticket prices, geographical location, and attraction category.

[0131] The nodes of a knowledge graph include, but are not limited to, attraction entities, category entities, time entities, price range entities, etc., and edges are used to represent semantic relationships between different entities, such as "belongs to category", "is located in location", "price is", etc.

[0132] S2. Constructing a visual feature embedding module

[0133] Based on a multimodal knowledge graph constructed using S1, visual information is extracted from scenic spot photos. A pre-trained CLIP (Contrastive Language–Image Pretraining) model is employed to obtain high-dimensional visual feature vectors for each photo, achieving embedded representation of visual modalities. This method encodes the visual semantics of scenic spot images into the graph structure, facilitating multimodal fusion and understanding of scenic spots.

[0134] The specific steps are as follows:

[0135] 1) Enter the image (JPEG / PNG format) corresponding to each attraction;

[0136] 2) Use the image encoder (ViT) in the CLIP model to extract visual vectors and obtain fixed-dimensional visual embeddings. ;

[0137] (1)

[0138] The extracted visual features were dimensionality reduced using principal component analysis (PCA); then, a fully connected layer (FC) was used. img The purpose of performing nonlinear mapping on these features is to project visual data into the embedding space of structured entities, where ReLU is the activation function.

[0139] 3) Embed the acquired visual data Mapped to the scenic spot nodes in the knowledge graph, serving as one of the initial node representations.

[0140] S3. Constructing a text feature embedding module

[0141] Based on the multimodal knowledge graph constructed using S1, the textual descriptions of scenic spots are processed using the BERT (Bidirectional Encoder Representations from Transformers) model to extract semantic embedding vectors, forming the textual semantic representations of the scenic spots. The specific steps are as follows:

[0142] 1) Enter the attraction's title, description, and summary of user reviews;

[0143] 2) Obtain the contextual semantic embedding through BERT's Token Embedding, Position Embedding, and other layers; represent the text embedding using a vector at the [CLS] position or average pooling. ;

[0144] (2)

[0145] 3) Obtain the text vector, which serves as the node embedding representation of the text modality.

[0146] S4. Construct a knowledge-aware graph convolutional neural network

[0147] This embodiment proposes a Knowledge-Aware Graph Convolutional Network (KA-GCN) for constructing multimodal knowledge graphs in tourism recommendation scenarios. This network effectively models triples containing user interests within structured graphs. Unlike traditional graph convolutional neural networks, KA-GCN integrates user interests, entity semantics, and relational semantics, possessing both structure and personality perception capabilities. Its modeling steps include:

[0148] 1) Input Structure: The input is a multimodal knowledge graph G=(E,R,U) constructed from S1, where: E: set of entity nodes (e.g., scenic spots, categories, locations); R: set of relations (e.g., "belongs to category", "located in city"); U: set of user nodes; each edge constitutes a quadruple (u,e) i ,e j ,r), representing user u and triple entity e i and entity e j And the interactive correlation of their relationship r.

[0149] 2) Message Passing Mechanism and Attention Modeling: Message propagation in KA-GCN is based not only on relationships between entities but also on user interests and preferences. For any edge (e... i ,r,e j In the context of user u, a multi-dimensional interactive attention mechanism is adopted, introducing joint representations of entity pairs, relations, and users, with attention weights. The calculation process is as follows:

[0150] (3)

[0151] in, It is an activation function. It is an attention vector. , , , and It is a learnable linear transformation matrix; This represents vector concatenation. Indicates with entity e i The set of connected neighboring nodes.

[0152] 3) Node information aggregation: Neighbor node information is weighted and aggregated based on attention weights to update node e.i The embedding is represented as :

[0153] (4)

[0154] σ(⋅) represents the activation function (such as ReLU); This is a trainable weight matrix.

[0155] A higher-order graph structure perception representation can be obtained by stacking multiple layers. Introducing user information not only guides preferences but also achieves the following technical advantages: 1) It can dynamically adjust the perception weights of different users on the same relationship; 2) The same triple can be embedded and updated in different directions under different user backgrounds, satisfying personalization; 3) It preserves the original graph structure while introducing user semantics, adapting to cold start and personalized recommendation.

[0156] S5. Comparative Learning Mechanism for Knowledge Graph Structure Awareness

[0157] To address the common problems in large-scale knowledge graphs, such as sparse node features, imbalanced connectivity, and cold start, this step employs a structure-aware graph structure enhancement and contrastive learning mechanism to improve the robustness, generalization ability, and discriminative power of the embedding model in weakly supervised environments. This mechanism mainly includes the following three steps:

[0158] 1) Edge Importance Assessment Mechanism: Traditional graph contrastive learning methods typically employ random edge discarding strategies when constructing structurally enhanced views. However, random perturbations can disrupt key semantic paths and structural features in the graph, thus affecting model performance. To address this issue, this embodiment introduces a structure-aware edge discarding strategy. By calculating the structural importance score of each edge, guided perturbations are applied to the graph structure: Node Degree: Reflects the connection strength of a node in the graph, used to assess its "centrality"; Edge PageRank: Measures the propagation influence of an edge in the overall graph, reflecting its semantic value. By guiding edge discarding through edge importance assessment (based on degree and PageRank), enhanced views are generated. Combined with InfoNCE loss for contrastive learning, the stability of node representations is improved.

[0159] First, for each edge e=(e) in the original multimodal knowledge graph G=(E,R,U) i ,r,e j Calculate its structural importance score S. e To guide subsequent graph structure perturbations, considering both node connectivity and global propagation impact, the importance calculation formula is as follows:

[0160] (5)

[0161] in, and The degree of the node. Used to measure the strength of its connection in the graph; the edge-level PageRank value reflects the propagation influence of the edge throughout the entire graph;

[0162] 2) Enhanced view generation: After obtaining the importance score of the edge, this embodiment introduces an importance ranking interval filtering mechanism in the edge perturbation stage, sorts all edges according to their importance scores, and selects only the edges located in the quantile interval [0.15, 0.75] as candidate perturbation edges to avoid destroying the key structure of the graph (such as high PageRank core paths), ensure that low connection frequency nodes (cold start nodes) do not have information breaks due to structural weakening, and reduce the overfitting risk of high frequency nodes.

[0163] Under this mechanism, two different but structurally similar graph-enhanced views, G1 and G2, are constructed from the original knowledge graph G to ensure semantic consistency and structural comparability. Subsequently, the two views are input into a graph encoder (KA-GCN of this invention) for encoding, yielding the embedding representations of nodes in different views:

[0164] (6)

[0165] in, and Node e i Embedded representation in two views.

[0166] 3) Contrastive Loss Training: To enhance the consistency of node representations across different views, this invention employs the InfoNCE (Noise Contrastive Estimation) loss function based on contrastive learning for embedding training. For each node e... i The representation of a node in each of the two views is considered a positive sample pair; representations between other different nodes are considered negative sample pairs. Contrast loss function. The definition is as follows:

[0167] (7)

[0168] Where sim() is the cosine similarity; Temperature parameters The set of all nodes in the graph.

[0169] By minimizing this contrast loss, the model learns that the same node should have a consistent representation in different views, while different node representations should be distinguished as much as possible, thereby enhancing the robustness and discriminativeness of the embedding model in diverse graph structures.

[0170] S6. Multimodal fusion:

[0171] In the multimodal knowledge graph constructed in this embodiment, tourist attraction nodes typically contain information modalities from multiple sources, including visual, textual, and structural information. Each of these modalities possesses different semantic expressive capabilities. The visual modality reflects the intuitive appearance and environmental characteristics of the attraction; the textual modality carries descriptive semantic information, such as attraction descriptions and user reviews; and the structural modality originates from entity relationships within the knowledge graph, revealing semantic connections and category affiliations.

[0172] Because different modal information differs in quality, coverage, and semantic level, direct concatenation or averaging will lead to information redundancy or representation shift. Therefore, this embodiment employs a gated fusion network to dynamically adjust the importance of each modal information, thereby generating a unified and highly discriminative multimodal embedding representation. Specifically, it includes the following steps:

[0173] 1) Input modal representation: Let e ​​be the entity of each scenic spot. i Its representation vectors in different modalities are as follows:

[0174] Visual embedding: Image features are extracted using the CLIP model;

[0175] Text embedding: The BERT model extracts the semantics of the text.

[0176] Graph structure embedding: The structured knowledge representation obtained from KA-GCN.

[0177] 2) Gated Weight Generation Mechanism: A gated fusion network is introduced to dynamically generate the weight coefficients of each modality during the fusion process, achieving "adaptive adjustment of information importance." The gated weights consider not only the representation of the modality itself but also its consistency and complementarity. The calculation process is as follows:

[0178] (8)

[0179] in, Gated weight generation matrix; Bias term; This represents the weighting coefficients corresponding to the three modes, satisfying α v +α t +α s =1.

[0180] 3) Fusion Embedding Generation: The final fused unified scenic spot embedding representation is as follows: :

[0181] (9)

[0182] This representation integrates the semantic expression capabilities, visual perception capabilities, and structural features of multimodal information, which is beneficial for improving the accuracy of downstream tasks such as recommendation and classification.

[0183] S7. User Interest Modeling:

[0184] User interests are the core basis for generating personalized results in tourism recommendation systems. This embodiment uses a dynamic interest modeling method based on Graph Attention Network (GAT) to model user preferences personalizedly, based on users' historical interaction data on tourism platforms. This method constructs a graph structure of users and the attractions they interact with, and introduces an attention mechanism to differentiate the attractions in users' historical behavior, highlighting representative and timely behaviors, thereby learning high-quality user interest embeddings. The specific steps are as follows:

[0185] 1) Construction of historical interaction graph

[0186] Constructing a user interest subgraph G u =(V u E u ), V u This includes user nodes and their interactive attraction nodes; E u This subgraph represents the interaction relationship between a user and the interactive attractions; it is used to locally model the user's individual interests.

[0187] 2) User access intent recognition

[0188] When scoring a candidate attraction i, embed it As a query, the structure-enhanced representation of each historical node k Perform semantic alignment calculations to assess query-aware attention, evaluate the influence weight of historical attractions on the current attraction, and characterize the user's primary access intent. :

[0189] (10)

[0190] in, and Learnable weights, q is used as the matching readout vector to output a scalar score. Softmax normalization is applied to the scores of all historical nodes to obtain the candidate relevant attention weights:

[0191] (11)

[0192] in, This represents the user's attention weight. The denominator is all historical sites. The sum of the scores of candidate site i and the total score of candidate site i is used to ensure that the total weight is 1, thus highlighting the influence of highly relevant historical nodes.

[0193] 3) User interest representation aggregation

[0194] The final user embedding vector is generated by weighting and summing the embedding representations of all historical interaction sites using the aforementioned attention weights. :

[0195] (12)

[0196] Extract the most relevant parts of the user's past behavior to the candidates and use them as the user representation for rating and ranking.

[0197] S8. Recommended generation

[0198] After completing user interest modeling and multimodal fusion embedding of tourist attractions, this embodiment provides an efficient and scalable recommendation generation mechanism to achieve personalized content delivery in tourism scenarios. The recommendation generation mainly includes the following steps:

[0199] 1) User embedding and scenic spot embedding calculation

[0200] User embedding representation: Based on the graph attention network (GAT), the user's historical interaction graph is processed to obtain the user embedding vector. This includes modeling results of preferences such as recent behavior and high-weight attractions. A graph attention network (GAT) is used to model user historical interactions, highlighting recent behavior and high-weight attraction nodes to generate dynamic user interest embeddings, preventing bypassing the weakened version that only uses average modeling.

[0201] Scenic spot embedding representation: For candidate scenic spot nodes, a fused multimodal embedding vector is generated through a gated fusion network. It includes visual, textual, and structural semantic features.

[0202] 2) Similarity scoring calculation

[0203] Cosine similarity is used as a metric for interest matching between users and attractions. Given a user embedding... and candidate attractions embedded The recommendation score is defined as:

[0204] (13)

[0205] A higher Score() score indicates a greater likelihood of user interest in the attraction. The attractions are then sorted from highest to lowest score, and the Top-N highest-scoring attractions are selected to form the final recommendation list.

[0206] The above embodiments are merely preferred embodiments of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A personalized tourist attraction recommendation method based on multimodal knowledge graph comparative learning, characterized in that: Includes the following steps: S1. Construct a multimodal knowledge graph and collect multi-source heterogeneous data of tourist attractions, including visual modal information, text modal information and structured attribute information; S2. Construct a visual feature embedding model to extract visual information from the scenic spot photos of the visual modality information. Use a pre-trained CLIP model to obtain the high-dimensional visual feature vector corresponding to each photo to realize the embedded representation of the visual modality. S3. Construct a text feature embedding model, process the text description information of the scenic spots in the text modality using the BERT model, extract semantic embedding vectors, and form the text semantic representation of the scenic spots; S4. Construct a knowledge-aware graph convolutional neural network model, and model the triples containing user interests in the structured graph using the knowledge-aware graph convolutional neural network KA-GCN; the KA-GCN integrates user interests, entity semantics and relational semantics, and has dual perception capabilities of structure and personality; S5. Comparative Learning Mechanism for Knowledge Graph Structure Awareness The multimodal knowledge graph is subjected to structure-aware graph structure enhancement and contrastive learning to improve the robustness, generalization ability and discriminativeness of the knowledge-aware graph convolutional neural network model embedded in the weakly supervised environment. S6. Multimodal fusion The features embedded from the visual feature embedding model, the text feature embedding model, and the knowledge-aware graph convolutional neural network model are adaptively weighted and fused to generate a unified and highly discriminative multimodal attraction representation, providing input for subsequent user interest modeling and recommendation ranking. S7. Construct a user interest model, combining user-attraction historical interaction information with candidate attraction features to build a personalized interest representation model, thereby capturing user access intent and generating targeted user embedding representations during the recommendation process; S8. Recommendation generation: Based on the matching relationship between user interest model embedding and candidate scenic spot multimodal embedding, calculate recommendation scores and output the final Top-N recommendation results; Step S5 includes the following steps: S5.1 Edge Importance Assessment Mechanism: First, for each edge e=(e...) in the original multimodal knowledge graph G=(E,R,U)... i ,r,e j Calculate its structural importance score S. e This guides subsequent graph structure perturbations, where E is the set of entity nodes, R is the set of relations, U is the set of user nodes, and r is the number of entity nodes. i and entity e j The relationship between nodes; taking into account both the connection strength of nodes and their global propagation influence, the importance calculation formula is as follows: （5） in, and The degree of the node. Used to measure the strength of its connection in the graph; the edge-level PageRank value reflects the propagation influence of the edge throughout the entire graph; S5.2 Structural Enhancement View Generation: After obtaining the importance scores of the edges, an importance ranking interval filtering mechanism is introduced during the edge perturbation stage. All edges are sorted according to their importance scores. Only edges located in the quantile interval [0.15, 0.75] are selected as candidate perturbation edges to avoid deleting overly critical or redundant edges. Random edge deletion is performed on the multimodal knowledge graph G to construct two different but structurally similar graph augmented views G1 and G2 to ensure semantic consistency and structural comparability. Subsequently, the two views are input into the graph encoder KA-GCN constructed in S4 for encoding to obtain the embedding representations of nodes in different views. （6） in, and For node e i Embedded representation in two views; S5.3 Contrastive Loss Training: To enhance the consistency of node representations across different views, the InfoNCE loss function based on contrastive learning is used for embedding training; for each node e i The representations of the nodes in the two views are considered positive sample pairs; the representations between other different nodes are considered negative sample pairs, and the loss function is compared accordingly. The definition is as follows: （7） Where sim() is the cosine similarity; For temperature parameters, Given the set of all nodes in the graph; obtain the node structure embedding optimized by structure contrast learning, which is used in the subsequent multimodal fusion step S6 to improve the robust representation and generalization performance of the scenic spot nodes.

2. The personalized tourist attraction recommendation method based on multimodal knowledge graph comparative learning according to claim 1, characterized in that: Step S1 includes the following steps: S1.1 Collect multi-source heterogeneous data in the tourism sector, including: Visual modality refers to a set of images corresponding to the scenic spots. Text modality, consisting of attraction titles, descriptions, and summaries of user reviews; Structured attributes include category, geographic location, time tag, and price range; User's historical interactions with attractions include browsing, rating, favorites, and check-in sequences; S1.2 Constructing a multimodal knowledge graph: The collected multimodal information of scenic spots is used to construct a multimodal knowledge graph G for the tourism field in the form of triples. The nodes include scenic spot entities, category entities, time entities, price entities, and user entities, which are used to describe multidimensional semantic features; the edges are used to represent the semantic or behavioral relationships between different entities. The multimodal knowledge graph is used to uniformly represent and organize scenic spot information from multiple information sources.

3. The personalized scenic spot recommendation method based on multimodal knowledge graph comparative learning according to claim 2, characterized in that: Step S2 includes the following steps: S2.1 Obtain the image set corresponding to each scenic spot entity from the multimodal knowledge graph G constructed in step S1, and input the image Image corresponding to each scenic spot; S2.2 utilizes the image encoder ViT in the pre-trained CLIP model to extract visual features from each image, obtaining a high-dimensional visual vector. Multiple images of the same scenic spot are averaged or weighted to obtain the visual representation of that scenic spot. To unify the embedding dimension, principal component analysis (PCA) is used to reduce the dimensionality of the extracted feature vectors, and then the vectors are passed through a fully connected layer (FC). img Achieve nonlinear mapping to obtain visual embedding vectors ; （1） Where ReLU is the activation function; S2.3 Visual Embedding The site nodes are mapped to the knowledge graph and used as one of the initial node representations to provide input features for the subsequent multimodal feature fusion step S6.

4. The personalized scenic spot recommendation method based on multimodal knowledge graph comparative learning according to claim 3, characterized in that: Step S3 includes the following steps: S3.1 From the multimodal knowledge graph G constructed in step S1, obtain the text information corresponding to each scenic spot node, including title, introduction and user comment summary, and combine them to form a complete text input sequence Text; S3.2 Employs a pre-trained BERT model to encode the input text; obtains contextual semantic representations through Token Embedding, Position Embedding, and Segment Embedding layers; and takes the output vector at the [CLS] position or performs average pooling on all token representations as the overall semantic vector of the text. The specific calculations are as follows: （2） S3.3 embeds the obtained text semantics The corresponding scenic spot nodes are mapped to the knowledge graph and used as the initial modal representation of the nodes, providing input features for the subsequent multimodal feature fusion step S6.

5. The personalized scenic spot recommendation method based on multimodal knowledge graph comparative learning according to claim 4, characterized in that: Step S4 includes the following steps: S4.1 Input Structure: The input is the multimodal knowledge graph G=(E,R,U) constructed in step S1, where E is the set of entity nodes, R is the set of relations, and U is the set of user nodes; in the graph, each edge constitutes a quadruple (u,e). i ,e j ,r), representing user u and triple entity e i and entity e j And the interactive relevance of their relationship r; used for modeling incorporating user personality preferences; S4.2 Message Passing Mechanism and Attention Modeling: The message propagation in the KA-GCN is based not only on inter-entity relationships but also jointly considers user interests and preferences; for any edge (e i ,r,e j In the context of user u, a multi-dimensional interactive attention mechanism is adopted, introducing joint representations of entity pairs, relations, and users, with attention weights. The calculation process is as follows: （3） in, It is an activation function. It is an attention vector. , , , and It is a learnable linear transformation matrix; This represents vector concatenation. Indicates with entity e i The set of connected neighboring nodes; S4.3 Node Information Aggregation: Based on the calculated attention weights, perform weighted aggregation of neighbor node information and update node e. i The embedding representation h i for: （4） In the above formula, σ() represents the activation function; The weight matrix is ​​used; through multi-layer stacking and message passing, a high-order structured graph representation can be obtained, resulting in the final structure embedding vector of the node.

6. The personalized scenic spot recommendation method based on multimodal knowledge graph comparative learning according to claim 5, characterized in that: Step S6 includes the following steps: S6.1 Input modal representation, for each attraction entity e i The following three modal embeddings are obtained from the previous steps: Visual embedding: Image feature representation extracted by the CLIP model in step S2; text embedding: The text semantic representation generated by the BERT model in step S3; graph structure embedding: The structured knowledge representation output by the KA-GCN model in step S5; S6.2 Gating Weight Generation: The gating fusion network dynamically generates the weight coefficients of each modality during the fusion process to achieve adaptive adjustment of information importance. The gating weights consider not only the representation of the modality itself, but also the consistency and complementarity between them. The calculation process is as follows: （8） in, Generate a matrix for the gate weights; For bias terms; This represents the weighting coefficients corresponding to the three modes, satisfying α v +α t +α s =1; S6.3 Fusion Embedding Generation, the fused unified scenic spot embedding representation is as follows: : （9） This representation integrates the semantic expressive power, visual perception ability, and structural features of multimodal information; the resulting fusion embedding It integrates the perceptual features of the visual modality, the semantic information of the text modality, and the relational knowledge of the structural modality, providing input for the subsequent user interest modeling step S7 and recommendation generation step S8.

7. The personalized scenic spot recommendation method based on multimodal knowledge graph comparative learning according to claim 6, characterized in that: Step S7 includes the following steps: S7.1 Construction of Historical Interaction Graph Extract user nodes and their interaction records from the knowledge graph G, and construct a user interest subgraph G. u =(V u E u ), V u This includes user nodes and their interactive attraction nodes; E u An edge is represented between a user and the interactive attractions, indicating the interaction relationship; this subgraph is used for local modeling of the user's individual interests. S7.2 User Access Intent Recognition When scoring a candidate attraction i, the score is embedded from step S6. As a query, the structure-enhanced representation from each historical node k in step S6 is used. Perform semantic alignment calculations to assess query-aware attention, evaluate the influence weight of historical attractions on the current attraction, and characterize the user's primary access intent. : （10） in, and Learnable weights, q is used as the matching readout vector to output a scalar score. ; Softmax normalization is applied to the scores of all historical nodes to obtain the candidate relevant attention weights: （11） in, The weights representing user attention; the denominator is all historical sites. The sum of the scores of candidate site i and the total score of candidate site i is used to ensure that the total weight is 1, thus highlighting the influence of highly relevant historical nodes. S7.3 User Interest Representation Aggregation The attention weights obtained in S7.2 are used to perform a weighted summation of the embedding representations of all historical interaction sites to generate the final user interest embedding vector. : （12） The most relevant parts of the user's past behavior to the candidates are extracted and used as the user representation for rating and ranking. This representation dynamically aggregates the interest patterns most relevant to the current candidate attractions from the user's past behavior, providing input features for the subsequent recommendation generation step S8.

8. The personalized scenic spot recommendation method based on multimodal knowledge graph comparative learning according to claim 7, characterized in that: Step S8 includes the following steps: S8.1 User Embedding and Scenic Spot Embedding Calculation User embedding representation: Based on the user interest embedding vector output in step S7 ; Scenic spot embedding representation: For candidate scenic spot nodes, a fused multimodal embedding vector is generated through an S6 gated fusion network. It includes visual, textual, and structural semantic features; S8.2 Similarity Scoring Calculation Using cosine similarity as a measure of interest matching between users and attractions, given user embeddings and candidate attraction embeddings, the recommendation score is defined as: （13） A higher score indicates that the user is more likely to be interested in the attraction. Then, the attractions are sorted from highest to lowest score, and the top-N highest-scoring attractions are selected to generate the final recommendation results.

Images